NWN2 Datafile Accessor Library

I'm making available some of the supporting libraries that the Client Extension uses in order to interface with many of the game's data files.

This library may be of use to programmers and developers of NWN2 tools and utilities. You can get a copy of it here:

http://www.nynaeve.net/Skywing/nwn2/nwn2dev/public_nwn2dev.zip (341)
Forum thread: http://forum.bioware.com/topic/124122-nwn2-datafile-accessor-library/

An example tool (a simple MDB renderer against GDI) that functions as a demonstration of how to use the libraries is also included.

The library is set to be built with the Windows Driver Kit (WDK). It will also build with Visual Studio (VC++), but I don't maintain a VS development environment for the project any more, so you'll have to set up the projects yourself. The WDK's freely downloadable and self-contained, and a link to it is included in the readme file.

Including the blurb from the original post describing the library:

The library consists of heavily commented C++ code that provides a suite of file format parsers for various NWN2 formats:

- TRX format (all packets/records are fully decoded), including the complete area surface walkmesh and navigational node format. Meshes and navigational data are deserialized to a convenient set of data structures suitable for drawing or interacting with the walkmesh for pathing traversal. A partial set of APIs for interacting with the walkmesh structures (such as visibility and straight path computation) are also included.
- MDB format (all packets/records are fully decoded). Meshes are deserialized into convenient data structures suitable for drawing. APIs for performing C2/C3 collision mesh intersection in the same fashion as the game engine does are provided.
- TLK format (localized string lookup).
- 2DA format (all 2DA format features except the 2DA default value option -- which seems to not be used by NWN2 -- are supported).
- GR2 format, in the form of a wrapper for tazpn's granny2 reading code for skeletons, plus various fixes to tazpn's underlying granny2 reading code. Eventually, animation reading will be added.
- DDS images, for simple 32-bpp DDS images. The DDS reader was designed only to support enough DDS features to process the inline texture blending maps embedded in TRX files, although other 32-bpp simple DDS images could be read via it.

At present, only readonly access to these resources is available; however, as the resource formats are generally (with the exception of GR2) completely deserialized, all of the information necessary to create writer APIs is available. I have not done so yet as my projects have not yet required writing to these files, only reading.

The NWN2 datafile lib also includes a complete (readonly) implementation of an NWN2-compatible resource loader. The resource loader supports the following features:

- Directory mode file support
- ERF file support (MOD, PWC, HAK, ERF, ...)
- Zip file support (indexes all ~100K of the in-game zip resources in on the order of 200-400ms on bootup, as opposed to the 3-6 seconds that the NWN2 native resource loader burns searching through zip files at game initialization)

The resource manager provides both direct in-memory access APIs to read resource files, as well as optimized APIs for pulling resources files out to disk and allowing them to be accessed via any standard file I/O API.

A search strategy compatible with the game's native resource manager has been implemented based on the original BioWare Aurora documentation and further observations of the NWN2 engine behavior. The search strategy is believed to be complete (or nearly so) and can successfully load custom content with the correct priorities in multiplayer PWs, to the point of being able to load custom models, 2DAs, etc.

A uniform interface for accessing resource files is provided (uniform regardless of whether they are backed by directory mode filesystem files or embedded in ERFs or compressed in Zip files).

The Client Extension internally uses this library for all its resource loading needs, including model/terrain/game rules processing support.

If there are any projects which could make real use of such a library for purposes of custom content tools creation or the like, I'm open to doing the work necessary to release a version of the library for public consumption.

The code is currently written for 32-bit and 64-bit Windows, although it should be trivially portable to any other operating system. Note that granny2 reading requires calling a granny2 support function to decompress the .gr2 files to their raw format and thus granny2-related features are regrettably tied to 32-bit Windows.

Core dependencies include a zlib, the minzip library shipping with zlib in the contrib distribution (plus some minor changes for significant performance improvements in zip enumeration), an C-interface internal math library, granny2.dll (only if .gr2 support is desired), C++ STL, and basic file I/O functions and directory enumeration APIs (memory mapped I/O used in some cases, though the code is trivially portable to operate without memory mapped I/O and could easily function with standard C CRT file I/O APIs). No MFC or other framework libraries are used.

All data file formats are validated when parsed, and as such malformed data files should not introduce crashes or other security bugs. An exception to this rule is that loops in the navigational node structures in the area surface walkmesh must be handled during pathing traversal instead of at load time, for performance reasons.

The datafile library is heavily tested under load with pageheap and is believed to provide a robust foundation.

Possibly uses for the library might include custom content creation, viewing, or manipulation tools, or certain toolset plugins.