Share this post

Link to post

Share on other sites

Most I've seen, the header is the file table ;)
This part is read into Ram on startup and kept there. It lets you perform a lookup by filename and retrieve the offset and size of a file within the archive.

The last engine that I used, used paths within filenames (e.g. An asset might be called "foo/bar/baz.type.platform") - so just a flat/non-hierarchical table containing these long names.

On my current engine, I actually ignore paths completely, basically moving all assets into a single directory when building an archive (e.g. The above would just be "baz.type.platform"). This means that you can't have two assets with the same name, but I see this as a positive feature rather than a negative ;-)
During development, assets are stored as individual files (not packaged) so it's easy to make changes to the game. Also, any folder structure can be used during development (during dev, that file might be stored in the "foo/bar" directory, the game doesn't care).

Share this post

Link to post

Share on other sites

the header/table should only be a few KB. Easy to 'waste' RAM on storing that whole table, to make loading files easier.

How can you differentiate if it has the same name ?

I don't. If the artists have "level1/concrete.png" and "level2/concrete.png", then the engine tools gives an error, asking them to delete or rename one of them.

What's about the compression ? That still a question.

Using zlib or LZMA SDK is pretty common. Use it to compress each individual file, then in the header/table you can store the offset, compressed size and uncompressed size of each file. When loading a file, malloc the uncompressed size, then stream the compressed data off disk and through your decompression library into the malloc'ed buffer.
Unless the user has an SSD or RAM-disk, this should actually be a lot faster than loading uncompressed files! (As long as you've got the spare CPU time to do the decompression)

3

Share this post

Link to post

Share on other sites

We call our packages "archives". The format is basically a table of contents containing a map of symbols (hashed string asset names) to a struct containing the offset + size of the actual asset data. All of our asset ID's are flat like in Hodgman's setup. The whole archive is compressed using Oodle (compression middleware by RAD Game Tools), and when we load an archive we stream in chunk by chunk asynchronously and pipeline the decompression in parallel. Once that's done we have to do a quick initialization step, where we mostly just fixup pointers in the data structures (on Windows we also create D3D resources in this step, because you have to do this at runtime). Once this is done the users of the assets can load assets individually by asset ID, which basically just amounts to a binary search through the map and then returning a pointer once the asset is found.

As for loose files vs. packages, we support both for development builds. Building a level always triggers packaging an archive, but when we load an archive we check the current status of the individual assets and load them off disk if we determine that the version on disk is newer. That way you get fast loads by default, but you can still iterate on individual assets if you want to do that.

Edited April 23, 2014 by MJP

4

Share this post

Link to post

Share on other sites

Once this is done the users of the assets can load assets individually by asset ID, which basically just amounts to a binary search through the map and then returning a pointer once the asset is found.

Do you keep loaded the file or use fopen when you want to keep a file data ?

Is it bad to fopen/fclose for each file in a GetMemoryFile( const int EntryIndex ) ?