Allegro Manual: Datafile routines

# Datafile routines

- create_datafile_index — Creates an index for a datafile.
- DAT_ID — Makes an ID value from four letters.
- destroy_datafile_index — Destroys a datafile index.
- find_datafile_object — Searches a datafile for an object with a name.
- fixup_datafile — Fixes truecolor images in compiled datafiles.
- get_datafile_property — Returns the property string for the object.
- load_datafile — Loads a datafile into memory.
- load_datafile_callback — Loads a datafile into memory, calling a hook per object.
- load_datafile_object — Loads a specific object from a datafile.
- load_datafile_object_indexed — Loads a single object from a datafile index.
- register_datafile_object — Registers load/destroy functions for custom object types.
- unload_datafile — Frees all the objects in a datafile.
- unload_datafile_object — Frees an object previously loaded by load_datafile_object().

Datafiles are created by the grabber utility (see grabber.txt for more information), and have a ``.dat'` extension. They can contain bitmaps, palettes, fonts, samples, MIDI music, FLI/FLC animations, and any other binary data that you import. You could distribute your bitmaps and samples in a myriad of separate files, but packing them in a few ``.dat'` binaries has a few advantages:

- On some platforms loading a single big datafile at once is faster than loading individual resources one by one.
- Instead of several loops for your resources, you can write a single line of code with just a single point of failure to take care of.
- You can potentially reduce the size of your data by enabling compression on your datafiles. Less download time for your end users, less wait during loading screens!
- If you don't need to load the whole datafile at once, you can still enable individual file compression. It is slightly worse than global compression, but it is very fast with loading times because Allegro can easily seek inside the datafile to find a specific object.
- Even without encryption, most end users of your application won't be able to look at or modify the resources for your game. A missing sound file or a modified bitmap could potentially crash the game if you haven't considered this in your loading code!
- It looks much more professional and convenient to distribute levels! For example, if you found a bug in a level of your game, just distribute your new ``level4.dat'` and tell users to overwrite their old version.

Allegro allows you to load datafiles once and forget about them. But if you have many levels it can be wise to load only the resources required for the current level. You can accomplish the later by separating levels in different datafiles, or using functions like load_datafile_object() to avoid loading everything at once. You can even read directly from a specific datafile object with the pack_fopen() function.

On some platforms you can attach datafiles to your binary, potentially reducing your game distribution down to a single executable file. Try the example exexedat on your platform to see if this is possible. However, this is not recommended for big programs: a single update to your code or binary data would force end users to download again a potentially big file, no matter how small your change is. The same warning goes for the tools dat2s or dat2c, which convert datafiles into assembler and C code respectively, prepared to be included directly in your binary.

Remember that with Allegro truecolor images can only be loaded after you have set a graphics mode. This is true for datafiles too. Load all your data after you have set the graphics mode, otherwise the pixel format (RGB or BGR) will not be known and the datafile may be converted wrongly. Oh, and read carefully the warning of fixup_datafile() if you plan on switching resolutions during runtime.

Note: even though Allegro datafiles provide encryption, you should consider it weak, so don't plan on hiding there the plans for a Death Star or something. Determinate knowledgeable users will be able to rip your resources no matter how hard you try to hide them! Use the encryption only as a slight deterrent towards unwanted tampering of your data. How to crack an encrypted datafile is left as an exercise to the reader, though.

Using datafiles once they are loaded is quite simple: you access the elements of the DATAFILE as a normal array. Read below the section "Using datafiles" below for several examples on how to access their data.

DATAFILE *load_datafile(const char *filename);
Loads a datafile into memory in one go. If the datafile has been encrypted, you must first call packfile_password() to set the appropriate key. If the datafile contains truecolor graphics, you must set the video mode or call set_color_conversion() before loading it. Example:

>           /* Load the resources for our game. */
>           DATAFILE *dat = load_datafile("game.dat");
>           if (!dat)
>              abort_on_error("Couldn't load sound resources!");
>           /* Use resources. */
>           ...
>           /* Destroy them when we don't need them any more. */
>           unload_datafile(dat);

Return value: Returns a pointer to the DATAFILE, or NULL on error. Remember to free this DATAFILE later to avoid memory leaks.

> See also: load_datafile_callback, unload_datafile, load_datafile_object, set_color_conversion, fixup_datafile, packfile_password, find_datafile_object, register_datafile_object, Using datafiles.

> Examples using this: excustom, exdata, exexedat, exgui, exsprite, exunicod.

DATAFILE *load_datafile_callback(const char *filename, void (*callback)(DATAFILE *d));
Loads a datafile into memory, calling the specified hook function once for each object in the file, passing it a pointer to the object just read. You can use this to implement very simple loading screens where every time the hook is called, the screen is updated to let the user know your program is still loading from disk:

>           void load_callback(DATAFILE *dat_obj)
>           {
>              static const char indicator[] = "-\\|/-.oOXOo.";
>              static int current = 0;

>              /* Show a different character every time. */
>              textprintf_ex(screen, font, 0, 0, makecol(0, 0, 0),
>                            makecol(255, 255, 255), "%c Loading %c",
>                            indicator[current], indicator[current]);
>              /* Increase index and check if we need to reset it. */
>              current++;
>              if (!indicator[current])
>                 current = 0;
>           }
>              ...
>              dat = load_datafile_callback("data.dat", load_callback);

Return value: Returns a pointer to the DATAFILE or NULL on error. Remember to free this DATAFILE later to avoid memory leaks.

> See also: load_datafile, unload_datafile, load_datafile_object, set_color_conversion, fixup_datafile, packfile_password, find_datafile_object, register_datafile_object.

void unload_datafile(DATAFILE *dat);
Frees all the objects in a datafile. Use this to avoid memory leaks in your program.

> See also: load_datafile.

> Examples using this: excustom, exdata, exexedat, exgui, exsprite, exunicod.

DATAFILE *load_datafile_object(const char *filename, const char *objectname);
Loads a specific object from a datafile. This won't work if you strip the object names from the file, and it will be very slow if you save the file with global compression. Example:

>           /* Load only the music from the datafile. */
>           music_object = load_datafile_object("datafile.dat",
>                                               "MUSIC");
>           /* Play it and wait a moment for it. */
>           play_midi(music_object->dat);
>           ...
>           /* Destroy unneeded music. */
>           unload_datafile_object(music_object);

Return value: Returns a pointer to a single DATAFILE element whose ``dat'` member points to the object, or NULL if there was an error or there was no object with the requested name. Remember to free this DATAFILE later to avoid memory leaks, but use the correct unloading function!

> See also: unload_datafile_object, load_datafile, set_color_conversion, find_datafile_object, register_datafile_object, Using datafiles.

void unload_datafile_object(DATAFILE *dat);
Frees an object previously loaded by load_datafile_object(). Use this to avoid memory leaks in your program.

> See also: load_datafile_object.

DATAFILE *find_datafile_object(const DATAFILE *dat, const char *objectname);
Searches an already loaded datafile for an object with the specified name. In the name you can use ``/'` and `#' separators for nested datafile paths. Example:

>           char level_name[10];
>           DATAFILE *dat, *level;
>           ...
>           uszprintf(level_name, sizeof(buffer),
>                     "LEVEL_%02d", level_number);
>           level = find_datafile_object(dat, level_name);
>           if (!level)
>              abort_on_error("That level doesn't exist!");

Return value: Returns a pointer to a single DATAFILE element whose ``dat'` member points to the object, or NULL if the object could not be found.

> See also: load_datafile, load_datafile_object.

DATAFILE_INDEX *create_datafile_index(const char *filename);
Creates an index for a datafile, to speed up loading single objects out of it. This is mostly useful for big datafiles, which you don't want to load as a whole. The index will store the offset of all objects inside the datafile, and then you can load it quickly with "load_datafile_object_indexed" later. Use destroy_datafile_index to free the memory used by it again.

Note: If the datafile uses global compression, there is no performance gain from using an index, because seeking to the offset still requires to uncompress the whole datafile up to that offset. Example:

>        DATAFILE_INDEX *index = create_datafile_index("huge.dat");
>        DATAFILE *object = load_datafile_object_indexed(index, 1234);
>        ...
>        unload_datafile_object(object);
>        destroy_datafile_index(index);

Return value: A pointer value which you can pass to load_datafile_object_indexed.

> See also: destroy_datafile_index, load_datafile_object_indexed, Using datafiles.

DATAFILE *load_datafile_object_indexed(const DATAFILE_INDEX *index, int item)
This loads a single object, using the index created previously with create_datafile_index. See create_datafile_index for an example.

Return value: Returns a pointer to a single DATAFILE element whose "dat" member points to the object, or NULL if the object could not be loaded.

> See also: create_datafile_index, load_datafile_object, unload_datafile_object.

void destroy_datafile_index(DATAFILE_INDEX *index)
This function frees the memory used by a datafile index created with create_datafile_index earlier.

> See also: create_datafile_index.

const char *get_datafile_property(const DATAFILE *dat, int type);
Finds the property type of a DATAFILE object. The type parameter must be a value created with the DAT_ID() macro. Example:

>           const char *name;
>           ...
>           name = get_datafile_property(game_data,
>                                        DAT_ID('N','A','M','E'));
>           if (name == empty_string)
>              abort_on_error("Object doesn't have a name!");

Return value: Returns a pointer to the text string for the object, or a pointer to the variable empty_string if the property isn't present.

> See also: Using datafiles, DAT_ID, empty_string.

void register_datafile_object(int id, void *(*load)(PACKFILE *f, long size), void (*destroy)(void *data));
Used to add custom object types, specifying functions to load and destroy objects of this type.

> See also: load_datafile, load_datafile_object, DAT_ID, Custom datafile objects.

void fixup_datafile(DATAFILE *data);
If you are using compiled datafiles (produced by the dat2s and dat2c utilities) on a platform that doesn't support constructors (currently any non GCC-based platform), or if the datafiles contain truecolor images, you must call this function once after your set the video mode that you will be using. This will ensure the datafiles are properly initialised in the first case and convert the color values into the appropriate format in the second case. It handles flipping between RGB and BGR formats, and converting between different color depths whenever that can be done without changing the size of the image (ie. changing 15<->16-bit hicolor for both bitmaps and RLE sprites, and 24<->32-bit truecolor for RLE sprites).

Note that you can only call this once and expect it to work correctly, because after the call the DATAFILE you fixed up is permanently converted to whatever is the current component ordering for your screen mode. If you call fixup_datafile again, the function assumes you have a freshly loaded datafile. It cannot "undo" the previous conversion.

If your program supports changing resolution and/or color depth during runtime, you have two choices: either call fixup_datafile() just once and hope that the component ordering and bit depth doesn't change when the screen mode changes (unlikely). Or, you can reload your datafiles when the screen mode changes.

> See also: set_gfx_mode, set_color_conversion, Differences between platforms.

Macro DAT_ID(a, b, c, d);
Every object or property in a datafile is identified by a 4 letter ID, which can be created with this macro. For example, to access the NAME property of a datafile object, you could use:

>           get_datafile_property(datob, DAT_ID('N','A','M','E'));

> See also: register_datafile_object, get_datafile_property, Custom datafile objects, Using datafiles.

## Using datafiles

In order to access the contents of a datafile, you will need to know where each object is located. The easiest way to do this is by integer index, using an automatically generated header file. With the grabber, type a name into the "Header:" field, and the object indexes will be written to this file whenever the datafile is saved. With the dat utility, use the '-h' option, eg. "dat filename.dat -h filename.h". The header will define C preprocessor symbols for each object in the datafile, for example:

>        #define SOME_DATA                        0        /* DATA */
>        #define SOME_MORE_DATA                   1        /* DATA */

To prevent name conflicts, you can specify a prefix string for these definitions by typing it into the "Prefix:" field in the grabber or using the '-p' option to dat.

To load a datafile into memory, call the function:

>        DATAFILE *load_datafile(char *filename);

This will load the entire file, returning a pointer to it, or NULL on error. When the data is no longer required, the entire thing can be destroyed by calling:

>        void unload_datafile(DATAFILE *dat);

When you load a datafile, you will obtain a pointer to an array of DATAFILE structures:

>        typedef struct DATAFILE
>        {
>           void *dat;                    - pointer to the actual data
>           int type;                     - object type ID
>           long size;                    - size of the data, in bytes
>           DATAFILE_PROPERTY *prop;      - list of object properties
>        } DATAFILE;

The only really important piece of information here is the ``dat'` field, which points to the contents of the object. What type of data this is will depend on the type of object: for bitmaps it will be an Allegro BITMAP structure, for RLE sprites an RLE_SPRITE, for fonts a FONT structure, etc. If you are programming in C you can pass this pointer directly to the relevant Allegro library functions, but if you are using C++ you will need to cast it to the appropriate type to prevent the compiler giving a warning.

For example, if you have a datafile called ``myfile.dat'`, which contains a bitmap called COOL_PICTURE, and you have used it to produce a header called ``myfile.h'`, you could display the bitmap with the code:

>        #include "myfile.h"

>        void show_the_bitmap()
>        {
>           DATAFILE *dat;
>           BITMAP *bmp;

>           dat = load_datafile("myfile.dat");
>           if (!dat) {
>              /* report an error! */
>              return;
>           }

>           bmp = (BITMAP *)dat[COOL_PICTURE].dat;
>           blit(bmp, screen, 0, 0, 0, 0, bmp->w, bmp->h);
>           unload_datafile(dat);
>        }

If a datafile contains nested child datafiles, the header will prefix the names of objects in the sub-files with the name of their parent datafile. It will also define a count of the number of objects in the child file, which may be useful if for example the child datafile contains several bitmaps which form a 'run' animation, and you want your code to automatically adjust to the number of frames in the datafile.

For example, the following datafile:

>        "FILE" - NESTED_FILE
>                 |- "BMP" - A_BITMAP
>                 |- "FONT" - A_FONT
>        "DATA" - SOME_DATA
>        "DATA" - SOME_MORE_DATA

Will produce the header:

>        #define NESTED_FILE                      0        /* FILE */

>        #define NESTED_FILE_A_BITMAP             0        /* BMP  */
>        #define NESTED_FILE_A_FONT               1        /* FONT */
>        #define NESTED_FILE_COUNT                2

>        #define SOME_DATA                        1        /* DATA */
>        #define SOME_MORE_DATA                   2        /* DATA */

The main datafile contains three objects (NESTED_FILE, SOME_DATA, and SOME_MORE_DATA) with consecutive indexes, while the child datafile contains the two objects A_BITMAP and A_FONT. To access these objects you need to reference both the parent and child datafiles, eg:

>        DATAFILE *dat = load_datafile("whatever.dat");
>        DATAFILE *nested = (DATAFILE *)dat[NESTED_FILE].dat;
>        FONT *thefont = (FONT *)nested[NESTED_FILE_A_FONT].dat;

If you need to access object property strings from within your program, you can use the function:

>        char *get_datafile_property(DATAFILE *dat, int type);

This will return a pointer to the property string if it can be found, and an empty string (not null!) if it does not exist. One possible use of this function is to locate objects by name, rather than using the indexes from a header file. The datafile array is ended by an object of type DAT_END, so to search the datafile dat for the object "my_object" you could use the code:

>        const int name_type = DAT_ID('N','A','M','E');
>        for (i=0; dat[i].type != DAT_END; i++) {
>           if (stricmp(get_datafile_property(dat+i, name_type),
>                       "my_object") == 0) {
>              /* found the object at index i */
>           }
>        }
>        /* not found... */

If you prefer to access objects by name rather than index number, you can use the function:

>        DATAFILE *find_datafile_object(DATAFILE *dat, char *objectname);

This will search an already loaded datafile for an object with the specified name, returning a pointer to it, or NULL if the object cannot be found. It understands '/' and '#' separators for nested datafile paths.

It is also possible to selectively load individual objects from a datafile, with the function:

>        DATAFILE *load_datafile_object(char *filename, char *objectname);

This searches the datafile for an object with the specified name, so obviously it won't work if you strip the name properties out of the file. Because this function needs to seek through the data, it will be extremely slow if you have saved the file with global compression. If you are planning to load objects individually, you should save the file uncompressed or with individual compression per-object. Because the returned datafile points to a single object rather than an array of objects, you should access it with the syntax datafile->dat, rather than datafile[index].dat, and when you are done you should free the object with the function:

>        void unload_datafile_object(DATAFILE *dat);

Example:

>        music_object = load_datafile_object("datafile.dat", "MUSIC");
>        play_midi(music_object->dat);
>        ...
>        unload_datafile_object(music_object);

Alternatively, the packfile functions can open and read directly from the contents of a datafile object. You do this by calling pack_fopen() with a fake filename in the form "filename.dat#object_name". The contents of the object can then be read in an identical way to a normal disk file, so any of the file access functions in Allegro (eg. load_pcx() and set_config_file()) can be used to read from datafile objects. Note that you can't write to datafiles in this way: the fake file is read only. Also, you should save the file uncompressed or with per-object compression if you are planning on using this feature. Finally, be aware that the special Allegro object types aren't the same format as the files you import the data from, so if for example you want to use load_pcx to read an image from a datafile, you should import it as a binary data chunk rather than as a BITMAP object.

If you have appended a datafile to the end of your executable with the exedat utility, use load_datafile("#") to read the entire thing into memory, load_datafile_object("#", "object_name") to load a specific object, and pack_fopen("#object_name", F_READ) to read one of the objects directly with your own code. Note that unless you use the previous functions to load the appended data, the OS will not load it into memory just because you are running the program, so you shouldn't have problems attaching datafiles to your binary larger than the available system memory.

By default, all graphic objects loaded from a datafile will be converted into the current color depth. This conversion may be both lossy and very slow, particularly when reducing from truecolor to 256 color formats, so you may wish to disable it by calling set_color_conversion(COLORCONV_NONE) or set_color_conversion(COLORCONV_PARTIAL) before your call to load_datafile().

## Custom datafile objects

Some of the objects in a datafile, for example palettes and FLI animations, are simply treated as blocks of binary data, but others are loaded into special formats such as bitmap structures or compiled sprites. It is possible to extend the datafile system to support your own custom object types, eg. map objects for a tile based engine, or level data for a platform game. Obviously the grabber has no way of understanding this data, but it will allow you to import binary data from external files, so you can grab information produced by your own utilities. If you are happy with the data being loaded as a simple binary block, that is all you need to do, but if you need to load it into a specific structure, read on...

Your custom objects must be given a unique type ID, which is formed from four ASCII characters (by convention all uppercase A-Z). If you don't use all four characters, the string should be padded with spaces (ASCII 32). You should use this ID when creating the objects in the grabber (select New/Other and type in the ID string), and in your code you should define an identifier for the type, eg:

>        #define DAT_MAPDATA  DAT_ID('M','A','P','D')

You then need to write functions for loading and destroying objects of this type, in the form:

>        void *load_mapdata(PACKFILE *f, long size)
>        {
>           /* Allegro will call this function whenever an object of your custom
>            * type needs to be loaded from a datafile. It will be passed a
>            * pointer to the file from which the data is to be read, and the size
>            * of the object in bytes. It should return a pointer to the loaded
>            * data, which will be stored in the dat field of the datafile object
>            * structure, or NULL if an error occurs. The file will have been
>            * opened as a sub-chunk of the main datafile, so it is safe to read
>            * past the end of the object (if you attempt this, Allegro will
>            * return EOF), and it is also safe to return before reading all the
>            * data in the chunk (if you do this, Allegro will skip any unused
>            * bytes before starting to read the next object). You should _not_
>            * close the file when you are done: this will be handled by the
>            * calling function. To clarify how all this works, here's an example
>            * implementation of a null-terminated string object:
>            */

>           #define MAX_LEN  256

>           char buf[MAX_LEN];
>           char *p;
>           int i, c;

>           for (i=0; i<;MAX_LEN-1; i++) {
>              if ((c = pack_getc(f)) == EOF)
>                 break;

>              buf[i] = c;
>           }

>           buf[i] = 0;

>           p = malloc(i+1);
>           strcpy(p, buf);

>           return p;
>        }

>        void destroy_mapdata(void *data)
>        {
>           /* Allegro will call this function whenever an object of your custom
>            * type needs to be destroyed. It will be passed a pointer to the
>            * object (as returned by the load function), and should free whatever
>            * memory the object is using. For example, the simple string object
>            * returned by the above loader could be destroyed with the code:
>            */

>           if (data)
>              free(data);
>        }

Finally, before you load your datafile you must tell Allegro about the custom format, by calling:

>        register_datafile_object(DAT_MAPDATA, load_mapdata, destroy_mapdata);

It is also possible to integrate support for custom object types directly into the grabber and dat utilities, by copying some special files into the tools/plugins directory. This can be used to add whole new object types and menu commands, or to provide additional import/export routines for the existing formats. See ``tools/plugins/plugins.txt'` for an overview of how to write your own grabber plugins.

Back to contents