Lately, I've been trying to condense my levels into compressed files that sort of encapsulate everything about them, including their tile information, the world objects within, anything pertaining to quests, etc. As of right now, I only have incorporated tile information because I haven't quite gotten the other aspects solidified in the engine to where they are ready to be hard set into the level specs (world objects are coming along, but it'll be a little bit yet). But I digress!

Basically, I wanted to compare my methods to all of yours. Before, I was storing my level files as .png images where all of the tile information was thus very visible and fairly condensed (I find .png files are pretty tiny most of the time). However, this is somewhat ungainly for my game, as a .png can't by itself encompass things like world objects, chests with their contents, data for liquid bodies, NPCs, and so on and so forth. Thus, I figured I would develop a level metafile that contains all of this information similar to how, I guess, Doom would store their files or whatever other example fits.

Down to process and numbers:

Step One: Convert the map to Unicode so that it can be stored in a file as a character matrix, with each character representing a relevant tile from the game engine.Step Two: This could encompass multiple steps, but essentially this is where everything else pertinent to the level gets stored besides the tiles (i.e., world objects, liquids, quest information, NPC information, etc.).Step Three: Convert the Unicode map and other information to a compressed ".lev" file using ZipStream. Not elegant or fancy, by any means, but it works.

Using the aforementioned process, I get about a 15-20% smaller size of the level than I would have with .png images comprising them, and not to mention the benefit of encapsulating the level's information to boot, which will probably augment the file by maybe another 5% at most, considering the sheer bulk of the files will be the tile information.

I'm not sure if this process is exactly the most efficient or best, and would love to know of what you guys have done or what you would recommend! I'm open to all sorts of suggestions and would like the best for my level data! Thanks for all your time! :]

Storing tile information is probably going to be highly inefficient in and of itself unless it's little more than 'Passable/Impassable', 'This collision profile' and 'This tile sprite/visual representation'. If you're storing any other information, then you might want to see what you could pull out and store separately (Especially if multiple maps end up making use of basically the same information).

Storing tile information is probably going to be highly inefficient in and of itself unless it's little more than 'Passable/Impassable', 'This collision profile' and 'This tile sprite/visual representation'. If you're storing any other information, then you might want to see what you could pull out and store separately (Especially if multiple maps end up making use of basically the same information).

Oh no, there isn't anything about the tiles being stored other than their visual information, basically (that's what I meant when I described the character matrix as each character being equivalent to one of the engine's tiles). For example:

AAAAAAABA

Would just basically mean that all A tiles are "dirt" tiles and B is a "stone" tile (for example's sake). Anything else in the level's metafile is basically going to be about macroscopic level features, like what objects are stored within it at what coordinates, which items are stored in what chests at which spots, and so forth. "Passable/impassable" functionality is coded into the engine and is relative to the tile, so it's just calculated at runtime.

Well then, I don't see your files being completely tile based information, (Say 100x100 is just 10k characters, and you could probably condense that considerably by limiting the number of tiles and then do some fancy masking when you load it).

Figuring out how you'll store data like item lists for certain tiles, enemy positions and the like will probably end up costing you a lot more (Position + item ID at the very least for each item on the screen). I mean, maybe not a large amount of the file, but most likely in excess of 5% of the total data you're going to be storing (This statement is less true if your maps are going to be highly sparse).

However, how large do you see all of these levels being? And how many do you think you'll have? Further, how much conversion back is going to be needed to make them easily/quickly parsed by your game's engine? Figure out of the trade offs there as well (I mean, I don't mind 200kb per 1mb of bloat, but I've gotten to the point where I glare in offended silence at long loading screens. Not that I believe that it'd end up with one, but really, you might be worrying about file sizes too soon.)

Well then, I don't see your files being completely tile based information, (Say 100x100 is just 10k characters, and you could probably condense that considerably by limiting the number of tiles and then do some fancy masking when you load it).

Figuring out how you'll store data like item lists for certain tiles, enemy positions and the like will probably end up costing you a lot more (Position + item ID at the very least for each item on the screen). I mean, maybe not a large amount of the file, but most likely in excess of 5% of the total data you're going to be storing (This statement is less true if your maps are going to be highly sparse).

However, how large do you see all of these levels being? And how many do you think you'll have? Further, how much conversion back is going to be needed to make them easily/quickly parsed by your game's engine? Figure out of the trade offs there as well (I mean, I don't mind 200kb per 1mb of bloat, but I've gotten to the point where I glare in offended silence at long loading screens. Not that I believe that it'd end up with one, but really, you might be worrying about file sizes too soon.)

Well, the levels are procedurally generated. At this point, the hugest size world one can generate creates roughly 30,000 levels. These levels aren't actually instantiated and specified until the player ventures into them, at which point it will make the level and save it and all that; before this, nothing except the name of the level will exist on disk. If the user really wanted to, though, they could spend a very long time exploring the world and thus instantiating all of these levels. The levels themselves can also grow to be fairly large, hypothetically, though I haven't a solid idea of just how big they will be at maximum because not everything is implemented yet, though I can make rough estimates; I highly doubt levels will exist over a meg in size.

Enemies will be instantiated at runtime, so that stuff won't be involved in the level files. Item lists will be stored for chests, and I'm not altogether sure just how much this will add to the file, though I'm willing to wager not an incredible amount.

It's not strictly about sizes; I'm also concerned about loading times as well. That's why I offered why I was choosing the implementation I am, in case anyone knows of a more sensible way of storing information in a way that will load and remain on disk efficiently; it would seem that having to parse less information to achieve similar results would involve less computation, and since worlds can hold so many levels, it makes sense to me that the data should be compressed so the game doesn't take up a huge amount of space.

I am interested in what you mean by using fancy masking to condense the tile information! Thanks for all your other advice, as well. I suppose I could have elaborated a little more on my game and its levels and so forth :p

so you have like two types of data- the level layout which is like a image- meta/object data

For the layout/tile data you can use simple compression on binary data.

give each tile type an ID

choose the smallest datatype to store the IDs, like if you won't have more then 255 types a single Byte is fine

save the level layout as a i.e. byte array

then you an compress the shit out of it like with LZMA, or build your own Huffman encoding^^

The other things like your game objects(NPC, cheast..) or meta data(time for level, level name ...)can be saved with normal serialisation(ByteStream, XML, JSON ...)

In the end pack everything together in a Zip file and you are finished

Wow, that sounds tantalizing! I'll have to look into all of that! I'm not sure yet if there will be less than 255 types of tiles (I hope so!), but if there are, then that sounds like a nice route! Thanks!

If you're interested in getting your map size as small as possible, why are you storing them in a text/ascii based format and not binary (and then compressed)?

Although to be honest I'm not sure that optimising levels for storage size is a worthwhile use of time. Storage space is cheap these days unless you're going to have humongous levels.

Well, to be honest, I'm not altogether sure how to go about compressing levels in binary format! If you could offer me a reference source for such a thing or offer some more tips, I would be grateful! Thank you!

Well, to be honest, I'm not altogether sure how to go about compressing levels in binary format! If you could offer me a reference source for such a thing or offer some more tips, I would be grateful! Thank you!

Basically just use a DataOutputStream and write bytes/ints/floats/whatever as appropriate, and pipe it through a zip output stream. And use the smallest data type for each bit of data - so if you only have a few different tile types then just use a byte (or even the lower few bits of the byte and pack some other info in the spare bits).

Well, to be honest, I'm not altogether sure how to go about compressing levels in binary format! If you could offer me a reference source for such a thing or offer some more tips, I would be grateful! Thank you!

Basically just use a DataOutputStream and write bytes/ints/floats/whatever as appropriate, and pipe it through a zip output stream. And use the smallest data type for each bit of data - so if you only have a few different tile types then just use a byte (or even the lower few bits of the byte and pack some other info in the spare bits).

Wow, that's pretty tricky stuff! Sounds really cool, though; I'll have to experiment with it and see what I can come up with! Thanks a ton!

Same thing what is done in games like minecraft, you just save the things the player changes the rest gets generated every time the game starts.

Minecraft only generates the first time you see a piece of level, after that's it's saved and loaded to disk, regardless of whether the player changes it or not. I'm guessing Colton wants to do the same - it does make things a lot more straightforward and robust.

Same thing what is done in games like minecraft, you just save the things the player changes the rest gets generated every time the game starts.

Minecraft only generates the first time you see a piece of level, after that's it's saved and loaded to disk, regardless of whether the player changes it or not. I'm guessing Colton wants to do the same - it does make things a lot more straightforward and robust.

A lot of tile maps use repeating adjacent tiles (e.g. many grass tiles together) which can be "compressed" using the same filtering techniques that PNG and GIF encoders use. You would use an "indexed color" format, but your "palette" would be tailored to tiles (sprite index, blocked, flipped, etc) rather than colors. Writing this kind of complex encoder/decoder would be overkill for simple tiled maps, but it's a cool idea.

The easiest thing to do would just be to pack as much information as you can in each byte/int.

A lot of tile maps use repeating adjacent tiles (e.g. many grass tiles together) which can be "compressed" using the same filtering techniques that PNG and GIF encoders use. You would use an "indexed color" format, but your "palette" would be tailored to tiles (sprite index, blocked, flipped, etc) rather than colors. Writing this kind of complex encoder/decoder would be overkill for simple tiled maps, but it's a cool idea.

The easiest thing to do would just be to pack as much information as you can in each byte/int.

I had seen something similar to that in a compression post I saw somewhere a few days ago, and indeed, I think it would be very interesting! I'm still slightly confused about how to go about packing more information in a byte than the tile's ID!

A lot of tile maps use repeating adjacent tiles (e.g. many grass tiles together) which can be "compressed" using the same filtering techniques that PNG and GIF encoders use. You would use an "indexed color" format, but your "palette" would be tailored to tiles (sprite index, blocked, flipped, etc) rather than colors. Writing this kind of complex encoder/decoder would be overkill for simple tiled maps, but it's a cool idea.

The easiest thing to do would just be to pack as much information as you can in each byte/int.

I had seen something similar to that in a compression post I saw somewhere a few days ago, and indeed, I think it would be very interesting! I'm still slightly confused about how to go about packing more information in a byte than the tile's ID!

You can do all this encoding by hand, or just use simple run length encoding, but really, are you learning how compression works, or do you want a compressed map? Use GZIPInput/OutputStream and be done with it.

You can do all this encoding by hand, or just use simple run length encoding, but really, are you learning how compression works, or do you want a compressed map? Use GZIPInput/OutputStream and be done with it.

I'm already using GZInput/OutputStream for all my compression, but I figured the guys above were referring to a way to get compression down even further. :]

You said your levels are procedurally generated and that you are more concerned about loading times than size, so for that reason I am going to ignore your first post and suggest something different. The best way to "compress" procedurally generated content is to only store the seed of your PRNG. Use java.util.Random's Random(long seed) constructor and save the 64-bit value you pass to it. In terms of storage size that's as good as you can get, but if it takes a long time to generate your level you may want to plan ahead.

I advise against storing the level as a file during run time. It would be appropriate if you wanted to let players save mid game, but there are too many drawbacks and not enough benefits. It's slightly inconsiderate, too. Your users will either have a hard drive and get annoyed by the constant whirring and clicking caused by the game, or have a solid state drive that suffers unnecessary wear. Or they might use some form of ram based file system and then you're just wasting your time. Store your current level and whatever levels that are accessible from that location in RAM. As long as you can preload those levels in the background faster than the player can get to them then the player won't experience any extra pauses between levels.

Another good way to save space is to use delta encoding. Track whatever changes the player makes to the level and only store the data necessary to recreate that level from a previously known state (such as the original procedurally created level.) For example, in a Legend of Zelda type game that didn't regenerate monsters and obstacles after you left the area... store only the relevant information, not a copy of the entire level. Store the damage dealt to each monster instead of the entire Monster object. Store boolean values (in bit mask form to save even more space) to record which doors/chests are unlocked, instead of storing Door and Chest complete with constant meta data such as references to Key or Treasure objects. This could be in the form of a list of changes (monster 4 move left, right, up, left, up, down, left) or only the change from start to finish (m4.x+=-1; m4.y+= 1;) For most games the final changes will be easier to calculate and faster to execute.

Finally, if you decide you need to save data in a different format because a feature requires it or because you change your mind and want to incorporate manually designed levels, then look into other simple methods. Some of the best compression methods have time/memory trade offs just like the potential time/memory trade offs between static and procedurally generated content. You might have surprisingly good results using images to store landscapes and meta data as plain text and using existing general purpose compression algorithms. Or you can write binary data using a data stream like Orangy Tang suggested (+1 for DataOutputStream). However combining both methods probably is not worthwhile because your binary data may already be compact. Try putting a .mp3 file in a zip file vs putting an xml file in a zip file. The latter will be much smaller because there is a shorter way to express the same data, but the former may just get you a zip file that's larger than its original contents because MP3s are already compressed to the point where they sacrifice quality to save storage space.

You said your levels are procedurally generated and that you are more concerned about loading times than size, so for that reason I am going to ignore your first post and suggest something different. The best way to "compress" procedurally generated content is to only store the seed of your PRNG. Use java.util.Random's Random(long seed) constructor and save the 64-bit value you pass to it. In terms of storage size that's as good as you can get, but if it takes a long time to generate your level you may want to plan ahead.

I advise against storing the level as a file during run time. It would be appropriate if you wanted to let players save mid game, but there are too many drawbacks and not enough benefits. It's slightly inconsiderate, too. Your users will either have a hard drive and get annoyed by the constant whirring and clicking caused by the game, or have a solid state drive that suffers unnecessary wear. Or they might use some form of ram based file system and then you're just wasting your time. Store your current level and whatever levels that are accessible from that location in RAM. As long as you can preload those levels in the background faster than the player can get to them then the player won't experience any extra pauses between levels.

Another good way to save space is to use delta encoding. Track whatever changes the player makes to the level and only store the data necessary to recreate that level from a previously known state (such as the original procedurally created level.) For example, in a Legend of Zelda type game that didn't regenerate monsters and obstacles after you left the area... store only the relevant information, not a copy of the entire level. Store the damage dealt to each monster instead of the entire Monster object. Store boolean values (in bit mask form to save even more space) to record which doors/chests are unlocked, instead of storing Door and Chest complete with constant meta data such as references to Key or Treasure objects. This could be in the form of a list of changes (monster 4 move left, right, up, left, up, down, left) or only the change from start to finish (m4.x+=-1; m4.y+= 1;) For most games the final changes will be easier to calculate and faster to execute.

Finally, if you decide you need to save data in a different format because a feature requires it or because you change your mind and want to incorporate manually designed levels, then look into other simple methods. Some of the best compression methods have time/memory trade offs just like the potential time/memory trade offs between static and procedurally generated content. You might have surprisingly good results using images to store landscapes and meta data as plain text and using existing general purpose compression algorithms. Or you can write binary data using a data stream like Orangy Tang suggested (+1 for DataOutputStream). However combining both methods probably is not worthwhile because your binary data may already be compact. Try putting a .mp3 file in a zip file vs putting an xml file in a zip file. The latter will be much smaller because there is a shorter way to express the same data, but the former may just get you a zip file that's larger than its original contents because MP3s are already compressed to the point where they sacrifice quality to save storage space.

Wow, what a long post! :]

The thing is, the levels are procedurally generated before the player enters them and then they persist after the player reaches them so that he/she can revisit them, thus making them sort of a "static" procedurally generated content. Thus, I would need them stored on disk inevitably, and they don't operate on a seed, per se, at least not the same one for any given level because it's a function of time. I am aware that it's not the smartest idea to have things stored on disk during the level's state in RAM; either I would implement an auto-save feature that would save to disk periodically or just save to disk when the player exits the level; either way, the level gets stored in very sparse intervals of time, and this is what I've originally had intended for this, but I think altogether that aspect of things is fairly incidental.

File loading itself is very fast in the engine right now, since the levels are relatively small and only really consist of tiles and entities (right now, level loading is instant, even from files on the fly, though with some changes I'm making to my code, I'm going to have all of the zones of a level loaded at once when the player enters rather than have an on-the-fly method like I currently have).

The way you described saving monsters and the like is smart, but it's unnecessary with the way my game is structured; all of the entities are determined upon entering the level, and this information essentially is lost when the player exits (though if a level is cleared, all of the enemies will take a long time to respawn, which is stored as a small flag value in the level file). Since enemies are dynamic, there are no delta values to be concerned about, and this information isn't even stored on disk anywhere. The locked and unlocked doors aspect will be saved on disk, but those will be, as you said, stored in simple values like booleans, but I may heed your advice and store them in bit mask form, though this may not be necessary given that they will already be stored as something simple, like one digit or character in a given location, which seems like it could end up being more trouble than the optimization is worth.

You're right about the compression, and I do intend to explore serialization methods to express all of the information in raw bytes because it sounds very tantalizing and efficient; however, this will take some getting used to and refactoring of my save methods.

(Glad I could help) I was afraid I would need to be more explicit on the seed thing. A lot of game programmers I've spoken to in just the past few weeks misunderstand how procedural content works. I think people automatically associate it with "random" content instead of a deterministic process. Everyone says "I can't do that, the numbers are different every time," or "The seed is based on time," or "No, the seed is a function of a random number and what keys the user has pressed so far." One of those programmers used an initialize_seed() function and didn't seem to grasp the idea that they could get the program to produce the same set of numbers without saving an ever expanding table of old numbers. Only one of those people shared their code with me, but it sounds like they are all using magical global/static random number generator sources. By any chance are you doing something along those lines? For example, calling Math.random() instead of using the Random class? (If not the spoiler text is a digression.) (Edit: The spoiler tag doesn't work how I thought it would.)

Conclusion: No matter how you derive your seed or what interface you use to access a pseudo random number generator, there is always some seed value even if it's hidden. (End digression)It sounds like in your game the only non-constant persistent data only includes the locked/unlocked state of each door and level clear time. Those things need to be saved on a disk (128 bits should be enough.) But it doesn't sound like you are changing tile data for example. Why does that need to be saved? If you have deformable terrain you would need a modifiable and persistent set of data (that's why it's necessary for Minecraft but not other types of games.) On the other hand, if your terrain data is constant then it is inherently persistent.

Procedural content generation is deterministic because the underlying RNG is also deterministic (see the introduction to the 'Pseudorandomness' Wikipedia article.) The first 8 ints generated by a new Random(123456) object is always 1774763047, -506496402, -41169962, 2018695370, 1083428877, 298499967, -1264165101, 1316144193 the same way "String value".hashCode() is always -1213577022 and the same reason a cryptographic hash of a password is always the same and the same reason 2 + 2 always equals 4. Am I missing any reason why even if you derive your PRNG seed from the time the user first visits an area you cannot save that time or the seed derived from that time? Even procedural level generators can be made to create the same output given the same input just like arithmetic should always produces the same results given the same expression in a given number system. Using the same input and the same recipe means you get the same results. All you need to save is the ingredients list because you already know a recipe.

And yes, I do understand that underneath everything there is a seed value no matter what; in Java, the Random class just takes the current time in (I believe) milliseconds since the creation of the UNIX operating system or some event like that, and I use the Random class for all of my randomizations; I kind of forgot this fact in my last post! However, it wouldn't matter anyway, because not only is the terrain deformable, but the levels themselves will contain other data that may tie into, say, quests given from other locations, all of which is procedurally generated, and I could see it growing unfeasible to regenerate the level every time when there are multiple variables involved which can not only be changed from the game level but from external sources in the overall game world (my game is a fantasy RPG, which is comprised both of an overworld and many levels scattered about it). If I were shooting for a basic platformer, however, then your idea would be excellent and very disk-space effective! :p

I definitely do see what you mean, though, and I apologize for not fully grasping everything you were saying in your prior post, because it's absolutely true. Thanks for your time and the information!

java-gaming.org is not responsible for the content posted by its members, including references to external websites,
and other references that may or may not have a relation with our primarily
gaming and game production oriented community.
inquiries and complaints can be sent via email to the info‑account of the
company managing the website of java‑gaming.org