Changing Sonic 2's level format

As one of the preparations for my work on Sonic 2, I'm entertaining the idea of changing the level format so that levels use 64x64 level chunks instead of the 128x128 chunks we're all used to. Now, I'm not going to create some facade of knowing Sonic 2's code like the back of my hand (I don't), which is sort of why I'm making this topic in the first place. Also note that my definition of success is the scheme working period; it doesn't need to be able to handle the levels Sonic 2 uses, since they're all going to be gone anyway. Also, if anyone knows how Sonic 3 handles level layouts, since I understand they are much larger in that game, that'd also be helpful.

The first obstacle is, naturally, RAM usage. From what I understand and see in the code, there isn't exactly a lot of free space to work with. That's sort of a bad thing considering how inflated the level layout size would be if this were implemented. Basically, every level would definitely need more than the 256 maximum chunk definitions we have to work with. Fortunately, reducing the size of the chunks will give me 1024 possible chunk combinations in the exact same memory space ($0000-$8000). However, the level layout format would then need two bytes instead of one to address each chunk. That multiplies the level layout size by 2. Then of course since each chunk takes up a smaller space, in order to create a $4000 by $800 sized level of both planes(in game XY size), you'd need 4 times the space. So instead of the original $1000 bytes in memory, we instead would need $8000 bytes to create a level of the same size using the scheme described above. Obviously, that won't do.

So far, these are the solutions I've come up with.

1. Keep the chunk data as is, and create a sort-of dynamic mapping system. Only a small portion of the level will ever be loaded at any given time. The contents of this level data would be updated in real-time depending on where the camera is located. Some issues I've had with that is that the level layout data would probably need to be uncompressed in ROM, since dynamically decompressing small sections of data isn't efficient at all. Also, I'm not sure exactly how objects work in the context of the game system, whether some of them need to know the whole level layout to do what they need to do and such. Also, I'm not sure how loading bits and pieces of the level would affect all the crazy BG effects that go on behind the scenes. Of course, the significant advantage of this scheme would be that it would allow me to create levels as big as I like.

2. For this, instead of reducing the level layout size, I would instead reduce the size of the chunk definitions in memory. Set it up so that only 256 definitions are loaded at a time in $2000 bytes of memory. This would 1) Allow the level layouts to only have to use one byte to reference a chunk, saving space, and 2) give me a whole bunch of extra free RAM to place the level data in. If my math is correct, I'd have $7000 bytes to work with, which would allow me to expand the level layout size a bit further. As far as using more chunk definitions, this could be handled in a number of ways, the foremost being an object similar to the path swappers controlling the dynamic switching of chunk definition sets. Not sure if they would need to be uncompressed for it to work without sacrificing speed.

I may be wrong on some of my observations, or I may just flat out be crazy. That's why I'm submitting this idea, to listen to any suggestions, comments, criticisms, or outright flames anyone might give me. Any help would be much appreciated

My best advice would be to look at the original code that interprets the 128x128 block mappings and figure out how it determines the size of these mappings. Once done, you can work on the math invloved that downsides the read limit for these chunks.

Then, you have to edit how the layout grid is set up for level layout. In what I believe may be a similar way of calculating the size of the block mappings, you should have it allocate for smaller size tiles by tweaking the correct part of the code.

As for the revised level layout data... I'd go with option 2. It may be more data, but perhaps a denser compression could be used to pack that down. Then again, you did say that the data may need to be uncompressed. I'm not too sure, as it isn't my forte.

Good luck in your exploits. You seem to know what you're talking about and made sure you gave specific details when explaining your problems and ideas. I like that.

First of all it is a great idea. You can simply change the size of the meta block into 64x64 tiles. It's a macro in the Sonic source that define the formula for 256 meta or 128 meta, which you can choose. But you will still lose RAM because you need more bytes to fill up the level layout.

Having the level uncompressed in ROM is another idea that I have also thought about. The only reason for having the level layout in RAM is so that you can do Labyrinth Act 3/Wing Fortress level updates, and of course having an uncompress buffer.
It will be worth it if you want to use it for Genesis only, but if you plan to port it to Sega CD (which is the next big thing 2006), then you are better off using as much compressed data as possible.

EDIT: And one more thing. If you choose to go with 64x64 meta blocks, you will have problems editing them since there is no editor supporting this format. It will be a hard task.

You'd need to rewrite parts of the graphics engine, and the collision detection. I think the RAM problem could be resolved by keeping the layouts uncompressed in the ROM, and have the game read data directly from the ROM.

First of all it is a great idea. You can simply change the size of the meta block into 64x64 tiles. It's a macro in the Sonic source that define the formula for 256 meta or 128 meta, which you can choose. But you will still lose RAM because you need more bytes to fill up the level layout.

That's a huge relief. I didn't see that in my first see-through, I'll be sure to check it out.

LOst said:

Having the level uncompressed in ROM is another idea that I have also thought about. The only reason for having the level layout in RAM is so that you can do Labyrinth Act 3/Wing Fortress level updates, and of course having an uncompress buffer.

Yeah, I was planning on having dynamic level changes, so I guess I'll have to think of some way to handle that if most of the level stays in ROM. Perhaps have all dynamic changes stored in ROM, and use the extra RAM to hold information about which level change data to be used and where?

Quote

It will be worth it if you want to use it for Genesis only, but if you plan to port it to Sega CD (which is the next big thing 2006), then you are better off using as much compressed data as possible.

Fortunately, I don't ave much interest in a project like that, though I'll take the opportunity to wish goodwill to everyone working on that. I take pride in my programming skills, but I respet people who have the patience and fortitude for that type of project

LOst said:

And one more thing. If you choose to go with 64x64 meta blocks, you will have problems editing them since there is no editor supporting this format. It will be a hard task.

No problem, I've programmed rudimentary editors before in my work on Super Metroid, I can do it again. Obviously, that's extra time spent, but if worse comes to worse (read: if I screw up) I can alsways ask Stealth for his source >_>

Hivebrain said:

You'd need to rewrite parts of the graphics engine, and the collision detection. I think the RAM problem could be resolved by keeping the layouts uncompressed in the ROM, and have the game read data directly from the ROM.

And, being who I am, I miss the simplest solution to the problem: keep everything in ROM. The one major concern I have is how the BGs will be affected. I took one look at all the BG warping code and... well, let's just say I'll get back to that >_>

Hopefully collision detection won't be too much of a problem, if all it uses is simple math to determine which collision block to use.

So if I'm correct, I can avoid loading the level into RAM if I keep it uncompressed. Obviously this will waste space, but I was planning on expanding the rom anyway

Tweaker said:

Good luck in your exploits. You seem to know what you're talking about and made sure you gave specific details when explaining your problems and ideas. I like that. smile.gif

Thank you, I look forward to more discussion on this and other subjects

As I said before, the new size of the level data if it were a simple array of two-byte chunk references, for both planes would be $8000 bytes... or 32KB in other words. Having each level in the rom uncompressed would take up quite a bit of space as I've said. From 1/2 to 3/4 a MB depending on how many levels/acts you want. This PROBABLY won't be a problem at this size.

However...

For my purposes I want the level size to be bigger. To the tune of $8000 by $1000 in most cases. Now, the problem wouldn't be with managing the extra space, it'd be with the amount of space. Expanding the level layout using the raw format would take up 128 KB per level uncompressed. That's why I'm so interested in seeing how Sonic 3 handled its level format, since it too had the same size levels, or so I presume, compressed. Now I *could* waste space by keeping my levels uncompressed, my plans could still proceed, but it wouldn't help out everyone who wants to add a million more levels to their hack :P Thus I want to reduce the size somehow. The optimal solution would be compression for the levels, with the engine being written to decompress the levels in real time. Don't know how well that would work though. The other solution I have is to employ option #2 above for the chunks, and make the levels only 64KB, but that's still a lot of space per level for a potential 4MB rom.

As I said before, the new size of the level data if it were a simple array of two-byte chunk references, for both planes would be $8000 bytes... or 32KB in other words. Having each level in the rom uncompressed would take up quite a bit of space as I've said. From 1/2 to 3/4 a MB depending on how many levels/acts you want. This PROBABLY won't be a problem at this size.

However...

For my purposes I want the level size to be bigger. To the tune of $8000 by $1000 in most cases. Now, the problem wouldn't be with managing the extra space, it'd be with the amount of space. Expanding the level layout using the raw format would take up 128 KB per level uncompressed. That's why I'm so interested in seeing how Sonic 3 handled its level format, since it too had the same size levels, or so I presume, compressed. Now I *could* waste space by keeping my levels uncompressed, my plans could still proceed, but it wouldn't help out everyone who wants to add a million more levels to their hack :P Thus I want to reduce the size somehow. The optimal solution would be compression for the levels, with the engine being written to decompress the levels in real time. Don't know how well that would work though. The other solution I have is to employ option #2 above for the chunks, and make the levels only 64KB, but that's still a lot of space per level for a potential 4MB rom.

Any other thoughts?

First just one comment on asking Stealth for source code. That's not an option.

Sonic 3 uses LZSS frame decompression. It's specially developed to decompress small segments of data during every frame. Sonic CD uses the MD compression for this (since this is the only option for frame decompression for Genesis games created before Sonic 3) to change backgrounds dynamically during the level, but it lags way too much to give a good feeling.

The problem with LZ frame is that LZ compression is so random for random data. The segments size must be computed by knowing how long you can decompress during that little period of one 1/60 second. If you decide to use MD frame compression for the meta blocks, then you may be able to do a smooth level update as long as you overlap the active camera. It's really a hack though, and the size of the level will be a lot bigger.

You have more problems though. Sonic is as developed as you can possibly get. Everytime you want to add something, you have to remove something, or else you will miss the frame.
An example: If you want to add more level drawing to the existing one, the water has to go, or the music has to go, or the frame decompression (okay that is not an option) has to go.
If you choose to keep all stuff, you will get random lag.
Just look at Sonic 2. The music driver had to be moved to Z80 completely, or else Tails and split screen would never have been possible. Sonic 3 went even futher and changed the level drawing code completely.

Ideas to make Sonic better are always popping up in our minds, but applying them without totally messing up the normal gameplay is unavoidable. You can almost always skip any hardware stuff if it hasn't already been used in a previous Sonic game.

^ he makes a good point. Sonic 2 is surprisingly close to the processing time limit... in places (especially 2 player) it already lags a little.

Reading tile data directly from ROM may be problematic due to extra time needed to read from ROM over RAM (especially the amount of reads done to assemble a screen worth of tiles). It's not to say you can't try - by all means, go for it, I'd personally love to see it, just bear this kinda stuff in mind. =P

Thanks. It's good to have these type of thoughts from people with so much experience with the code. After all, this is only an attempt, and while I really would like to create a system of this, if it turns out to sacrifice too much, then I'll just have to suck it up and work with what I have. Nevertheless, I certainly will try.

LOst said:

First just one comment on asking Stealth for source code. That's not an option.

Figured as much. A custom editor would probably better suit my needs anyway if I really end up screwing with the format.

LOst said:

Sonic 3 uses LZSS frame decompression. It's specially developed to decompress small segments of data during every frame. Sonic CD uses the MD compression for this (since this is the only option for frame decompression for Genesis games created before Sonic 3) to change backgrounds dynamically during the level, but it lags way too much to give a good feeling.

The problem with LZ frame is that LZ compression is so random for random data. The segments size must be computed by knowing how long you can decompress during that little period of one 1/60 second. If you decide to use MD frame compression for the meta blocks, then you may be able to do a smooth level update as long as you overlap the active camera. It's really a hack though, and the size of the level will be a lot bigger.

I'll do more research on both of these, see if there are concepts and techniques that'd be useful to me. Keeping the game smooth is of utmost concern.

LOst said:

You have more problems though. Sonic is as developed as you can possibly get. Everytime you want to add something, you have to remove something, or else you will miss the frame.
An example: If you want to add more level drawing to the existing one, the water has to go, or the music has to go, or the frame decompression (okay that is not an option) has to go.
If you choose to keep all stuff, you will get random lag.
Just look at Sonic 2. The music driver had to be moved to Z80 completely, or else Tails and split screen would never have been possible. Sonic 3 went even futher and changed the level drawing code completely.

I was afraid of this. Guess this almost rules out the possibility of a system that would work with Sonic 2 as is. The best I'll be able to do is attempt to juggle everything around on a very specific basis making sure that there's never too much going on at one time. That's easier said than done though. Fortunately in my case, I'll be getting rid of some stuff right off the bat (Tails and split screen, interestingly enough, are the first to go), but that's not going to fix the problem completely. Maybe I should've hacked Sonic 3 instead :P

Quote

Reading tile data directly from ROM may be problematic due to extra time needed to read from ROM over RAM (especially the amount of reads done to assemble a screen worth of tiles). It's not to say you can't try - by all means, go for it, I'd personally love to see it, just bear this kinda stuff in mind. =P

I will. Nobody said this would be easy. If it were it would've been done already. (or not? Who knows...) let's just hope I don't end up doing something to the code I'll regret.

You have more problems though. Sonic is as developed as you can possibly get. Everytime you want to add something, you have to remove something, or else you will miss the frame.
An example: If you want to add more level drawing to the existing one, the water has to go, or the music has to go, or the frame decompression (okay that is not an option) has to go.
If you choose to keep all stuff, you will get random lag.
Just look at Sonic 2. The music driver had to be moved to Z80 completely, or else Tails and split screen would never have been possible. Sonic 3 went even futher and changed the level drawing code completely.

I was afraid of this. Guess this almost rules out the possibility of a system that would work with Sonic 2 as is. The best I'll be able to do is attempt to juggle everything around on a very specific basis making sure that there's never too much going on at one time. That's easier said than done though. Fortunately in my case, I'll be getting rid of some stuff right off the bat (Tails and split screen, interestingly enough, are the first to go), but that's not going to fix the problem completely. Maybe I should've hacked Sonic 3 instead :P

Same with Sonic 3 I am afraid. Maybe even worse with Sonic 3 since so much is specially designed like level swapping and so much pattern animations. If I have to guess, Sonic 3 doesn't leave any room for people to play around in RAM. But no Sonic game does that.

When I had to choose between 256x256 metas or 128x128 metas, I had to ask myself why there were two different format to begin with.

256x256 is good because you have less work to fill out a whole level. They are bad because there are less combinations in the level. They are also bad because they take up more a great amount of RAM. But they are working! They are good because they are working and people like playing levels that uses pieces that repeate themselves more.

128x128 gives a lot more choices and designs to a level. They are good because they give free RAM that can be used to store cross-paths. Sonic 2 was first with cross-paths. It requires two sets of collision arrays that can be swapped so that Sonic can cross his old path. With cross-paths the levels get bigger. All of a sudden you can run back and forth over the same area which was impossible in Sonic 1. 128x128 has some bad things to remember, such as the level layout will become double in size. This is the same for backgrounds even if they are not using 90% of their layout info in 80% of the levels. The level layout needs to be compressed with LZ in ROM, and they will take up RAM...... So much that 128x128 and 256x256 will even out..... almost at least. The only thing you will win is cross-paths. But you will have more work designing all the meta blocks as well as layouting them, decompressing them, managing them, etc... One thing that I have a problem with, is to fit more meta blocks together. You still have the problem of finding one 128x128 block to fit another.

64x64 all of a sudden gives you the freedom to design levels with more slope combinations. Less meta blocks in RAM, and huge level layout buffers...... How much RAM you will win is unknown to me. It depend on if you want the backgrounds to update (Sonic Genesis/Sega CD style) or just wrap around (Sonic Advance style). I don't like wrapping backgrounds. It's nice but it makes the levels look so small. A great price will be the size of the level layout buffers. The other bad part is the slope connections. Now you also have to include palmtree connections, and all kinds of structures you want to make. It's a hard work but you get more variation in the level, and more design choices. The smaller the meta blocks are, the longer will it take to make a level complete.

And to include some info:
Genesis Sonic games are limited to a max speed of 16 pixels. That's why the 16x16 meta blocks have never changed in size. Why? Simply because you don't have time to draw more than one or two rows of 16x16 meta blocks as you move.

Sonic Advance series uses 12x12 meta blocks for level layout, and 8x8 tiles to update the screen. Many people in the GBA scene including me thinks 12 is an odd value for GBA. Divide must be used which is the slowest thing in GBA programming. But they are lazy when it comes to update the screen. When Sonic moves, the whole screen will update every time so that Sonic can run faster.... Well, you still don't know where you are going and end up into a wall or an enemy.

It's a great choice of design. Changing the way Sonic sees the world is as big as doing your own game from scratch.

Quote

Maybe it's possible to get the moduled compression stuff from S3 if you dissassemble it and add it onto S2?

It's possible. I have done it. I found no use of it since you can't compress your own art of meta blocks to use with it.

Very good advice LOst. I'm learning quite a bit about the whole Sonic methodology, and it's been a great help so far when I prepare for this.

LOst said:

It's possible. I have done it. I found no use of it since you can't compress your own art of meta blocks to use with it.

True. However, I am still quite interested in examining Sonic 3's code, notsomuch for the compression itself, but for the actual system it uses for all the art and meta block switching it does, since I can employ any sort of compression/decompression I like. It's clear that the only way I'm gonna pull this off is to employ a similar swapping system (will it be fast enough? Who knows until I try it). I wish Gens had some way of profiling Genesis games, though I suppose that's too much to ask for.

I've examined the code and I *think* I have a good idea as to how to do this. Just so I won't leave people in the dark, and also so that people who know more than I do would be able to tell me if this scheme has a major problem in it, I'll post the full theory here, coded for convinience:

NEW SONIC 2 LEVEL FORMAT
From my understanding, all pertinent data for levels as arranged in RAM is as so:
$0000-$8000 - 128x128 tile definitions $100 of them
$8000-$9000 - Main level layout where for each row of $100 bytes, the first $80 refers to the FG and the last $80 refers to the BG
$9000-$A800 - 16x16 tile definitions, $300 of them (although it seems the format would allow for up to $400, the data from $A800 to $B000 is apparantly used. Furthermore, the only block set in the game with more than $300 16x16 tiles is ARZ by a slight margin, and even then the tiles that do spill over $A800 are null and unused. Perhaps Sonic Team wanted $400 tiles for RAM, but found out they couldn't do so and thus had to cut down some extra tiles from ARZ? Who knows, but I digress.
Anyway, as can be seen, the big RAM hog in this scheme is the 128x128 tile definitions. So the biggest priority was trying to cut that down somehow in the move to 64x64 tiles. However, there were two more major problems: 1, the extra space that would be required to address 64x64 tiles as opposed to 128x128 ones for the same sized level, and the fact that I wanted my levels to be much bigger than the maximum $4000x$800 pixel cap in Sonic 2. Thus, I came up with the following solution:
$0000-$2000 - Primary 64x64 tile definitions ($100 of them)
$2000-$4000 - Secondary 64x64 tile definitions ($100 of them, see *A* for details)
$4000-$5000 - Level layout data square section 1 ($40x$40)
$5000-$6000 - Level layout data square section 2 ($40x$40) <--- See *B* for details of both
$6000-$6200 - Level layout section 1 Primary/Secondary tile switches
$6200-$6400 - Level layout section 2 Primary/Secondary tile switches <--- See *C* for details of both
$6400-$7400 - Exclusive 64x64 BG tile definitions ($80 of them)
$7400-$7800 - 64x64->128x128 BG tile redefinitions <--- See *D* for both
$7800-$9800 - 16x16 tile definitions, extended to $400 tiles
$9800-$A800 - Extra RAM
Now for explanations:
*A* - This scheme is used to not only lower the amount of RAM usage by the 64x64 tiles, but also the RAM usage of the level layout itself (more on that later). The primary 64x64 tile definitions are ones that are ALWAYS loaded, no matter what. They will be, obviously, used for the most common level elements. The secondary 64x64 tile definitions, however, are swapped in and out of each other in real-time. How? I haven't decided yet. it'll most likely be tags within the level itself similar to the path switcher that will move the data into RAM when crossed, probably via DMA in the VBlank routine (not all at once, but sections at a time). Regardless of how it'll be done, the big limitation will be that levels will have to be designed with this scheme in mind.
*B* - Now for the interesting part. Instead of loading the entire level into RAM, instead the level will be loaded in sections, using a sort of page fliiping scheme to keep things moving in real-time. Each page of a level is $40x$40 tiles big, which amounts to $1000x$1000 pixels of level per page. Every attempt will be made to keep ALL the level pages compressed. So how will this work?
1 2 1 2 1 2 1 2 etc
____________________________________________________________________
| | | ! ! | | | | | |
| | | ! ! | | | | | |
| | | ! ! | | | | | |
| | | ! ! | | | | | |
| | | ! ! | | | | | |
| | | ! ! | | | | | |
| | | ! ! | | | | | |
| | | ! ! | | | | | |
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
A B C D E F G H
Each section of this chart represents a section of the level (yes, I know they aren't square, this is for demonstration purposes only). Each level page is connected to each other horizontally to form an entire level. The big advantage of this is that I can make each level as big or as small as I please (up to $10000x$1000 pixels of course so I don't break the engine) in ROM. The numbers on top indicate which section of RAM each portion of the game refers to when determining tile data. It's a simple 1-2 pattern, and actually allows me to do tile position calculations a bit more easily (in theory). The letters on the bottom are just arbitrary section numbering for easy reference.
As for swapping, take a look at the lines designated by l and r. Assume sections B and C are currently loaded. In Section C, Sonic passes through the line marked by 'r'. Once that occurs,the engine will see if section D is already loaded. Since it is not, the engine will now fill the space currently occupied by section B (in other words, the section 2 RAM data) with Section D's data. How? It'll begin a sort of dynamic decompression. Instead of decompressing all $1000 bytes of data at once, it will instead decompress a small amount of data each frame, pausing the decompression each time until it's done. This means that there has to be a suitable amount of space between where Section C ends and where the point Sonic crosses to begin the decompression is. These values haven't been exactly determined yet. Now section D is loaded, but you need to go back. Once Sonic crosses the line marked by 'l' in section C, the game will reload section B into section D's space. Repeat ad infinium.
*C* - These do have to be noted though.These are the primary/secondary tile bit switches. The normal level layout sections described above use a single byte to reference a tile. But as you've noticed, there are indeed $200 64x64 tiles loaded in RAM at a time. So we need these sections of memory. Basically, both of these refer to their companion level layouts in RAM. Each bit represents whether to use the primary or seconday tiles in RAM. Each byte will take care of 8 bytes in the normal layout, conserving space. They will most likely stay decompressed in ROM since they're so small and so that they can be swapped in and out in tandem with the main level layout sections without skipping a beat.
*D* - Ah, the BGs. The layouts used for BGs will not be nearly as varied as the normal layouts. So the BG for each level is going to use makeshift 128x128 tiles. Also, each BG will be stored and accessed in ROM uncompressed (though they will most likely be segmented into pages like the normal layouts, for convinience sake). how will it work? The first area in RAM designated here is simply an array of BG exclusive 64x64 tiles. As far as referencing goes, these tiles will 'replace' the first $80 tiles in the primary 64x64 tile definitions. The last $80 are as-is. The redefinitions RAM area is how the BG is going to be able to use 128x128 tiles. Each byte in a 4-byte section will reference a 64x64 tile. If less than $80, it'll refer to the 'exclusive tiles' as mentioned earlier. If $80 or higher, it'll refer to the last $80 tiles in the primary definitions. The BG layout, therefore, refers to each of these 4-byte sections ($100 of them, so one byte per section in the BG), which are then used to determine the 64x64 tiles used to draw the BG. It's hard to explain, so if you don't understand something, ask for clarification. Also, it is probably possible to store the 4-byte sections in ROM uncompressed, but I'll see about that.
Also, as you'll note, the 16x16 tile definitions have been augmented to $400 instead of $300 to give more breathing room. Also, there is at least $1000 bytes of unused RAM that I have to decide how to use. Possibilities include an array of addresses to custom dynamic level layout changes stored in ROM. as well as a decompress buffer in-game, so I can for example switch art mid-level. Any other suggestions would be appreciated.

Longer than I wanted, but it works. I probably made several mistakes, so fire at will. I'll begin implementing it once I get an editor running to actually make all the data this scheme uses.