Automatic Charset Extraction

I mentioned in an earlier post already, that the ToS 2 game engine on the C64 is pretty flexible regarding the handling of character sets. The character mode on the C64, which is more or less required to create games with scrolling, makes use of one set of 256 characters at any given time. In ToS 1, every level consisted of two graphic sets, one for the outside world and one for all inside rooms. These two graphic sets had each a completely independent character set.

Now with the ToS 2 engine, I decided to allow to use a different graphics set for each room (with the outside world being a “room”, too). This allows for much more flexibility when designing rooms, e.g. rooms inside wooden huts, caves or palaces all in the same level. Obviously, defining the related character sets completely independently would need a huge amount of memory. Every full character set needs 2kb, so a dozen rooms would already use more than 20kb of charset data, which is by far too much. Therefore, the character sets are assembled from segments of characters, which can be combined in a mostly arbitrary manner. The picture below shows the principle.

Originally I thought that one could place these segments of characters at any location in the character sets, but using arbitrary positions for the segments creates a problem: The tiles used in the engine reference the characters in the charset by index. If the same segment appears at different places in different charsets, all the characters in the segment suddenly get different indices. So if a tile that uses a character from such a segment is used in different rooms, the index in the tile definition becomes ambigious. This would mean that the tile has to be defined several times, for each of the different character sets. As tile definitions are comparably expensive (16 bytes per tile), this did not seem a good thing to do. So all segments have a fixed position that is valid for all charsets in which they are used.

Managing the segments and character sets looks complicated, and it is. I did not want to spend time thinking about these things while creating levels, because that can really hamper creativity. So I decided to design an automatic mechanism in the editor, that would take care of deciding about which character to put into which segment, and how to combine the segments into character sets so that they fit for the rooms and need the smallest possible amount of memory at the same time. I did not find a suitable algorithm on the Internet, although it might well be that there are some. But I had a hard time coming up with suitable search terms, as the problem to solve is rather peculiar. In the end I came up with something myself after lots of head-scratching.

Unfortunately, a brute force approach that simply tries out all possible solutions and selects the best one does not work, because even for 256 characters the possible number of permutations is in the order of 10506. And that is only all possibilities for one single segment, the number gets far bigger when all potential combinations of segments are taken into account. Even for modern PCs this is far from doable in a reasonable amount of time. Instead, the algorithm that I came up with starts with a trivial solution: every single character is put in its own segment. After that, two segments are merged to a larger segment and it is verified, if the resulting segments can still be combined to character sets such that they cover every existing room, while still staying in the limit of 256 characters. The two segments that are chosen for merging are selected after evaluating, by how many rooms they both are needed. The two segments that are used together by the highest number of rooms are the best candidates. If merging them does not result in character sets that work for all rooms, the pair of segments with the next highest number of rooms that use both of them is tried. The merging is done again and again with the best candidates, until finally no further merge can take place because the resulting character sets get too large. The picture below shows the first three iterations in an example.Having this algorithm in the editor allows for a very memory efficient storage of flexible character sets, so that I can design the rooms in a level much more interesting. This solves one aspect of the handling of more than 256 characters. The other aspect is to identify areas on the map of a single room, for which the character sets can be adapted on the fly in the background while moving on the map in the game. Obviously, I also did not want to handle this manually, so I wrote another algorithm. But more on this in the next post…