Category: code

(10/23/2012) WARNING! As Tomaitheous pointed out, there’s a problem with this board! Please refer to comments for more informations.

Yeah a new post!

Some years ago (more like a decade than a year), I bought a Turbo Stick. It’s the standard 2 buttons stick for NEC PC Engine console. Unfortunately the stick makes an awfull squealing nose. Forget 1 life run with this plastic beast. After 5 minutes all I wanted to do was to use it as a soccer ball. According to the Wataru’s PC Engine hardware list, 4 six buttons sticks were released. They are not small and given shipping costs I decided to build one. So I bought a stick and some buttons from Sparkfun. Later a friend gave me a Sanwa stick and buttons.

There are several places on the internet where to find PC Engine pad schematics. The main source are Emanuele Bettidi‘s pad schematics, PCE Wiki, NFG PC engine/TG 16 article. I first built a 2 buttons circuit with no auto fire. But there are a few games that requires a 6 buttons stick (Street Fighter 2 for example). The Avnue Pad 6 comes with auto fire and slow mode. The later is a particular ugly hack. The slow mode is acheived by toggling the pause rapidly. It’s as if you drank too much coffee and go berserk on the RUN button. Here are the guts shots and schematic of an Avenue Pad 6. I spent hours trying to make PCB board in Eagle for the “complete” version (with auto fire and slow mode). I didn’t find a local source for the DTC114Y. A replacement was found on the interweb. It was told that this component can be replaced by a 2N3904 and 2 resistors. This is rather straightforward if you look at the DTC114Y datasheet. Anyway, in the end I didn’t liked the way the board looked. The most logical step was to remove the rapid fire and slow mode (schematic). And by the way, I think I’ll never use slow mode and barely ever used auto-fire.

Here comes level names. This was the most painful part of the whole translation hacking. First, they are made of sprites. Second, there is a limited number of them. And last but not least, the table that tells which sprites to use for a given level name can’t be expanded.
The LSB of the pattern indices for the first level are stored at $5196 (file offset) and $519C for the second. After putting a read breakpoint on $*4f96 (remember that there’s usually a 512 bytes header to PC Engine roms) and $*4f9C, I ended up at $8E3F. A quick look at the code show me there is a address table stored at $8F81 where each address starts 1 byte before the LSB pattern index list. The first byte indicates the number of sprites. Here’s the complete index list:

At this point, I managed to avoid using compressed data. Unfortunatelly the whole sprites were compressed. In fact, there are 4 “decompression” schemes. The first one is a direct transfer to VRAM using the tia instruction. The second is using a simple RLE. The third one and the most tricky mixes RLE and some XOR. The last one simply copies the data to VRAM using a handcrafted loop.

The XOR is used here to maximize the number of repetitions in order to make the RLE more efficient. Well, that’s how I understood it. I didn’t make any benchmark to verify if it was really the case. The encoder was written pretty quickly. I reworked the font so that it more or less matches the ASCII set. This simplified the script encoder and the display routine.

Back to level names! I got help from a bunch of people to translate them. The last character of each level can be translated to country. Unfortunately that was far too big so I switched to “land”.

Candy Land

Toy Land

Greenery Land

Ice Land

Time Land

Water Land

Sky Land

Mirror Land

Queen Land

Here’s the actual “font” made of 16×16 sprites.
The zigzag lines on the last 16×16 sprite are there to make the compressed data fits into 504 bytes. I must also keep an empty sprite for spacing.
I had the sprite index list done pretty quickly. But when I started working on the index pointer table, I realized that there were 9 entries. 9? Did I forgot one?
Yes, I forgot one. The water land… And I didn’t have enough room to put the missing letters (bloody size limit).

After a cautious review, it was decided to rename “Greenery land” into “Green land”. As There were already a “Queen land”, there’s now enough room for the missing “Water”.
The only issue remaining was that the sprite index list is 42 bytes long, and mine was 43. As each level name ends with the same sequence I can use the last index as the beginning of the next one (each sequence starts with the length of the index list). So I swapped the sprite for “nd” with the beginning of the one whose index was equal to a sequence length, namely “To” which was at index 6. Why 6? First because it’s the length of the longest index list and also because sprite indexes are always even.

Shortly after beating the evil level names I received the first version of the translated script. Unfortunately the first tests revealed that the current text layout was unreadeable. There were only 2 or 3 words per line. So I had to crawl brack into the rom and change the text box position and size. I was only some couples of values to change.

So the 24th of July 2011, the english patch was released. Nearly 2 years after I started working on it… Anyway, some days after the release a couple of angry frenchmen complained that we didn’t release a french patch. In order to silence those annoying creeters I had to modify the font and the script encoder in order to add accents. That was not a big deal. But I let the french translators handle the level names (with some kind of sadistic grin). After some weeks of hard work, they decided to give up and use the english names. The french translation was released in December 2011.

Oh hy! I didn’t notice you were here. Please take a seat.
So I were talking about the font. As shown in the following image, the original font already contained uppercase letters.
As I wanted to add lowercases and some punctuations, I had to find where and how the font was stored in ROM. The visual editor told me that the font was stored from $7300 to $7a10 in VRAM. So under the code debugger I set an aux write breakpoint (ctrl+w) to these addresses. The breakpoint was triggered at $f7c6 on :

TIA $3740, $0002, $0020

If you look closely at the following image, you’ll notice that there are 2 similar preceding occurrences. This is it! You are looking at the piece of code in charge of decompressing the tile data into VRAM.
This code supports 4 kind of compression:

Laziness told me to avoid writing a little tool to compress the font. So I checked how many uncompresserd char were used and if it was enough to store lowercases. Hopefully there was enough room for all of them.

At this point I had nearly everything I wanted. There were no need to modify the string display routine. Most of the work done would have been done by the script encoder as the lowercases characters where not contigous. It was not a big deal. Nevertheless I wanted to check if there will be enough room for the translated texts and how much characters could be displayed per screen. Basically you had 16 characters per line and 7 or 11 line depending on the screen. This gave me an idea of the maximal script size. Unfortunately it was too big. So I looked for a fast/small/simple text compression/decompression for 6502 based cpu. I had a quick look at the romhacking.net forum. And it seems that one of the most used technique is DTE (Dual Tile Encoding). It’s a pretty simple approach. A text does not use all the ascii values. Basically you only use upper and lowercases (52 chars), numbers (10) and a dozen punctuation marks. This leaves you with 182 unused values. You can use them to store couples of characters. Bytes with values between 0 and 74 represent a single character and all the others represent a tuple. For example 182 represents ‘ab’, 183 ‘en’ and so on. Here is quick example: Let’s compress the string “abcacbabcabcabba”. First we extract all the characters couples and count them:

ab : 4

bc : 3

ca : 3

ac : 1

cb : 1

ba : 2

bb : 1

“ab” is the tuple with the highest occurence. We insert it in the symbol table: a,b,c,d=’ab’. And let’s replace it in the original string:“dcacbdcdcdba”. DTE only works on the original symbols. So in the second pass we must avoid the newly inserted symbols. The remaining tuples are :

ca : 1

ac : 1

cb : 1

ba : 1

The new symbol table is : a,b,c,d=’ab’,e=’ca’,f=’cb’,g=’ba’. In the end “abcacbabcabcabba” becomes “defdcdcdg”. The tricky part is to choose the right substitution sequence that will lead to the smallest possible string. It’s a whole new game and you can stick with choosing the tuple with the highest occurence.
Let’s get back to Alice. I grabbed the infamous lorem ipsum and made a script out of it. I compressed it with DTE. And I still had a lot of unused values. Remember the second pass of our example when I excluded the newly inserted symbol for the occurence counting. If I included it, DTE would in fact have been what is called BPE (Binary Pair Encoding). Here’s the mandatory academical reference : Byte Pair Encoding: A Text Compression Scheme That Accelerates Pattern Matching (1999). Using BPE we would have had for “dcacbdcdcdba”.

dc : 3

ca : 1

ca : 1

ac : 1

cb : 1

bd : 1

cd : 2

db : 1

ba : 1

Let’s add “dc” in the symbol table a,b,c,d=’ab’,e=’dc’. We now have “eacbeedba”. BPE is basically a recursive algorithm. So when we are decoding a string, we must check if it’s part of the tuple symbols (let’s call them “special” symbols). If that’s the case we will have to loop until the decoded symbol does not contain any special symbol. If speed is a concern we may limit the depth of recursion. This means that when we must take the depth of each symbol into consideration. For example imagine we have 2 tuples “dc” and “ca”. Both have the same number of occurence in the string. We may choose “ca” instead of “dc” because it only contains “base” symbols.
Using BPE the “maximal” script was small enough to fit. I made a silly test using some part of Hamlet (here’s the script). You can find the BPE compression/decompression as long as code for generating IPS files in this [archive].

Next : Levels names. Shitty part : they are sprites. Describe pointer tables and all.
All compressed. So I can’t avoid it anymore. Describe the compression scheme.
Got help for level name translation. (show the level names and their translation).
As I have a compressor, reworked the font so that it more or less match ascii. Had to spot where the font was rendered to vram. (link to final font). Simplify script encoder and display routine.

Or more humbly, the awesome Märchen Maze translation postmortem. It’s more or less a compilation/addendum for this pcedev forum [post].

Märchen Maze is a PCEngine HuCard game released by Namco the 11th of December 1990. It’s a conversion of the 1988 arcade game. It was also ported to the almighty Sharp X68000. As you may have guessed the X68000 version is a near perfect conversion of the original arcade game. It features some jolly 3D isometric view and colourfull introduction. Well, things are not that fancy on the PCEngine version. The 3D isometric view was changed to a top view (à la Mercs or Commando). And the intro is now in sepia tones. What about the story and gameplay? Check the Video Game Den review [here].

Ok! Back to business.
The PCEngine has some neat RPGs. Unfortunately only a few of them were translated into the idiom of the perfidious Albion (English). This was a perfect excuse for trying out that black art called romhacking. Monster Maker was my first choice. I must admin that cdrom games are not a really good choice for your first romhacking experience. A HuCard game is far more simpler to hack. So I went for Momotarō Katsugeki. It’s a cool platform game with tons of text. Unfortunately the text is displayed top to bottom and from left to right in bubbles. It was clearly out of my league. So I asked some people if they knew some easily translatable games. I can’t remember who suggested it. But Märchen Maze was designated as it doesn’t have too much text and no apparent string compression.

The first thing I had to do was to find where the text was stored in the ROM. After hours of pain, David Shadoff took pity. And like a mighty wizard, he managed to extract the whole script in no time. So I had the rom location of the strings. All I had to do was to put read breakpoint in Mednafen. I was able to locate the text output routine ($87D6). The string “parsing” was spotted from $89E5 to $8CD8 using step-by-step debugging starting from the the output routine. It made me realized that $5C,$5D held the pointer to the string to be displayed. The parsing routine was called from $E377 using jmp ($202C) instead of jsr. The return address was pushed by hand onto the stack. Here’s an snippet of the parsing routine. $ff is a special value. More on this later.

l_89e2: LDY $271b
LDA [$5c], Y
CMP #$ff
BNE l_89ee

The most important information was that I now knew the zero page location holding the string pointer. Setting a write breakpoint to $5C:$5D led me to $8C4B.

So $5c:$5d is set using the data pointed by $52:$53 indexed by $2716. Let’s put $2716 aside. This is clearly the string index. So $52:$53 contains our string pointer. And the preceding lines tells us that the string table address is stored at $560C. A quick revealed that the string table was stored from $5614 to $5849. If we translate the address, it’s at the file offsets $11814 to $11A49 (this is with the annoying 512 bytes offset). Now that I had the string pointer table, I could make it point to whenever I want (technically). So the translated texts may not have to fit into the original string locations. I checked that I got it right by swapping the pointer table entries. And as expected the intro started with the second text and so on.
A quick look at the tile vram showed me that the font already contains capital letters. So I decided to manually edit the ROM and managed to get this:

The next step was to modify the font in order to have lowercase letters.
To continue…

This is what is supposed to be the shortest post on this blog.
Remember last post where I used __builtin_ctz to compute GCD? Well… Here’s a way to compute the log2 of an integer using another gcc builtin. Let me introduce __builtin_clz. clz stands for count leading zeroes. So this wonderful thing counts 0 bits starting from the most significant bit. Please note that the result is undefined for 0.

Some weeks ago I decided to fix once and for all this bloody hdrr. As it requires the framebuffer average log luminance, I decided to compute it be generating a pixel accurate mipmap. So for a 320×240 framebuffer we will have the following downsampling steps : (4,4) (4,4) (5,5) (4,3).
Don’t ask me why but instead of working on a way to decompose a number, I decided that I needed to find the greatest common divisor of width and height! 10 seconds later thanks to the almighty web search engine I started reading the Wikipedia entry about Binary GCD. And after some interweb jumps I ended on this article. The Stein’s algorithm looked cool.
If you look at Sputsoft code, you’ll realize that the shift_to_uneven function removes trailing zeros. This can be optimized using GCC builtins. The builtins we need is __builtin_ctz that counts trailing zeros.
We can then replace shift_to_uneven by:

I spent the whole afternoon trying to make the arduino works with 2 dataflash chips. A coupe of really stupid code mistakes and some weird stuffs later, it worked. You can now have 2 instances of ATD45DB161D class. For example one will use the pins 2, 4 and 5 for SS, RESET and WP and the other one will use pins 10,8 and 7.
I couldn’t make SS work with an other pin than the pin 10 until I ran across forum post. So I have to set both the requested SS pin and pin 10 (hardwired SS pin) high before setting the SPCR register so that the atmega168 becomes master.
So I moved the SPCR initialization to a new method astutely called Enable. And I added its counterpart called Disable which drives the SS pin high.
So before using a given Dataflash chip, you’ll have to call Enable before using it. And call Disable when you are done with it.

As I’m not happy with the current design, I created a new branch for this dev.

You can find it [here].
Any ideas/suggestions for a cleaner/safer way to handle multiple SPI device are welcome 🙂

[edit]: Arg, I should have checked Atmel site. They have an application note called AVR151: Setup And Use of The SPI.[edit] (it’s the last one. I swear 🙂 ) I moved the SPI master init into a new function (spi_init). It’s a little bit cleaner now. But it’ll stay in a branch until I run more tests and have some feedback.

Some of the stuffs I posted here are now on [GitHub]. Namely, the [arduino dataflash library] and the [ips patcher]. Feel free to clone them 🙂
I’m taking a little break from OpenGL stuffs and the pcengine tracker. To relax I’m working on Marchen Maze translation. I’m nearly done with the introduction. I’m currently working on the insertion software. But I need to find someone to translate the script… Anyway you can check my progress on [pcedev].

It’s not that I had nothing to write. On the contrary, I have a lot to post. I just didn’t found the time to write… Erm.. Let’s go with a 7 months old stuff.

While working on the tracker, I wanted to add some kind of “live” mode. The user could choose which sequence to play, modify it, control volume, add an effect etc… So I came with the idea to associate a command to a key combo.

Combos are detected by walking down a tree. Each time a button is pressed, we look at the current node and check if one of its children is associated to this button. If that’s the case, this child becomes the new current node. Otherwise there’s no matching combo and we reset the current node to the tree root. If the button pressed didn’t not changed for a certain amount of time (timeout), we check there’s a child corresponding to the current button. This way, we can detect combo containing a sequence like “press A button during t seconds”.

The root is always at index 0 and we associate the key value 255 to it.
The tree traversal algorithm is very simple. For a given node i, we know that its children are at indices j from combo_tree[i] to combo_tree[i+1]-1. And that we can reach them only if the current key is combo_key[j]. If the bit 7 of a value in the children list is set, this means that we have a transition a leaf. The remaining bits are used as an index to a jump table.
The traversal looks like this:

i // current node index
for(j=combo_tree[i]; jCOMBO_DEFAULT_DELAY in combo.asm). Note that in current code combo_tree, combo_keys and combo_jumpTable are in RAM. So you’ll have to initialize them before-hand.

Some months ago, I bought a 4DSystem’s µDRIVE-uSD-G1 (microDRIVE) from Sparkfun. It’s basically a serial module which let you access a microSD memory card (from 512Mb to 2Gig). You may wonder why use an external module as SD cards have a SPI interface? The main reason is … laziness. All file-system operations are performed by the module. So why bother coding fat16 support or whatever? Unfortunately, the current microDRIVE firmware only supports raw access.

Anyway, I wrote a small module for the current firmware. It was implemented as a template in order to support various serial implementations. The serial class must provide the following methods :

uint8_t read()

void print(uint8_t)

int available()

It’d have been better if all the serial implementations derived from a same interface. But as I didn’t want to modify the base Arduino code (and as i said earlier, because I’m lazy), I went for the easy/brutal way. And you’d probably never have more than one microDRIVE. At least you want raid 😀

I tested Arduino‘s HardwareSerial and SoftwareSerial, as long as ladyada’sAFSoftSerial. None of the software implementations work 😐 Only soft read is working (hardware tx was used). As software serial is working for the serial LCD, I think the problem might be the autobaud detection during microDRIVE initialization… Or not…