That’s right, Wallace can now screen scrape games while they’re being played in order to determine how well the “player” is doing. Sort of.

Here’s what the configuration for this looks like:

Intuitively, we need two pieces of information in order to screen scrape: the part of the screen where the data is, and how to convert the raw pixels into a number.

The first part of that is easy enough; load a screen dump into the viewer and draw a rectangle around the part of the screen you want. To make things easier, the current code assumes each character will be on an 8-pixel boundary, so the box “snaps” along the grid lines accordingly. (More on this later….)

The second part is a bit trickier, as we need to sort of do OCR on the image to figure out what characters are there. Luckily, part of NES video memory is the “pattern table”, where the currently loaded patterns are stored. A pattern is simply an 8×8 bitmap with a color depth of 2 bits. NES games use the pattern table as the building block for backgrounds and sprites. For example, here’s the pattern table for Super Mario Bros:

The colors are arbitrary — the actual color you see during the game is a result of combining the pattern’s two color bits with two additional color bits associated with the particular part of the screen it’s being drawn at. If you look carefully, you can see that the first set of patterns are pieces of Mario sprites.

Of course, what we’re really interested in are the patterns for each number, as those tell us what each digit will look like (modulo palette rotation) on the screen. Once we assign each digit to its corresponding pattern (assuming a base-10 number system for the time being), converting a part of the screen to a numerical value is relatively easy.

“Relatively” being the operative word here, unfortunately. For starters, I’ve only been able to figure out how to access the current contents of the pattern table from the FCEU store. For Super Mario Bros this is good enough, since it never changes. However, games with more graphics data will swap different pattern data in and out — for example, the contents of the pattern table in Mega Man 2 change depending on which stage you’re in. I think this might be done with the different mappers in NES games, but the FCEU code here is particularly inscrutable, so I haven’t figured out any way to access the “other” pattern tables without playing to the point where they’re loaded.

Also, remember how I mentioned the code assumes that scores and other numbers you want to screen-scrape occur on 8-pixel boundaries? If you look very carefully at the screen shots above, it looks like that’s the case, doesn’t it? After all, the upper-left corner of the score display starts at pixel (24, 24) (where pixel (0, 0) is the upper-left corner), right?

Well, no. Look very carefully at the patterns for each digit. The digits themselves are only 7 pixels wide, so one column of pixels on each is “blank”. This margin is on the left side of each digit.

In other words, the score actually starts at pixel (23, 24).

The code currently hacks around this by adjusting all your screen scraping bounding boxes by one pixel to the left. Which, of course, breaks pretty much any other game you try.

Plus, if you want to scrape multiple values, you need to set up an action for each separately, which is very much a pain.

In other words, the code is functional, but the interface has got to go. While the “one action per metric adjustment” sounded nice back when I made that design decision, it’d be much nicer to take one screen dump and specify all the parts of the screen we want.

Additionally, since we need pixel-perfect precision for specifying what parts of the screen should be scraped, it’d be really nice to have the program figure out the bounding rectangles for us. As a one-time sort of thing when defining an action, it could take the digit-to-pattern mapping we specify and scan the entire screen to deduce where the digits appear, assemble them into rectangles, and let those rectangles be mapped to metrics.

Just another example of not learning design-relevant issues until actually writing code. In your face, waterfall model!