Category: Technical Stuff

I’ve wanted to monitor room temperature and humidity in my home for quite some time and I recently came across the ESP8266 Wi-Fi microcontrollers that were ideal for the job. Prior to the ESP8266 becoming mainstream, I was considering running a 1wire network over spare phone wires, similar to this weather station setup.

The ESP8266 chip itself includes a Tensilica Diamond Standard core, although documentation is a little sparse. Fortunately, setting up a toolchain is relatively straight forward, and there is a (proprietary) SDK available that allows custom firmware to be built and flashed onto the modules.

Hardware

The hardware I threw together is very simple, and hooks up a DS18B20 1wire temperature sensor and a DHT11 temperature and humidity sensor up to the two GPIOs hooked up to pins on the module. Since I wanted a handful of the the boards, I opted to design the circuit in Fritzing and got a pack of boards from DirtyPCBs. The schematic and board are below:

The schematic is straight-forward; 5V comes in from a USB socket and goes through a 250mA fuse and gets dropped to 3.3V by a TS1086CZ or AMS1117 module. The switch S1 allows the GPIO pin to be grounded to program the module, with the UART pins broken out to a separate header.

Software – ESP8266

The software is quite simple. Using sprite_tm’s webserver code and building on Martin’s additions for the DHT sensor, my code adds a DS18B20 driver and support for the cheaper DHT11 sensor.

In addition to serving the sensor values over HTTP, the code periodically sends a JSON packet over UDP to a logging host (a Raspberry Pi) which stores it in a MySQL database and an rrdtool database.

I noticed after a few days of running, the ESP boards can get quite warm. Although I don’t think it affected the sensors, it seemed sensible to avoid leaving the ESP8266 online if it only needed to periodically log data. Fortunately, there is a “deep sleep” function in the SDK, but unfortunately this requires the RESET pin to be soldered to the RTC pin to trigger wakeups. With the chips spending most of the time asleep, they don’t get warm at all.

Software – Logging

A python script listens on the UDP port for incoming packets, deserialises them, and logs them. The readings from the DHT sensor are susceptible to noise (possibly a bug in the driver), so the script includes logic to reject samples that are outside a window.

RRDTool expects updates for every value at once, so it’s not possible to use a single RRD to store the data from multiple hosts as it comes in at different times. Instead, each host needs its own RRD, so I use a script to create them as I add hosts to the network.

Software – Visualising

I initially used a Shiny app to load the data from the SQL database, but this became unworkable quite quickly; without a lot of spare RAM, MySQL resorts to creating temporary tables on the SD card which is tragically slow.

For quick at-a-glance readouts, I switched to RRDTool. Although it’s not as pretty as ggplot2, it’s lightweight enough to render graphs regularly and serve them as static images, which fits far more comfortably on the Raspberry pi. The script just iterates over each host and plots it for various intervals, saving the images to a webserver’s directory.

Bill of Materials & Cost

PCBs: $14/12 – £0.78ea

USB B Connector: £2.51/5 – £0.50ea

Fuse holder: £1.29/10 – £0.13ea

Fuse: ~£0.10ea

ESP8266: £8.36/4 – £2.09ea

DS18B20: £4.99/5 – £1.00ea

DHT11: £3.04/5 – £0.61ea

AMS1117 module: £2.62/5 – £0.52ea

Switch: £1.99/5 – £0.40ea

4.7kR resistor: 2x~£0.02ea

Project box: £3.36/5 – £0.67ea

That works out at approximately £6.84 (amortising the cost of the PCBs), or £7.93 each for the 5 nodes I built. I’m ignoring the cost of power supplies as I have at least one USB power source and cable in the rooms I’m monitoring. Given there are more GPIOs on board to the ESP8266, there’s also scope for adding more sensors to them later on.

Vircurex is an online exchange for various cryptocurrencies. They have an accessible API which is easily queried with a small bit of R code.

Get Data

Pulling data from HTTP URLs is straightforward in R. HTTPS support is a slightly different story, but the RCurl package easily solves the problem, using the getURL function. Querying the various endpoints becomes fairly straightforward:

Almost all API calls follow the same sequence; build a query string and send it. R provides some interesting reflection capabilities that make implementing this particularly easy. Firstly, match.call() gives the call for the function (name and arguments). In addition, named arguments can be extracted using formals(). Between those two functions, we can determine argument names and values dynamically. These will be the parameters used in the query.

This defines a function, query_trade that calls out to the Vircurex API (via request). Since most of the API calls take the same arguments, we can reuse this function for most of the calls by abusing sys.calls() which gives access to the call stack.

Visualising MtGox’s impact on BTC prices

Of the various API calls supports, the trades function gives all the transactions completed in the last 7 days for a given currency pair. The fromJSON function returns a list of lists which need to be munged into a data frame. The easiest way to do this is to convert it to a matrix, then to a data frame.

I use JGAP as part of my Crunch clustering tool. For my thesis writeup, I needed to describe the specific parameters to the genetic algorithm it used. This post details how JGAP starts up, and where to find the defaults, what they are and what the values actually mean. These values are probably specific to the version I’m looking at (3.62), although they haven’t changed in roughly 4 years at the time of writing.

Startup

The search parameters are contained in a Configuration object. The configuration is initialised with various parameters and the type of chromosome to produce. The Genotype represents the population, and exposes the GA component to the user (via the evolve and getFittestIndividual methods).

Genotype

Genotypes can be bootstrapped from a Configuration via the randomInitialGenotype method. This constructs the population (which uses randomInitialChromosome to produce random individuals) and returns the initialised Genotype object.

Genotype.evolve()

evolve has three overloaded methods, the two that take arguments internally call the no-argument version.

The evolve method simply obtains an IBreeder object from the Configuration and uses it to construct a new Population, before setting the current population to the newly constructed version.

IBreeder

IBreeders coordinate the evolution step, however they do not actually perform or store any of the genetic operators. These are kept in the Configuration object’s m_geneticOperators field.

GABreeder

The default Breeder used is the GABreeder. The following stages are applied to the population:

Fitness Evaluation

The fitness of each individual in the population is first calculated, using the bulk fitness function, or via updateChromosomes if one isn’t provided.

Natural Selectors (Before Genetic Operators)

Each of the NaturalSelectors registered for application before the genetic operator chain runs is run on the population. The default configuration does not set any of these selectors.

Genetic Operators

Each of the genetic operators registered in the configuration is run, one at a time, in order of addition to the Configuration. Implementation for this step appears in BreederBase.applyGeneticOperators. The genetic operators implement an operate method, which runs over the population. These may only add to the population, however; an operator should never modify an individual in the population. A List of IChromosomes is passed as well as a Population, although this is just a reference to the Population‘s internal list.

DefaultConfiguration uses a Crossover operator, followed by a Mutation operator. Parameters to these are given at the end of this post.

Evaluation

The population is then re-evaluated, as it contains new individuals not present before the first round of selection.

Natural Selectors (After Genetic Operators)

Finally, the new population is produced by applying a NaturalSelector to it. If the selector selects too few individuals, JGAP can optionally fill the remainder with random individuals – change MinimumPopSizePercent to turn this on, but it’s off by default.

The Default JGAP Configuration

DefaultConfiguration sets up the following operator chain:

Genetic Operators

Desired Rate: 12. The rate for this one is a little different; mutation occurs if a random integer between 0 and the rate (12) is 0, thus it equates to a probability of 1/12. This operation is applied to each gene in each element of the population (excluding those produced by crossover), so the rough likelihood of a mutation being applied is (1/rate) * popSize * chromosomeSize. For each gene that is to be mutated, a copy of the whole IChromosome that contains it is made, with the selected Gene mutated via its applyMutation method.

Natural Selector

One selector is defined by default, which is the BestChromosomesSelector. This selector is given the following parameters:

Original Rate: 0.9. 90% of the population size allotted to this selector will actually be used. The selector is elitist, so this will return the top populationSize * 0.9 elements. The remaining 10% is filled by cloning the selected elements, one by one, in order of fitness until the population size reaches the limit.

Configuration Parameters

The configuration specifies that the pool of selectors will select 100% of the population size. However, the only selector is configured to produce only 90% of the population size each time.

keepPopulationSizeConstant is also set to true, which dictates that the population is trimmed to size at each iteration. Somewhat counter-intuitively, this does not cause the population size to be increased if it falls below the user supplied value; it doesn’t enforce a minimum, only a maximum

minimumPopSizePercent is set to zero, so random individuals will never be used to fill the population if it declines below the limit (won’t ever happen under these conditions).

So What is the Default?

The default configuration is an elitist ranking selector that clones the top 90% of the user-specified population size, repeating it to fill the rest. Crossover is random point with a rate of populationSize * 0.35. Mutation is random and applied at to 1 in 12 genes in the whole population (i.e. the rate is dictated by chromosome size * population size / 12).

I recently brought back an old mini ITX box which had an unsupported Ubuntu version on it (last booted well over a year ago). During the upgrade process from Maverick to Natty, one of the scripts asked if I wanted a new /etc/sudoers file. Stupidly, I assumed that my user was in the correct group and took the new one.

On rebooting it turned out that my choice was unwise – I didn’t have sudo access nor did I have the root password for recovery. Unfortunately, the trick of using /bin/bash as an init replacement to get a root shell didn’t work either (it’s a common problem).

The fix was to write a small C program which just executed a script with /bin/sh to replace the sudoers file:

Now, at the GRUB boot menu, edit the first kernel entry. Instead of using init=/bin/sh, init=/home/mat/replace can be used which will launch the script and overwrite the old sudoers file. On rebooting, you’ll have your sudo privileges back.

Early Game Boy cartridges that save games use battery-backed SRAM cartridges to hold data between plays. It’s recently arisen that the batteries in them are expected to start dying soon. Critically: a lot of Pokemon from a lot of childhoods are facing extinction.

Luckily, thanks to the Pandocs [1] and a lot of patience from other developers, people have started building their own dumpers. This post details another implementation, built for PIC16 devices instead of the existing Arduino and AVR implementations. It is heavily based on the InsideGadgets Arduino GBCartRead project [2].

There's a ROM dumper under this rat's nest, honest.

The Hardware

I built my dumper around a PIC16F690. It’s got just enough pins to drive a couple of shift registers, a UART and to leave enough pins left over for I/O. The most difficult part to get is the cartridge connector – a donor Game Boy is required unfortunately.

The schematic is fairly straightforward. The PIC communicates with the PC via a MAX232. The address lines are driven via two 74HC594 shift registers. Data and control pins take up the remaining pins. Power is supplied via a LM7805 linear regulator (not shown).

The cartridge requires a connection to GND and +5V, as well as a pull-up resistor from +5V to the /RST pin (30)

The Software

Game Boy cartridges are fairly simple devices. They use a 32-pin connector which consists of power, three control lines, a 16-bit address bus and an 8-bit data bus. The cartridge shares the address bus with other devices and is mapped to the region 0x0000 to 0x7FFFF [3].

Talking to a cartridge is fairly simple:

Write the address you want to read or write to the address bus

(optional) Write the data you want to write to the data bus

(optional) Raise MREQ if you’re reading SRAM (Pokemon live there)

Raise RD or WR to read or write respectively

Drop RD, WR and MREQ

(optional) Read the data from the address bus

The WR, RD and MREQ pins are all active high.

Cartridges larger 16KB require bank switching which is handled by a device called the memory bank controller (MBC). The MBC also controls access to the SRAM, as well as any other peripherals on the cartridge.

The MBC is controlled by writing commands to specific addresses. These addresses vary between MBCs, although most seem to enable RAM when 0xA0 is written to 0x0000. The MBC type is readable from the cartridge descriptor which is at 0x0134-0x0148 [1].

The firmware for the PIC is available at GitHub [4].

Rescuing the Pokemon

To use the dumper just connect to it via the serial port. The interface is fairly straight forward. It waits for confirmation before reading the descriptor, it’s pretty straight forward from there. The most important mode to use first is the diagnostics mode, which needs to be run without a cartridge connected.

On the Importance of Testing

The diagnostics mode is ideal for catching faults before they have real implications.

Faults like this one. In this case the WR pin hadn’t been properly soldered. Since the pins are active low and they’re connected via pull-down resistors. That means that the WR signal was 1, so the attempt to read RAM ended up writing garbage to it. I euthanised my Pokemon.

Dumping Data

Retrieving data (hopefully having not destroyed it) can be done using RealTerm [5]. It features true raw logging which is necessary, as, for example, PuTTY interprets some control characters, even in raw logging mode.

Future Work

At the moment the interface is a bit clunky, as is having to use RealTerm to dump things. I’ll be changing it to speak the same protocol as the original Arduino GBCartRead [2] which will make dumping things easier.

ApectJ is a project to bring aspect oriented programming to Java. Sidestepping the whole issue around whether or not aspect-oriented programming is a good idea; it can be used to insert code at points you define and is a very useful tool for dynamic program analysis.

Some terminology needs explaining here before we go further. AspectJ defines a point in the source to stop execution and pass it to your code as a “Pointcut”, the actual code executed on this pause is termed “advice”. Although I use the word “pause” it’s not strictly correct; AspectJ “weaves” the advice code for matching pointcuts at compile time. Additionally (and perhaps most significantly) AspectJ can weave on class load time (“Load Time Weaving”), this is what we’ll be using.

So we have a way of inserting code in places in an existing code base, what can we do with it? A lot of horrible things (playing around with returned values and parameters to name a few); but for now we’ll stick to what we want to do, tracing method calls.

The most obvious way to instrument a program is to insert calls to some logging library at the top of every method. Although this works it takes a lot of time, either manually logging everything or figuring out how to use a regex/parser to do it for you. With AspectJ we can simply weave code that logs the current method at every method execution point.

So how do we do that? First we need to get AspectJ working; Eclipse will do that for you, alternatively you can do everything from the command line (assuming you have AspectJ installed, including the ajc command on your PATH).

We’ll define an Aspect that defines a pointcut on every method execution, as well as some advice to run when they turn up when the code is executed.

So what exactly does this do? Firstly, the pointcut traceMethods() defines a new pointcut called traceMethods. This pointcut matches execution of every method in every class, as long as the control flow isn’t in the current class (Trace). The latter constraint is to stop an infinite loop occurring.

The before(): part of the class defines advice. This is the code that gets inserted just before the execution of the method. Advice can also be given after a pointcut is hit (using the after keyword instead). Our “advice” doesn’t modify execution flow, it just logs some information about the state of the program when the pointcut was hit, but it could quite easily start modifying control flow.

This compiles the aspect and puts it into a jar ready for use. The -outxml parameter will cause ajc to automatically generate an aop.xml file and save it in the jar’s META-INF directory. The load time weaving agent will read this to determine which aspects to weave with the classes it loads. The aop.xml file can do more than that, it can be used to set namespaces to ignore when weaving (such as java.*) to avoid mucking with the control flow of libraries.

So now we have our aspects.jar how do we use it? The AspectJ weaving agent must be available somewhere on the filesystem (aspectjweaver.jar) and your aspect.jar (and the target application) must be on the classpath.

Once that’s sorted to trace the application, use

java -javaagent:<path to aspectjweaver.jar> -cp <path to aspects.jar>:<path to target jar/folder> <name of main class to run>

Hopefully that will run and you should see a large amount of console output:
INFO: Call from main.RunFile line 206 to main.RunFile.main
Mar 31, 2011 2:52:53 PM aspects.Trace ajc$before$aspects_Trace$1$b314f86e
INFO: Call from main.RunFile line 186 to main.RunFile.runList
Mar 31, 2011 2:52:53 PM main.RunFile main
INFO: Starting clustering of 0 files
Mar 31, 2011 2:52:53 PM aspects.Trace ajc$before$aspects_Trace$1$b314f86e
INFO: Call from main.ExperimentRunner line 59 to main.ExperimentRunner.runExperiments

AspectJ lets you do far more than this, it’s an understatement to say this is all it can be used for. Using LTW does make for some interesting possibilities, one obvious use would be monkey patching existing projects to fix bugs where source isn’t available (and the license permits it, of course).

AspectJ is an interesting piece of work; the official project page has far more resources on the types of pointcuts you can define and the more nitty-gritty details of making it mess with the traced program.