Musings and Projects from Kevin Cuzner

One of the things that has intrigued me for the past couple years is making embedded USB devices. It’s an industry standard bus that just about any piece of computing hardware can connect with yet is complex enough that doing it yourself is a bit of a chore.

Traditionally I have used the work of others, mainly the V-USB driver for AVR, to get my devices connected. Lately I have been messing around more with the ARM processor on a Teensy 3.1 which has an integrated USB module. The last microcontrollers I used that had these were the PIC18F4550s that I used in my dot matrix project. Even with those, I used microchip’s library and drivers.

Teensy 3.1

Over the thanksgiving break I started cobbling together some software with the intent of writing a driver for the USB module in the Teensy myself. I started originally with my bare metal stuff, but I ended up going with something closer to Karl Lunt’s solution. I configured code::blocks to use the arm-none-eabi compiler that I had installed and created a code blocks project for my code and used that to build it (with a post-compile event translating the generated elf file into a hex file).

This is a work in progress and the git repository will be updated as things progress since it’s not a dedicated demonstration of the USB driver.

The github repository here will be eventually turned in to a really really rudimentary 500-800ksps oscilloscope.

This site explained very clearly exactly what was going on with USB. Coupled with my previous knowledge, it was almost all I needed in terms of getting the protocol.

The Freescale K20 Family and their USB module

The one thing that I don’t like about all of these great microcontrollers that come out with USB support is that all of them have their very own special USB module which doesn’t work like anyone else. Sure, there are similarities, but there are no two exactly alike. Since I have a Teensy and the K20 family of microcontrollers seem to be relatively popular, I don’t feel bad about writing such specific software.

There are two documents I found to be essential to writing this driver:

The family manual. Getting a correct version for the MK20DX256VLH7 (the processor on the Teensy) can be a pain. PJRC comes to the rescue here: http://www.pjrc.com/teensy/K20P64M72SF1RM.pdf (note, the Teensies based on the MK20DX128VLH5 use a different manual)

It needs a specific memory layout. Since it doesn’t have any dedicated user-accessible memory, it requires that the user specify where things should be. There are specific valid locations for its Buffer Descriptor Table (more on that later) and the endpoint buffers. The last one bit me for several days until I figured it out.

It has several different clock inputs and all of them must be enabled. Identifying the different signals is the most difficult part. After that, its not hard.

The module only handles the electrical aspect of things. It doesn’t handle sending descriptors or anything like that. The only real things it handles are the signaling levels, responding to USB packets in a valid manner, and routing data into buffers by endpoint. Other than that, its all user software.

The module can act as both a host (USB On-the-go (OTG)) and a device. We will be exclusively focusing on using it as a device here.

In writing this, I must confess that I looked quite a lot at the Teensyduino code along with the V-USB driver code (even though V-USB is for AVR and is pure software). Without these “references”, this would have been a very difficult project. Much of the structure found in the last to parts of this document reflects the Teensyduino USB driver since they did it quite efficiently and I didn’t spend a lot of time coming up with a “better” way to do it, given the scope of this project. I will likely make more changes as I customize it for my end use-case.

Part 1: The clocks

The K20 family of microcontrollers utilizes a miraculous hardware module which they call the “Multipurpose Clock Generator” (hereafter called the MCG). This is a module which basically allows the microcontroller to take any clock input between a few kilohertz and several megahertz and transform it into a higher frequency clock source that the microcontroller can actually use. This is how the Teensy can have a rated speed of 96Mhz but only use a 16Mhz crystal. The configuration that this project uses is the Phase Locked Loop (PLL) from the high speed crystal source. The exact setup of this configuration is done by the sysinit code.

The PLL operates by using a divider-multiplier setup where we give it a divisor to divide the input clock frequency by and then a multiplier to multiply that result by to give us the final clock speed. After that, it heads into the System Integration Module (SIM) which distributes the clock. Since the Teensy uses a 16Mhz crystal and we need a 96Mhz system clock (the reason will become apparent shortly), we set our divisor to 4 and our multiplier to 24 (see common.h). If the other type of Teensy 3 is being used (the one with the MK20DX128VLH5), the divisor would be 8 and the multiplier 36 to give us 72Mhz.

Every module on a K20 microcontroller has a gate on its clock. This saves power since there are many modules on the microcontroller that are not being used in any given application. Distributing the clock to each of these is expensive in terms of power and would be wasted if that module wasn’t used. The SIM handles this gating in the SIM_SCGC* registers. Before using any module, its clock gate must be enabled. If this is not done, the microcontroller will “crash” and stop executing when it tries to talk to the module registers (I think a handler for this can be specified, but I’m not sure). I had this happen once or twice while messing with this. So, the first step is to “turn on” the USB module by setting the appropriate bit in SIM_SCGC4 (per the family manual mentioned above, page 252):

C

1

SIM_SCGC4|=SIM_SCGC4_USBOTG_MASK;

Now, the USB module is a bit different than the other modules. In addition to the module clock it needs a reference clock for USB. The USB module requires that this reference clock be at 48Mhz. There are two sources for this clock: an internal source generated by the MCG/SIM or an external source from a pin. We will use the internal source:

C

1

2

SIM_SOPT2|=SIM_SOPT2_USBSRC_MASK|SIM_SOPT2_PLLFLLSEL_MASK;

SIM_CLKDIV2=SIM_CLKDIV2_USBDIV(1);

The first line here selects that the USB reference clock will come from an internal source. It also specifies that the internal source will be using the output from the PLL in the MCG (the other option is the FLL (frequency lock loop), which we are not using). The second line sets the divider needed to give us 48Mhz from the PLL clock. Once again there are two values: The divider and the multiplier. The multiplier can only be 1 or 2 and the divider can be anywhere from 1 to 16. Since we have a 96Mhz clock, we simply divide by 2 (the value passed is a 1 since 0 = “divide by 1”, 1 = “divide by 2”, etc). If we were using the 72Mhz clock, we would first multiply by 2 before dividing by 3.

With that, the clock to the USB module has been activated and the module can now be initialized.

Part 2: The startup sequence

The Peripheral Module Quick Reference guide mentioned earlier contains a flowchart which outlines the exact sequence needed to initialize the USB module to act as a device. I don’t know if I can copy it here (yay copyright!), but it can be found on page 134, figure 15-6. There is another flowchart specifying the initialization sequence for using the module as a host.

USB0_USBTRC0|=0x40;//a hint was given that this is an undocumented interrupt bit

//6: Enable USB reset interrupt

USB0_CTL=USB_CTL_USBENSOFEN_MASK;

USB0_USBCTRL=0;

USB0_INTEN|=USB_INTEN_USBRSTEN_MASK;

//NVIC_SET_PRIORITY(IRQ(INT_USB0), 112);

enable_irq(IRQ(INT_USB0));

//7: Enable pull-up resistor on D+ (Full speed, 12Mbit/s)

USB0_CONTROL=USB_CONTROL_DPPULLUPNONOTG_MASK;

The first two steps were covered in the last section. The next one is relatively straightfoward: We ask the module to perform a “reset” on itself. This places the module to its initial state which allows us to configure it as needed. I don’t know if the while loop is necessary since the manual says that the reset bit always reads low and it only says we must “wait two USB clock cycles”. In any case, enough of a wait seems to be executed by the above code to allow it to reset properly.

The next section (4: Set BDT base registers) requires some explanation. Since the USB module doesn’t have a dedicated memory block, we have to provide it. The BDT is the “Buffer Descriptor Table” and contains 16 * 4 entries that look like so:

C

1

2

3

4

typedefstruct{

uint32_t desc;

void*addr;

}bdt_t;

“desc” is a descriptor for the buffer and “addr” is the address of the buffer. The exact bits of the “desc” are explained in the manual (p. 971, Table 41-4), but they basically specify ownership of the buffer (user program or USB module) and the USB token that generated the data in the buffer (if applicable).

Each entry in the BDT corresponds to one of 4 buffers in one of the 16 USB endpoints: The RX even, RX odd, TX even, and TX odd. The RX and TX are pretty self explanatory…the module needs somewhere to read the data its going to send and somewhere to write the data it just received. The even and odd are a configuration that I have seen before in the PIC 18F4550 USB module: Ping-pong buffers. While one buffer is being sent/received by the module, the other can be in use by user code reading/writing (ping). When the user code is done with its buffers, it swaps buffers, giving the usb module control over the ones it was just using (pong). This allows seamless communication between the host and the device and minimizes the need for copying data between buffers. I have declared the BDT in my code as follows:

One caveat of the BDT is that it must be aligned with a 512-byte boundary in memory. Our code above showed that only 3 bytes of the 4 byte address of “table” are passed to the module. This is because the last byte is basically the index along the table (the specification of this is found in section 41.4.3, page 970 of the manual). The #define directly above the declaration is a helper macro for referencing entries in the table for specific endpoints (this is used later in the interrupt). Now, accomplishing this boundary alignment requires some modification of the linker script. Before this, I had never had any need to modify a linker script. We basically need to create a special area of memory (in the above, it is called “.usbdescriptortable” and the attribute declaration tells the compiler to place that variable’s reference inside of it) which is aligned to a 512-byte boundary in RAM. I declared mine like so:

1

2

3

4

.usbdescriptortable(NOLOAD):{

.=ALIGN(512);

*(.usbdescriptortable*)

}>sram

The position of this in the file is mildly important, so looking at the full linker script would probably be good. This particular declaration I more or less lifted from the Teensyduino linker script, with some changes to make it fit into my linker script.

Steps 5-6 set up the interrupts. There is only one USB interrupt, but there are two registers of flags. We first reset all of the flags. Interestingly, to reset a flag we write back a ‘1’ to the particular flag bit. This has the effect of being able to set a flag register to itself to reset all of the flags since a flag bit is ‘1’ when it is triggered. After resetting the flags, we enable the interrupt in the NVIC (Nested Vector Interrupt Controller). I won’t discuss the NVIC much, but it is a fairly complex piece of hardware. It has support for lots and lots of interrupts (over 100) and separate priorities for each one. I don’t have reliable code for setting interrupt priorities yet, but eventually I’ll get around to messing with that. The “enable_irq()” call is a function that is provided in arm_cm4.c and all that it does is enable the interrupt specified by the passed vector number. These numbers are specified in the datasheet, but we have a #define specified in the mk20d7 header file (warning! 12000 lines ahead) which gives us the number.

The very last step in initialization is to set the internal pullup on D+. According to the USB specification, a pullup on D- specifies a low speed device (1.2Mbit/s) and a pullup on D+ specifies a full speed device (12Mbit/s). We want to use the higher speed grade. The Kinetis USB module does not support high speed (480Mbit/s) mode.

Part 3: The interrupt handler state machine

The USB protocol can be interpreted in the context of a state machine with each call to the interrupt being a “tick” in the machine. The interrupt handler must process all of the flags to determine what happened and where to go from there.

C

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

88

89

#define ENDP0_SIZE 64

/**

* Endpoint 0 receive buffers (2x64 bytes)

*/

staticuint8_t endp0_rx[2][ENDP0_SIZE];

//flags for endpoint 0 transmit buffers

staticuint8_t endp0_odd,endp0_data=0;

/**

* Handler functions for when a token completes

* TODO: Determine if this structure really will work for all kinds of handlers

The above code will be executed whenever the IRQ for the USB module fires. This function is set up in the crt0.S file, but with a weak reference, allowing us to override it easily by simply defining a function called USBOTG_IRQHandler. We then proceed to handle all of the USB interrupt flags. If we don’t handle all of the flags, the interrupt will execute again, giving us the opportunity to fully process all of them.

Reading through the code is should be obvious that I have not done much with many of the flags, including USB sleep, errors, and stall. For the purposes of this super simple driver, we really only care about USB resets and USB token decoding.

The very first interrupt that we care about which will be called when we connect the USB device to a host is the Reset. The host performs this by bringing both data lines low for a certain period of time (read the USB basics stuff for more information). When we do this, we need to reset our USB state into its initial and ready state. We do a couple things in sequence:

Initialize the buffers for endpoint 0. We set the RX buffers to point to some static variables we have defined which are simply uint8_t arrays of length “ENDP0_SIZE”. The TX buffers are reset to null since nothing is going to be transmitted. One thing to note is that the ODDRST bit is flipped on in the USB0_CTL register. This is very important since it “syncronizes” the USB module with our code in terms of knowing whether the even or odd buffer should be used next for transmitting. When we do ODDRST, it sets the next buffer to be used to be the even buffer. We have a “user-space” flag (endp0_odd) which we reset at the same time so that we stay in sync with the buffer that the USB module is going to use.

We enable endpoint 0. Specifically, we say that it can transmit, receive, and handshake. Enabled endpoints always handshake, but endpoints can either send, receive, or both. Endpoint 0 is specified as a reading and writing endpoint in the USB specification. All of the other endpoints are device-specific.

We clear all of the interrupts. If this is a reset we obviously won’t be doing much else.

Set our USB address to 0. Each device on the USB bus gets an address between 0 and 127. Endpoint 0 is reserved for devices that haven’t been assigned an address yet (i.e. have been reset), so that becomes our address. We will receive an address later via a command sent to endpoint 0.

Activate all necessary interrupts. In the previous part where we discussed the initialization sequence we only enabled the reset interrupt. After being reset, we get to enable all of the interrupts that we will need to be able to process USB events.

After a reset the USB module will begin decoding tokens. While there are a couple different types of tokens, the USB module has a single interrupt for all of them. When a token is decoded the module gives us information about what endpoint the token was for and what BDT entry should be used. This information is contained in the USB0_STAT register.

The exact method for processing these tokens is up to the individual developer. My choice for the moment was to make a dynamic jump table of sorts which stores 16 function pointers which will be called in order to process the tokens. Initially, these pointers point to dummy functions that do nothing. The code for the endpoint 0 handler will be discussed in the next section.

Our code here uses USB0_STAT to determine which endpoint the token was decoded for, finds the appropriate function pointer, and calls it with the value of USB0_STAT.

Part 4: Token processing & descriptors

This is one part of the driver that isn’t something that must be done a certain way, but however it is done, it must accomplish the task correctly. My super-simple driver processes this in two stages: Processing the token type and processing the token itself.

As mentioned in the previous section, I had a handler for each endpoint that would be called after a token was decoded. The handler for endpoint 0 is as follows:

The very first step in handling a token is determining the buffer which contains the data for the token transmitted. This is done by the first statement which finds the appropriate address for the buffer in the table using the BDT_INDEX macro which simply implements the addressing form found in Figure 41-3 in the family manual.

After determining where the data received is located, we need to determine which token exactly was decoded. We only do things with four of the tokens. Right now, if a token comes through that we don’t understand, we don’t really do anything. My thought is that I should be initiating an endpoint stall, but I haven’t seen anywhere that specifies what exactly I should do for an unrecognized token.

The main token that we care about with endpoint 0 is the SETUP token. The data attached to this token will be in the format described by setup_t, so the first step is that we dereference and cast the buffer into which the data was loaded into a setup_t. This token will be stored statically since we need to look at it again for tokens that follow, especially in the case of the IN token following the request to be assigned an address.

One part of processing a setup token that tripped me up for a while was what the next DATA state should be. The USB standard specifies that the data in a frame is either marked DATA0 or DATA1 and it alternates by frame. This information is stored in a flag that the USB module will read from the first 4 bytes of the BDT (the “desc” field). Immediately following a SETUP token, the next DATA transmitted must be a DATA1.

After this, the setup function is run (more on that next) and as a final step, the USB module is “unfrozen”. Whenever a token is being processed, the USB module “freezes” so that processing can occur. While I haven’t yet read enough documentation on the subject, it seems to me that this is to give the user program some time to actually handle a token before the USB module decodes another one. I’m not sure what happens if the user program takes to long, but I imagine some error flag will go off.

The guts of handling a SETUP request are as follows:

C

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

88

89

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

typedefstruct{

uint8_t bLength;

uint8_t bDescriptorType;

uint16_t bcdUSB;

uint8_t bDeviceClass;

uint8_t bDeviceSubClass;

uint8_t bDeviceProtocol;

uint8_t bMaxPacketSize0;

uint16_t idVendor;

uint16_t idProduct;

uint16_t bcdDevice;

uint8_t iManufacturer;

uint8_t iProduct;

uint8_t iSerialNumber;

uint8_t bNumConfigurations;

}dev_descriptor_t;

typedefstruct{

uint8_t bLength;

uint8_t bDescriptorType;

uint8_t bInterfaceNumber;

uint8_t bAlternateSetting;

uint8_t bNumEndpoints;

uint8_t bInterfaceClass;

uint8_t bInterfaceSubClass;

uint8_t bInterfaceProtocol;

uint8_t iInterface;

}int_descriptor_t;

typedefstruct{

uint8_t bLength;

uint8_t bDescriptorType;

uint16_t wTotalLength;

uint8_t bNumInterfaces;

uint8_t bConfigurationValue;

uint8_t iConfiguration;

uint8_t bmAttributes;

uint8_t bMaxPower;

int_descriptor_t interfaces[];

}cfg_descriptor_t;

typedefstruct{

uint16_t wValue;

uint16_t wIndex;

constvoid*addr;

uint8_t length;

}descriptor_entry_t;

/**

* Device descriptor

* NOTE: This cannot be const because without additional attributes, it will

* not be placed in a part of memory that the usb subsystem can access. I

* have a suspicion that this location is somewhere in flash, but not copied

This is the part that took me the longest once I managed to get the module talking. Handling of SETUP tokens on endpoint 0 must be done in a rather exact fashion and the slightest mistake gives some very cryptic errors.

This is a very very very minimalistic setup token handler and is not by any means complete. It does only what is necessary to get the computer to see the device successfully read its descriptors. There is no functionality for actually doing things with the USB device. Most of the space is devoted to actually returning the various descriptors. In this example, the descriptor is for a device with a single configuration and a single interface which uses no additional endpoints. In a real device, this would almost certainly not be the case (unless one uses V-USB…this is how V-USB sets up their device if no other endpoints are compiled in).

The SETUP packet comes with a “request” and a “type”. We process these as one word for simplicity. The above shows only the necessary commands to actually get this thing to connect to a Linux machine running the standard USB drivers that come with the kernel. I have not tested it on Windows and it may require some modification to work since it doesn’t implement all of the necessary functionality. A description of the functionality follows:

Set address (0x0500): This is a very simple command. All it does is wait for the next IN token. Upon receipt of this token, the address is considered “committed” and the USB module is told of its new address (see the endpoint 0 handler function above (not the setup handler)).

Set configuration (0x0900): This command can be complex, but I have stripped it down for the purposes of this example. Normally, during this command the USB module would be set up with all the requisite BDT entries for the endpoints described by the selected configuration. Since we only have one possible configuration and it doesn’t use any additional endpoints, we basically do nothing. Once I start added other endpoints to this, all of the setup for those endpoints will go in here. This is the equivalent of the RESET handler for non-zero endpoints in terms of the operations that occur. If the Set Interface command was implemented, it would have similar functionality. More about this command can be read in the referenced USB basics website.

Get descriptor (0x0680, 0x0681): In reality, this is two commands: Get descriptor and get interface. However, due to the structure we have chosen in storing the descriptors, these two commands can be merged. This is the most complex part of this particular driver and is influenced heavily by the way things are done with the Teensyduino driver since I thought they had a very efficient pattern. Basically, it uses the wIndex and wValue to find a pointer to some data to return, whether that be the device descriptor, the configuration descriptor, a string, or something else. In our case, we have only the device descriptor and the configuration descriptor. Adding a string would be trivial, however, and the exact wIndex and wValue combination for that is described in the USB basics. The wIndex for strings matches with any of the several i* (iManufacturer, iProduct, etc) which may be specified.

default: When an unrecognized command is received, we enter a stall. This is basically the USB way of saying “uhh…I don’t know what to do here” and requires the host to un-stall the endpoint before it can continue. From what I gather, there isn’t really much the user code has to do other than declare that a stall has occurred. The USB module seems to take care of the rest of that.

After handling a command and determining that it isn’t a stall, the transmission is set up. At the moment, I only have transmission set up for a maximum of 64 bytes. In reality, this is limited by the wLength transmitted with the setup packet (note the if statement before the call to usb_endp0_transmit), but as far as I have seen this is generally the same as the length of the endpoint (I could be very wrong here…so watch out for that one). However, it would be fairly straightfoward to allow it to transmit more bytes: Upon receipt of an IN token, just check if we have reached the end of what we are supposed to transmit. If not, point the next TX buffer to the correct starting point and subtract the endpoint size from the remaining length until we have transmitted all of the bytes. Although the endpoint size is 64 bytes, it is easy to transmit much more than that; it just takes multiple IN requests. The data length is given by the descriptors, so the host can determine when to stop sending IN requests.

During transmission, both the even and data flags are toggled. This ensures that we are always using the correct TX buffer (even/odd) and the DATA flag transmitted is valid.

The descriptors are the one part that can’t really be screwed up here. Screwing up the descriptors causes interesting errors when the host tries to communicate. I did not like how the “reference” usb drivers I looked at generally defined descriptors: They used a char array. This works very well for the case where there are a variable number of entries in the descriptor, but for my purposes I decided to use named structs so that I could match the values I had specified on my device to values I read from the host machine without resorting to counting bytes in the array. It’s simply for easier reading and doesn’t really give much more than that. It may even be more error prone because I am relying on the compiler packing the struct into memory in the correct order for transmission and in later versions I may end up using the char array method.

I won’t delve into a long and drawn out description of what the USB descriptor has in it, but I will give a few points:

In Linux, the device descriptor is requested first and then the configuration descriptor after that. They are two separate commands, hence the two separate descriptor entries in my descriptor table.

The device descriptor must NOT be “const”. For my compiler at least, this causes it to be placed into flash which, while a perfectly valid memory address that in general can be read, is inaccessible to the USB module. I spent a long time banging my head on this one saying “but it should work! why doesn’t it work???” Moral of the story: Anything that is pointed to by a BDT entry (transmit buffers, receive buffers) must be located in main RAM, not in the flash. It must not be const.

A device must have at least one configuration. Linux, at least, didn’t seem to like it very much when there were zero configurations and would put lots of errors into my log.

The configuration needs to have at least one interface. Specifying no interfaces caused the same problems as not specifying any configurations.

The configuration indices (bConfigurationValue) are 1-based and the interface indices (bInterfaceNumber) are zero based. I haven’t fooled around with these enough to test the veracity of this claim fully, but it was the only configuration that I managed to get things working in.

The length values are very important. If these are not correct, the host will have some serious troubles reading the descriptors. I spend a while troubleshooting these. The main one to make sure of is the wTotalLength value in the configuration descriptor. Most of the others are pretty much always going to be the same.

Where to go from here

The driver I have implemented leaves much to be desired. This isn’t meant to be a fully featured driver. Instead, its meant to be something of an introduction to getting the USB module to work on the bare metal without the support of some external dependency. A few things that would definitely need to be implemented are:

The full set of commands for the endpoint 0 SETUP token processing

A more expansive configuration that allows for having some bulk endpoints for sending data. The 64-byte limitation of packet size for endpoint 0 can cause some issues when attempting to actually utilize the full 12Mbit/s bandwidth. The USB protocol does actually add overhead and the less times that a token has to be invoked, the better.

Strings in the configuration. Right now, the configuration is essentially “blank” because it uses a shared VID/PID and doesn’t specify a manufacturer, product, or serial number. It would be rather hard to identify this device using libusb on a system with multiple devices using that VID/PID combination.

Real error handling. Right now, the interrupt basically ignores the errors. In a real application, these would need to be handled.

A better structure. I am not a real fan of how I have structured this, but my idea was to make it “expandable” without needing to recompile usb.c every time a change was made. It doesn’t achieve that yet, but in future iterations I hope to have a relatively portable usb driver module that I can port to other projects without modification, placing the other device-specific things into another, mimimalistic, file.

Conclusion

I can only hope that this discussion has been helpful. I spent a long time reading documentation, writing code, smashing my keyboard, and figuring things out and I would like to see that someone else could benefit from this. I hope as I learn more about using the modules on my Teensy that I will become more competent in understanding how many of the systems I rely on on a daily basis function.

The code I have included above isn’t always complete, so I would definitely recommend actually reading the code in the repository referenced at the beginning of this article.

If there are any mistakes in the above, please let me know in the comments or shoot me an email.

So for the past couple months my server has been going on and off due to the fact that rackspace increased their retirements of swapping and such. I made the swap to Amazon EC2 today and so over the next couple weeks we’ll see how this works out.

Introduction

If you are anything like me, you love reflection in any programming language. For the last two years or so I have been writing code for work almost exclusively in C# and have found its reflection system to be a pleasure to use. Its simple, can be fast, and can do so much.

I recently started using Autofac at work to help achieve Inversion of Control within our projects. It has honestly been the most life changing C# library (sorry Autofac, jQuery and Knockout still take the cake for “life-changing in all languages”) I have ever used and has changed the way I decompose problems when writing programs.

This article will cover some very interesting features of the Autofac Attributed Metadata module. It is a little lengthy, so I have here what will be covered:

What is autofac?

Attributed Metadata: The Basics

The IMetadataProvider interface

IMetadataProvider: Making a set of objects

IMetadataProvider: Hierarchical Metadata

What is Autofac?

This post assumes that the reader is at least passingly familiar with Autofac. However, I will make a short introduction: Autofac allows you to “compose” your program structure by “registering” components and then “resolving” them at runtime. The idea is that you define an interface for some object that does “something” and create one or more classes that implement that interface, each accomplishing the “something” in their own way. Your parent class, which needs to have one of those objects for doing that “something” will ask the Autofac container to “resolve” the interface. Autofac will give back either one of your implementations or an IEnumerable of all of your implementations (depending on how you ask it to resolve). The “killer feature” of Autofac, IMO, is being able to use constructor arguments to recursively resolve the “dependencies” of an object. If you want an implementation of an interface passed into your object when it is resolved, just put the interface in the constructor arguments and when your object is resolved by Autofac, Autofac will resolve that interface for you and pass it in to your constructor. Now, this article isn’t meant to introduce Autofac, so I would definitely recommend reading up on the subject.

Attributed Metadata: The Basics

One of my most favorite features has been Attributed Metadata. Autofac allows Metadata to be included with objects when they are resolved. Metadata allows one to specify some static parameters that are associated with a particular implementation of something registered with the container. This Metadata is normally created during registration of the particular class and, without this module, must be done “manually”. The Attributed Metadata module allows one to use custom attributes to specify the Metadata for the class rather than needing to specify it when the class is registered. This is an absurdly powerful feature which allows for doing some pretty interesting things.

For my example I will use a “extendible” letter formatting program that adds some text to the content of a “letter”. I define the following interface:

C#

1

2

3

4

interfaceILetterFormatter

{

stringFormatLetter(stringcontent);

}

This interface is for something that can “format” a letter in some way. For starters, I will define two implementations:

Ok, so we have ran into a problem: We have a list of formatters, but we don’t know which is which. There are a couple different solutions:

Use the “is” test or do a “soft cast” using the “as” operator to a specific type. This is bad because it requires that the resolver know about the specific implementations of the interface (which is what we are trying to avoid)

Just choose one based on order. This is bad because the resolution order is just as guaranteed as reflection order in C#…which is not guaranteed at all. We can’t be sure they will be resolved in the same order each time.

Use metadata at registration time and resolve it with metadata. The issue here is that if we used RegisterAssemblyTyps like above, it makes registration difficult. Also, once we get any sizable number of things registered with metadata, it becomes unmanageable IMO.

Use attributed metadata! Example follows…

We define another class:

C#

1

2

3

4

5

6

7

8

9

10

[MetadataAttribute]

sealedclassLetterFormatterAttribute:Attribute

{

publicstringName{get;privateset;}

publicLetterFormatterAttribute(stringname)

{

this.Name=name;

}

}

Marking it with System.ComponetModel.Composition.MetadataAttributeAttribute (no, that’s not a typo) will make the Attributed Metadata module place the public properties of the Attribute into the metadata dictionary that is associated with the class at registration time.

We mark the classes as follows:

C#

1

2

3

4

5

6

7

[LetterFormatter("Impersonal")]

classImpersonalLetterFormatter:ILetterFormatter

...

[LetterFormatter("Personal")]

classPersonalLetterFormatter:ILetterFormatter

...

And then we change the builder to take into account the metadata by asking it to register the Autofac.Extras.Attributed.AttributedMetadataModule. This will cause the Attributed Metadata extensions to scan all of the registered types (past, present, and future) for MetadataAttribute-marked attributes and use the public properties as metadata:

C#

1

2

3

4

5

6

7

varbuilder=newContainerBuilder();

builder.RegisterModule<AttributedMetadataModule>();

builder.RegisterAssemblyTypes(typeof(MainClass).Assembly)

.Where(c=>c.IsAssignableTo<ILetterFormatter>())

.AsImplementedInterfaces();

Now, when we resolve the ILetterFormatter classes, we can either use Autofac.Features.Meta<TImplementation> or Autofac.Features.Meta<TImplementation, TMetadata>. I’m a personal fan of the “strong” metadata, or the latter. It causes the metadata dictionary to be “forced” into a class rather than just directly accessing the metadata dictionary. This removes any uncertainty about types and such. So, I will create a class that will hold the metadata when the implementations are resolved:

C#

1

2

3

4

classLetterMetadata

{

publicstringName{get;set;}

}

It would worthwhile to note that the individual properties must have a value in the metadata dictionary unless the DefaultValue attribute is applied to the property. For example, if I had an integer property called Foo an exception would be thrown when metadata was resolved since I have no corresponding Foo metadata. However, if I put DefaultValue(6) on the Foo property, no exception would be thrown and Foo would be set to 6.

So, we now have the following inside our using statement that controls our scope in the main method:

//just for kicks, we say the first argument is our letter, so we format it and output it to the console

Console.WriteLine(chosen.FormatLetter(args[0]));

The IMetadataProvider Interface

So, in the contrived example above, we were able to identify a class based solely on its metadata rather than doing type checking. What’s more, we were able to define the metadata through attributes. However, this is old hat for Autofac. This feature has been around for a while.

When I was at work the other day, I needed to be able to handle putting sets of things into metadata (such as a list of strings). Autofac makes no prohibition on this in its metadata dictionary. The dictionary is of the type IDictionary<string, object>, so it can hold pretty much anything, including arbitrary objects. The problem is that the Attributed Metadata module had no way to do this easily. Attributes can only take certain types as constructor arguments and that seriously places a limit on what sort of things could be put into metadata via attributes easily.

I decided to remedy this and after submitting an idea for autofac via a pull request, having some discussion, changing the exact way to accomplish this goal, and fixing things up, my pull request was merged into autofac which resulted in a new feature: The IMetadataProvider interface. This interface provides a way for metadata attributes to control how exactly they produce metadata. By default, the attribute would just have its properties scanned. However, if the attribute implemented the IMetadataProvider interface, a method will be called to get the metadata dictionary rather than doing the property scan. When an IMetadataProvider is found, the GetMetadata(Type targetType) method will be called with the first argument set to the type that is being registered. This allows the IMetadataProvider the opportunity to know which class it is actually applied to; something normally not possible without explicitly passing the attribute a Type in a constructor argument.

To get an idea of what this would look like, here is a metadata attribute which implements this interface:

C#

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

[MetadataAttribute]

classLetterFormatterAttribute:Attribute,IMetadataProvider

{

publicstringName{get;privateset;}

publicLetterFormatterAttribute(stringname)

{

this.Name=name;

}

#region IMetadataProvider implementation

publicIDictionary<string,object>GetMetadata(Type targetType)

{

returnnewDictionary<string,object>()

{

{"Name",this.Name}

};

}

#endregion

}

This metadata doesn’t do much more than the original. It actually returns exactly what would be created via property scanning. However, this allows much more flexibility in how MetadataAttributes can provide metadata. They can scan the type for other attributes, create arbitrary objects, and many other fun things that I can’t even think of.

IMetadataProvider: Making a set of objects

Perhaps the simplest application of this new IMetadataProvider is having the metadata contain a list of objects. Building on our last example, we saw that the “personal” letter formatter just said “Dear Individual” every time. What if we could change that so that there was some way to pass in some “properties” or “options” provided by the caller of the formatting function? We can do this using an IMetadataProvider. We make the following changes:

//note the lack of the [MetadataAttribute] here. We don't want autofac to scan this for properties

[AttributeUsage(AttributeTargets.Class,AllowMultiple=true)]

sealedclassStringOptionAttribute:Attribute,IFormatOptionProvider

{

publicstringName{get;privateset;}

publicstringDescription{get;privateset;}

publicStringOptionAttribute(stringname,stringdescription)

{

this.Name=name;

this.Description=description;

}

publicIFormatOption GetOption()

{

returnnewStringOption()

{

Name=this.Name,

Description=this.Description

};

}

}

publicclassStringOption:IFormatOption

{

publicstringName{get;set;}

publicstringDescription{get;set;}

//note that we could easily define other properties that

//do not appear in the interface

}

classLetterMetadata

{

publicstringName{get;set;}

publicIEnumerable<IFormatOption>Options{get;set;}

}

Ok, so this is just a little bit more complicated. There are two changes to pay attention to: Firstly, the FormatLetter function now takes a list of FormatOptionValues. The second change is what enables the caller of FormatLetter to know which options to pass in. The LetterFormatterAttribute now scans the type in order to construct its metadata dictionary by looking for attributes that describe what options it needs. I feel like the usage of this is best illustrated by decorating our PersonalLetterFormatter for it to have some metadata describing the options that it requires:

C#

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

[LetterFormatter("Personal")]

[StringOption(ToOptionName,"Name of the individual to address the letter to")]

When the metadata for the PersonalLetterFormatter is resolved, it will contain an IFormatOption which represents the To option. The resolver can attempt to cast the IFormatOption to a StringOption to find out what type it should pass in using the FormatOptionValue.

This can be extended quite easily for other IFormatOptionProviders and IFormatOption pairs, making for a very extensible way to easily declare metadata describing a set of options attached to a class.

IMetadataProvider: Hierarchical Metadata

The last example showed that the IMetadataProvider could be used to scan the class to provide metadata into a structure containing an IEnumerable of objects. It is a short leap to see that this could be used to create hierarchies of arbitrary objects.

For now, I won’t provide a full example of how this could be done, but in the future I plan on having a gist or something showing arbitrary metadata hierarchy creation.

Conclusion

I probably use Metadata more than I should in Autofac. With the addition of the IMetadataProvider I feel like its quite easy to define complex metadata and use it with Autofac’s natural constructor injection system. Overall, the usage of metadata & reflection in my programs has made them quite a bit more flexible and extendable and I feel like Autofac and its metadata system complement the built in reflection system of C# quite well.

Introduction

A couple of weeks ago I saw a link on hackaday to an article by Karl Lunt about using the Teensy 3.1 without the Arduino IDE and building for the bare metal. I was very intrigued as the Arduino IDE was my only major beef with developing stuff for the Teensy 3.1 and I wanted to be able to do things without having to use the IDE. I read through the article and although it was geared towards windows, I decided to try to adapt it to my development style. There were a few things I wanted to do:

No additional code dependencies other than the teensyduino installation which I already had

Use local binaries for compilation, not the ones included with teensyduino (it just felt uncomfortable to use theirs)

Separation of src, obj, and bin directories

Mixture of c and cpp files in the src directory

Not needing to explicitly list the files in the src directory to compile

Selective inclusion of features from the teensyduino installation

I have very little experience writing more complex Makefiles. When I say “complex” I am referring to makefiles which have the src, obj, bin separation and pull in objects from multiple sources. While this may not seem complex to many people, its something I have very little experience actually doing by hand (I would normally use a generator of some sort).

I’m writing this in the hope that those without mad Makefile skills, such as myself, can liberate themselves from the Arduino IDE when developing for an awesome platform like the Teensy 3.1.

Prerequisites

As my first order of business, I located the arm-none-eabi binaries for my linux distribution. These can also be found for Windows as noted in Karl Lunt’s article. Random sidenote: I found this description of why arm-none-eabi is called arm-none-eabi. Very informative. Anyway, for those who run archlinux, the following packages are needed:

arm-none-eabi-gcc (contains the compilers)

arm-none-eabi-binutils (contains the linker, objdump, and other things for manipulating the binaries into hex files)

make (we are using a makefile…)

Hopefully this gives a bit of a hint on what packages may need to be installed on other systems. For Windows, the compiler is here and make can be found here or by googling around. I haven’t tested any of this on Windows and would advocate using Linux for this, but it shouldn’t be hard to modify the Makefile for Windows.

My Flow

For C and C++ development I have a particular flow that I like to follow. This is heavily influenced by my usage of Code::Blocks and Visual Studio. I like to have a src directory where I put all of my sources, an include directory where I put all of my headers, an obj directory for all the obj, d, & lst files, and a bin directory for my executable output. I’ve always had such a hard time with raw Makefiles because I could never quite get that directory structure working. I was never quite satisfied with my feeble Makefile attempts which ended up placing the object files in the root directory where the sources had to be. This Makefile represents my first time I was ever able to actually have a real bin, obj, src structure that works.

Compiling object files to obj & looking in src for source

A working description of this can be found in the Makefile in my github repository I mentioned earlier.

Makefiles work by defining a series of “targets” which have “dependencies”. Every dependency can also be the name of a target and a target may have multiple ways of being resolved (this I never realized before). So, here is the parts of the Makefile which enable searching in src for both c and cpp and doing specific actions for each, comping them into the obj directory:

Each section above has a specific purpose and the order can be rather important. The first part uses $(wildcard …) to pick up all of the C++ and C files. The CPP_FILES variable, for example, will become “src/file1.cpp src/file2.cpp src/etc.cpp” if we had “file1.cpp”, “file2.cpp” and “etc.cpp” in the src directory. Similarly, the C_FILES would pick up any files in src with a c file extension. Next, the filenames are transformed into object filenames living in the obj directory. This is done by first changing the file extension of the files to .o using the $(CPP_FILES:.cpp=.o) or $(C_FILES:.c=.o) syntax. However, these files still look like they are in the src directory (e.g. src/file1.o) so the directory is next stripped off each file using $(nodir…). Removing the directory doesn’t allow for a nested src directory, but that wasn’t one of our objectives here. At this point, the files are just names with no directories (e.g. file1.o) and so the last step is to change them to live in the obj directory using $(addprefix $(OBJDIR)/,..). This completes our transformation, populating OBJ_FILES to look like “obj/file1.o obj/file2.o” etc.

The next part is where we take that list of object files and use them as dependencies for a target. Targets are defined by <target name>: <dependency list> followed by a list of commands to execute after resolving the dependencies. IMPORTANT: The list of commands needs to be indented by a tab (t) character. Spaces will not work (it will say something like “missing separator” with a line number). A target is anything that we pass into make. The default target is ‘all’. The “dependencies” are files which much be “up to date” before the target is run.

In our example, we use $(OBJ_FILES) as a dependency of “$(OUTPUTDIR)/$(PROJECT).elf” which is required as a dependency of “build”. This tells make that when we run “make build”, it needs to try to resolve the dependency of “bin/<project>.elf” which in turn needs to resolve “obj/file1.o”, “obj/file2.o”, and “obj/etc.o” (going from our example in the previous paragraph). This is where the next couple targets come in. A target will only be executed if it can find some rule to resolve all of the dependencies. We will use “obj/file1.o” as an example here. There are 2 targets with that name, actually: “$(OBJDIR)/%.o: $(SRCDIR)/%.c” and “$(OBJDIR)/%.o: $(SRCDIR)/%.cpp”. It would be good to note that the target names here the exact same even though the dependencies are different. Now, how does “$(OBJDIR)%.o” match “obj/file1.o”? A Makefile does something called “pattern matching” when the % sign is used. It says “match something that looks like $(OBJDIR)<some file>.o” which our “obj/file1.o” happens to match. The cool part is that once the target name is resolved using a %, the dependencies get to use % to substitute the exact same thing. Thus, our % here is “file1”, so it follows that its dependency must be “$(SRCDIR)/file1.c”. Now, our example used “file1.cpp”, not “file1.c” and this is where defining multiple targets with the same names but different dependencies comes in. A target will only be executed if the dependencies can be resolved to either an actual file and/or another target. Our first target won’t be a match since it says that the source file should be a C file. So, it goes to the next target that matches the name which has a dependency of “$(SRCDIR)/file1.cpp”. This one matches, and so commands following that target are executed.

When executing a target (“$(OBJDIR)/%.o: $(SRCDIR)/%.cpp” in our example), there are some special variables which are available for use. These are described here, but I will discuss two important ones that I used: $@ and $<. $@ is the name of the target (so, “obj/file.o” in our case) and $< is the name of the first dependency (“src/file.cpp” in our case). This lets us pass these arguments into the commands that we execute. Our Makefile will first create the obj directory by calling “mkdir -p $(dir $@)” which is translated into “mkdir -p obj” since $(dir $@) will give us “obj”. Next, we actually compile the $< (which is translated to “src/file.cpp”), outputting it to $< which is translated to “obj/file.o”.

Outputting everything to bin

Compared to the pattern matching and multiple target definitions that we discussed above, this is comparatively simple. We simply get to prefix all of our “binary” output files with some directory which is set as $(OUTPUTDIR) in my Makefile. Here is an example:

We see here that any output that we are creating as a result of the compilation (.elf, .hex, .bin) is going to end up in $(OUTPUTDIR). Futher, we see that our “all” target asks the Makefile to create both a bin file and a hex file along with two other targets called “stats” and “dump”. These are just scripts that execute the “size” and “objdump” commands on our bin file.

Using Teensyduino without compiling everything

This was by far the most frustrating part to get working. Everything about the makefiles was readily available online, with some serious googling. However, getting things to actually compile was a little different story.

The thing that makes this complex is the fact that it seems the Teensyduino libraries were not designed to be used independently of each other. I will cover, in order, what steps I had to take in order to get this to work.

The most important file we need is called “mk20dx128.c”. This sets up a lot of things relating to interrupts along with the Phase Lock Loop (PLL) which controls the speed of the Teensy’s processor. Without this configuration, we don’t get interrupts and the processor runs at a pitiful 16Mhz. The only problem is that “mk20dx128” references a few functions that are either part of the standard library and not used often (making them difficult to search for) or are defined in other files, increasing our dependency count.

My first mistake was explicitly using the linker to link all of my object files (wait…aren’t we supposed to use the linker? Read on.). Since arm-none-eabi is not dependent on a specific architecture, it doesn’t know which standard library (libc) to use. This results in an undefined reference to “__libc_init_array()”, a function used during the initialization phase of a program which is not often invoked in code outside the standard library itself. mk20dx128.c uses this function in its custom startup code which prepares the processor for running our program. To solve this, I wanted to tell the linker that I was using a cortex-m4 cpu so that it would know which libc to include and thereby resolve the reference. However, this proved difficult to do when directly invoking the linker. Instead, I took a hint from the Makefile that comes with Teensyduino and used the following command to link the objects:

We would have thought that we should be using arm-none-eabi-ld instead of arm-none-eabi-gcc. However, by using arm-non-eabi-gcc I was able to pass the argument “-mcpu=cortex-m4” which then allowed GCC to instruct the linker which standard library to use. Wonderful, right? So all of our problems are solved? Not yet.

The next thing is that mk20dx128.c has a lot of external dependencies. It uses a function defined in pins_teensy.c which in turn requires functions defined in both analog.c and usb_dev.c which opens another can of worms. Ugh. I didn’t want this many dependencies and I couldn’t see a way to escape compiling nearly the entire Teensyduino library just to run my simple blinking program. Then, it dawned on me: I could use the same technique that mk20dx128.c uses to define its ISRs to “define” the functions that pins_teensy.c was calling that I didn’t really want. So, I made a file called “shim.c” which contained the following:

1

2

3

voidunused_void(void){}

voidusb_init(void)__attribute__((weak,alias("unused_void")));

I decided that I would include “yield.c” and “analog.c” since those weren’t too big. This left just the usb stuff. The only function that was actually called from pins_teensy.c was “usb_init”. What the above statement says to the compiler is “I am defining usb_init(void) here (which points to unused_void(void)) unless you find another definition of usb_init(void) somewhere”. The “weak” attribute makes this “strong” symbol of usb_init a “weak” symbol reference to which is basically the same as just making a declaration (in contrast to the definition a function, which is usually a strong reference). Sidenote: A program can have any number of weak symbol references to a specific function/variable (declarations), but only one strong symbol reference (definition) of that function/variable. The “alias” attribute allows us to say “when I say usb_init I really mean unused_void”. The end result of this is that if nobody defines usb_init(void) anywhere, as would be situation if I were to decide not to include usb_dev.c, any calls to usb_init(void) will actually call unused_void(void). However, if somebody did define usb_init(void), my definition of usb_init would be ignored in favor of using their definition. This lets me include usb support in the future if I wanted to. Isn’t that cool? That fixed all of my reference issues and let me actually build the project.

Conclusion

Armed with my new Makefile and a better understanding of how the Teensy 3.1 works from a software perspective, I managed to compile and upload my “blinky” program which just blinks the onboard LED (pin 13) on and off every 1/4 second. The overall program size was 3% of the total space, which is much more reasonable compared to the 10-20% it was taking when compiled using the Arduino IDE.

I entered a contest of sorts this past week which involves building an autonomous turret which uses an ultrasonic sensor to locate a target within 10 feet and fire at it with a tiny dart gun. The entire assembly is to be mounted on servos. This is something my University is doing as an extra-curricular for engineers and so when a friend of mine asked if I wanted to join forces with him and conquer, I readily agreed.

The most interesting part to me, by far, is the processor to be used. It is going to be a Teensy 3.1:

Teensy 3.1

This board contains a Freescale ARM Cortex-M4 microcontroller along with a smaller non-user-programmable microcontroller for assistance in the USB bootloading process (the exact details of that interaction are mostly unknown to me at the moment). I have never used an ARM microcontroller before and never a microcontroller with as many peripherals as this one has. The datasheet is 1200 pages long and is not really even being very verbose in my opinion. It could easily be 3000 pages if they included the level of detail usually included in AVR and PIC datasheets (code examples, etc). The processor runs at 96Mhz as well, making it the most powerful embedded computer I have used aside from my Raspberry Pi.

The Teensy 3.1 is Arduino-compliant and is designed that way. However, it can also be used without the Arduino software. I have not used an Arduino before since I rather enjoy using microcontrollers in a bare-bones fashion. However, it is become increasingly more difficult for me to be able to experiment with the latest in microcontroller developments using breadboards since the packages are becoming increasingly more surface mount.

The Arduino IDE

Oh my goodness. Worst ever. Ok, not really, but I really have a hard time justifying using it other than the fact that it makes downloading to the Teensy really easy. This post isn’t meant to be a review of the arduino IDE, but the editor could use some serious improvements IMHO:

Tab indentation level: Some of us would like to use something other than 2 spaces, thank you very much. We don’t live in the 70’s where horizontal space is at a premium and I prefer 4 spaces. Purely personal preference, but I feel like the option should be there

Ability to reload files: The inability to reload the files and the fact that it seems to compile from a cache rather than from the file itself makes the arduino IDE basically incompatible with git or any other source control system. This is a serious problem, in my opinion, and requires me to restart the editor frequently whenever I check out a different branch.

Real project files: I understand the aim for simplicity here, but when you have a chip with 256Kb of flash on it, your program is not going to be 100 lines and fit into one file. At the moment, the editor just takes everything in the directory and compiles it by file extension. No subdirectories and every file will be displayed as a separate tab with no way to close it. I am in the habit of separating my source and not having the ability to structure my files how I please really makes me feel hampered. To make matters worse, the IDE saves the original sketch file (which is just a cpp file that will be run through their preprocessor) with its own special file extension (*.ino) which makes it look like it should be a project file, but in reality it is not.

There are few things I do like, however. I do like their library of things that make working with this new and foreign processor rather easy. I also like that their build system is very cross-platform and easy to use.

First impression of the processor

I must first say that the level of work that has gone into the surrounding software (the header files, the teensy loader, etc) truly shows and makes it a good experience to use the Teensy, even if the Arduino IDE sucks. I tried a Makefile approach using Code::Blocks, but it was difficult for me to get it to compile cross-platform and I was afraid that I would accidentally overwrite some bootloader code that I hadn’t known about. So, I ended up just going with the Ardiuno IDE for safety reasons.

The peripherals on this processor are many and it is hard at times to figure out basic functions, such as the GPIO. The manual for the peripherals is in the neighborhood of 60 chapters long, with each chapter describing a peripheral. So far, I have messed with just the GPIOs and pin interrupts, but I plan on moving on to the timer module very soon. This project likely won’t require the DMA or the variety of onboard bus modules (CAN, I2C, SPI, USB, etc), but in the future I hope to have a Teensy of my own to experiment on. The sheer number of registers combined with the 32-bit width of everything is a totally new experience for me. Combine that with the fact that I don’t have to worry as much about the overhead of using certain C constructs (struct and function pointers for example) and I am super duper excited about this processor. Tack on the stuff that PJRC created for using the Teensy such as the nice header files and the overall compatibility with some Arduino libraries, and I have had an easier time getting this thing to work than with most of my other projects I have done.

Conclusion

Although the Teensy is for a specific contest project right now, at the price of $19.80 for the amount of power that it gives, I believe I will buy one for myself to mess around with. I am looking forward to getting more familiar with this processor and although I resent the IDE I have to work with at the moment, I hope that I will be able to move along to better compilation options that will let me move away from the arduino IDE.

Every time I do one of these bus emulation projects, I tell myself that the next time I do it I will use an oscilloscope or DLA. However, I never actually break down and just buy one. Once more, I have done a bus emulation project flying blind. This is the harrowing tale:

Introduction

A couple of days ago, I was asked to help do some soldering for a modification someone was trying to do to a PS1 controller. He informed me that it was for the game Pop ‘n Music and that it required a special controller to be played properly. Apparently, official controllers can sell for $100 or more, so modifying an existing controller was the logical thing to do. After much work and pain, it was found that while modifying an existing controller was easy, it wasn’t very robust and could easily fall apart and so I built one using an ATMega48 and some extra components I had lying around. The microcontroller emulates the PSX bus which is used to communicate between the controller and the playstation/computer. As my reference for the bus, I used the following two web pages:

The first attempt: Controller mod

The concept behind the controller mod was simple: Run wires from the existing button pads to some arcade-style buttons arranged in the pattern needed for the controller. It worked well at first, but after a little while we began to have problems:

The style of pad that he purchased had conductive rubber covering all of the copper for the button landings. In order to solder to this, it was necessary to scrape off the rubber. This introduced a tendency for partially unclean joints, giving rise to cold connections. While with much effort I was able to mitigate this issue (lots of scraping and cleaning), the next problem began to manifest itself.

The copper layout for each button pad was of a rather minimalist design. While some pads shown online had nice large areas for the button to contact, this particular controller had 50-100 mil lines arranged in a circular pattern rather than one huge land. While I imagine this is either economical or gives better contact, it sure made soldering wires onto it difficult. I would get the wire soldered, only to have it decide that it wanted to come off and take the pad with it later. This was partly due to bad planning on my part and using wire that wasn’t flexible enough, but honestly, the pads were not designed to be soldered to.

With each pad that lifted, the available space for the wires on certain buttons to be attached to began to become smaller and smaller. Some buttons were in the large land style and were very easy to solder to and the joints were strong (mainly the arrow pad buttons). The issue was with the start and select buttons (very narrow) and the X, square, triangle, and O buttons (100mil spiral thing mentioned earlier). Eventually, I was resorting to scraping the solder mask off and using 30awg wire wrapping wire to solder to the traces. It just got ridiculous and wasn’t nearly strong enough to hold up as a game controller.

In order for the controller to be used with a real playstation, rather than an emulator, the Left, Right, and Down buttons had to be pressed at the same time to signify to the game that it was a Pop ‘n Music controller. Emulators generally can’t handle this sort of behavior when mapping the buttons around, so putting a switch was considered. However, any reliable switch (read: Nice shiny toggle switch) was around $3. Given the low cost aim of this project, it was becoming more economical to explore other options

So, we began exploring other options. I found this site detailing emulation of these controllers using either 74xx logic or a microcontroller. It is a very good resource, and is mostly correct about the protocol. After looking at the 74xx logic solution and totaling up the cost, I noticed that my $1.75 microcontroller along with the required external components would actually come out to be cheaper than buying 4 chips and sockets for them. Even better, I already had a microcontroller and all the parts on hand, so there was no need for shipping. So, I began building and programming.

AVR PSX Bus Emulation: The Saga of the Software

PSX controllers communicate using a bus that has a clock, acknowledge, slave select, psx->controller (command) line, and controller->psx (data) line. Yes, this looks a lot like an SPI bus. In fact, it is more or less identical to a SPI Mode 3 bus with the master-in slave-out line driven open collector. I failed to notice this fact until later, much to my chagrin. Communication is accomplished using packets that have a start signal followed by a command and waiting for a response from the controller. During the transaction, the controller declares its type, the number of words that it is going to send, and the actual controller state. I was emulating a standard digital controller, so I had to tell it that my controller type was 0x41, which is digital with 1 word data. Then, I had to send a 0x5A (start data response byte) and two bytes of button data. My initial approach involved writing a routine in C that would handle pin changes on INT0 and INT1 which would be connected to the command and clock lines. However, I failed to anticipate that the bus would be somewhere in the neighborhood of 250Khz-500Khz and this caused some serious performance problems and I was unable to complete a transaction with the controller host. So, I decided to try writing the same routine in assembly to see if I could squeeze every drop of performance out of it possible. I managed to actually get it to complete a transaction this way, but without sending button data. To make matters worse, every once in a while it would miss a transaction and this was quite noticeable when I made an LED change state with every packet received. It was very inconsistent and that was without even sending button data. I eventually realized the problem was with the fact that making the controller do so much between cycles of the clock line actually caused it to miss bits. So, I looked at the problem again. I noticed that the ATMega48A had an SPI module and that the PSX bus looked similar, but not exactly like, an SPI bus. However, running the bus in mode 3 with the data order reversed and the MISO driving the base of a transistor operating in an open-collector fashion actually got me to be able to communicate to the PSX bus on almost the first try. Even better, the only software change that had to be made was inverting the data byte so that the signal hitting the base of the transistor would cause the correct changes on the MISO line. So, I hooked up everything as follows:

Pop ‘n Music Controller Schematic

After doing that, suddenly I got everything to work. It responded correctly to the computer when asked about its inputs and after some optimization, stopped skipping packets due to taking too much time processing button inputs. It worked! Soon after getting the controller to talk to the computer, I discovered an error in the website I mentioned earlier that detailed the protocol. It mentioned that during transmission of the data about the buttons that the control line was going to be left high. While its a minor difference, I thought I might as well mention this site, which lists the commands correctly and was very helpful. As I mentioned before, one problem that was encoutered was that in order for the controller to be recognized as a pop-n-music controller by an actual playstation, the left, right, and down buttons must be pressed. However, it seems that the PSX->USB converter that we were using was unable to handle having those 3 pressed down at once. So, there needed to be a mode switch. The way for switching modes I came up with was to hold down both start and select at the same time for 3 seconds. After the delay, the modes would switch. The UI interaction for this is embodied in two LEDs. One LED is lit for when it is in PSX mode and the other when it is in emulator mode. When both buttons are pressed, both LEDs light up until the one for the previous mode shuts off. At first, I had the mode start out every time the controller was started in the same mode, no matter what the previous mode was before it was shut off. It soon became apparent that this wouldn’t do, and so I looked in to using the EEPROM to store the flag value I was using to keep the state of the controller. Strangely, it worked on the first try, so the controller will stay in the same mode from the last time it was shut off. My only fear is that switching the mode too much could degrade the EEPROM. However, the datasheet says that it is good for 100,000 erase/write cycles, so I imagine it would be quite a while before this happens and other parts of the controller will probably fail first (like the switches).

On to the hardware!

I next began assembly. I went the route of perfboard with individual copper pads around each hole because that’s what I have. Here are photos of the assembly, sadly taken on my cell phone because my camera is broken. Sorry for the bad quality…

Socket and PSX plug mouted

Wiring between PSX plug and socket

Adding some transistors…first try

Adding the transistors, try 2

Wiring almost done

Inside of switch box

The controller in action

Conclusion

So, with the controller in the box and everything assembled, it seems that all will be well with the controller. It doesn’t seem to miss keypresses or freeze and is able to play the game without too many hiccups (the audio makes it difficult, but that’s just a emulator tweaking issue). The best part about this project is that in terms of total work time, it probably took only about 16 hours. Considering that most of my projects take months to finish, this easily takes the cake as one of my quickest projects start to finish.

Introduction

Recently, I got my hands on a Raspberry Pi and one of the first things I wanted to do with it was to turn it into my complete AVR development environment. As part of that I wanted to make avrdude be able to program an AVR directly from the Raspberry Pi with no programmer. I know there is this linuxgpio programmer type that was recently added, but it is so recent that it isn’t yet included in the repos and it also requires a compile-time option to enable it. I noticed that the Raspberry Pi happens to expose its SPI interface on its expansion header and so I thought to myself, “Why not use this thing instead of bitbanging GPIOs? Wouldn’t that be more efficient?” Thus, I began to decipher the avrdude code and write my addition. My hope is that things like this will allow the Raspberry Pi to be used to explore further embedded development for those who want to get into microcontrollers, but blew all their money on the Raspberry Pi. Also, in keeping with the purpose that the Raspberry Pi was originally designed for, using it like this makes it fairly simple for people in educational surroundings to expand into different aspects of small computer and embedded device programming.

As my addition to avrdude, I created a new programmer type called “linuxspi” which uses the userspace SPI drivers available since around Linux ~2.6 or so to talk to a programmer. It also requires an additional GPIO to operate as the reset. My initial thought was to use the chip select as the reset output, but sadly, the documentation for the SPI functions mentioned that the chip enable line is only held low so long as the transaction is going. While I guess I could compress all the transactions avrdude makes into one giant burst of data, this would be very error prone and isn’t compatible with avrdude’s program structure. So, the GPIO route was chosen. It just uses the sysfs endpoints found in /sys/class/gpio to manipulate a GPIO chosen in avrdude.conf into either being in a hi-z input state or an output low state. This way, the reset can be connected via a resistor to Vcc and then the Raspberry Pi just holds reset down when it needs to program the device. Another consequence which I will mention here of choosing to use the Linux SPI drivers is that this should actually be compatible with any Linux-based device that exposes its SPI or has an AVR connected to the SPI; not just the Raspberry Pi.

Raspberry Pi Programming an AVR

Usage

So, down to the nitty gritty: How can I use it? Well, at the moment it is in a github repository at https://github.com/kcuzner/avrdude. As with any project that uses the expansion header on the Raspberry Pi, there is a risk that a mistake could cause your Raspberry Pi to die (or let out the magic smoke, so to speak). I assume no responsibility for any damage that may occur as a result of following these directions or using my addition to avrdude. Just be careful when doing anything involving hooking stuff up to the expansion port and use common sense. Remember to measure twice and cut once. So, with that out of the way, I will proceed to outline here the basic steps for installation and usage.

Installation

The best option here until I bother creating packages for it is to do a git clone directly into a directory on the Raspberry Pi and build it from there on the Raspberry Pi itself. I remember having to install the following packages to get it to compile (If I missed any, let me know):

bison

autoconf

make

gcc

flex

Also, if your system doesn’t have a header at “linux/spi/spidev.h” in your path, you probably need to install that driver. I was using Arch Linux and it already had the driver there, so for all I know its always installed. You also should take a look to make sure that “/dev/spidev0.0” and “/dev/spidev0.1” or something like that exist. Those are the sort of endpoints that are to be used with this. If they do not exist, try executing a “sudo modprobe spi_bcm2708”. If the endpoints still aren’t there after that, then SPI support probably isn’t installed or enabled for your kernel.

After cloning the repo and installing those packages, run the “./boostrap” script which is found in the avrdude directory. This will run all the autoconf things and create the build scripts. The next step is to run “./configure” and wait for it to complete. After the configure script, it should say whether or not “linuxspi” is enabled or disabled. If it is disabled, it was not able to find the header I mentioned before. Then run “make” and wait for it to complete. Remember that the Raspberry Pi is a single core ARM processor and so building may take a while. Afterwards, simply do “sudo make install” and you will magically have avrdude installed on your computer in /usr/local. It would probably be worthwhile to note here that you probably want to uninstall any avrdude you may have had installed previously either manually or through a package manager. The one here is built on top of the latest version (as of May 26th, 2013), so it should work quite well and be all up to date and stuff for just using it like a normal avrdude. I made no changes to any of the programmer types other than the one I added.

To check to see if the avrdude you have is the right one, you should see an output similar to the following if you run this command (tiny-tim is the name of my Raspberry Pi until I think of something better):

wiring=http://wiring.org.co/, Basically STK500v2 protocol, with some glue to trigger the bootloader.

Note that right under “linuxgpio” there is now a “linuxspi” driver. If it says “(not available)” after the “linuxspi” description, “./configure” was not able to find the “linux/spi/spidev.h” file and did not compile the linuxspi programmer into avrdude.

Configuration

There is a little bit of configuration that happens here on the Raspberry Pi side before proceeding to wiring it up. You must now decide which GPIO to sacrifice to be the reset pin. I chose 25 because it is next to the normal chip enable pins, but it doesn’t matter which you choose. To change which pin is to be used, you need to edit “/usr/local/etc/avrdude.conf” (it will be just “/etc/avrdude.conf” if it wasn’t built and installed manually like above). Find the section of the file that looks like so:

1

2

3

4

5

6

programmer

id="linuxspi";

desc="Use Linux SPI device in /dev/spidev*";

type="linuxspi";

reset=25;

;

The “reset = ” line needs to be changed to have the number of the GPIO that you have decided to turn into the reset pin for the programmer. The default is 25, but that’s just because of my selfishness in not wanting to set it to something more generic and having to then edit the file every time I re-installed avrdude. Perhaps a better default would be “0” since that will cause the programmer to say that it hasn’t been set up yet.

Wiring

After setting up avrdude.conf to your desired configuration, you can now connect the appropriate wires from your Raspberry Pi’s header to your microchip. A word of extreme caution: The Raspberry Pi’s GPIOs are NOT 5V tolerant, and that includes the SPI pins. You must do either one of two things: a) Run the AVR and everything around it at 3.3V so that you never see 5V on ANY of the Raspberry Pi pins at any time (including after programming is completed and the device is running) or b) Use a level translator between the AVR and the SPI. I happen to have a level translator lying around (its a fun little TSSOP I soldered to a breakout board a few years back), but I decided to go the 3.3V route since I was trying to get this thing to work. If you have not ever had to hook up in-circuit serial programming to your AVR before, perhaps this would be a great time to learn. You need to consult the datasheet for your AVR and find the pins named RESET (bar above it), MOSI, MISO, and SCK. These 4 pins are connected so that RESET goes to your GPIO with a pullup resistor to the Vcc on your AVR, MOSI goes to the similarly named MOSI on the Raspberry Pi header, MISO goes to the like-named pin on the header, and SCK goes to the SPI clock pin (named SCLK on the diagram on elinux.org). After doing this and double checking to make sure 5V will never be present to the Raspberry Pi, you can power on your AVR and it should be able to be programmed through avrdude. Here is a demonstration of me loading a simple test program I made that flashes the PORTD LEDs:

I set the programmer type (-c option) to be “linuxspi”. This tells avrdude to use my addition as the programming interface

I set the port (-P option) to be “/dev/spidev0.0”. On my Raspberry Pi, this maps to the SPI bus using CE0 as the chip select. Although we don’t actually use CE0 to connect to the AVR, it still gets used by the spidev interface and will toggle several times during normal avrdude operation. Your exact configuration may end up being different, but this is more or less how the SPI should be set. If the thing you point to isn’t an SPI device, avrdude should fail with a bunch of messages saying that it couldn’t send an SPI message.

Other than that, usage is pretty straightforward and should be the same as if you were using any other programmer type.

Future

As issues crop up, I hope to add improvements like changing the clock frequency and maybe someday adding TPI support (not sure if necessary since this is using the dedicated SPI and as far as I know, TPI doesn’t use SPI).

I hope that those using this can find it helpful in their fun and games with the Raspberry Pi. If there are any issues compiling and stuff, either open an issue on github or mention it in the comments here.

For the greater part of this week I have been trapped in a car driving across the United States for my sister’s wedding. Of course, I had my trusty netbook with me and so I decided to do some programming. After discovering that I had unwittingly installed tons of documentation in my /usr/share/doc folder (don’t you love those computer explorations that actually reveal something?), I started building yet another rendition of Tetris in Python. I did several things differently this time and the code is actually readable and I feel like some structure things I learned/discovered/re-discovered should be shared here.

Game State Management

One problem that has constantly plagued my weak attempts at games has been an inability to do menus and manage whether the game is paused or not in a reasonably simple manner. Of course, I could always hack something together, but I could also write the whole thing in assembly language or maybe write it all as one big function in C. The issue is that if I program myself into a hole, I usually end up giving up a bit unless I am getting paid (then I generally bother thinking before programming so that I don’t waste people’s time). After finishing the game logic and actually getting things to work with the game itself, I remembered a funny tutorial I was once doing for OGRE which introduced a very simple game engine. The concept was one of a “stack” of game states. Using this stack, your main program basically injects input into and then asks a rendering from the state on the top of the stack. The states manipulate the stack, pushing a new state or popping themselves. When the stack is empty, the program exits.

My game state stack works by using an initial state that loads the resources (the render section gives back a loading screen). After it finishes loading the resources, it asks the manager to pop itself from the stack and then push on the main menu state. The menu state is quite simple: It displays several options where each option either does something immediate (like popping the menu from the stack to end the program in the case of the Quit option) or something more complex (like pushing a sub-menu state on to create a new game with some settings). After that there are states for playing the game, viewing the high scores, etc.

The main awesome thing about this structure is that switching states is so easy. Want to pause the game? Just push on a state that doesn’t do anything until it receives a key input or something, after which it pops itself. Feel like doing a complex menu system? The stack keeps track of the “breadcrumbs” so to speak. Never before have I been able to actually create an easily extensible/modification-friendly game menu/management system that doesn’t leave me with a headache later trying to understand it or unravel my spaghetti code.

The second awesome thing is that this helps encourage an inversion of control. Rather than the state itself dictating when it gets input or when it renders something, the program on top of it can choose whether it wants to render that state or whether it wants to give input to that state. I ended up using the render function like a game loop function as well and so if I wanted a state to sort of “pause” execution, I could simply stop telling it to render (usually by just pushing a state on top of it). My main program (in Python) was about 25 lines long and took care of setting up the window, starting the game states, and then just injecting input into the game state and asking the state to render a window. The states themselves could choose how they wanted to render the window or even if they wanted to (that gets into the event bubbling section which is next). It is so much easier to operate that way as the components are loosely tied to one another and there is a clear relationship between them. The program owns the states and the states know nothing about about the program driving them…only that their input comes from somewhere and the renderings they are occasionally asked to do go somewhere.

Event bubbling

When making my Tetris game, I wanted an easy to way to know when to update or redraw the screen rather than always redrawing it. My idea was to create an event system that would “notice” when the game changed somehow: whether a block had moved or maybe the score changed. I remembered of two event systems that I really really liked: C#’s event keyword plus delegate thing and Javascript’s event bubbling system. Some people probably hate both of those, but I have learned to love them. Mainly, its just because I’m new to this sort of thing and am still being dazzled by their plethora of uses. Using some knowledge I gleaned from my database abstraction project about the fun data model special function names in Python, I created the following:

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

classEventDispatcher(object):

"""

Event object which operates like C# events

"event+=handler" adds a handler to this event

"event-=handler" removes a handler from this event

"event(obj)" calls all of the handlers, passing the passed object

Events may be temporarily suppressed by using them in a with

statement. The context returned will be this event object.

"""

def __init__(self):

"""

Initializes a new event

"""

self.__handlers=[]

self.__supress_count=0

def __call__(self,e):

ifself.__supress_count&gt;0:

return

forhinself.__handlers:

h(e)

def __iadd__(self,other):

ifother notinself.__handlers:

self.__handlers.append(other)

returnself

def __isub__(self,other):

self.__handlers.remove(other)

returnself

def __enter__(self):

self.__supress_count+=1

returnself

def __exit__(self,exc_type,exc_value,traceback):

self.__supress_count-=1

My first thought when making this was to do it like C# events: super flexible, but no defined “way” to do them. The event dispatcher uses the same syntax as the C# events (+=, -= or __iadd__, __isub__) to add/remove handlers and also to call each handler (() or __call__). It also has the added functionality of optionally suppressing events by using them inside a “with” statement (which may actually be breaking the pattern, but I needed it to avoid some interesting redrawing issues). I would add EventDispatcher objects to represent each type of event that I wanted to catch and then pass an Event object into the event to send it off to the listening functions. However, I ran into an issue with this: Although I was able to cut down the number of events being sent and how far they propagated, I would occasionally lose events. The issue this caused is that my renderer was listening to find out where it should “erase” blocks and where it should “add” blocks and it would sometimes seem to forget to erase some of the blocks. I later discovered that I had simply forgotten to call the event during a certain function which was called periodically to move the Tetris blocks down, but even so, it got my started on the next thing which I feel is better.

Javascript events work by specifying types of events, a “target” or object in the focus of the event, and arguments that get passed along with the event. The events then “bubble” upward through the DOM, firing first for a child and then for the parent of that child until it reaches the top element. The advantage of this is that if one wants to know, for example, if the screen has been clicked, a listener doesn’t have to listen at the lowest leaf element of each branch of the DOM; it can simply listen at the top element and wait for the “click” event to “bubble” upwards through the tree. After my aforementioned issue I initially thought that I was missing events because my structure was flawed, so I ended up re-using the above class to implement event bubbling by doing the following:

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

classEvent(object):

"""

Instance of an event to be dispatched

"""

def __init__(self,target,name,*args,**kwargs):

self.target=target

self.name=name

self.args=args

self.kwargs=kwargs

classEventedObject(object):

def __init__(self,parent=None):

self.__parent=parent

self.event=EventDispatcher()

self.event+=self.__on_event

def __on_event(self,e):

ifhasattr(self.__parent,'event'):

self.__parent.event(e)

@property

def parent(self):

returnself.__parent

@parent.setter

def parent(self,value):

l=self.parent

self.__parent=value

self.event(Event(self,"parent-changed",current=self.parent,last=l))

The example here is the object from which all of my moving game objects (blocks, polyominoes, the game grid, etc) derive from. It defines a single EventDispatcher, through which Event objects are passed. It listens to its own event and when it hears something, it activates its parent’s event, passing through the same object that it received. The advantage here is that by listening to just one “top” object, all of the events that occurred for the child objects are passed to whatever handler is attached to the top object’s dispatcher. In my specific implementation I had each block send an Event up the pipeline when the were moved, each polyomino send an Event when it was moved or rotated, and the game send an Event when the score, level, or line count was changed. By having my renderer listen to just the game object’s EventDispatcher I was able to intercept all of these events at one location.

The disadvantage with this particular method is that each movement has a potentially high computational cost. All of my events are synchronous since Python doesn’t do true multithreading and I didn’t need a high performance implementation. It’s just method calls, but there is a potential for a stack overflow if either a chain of parents has a loop somewhere or if I simply have too tall of an object tree. If I attach too many listeners to a single EventDispatcher, it will also slow things down.

Another problem I have here has to do with memory leaks which I believe I have (I haven’t tested it and this is entirely in my head, thinking about the issues). Since I am asking the EventDispatcher to add a handler which is a bound method to the object which owns it, there is a loop there. In the event that I forget all references to the EventedObject, the reference count will never decrease to 0 since the EventDispatcher inside the EventedObject still holds a reference to that same EventedObject inside the bound method. I would think that this could cause garbage collection to never happen. Of course, they could make the garbage collector really smart and notice that the dependency tree here is a nice orphan tree detached from the rest and can all be collected. However, if it is a dumb garbage collector, it will probably keep it around. This isn’t a new issue for me: I ran into it with doing something like this on one of my C# projects. However, the way I solved it there was to implement the IDisposable interface and upon disposal, unsubscribe from all events that created a circular dependency. The problem there was worse because there wasn’t a direct exclusive link between the two objects like there is with this one (here one is a property of the other (strong link) and the other only references a bound method to its partner (weak…kinda…link)).

Overall, even though there are those disadvantages, I feel that the advantage gained by having all the events in one place is worth it. In the future I will create “filters” that can be attached to handlers as they are subscribed to avoid calling the handlers for events that don’t match their filter. This is similar to Javascript in that handlers can be used to catch one specific type of event. However, mine differs in that in the spirit of Python duck typing, I decided to make no distinction between types of events outside of a string name that identifies what it is. Since I only had one EventDispatcher per object, it makes sense to only have one type of event that will be fed to its listeners. The individual events can then just be differentiated by a property value. While this feels flaky to me since I usually feel most comfortable with strong typing systems, it seems to be closer to what Python is trying to do.

Conclusion

I eventually will put up this Tetris implementation as a gist or repository on github (probably just a gist unless it gets huge…which it could). So far I have learned a great deal about game design and structure, so this should get interesting as I explore other things like networking and such.

As I was recently working on trying out the Flask web framework for Python, I ended up wanting to access my MySQL database. Recently at work I have been using entity framework and I have gotten quite used to having a good database abstraction that allows programmatic creation of SQL. While such frameworks exist in Python, I thought it would interesting to try writing one. This is one great example of getting carried away with a seemingly simple task.

I aimed for these things:

Tables should be represented as objects which each instance of the object representing a row

These objects should be able to generate their own insert, select, and update queries

Querying the database should be accomplished by logical predicates, not by strings

Update queries should be optimized to only update those fields which have changed

The database objects should have support for “immutable” fields that are generated by the database

I also wanted to be able to do relations between tables with foreign keys, but I have decided to stop for now on that. I have a structure outlined, but it isn’t necessary enough at this point since all I wanted was a database abstraction for my simple Flask project. I will probably implement it later.

Example

Before going into the code, here is an example of what this abstraction can do as it stands. It directly uses the DbObject and DbQuery-inheriting objects which are shown further down in this post.

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

from db import *

import hashlib

def salt_password(user,unsalted):

ifuser isNone:

returnunsalted

m=hashlib.sha512()

m.update(user.username)

m.update(unsalted)

returnm.hexdigest()

classUser(DbObject):

dbo_tablename="users"

primary_key=IntColumn("id",allow_none=True,mutable=False)

username=StringColumn("username","")

password=PasswordColumn("password",salt_password,"")

display_name=StringColumn("display_name","")

def __init__(self,**kwargs):

DbObject.__init__(self,**kwargs)

@classmethod

def load(self,cur,username):

selection=self.select('u')

selection[0].where(selection[1].username==username)

result=selection[0].execute(cur)

iflen(result)==0:

returnNone

else:

returnresult[0]

def match_password(self,password):

salted=salt_password(self,password)

returnsalted==self.password

#assume there is a function get_db defined which returns a PEP-249

#database object

def main():

db=get_db()

cur=db.cursor()

user=User.load(cur,"some username")

user.password="a new password!"

user.save(cur)

db.commit()

new_user=User(username="someone",display_name="Their name")

new_user.password="A password that will be hashed"

new_user.save(cur)

db.commmit()

print new_user.primary_key# this will now have a database assigned id

This example first loads a user using a DbSelectQuery. The user is then modified and the DbObject-level function save() is used to save it. Next, a new user is created and saved using the same function. After saving, the primary key will have been populated and will be printed.

Change Tracking Columns

I started out with columns. I needed columns that track changes and have a mapping to an SQL column name. I came up with the following:

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

88

89

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

classColumnSet(object):

"""

Object which is updated by ColumnInstances to inform changes

"""

def __init__(self):

self.__columns={}# columns are sorted by name

i_dict=type(self).__dict__

forattr ini_dict:

obj=i_dict[attr]

ifisinstance(obj,Column):

# we get an instance of this column

self.__columns[obj.name]=ColumnInstance(obj,self)

@property

def mutated(self):

"""

Returns the mutated columns for this tracker.

"""

output=[]

forname inself.__columns:

column=self.get_column(name)

ifcolumn.mutated:

output.append(column)

returnoutput

def get_column(self,name):

returnself.__columns[name]

classColumnInstance(object):

"""

Per-instance column data. This is used in ColumnSet objects to hold data

specific to that particular instance

"""

def __init__(self,column,owner):

"""

column: Column object this is created for

initial: Initial value

"""

self.__column=column

self.__owner=owner

self.update(column.default)

def update(self,value):

"""

Updates the value for this instance, resetting the mutated flag

"""

ifvalue isNone andnotself.__column.allow_none:

raise ValueError("'None' is invalid for column '"+

self.__column.name+"'")

ifself.__column.validate(value):

self.__value=value

self.__origvalue=value

else:

raise ValueError("'"+str(value)+"' is not valid for column '"+

self.__column.name+"'")

@property

def column(self):

returnself.__column

@property

def owner(self):

returnself.__owner

@property

def mutated(self):

returnself.__value!=self.__origvalue

@property

def value(self):

returnself.__value

@value.setter

def value(self,value):

ifvalue isNone andnotself.__column.allow_none:

raise ValueError("'None' is invalid for column '"+

self.__column.name+"'")

ifnotself.__column.mutable:

raise AttributeError("Column '"+self.__column.name+"' is not"+

" mutable")

ifself.__column.validate(value):

self.__value=value

else:

raise ValueError("'"+value+"' is not valid for column '"+

self.__column.name+"'")

classColumn(object):

"""

Column descriptor for a column

"""

def __init__(self,name,default=None,allow_none=False,mutable=True):

"""

Initializes a column

name: Name of the column this maps to

default: Default value

allow_none: Whether none (db null) values are allowed

mutable: Whether this can be mutated by a setter

"""

self.__name=name

self.__allow_none=allow_none

self.__mutable=mutable

self.__default=default

def validate(self,value):

"""

In a child class, this will validate values being set

"""

raise NotImplementedError

@property

def name(self):

returnself.__name

@property

def allow_none(self):

returnself.__allow_none

@property

def mutable(self):

returnself.__mutable

@property

def default(self):

returnself.__default

def __get__(self,owner,ownertype=None):

"""

Gets the value for this column for the passed owner

"""

ifowner isNone:

returnself

ifnotisinstance(owner,ColumnSet):

raise TypeError("Columns are only allowed on ColumnSets")

returnowner.get_column(self.name).value

def __set__(self,owner,value):

"""

Sets the value for this column for the passed owner

"""

ifnotisinstance(owner,ColumnSet):

raise TypeError("Columns are only allowed on ColumnSets")

owner.get_column(self.name).value=value

classStringColumn(Column):

def validate(self,value):

ifvalue isNone andself.allow_none:

print"nonevalue"

returnTrue

ifisinstance(value,basestring):

print"isstr"

returnTrue

print"not string",value,type(value)

returnFalse

classIntColumn(Column):

def validate(self,value):

ifvalue isNone andself.allow_none:

returnTrue

ifisinstance(value,int)orisinstance(value,long):

returnTrue

returnFalse

classPasswordColumn(Column):

def __init__(self,name,salt_function,default=None,allow_none=False,

mutable=True):

"""

Create a new password column which uses the specified salt function

salt_function: a function(self, value) which returns the salted string

"""

Column.__init__(self,name,default,allow_none,mutable)

self.__salt_function=salt_function

def validate(self,value):

returnTrue

def __set__(self,owner,value):

salted=self.__salt_function(owner,value)

super(PasswordColumn,self).__set__(owner,salted)

The Column class describes the column and is implemented as a descriptor. Each ColumnSet instance contains multiple columns and holds ColumnInstance objects which hold the individual column per-object properties, such as the value and whether it has been mutated or not. Each column type has a validation function to help screen invalid data from the columns. When a ColumnSet is initiated, it scans itself for columns and at that moment creates its ColumnInstances.

Generation of SQL using logical predicates

The next thing I had to create was the database querying structure. I decided that rather than actually using the ColumnInstance or Column objects, I would use a go-between object that can be assigned a “prefix”. A common thing to do in SQL queries is to rename the tables in the query so that you can reference the same table multiple times or use different tables with the same column names. So, for example if I had a table called posts and I also had a table called users and they both shared a column called ‘last_update’, I could assign a prefix ‘p’ to the post columns and a prefix ‘u’ to the user columns so that the final column name would be ‘p.last_update’ and ‘u.last_update’ for posts and users respectively.

Another thing I wanted to do was avoid the usage of SQL in constructing my queries. This is similar to the way that LINQ works for C#: A predicate is specified and later translated into an SQL query or a series of operations in memory depending on what is going on. So, in Python one of my queries looks like so:

This would print out a tuple (" WHERE x.column_1 = %s AND x.column_2 > %s", ["4", 5]). So, how does this work? I used operator overloading to create DbQueryExpression objects. The code is like so:

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

88

89

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

classDbQueryExpression(object):

"""

Query expression created from columns, literals, and operators

"""

def __and__(self,other):

returnDbQueryConjunction(self,other)

def __or__(self,other):

returnDbQueryDisjunction(self,other)

def __str__(self):

raise NotImplementedError

@property

def arguments(self):

raise NotImplementedError

classDbQueryConjunction(DbQueryExpression):

"""

Query expression joining together a left and right expression with an

AND statement

"""

def __init__(self,l,r):

DbQueryExpression.__ini__(self)

self.l=l

self.r=r

def __str__(self):

returnstr(self.l)+" AND "+str(self.r)

@property

def arguments(self):

returnself.l.arguments+self.r.arguments

classDbQueryDisjunction(DbQueryExpression):

"""

Query expression joining together a left and right expression with an

OR statement

"""

def __init__(self,l,r):

DbQueryExpression.__init__(self)

self.l=l

self.r=r

def __str__(self):

returnstr(self.r)+" OR "+str(self.r)

@property

def arguments(self):

returnself.l.arguments+self.r.arguments

classDbQueryColumnComparison(DbQueryExpression):

"""

Query expression comparing a combination of a column and/or a value

"""

def __init__(self,l,op,r):

DbQueryExpression.__init__(self)

self.l=l

self.op=op

self.r=r

def __str__(self):

output=""

ifisinstance(self.l,DbQueryColumn):

prefix=self.l.prefix

ifprefix isnotNone:

output+=prefix+"."

output+=self.l.name

elif self.lisNone:

output+="NULL"

else:

output+="%s"

output+=self.op

ifisinstance(self.r,DbQueryColumn):

prefix=self.r.prefix

ifprefix isnotNone:

output+=prefix+"."

output+=self.r.name

elif self.risNone:

output+="NULL"

else:

output+="%s"

returnoutput

@property

def arguments(self):

output=[]

ifnotisinstance(self.l,DbQueryColumn)andself.lisnotNone:

output.append(self.l)

ifnotisinstance(self.r,DbQueryColumn)andself.risnotNone:

output.append(self.r)

returnoutput

classDbQueryColumnSet(object):

"""

Represents a set of columns attached to a specific DbOject type. This

object dynamically builds itself based on a passed type. The columns

attached to this set may be used in DbQueries

"""

def __init__(self,dbo_type,prefix):

d=dbo_type.__dict__

self.__columns={}

forattr ind:

obj=d[attr]

ifisinstance(obj,Column):

column=DbQueryColumn(dbo_type,prefix,obj.name)

setattr(self,attr,column)

self.__columns[obj.name]=column

def __len__(self):

returnlen(self.__columns)

def __getitem__(self,key):

returnself.__columns[key]

def __iter__(self):

returniter(self.__columns)

classDbQueryColumn(object):

"""

Represents a Column object used in a DbQuery

"""

def __init__(self,dbo_type,prefix,column_name):

self.dbo_type=dbo_type

self.name=column_name

self.prefix=prefix

def __lt__(self,other):

returnDbQueryColumnComparison(self,"&lt;",other)

def __le__(self,other):

returnDbQueryColumnComparison(self,"&lt;=",other)

def __eq__(self,other):

op="="

ifother isNone:

op=" IS "

returnDbQueryColumnComparison(self,op,other)

def __ne__(self,other):

op="!="

ifother isNone:

op=" IS NOT "

returnDbQueryColumnComparison(self,op,other)

def __gt__(self,other):

returnDbQueryColumnComparison(self,"&gt;",other)

def __ge__(self,other):

returnDbQueryColumnComparison(self,"&gt;=",other)

The __str__ function and arguments property return recursively generated expressions using the column prefixes (in the case of __str__) and the arguments (in the case of arguments). As can be seen, this supports parameterization of queries. To be honest, this part was the most fun since I was surprised it was so easy to make predicate expressions using a minimum of classes. One thing that I didn’t like, however, was the fact that the boolean and/or operators cannot be overloaded. For that reason I had to use the bitwise operators, so the expressions aren’t entirely correct when being read.

This DbQueryExpression is fed into my DbQuery object which actually does the translation to SQL. In the example above, we saw that I just passed a logical argument into my where function. This actually was a DbQueryExpression since my overloaded operators create DbQueryExpression objects when they are compared. The DbColumnSet object is an dynamically generated object containing the go-between column objects which is created from a DbObject. We will discuss the DbObject a little further down

The DbQuery objects are implemented as follows:

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

88

89

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

classDbQueryError(Exception):

"""

Raised when there is an error constructing a query

"""

def __init__(self,msg):

self.message=msg

def __str__(self):

returnself.message

classDbQuery(object):

"""

Represents a base SQL Query to a database based upon some DbObjects

All of the methods implemented here are valid on select, update, and

delete statements.

"""

def __init__(self,execute_filter=None):

"""

callback: Function to call when the DbQuery is executed

"""

self.__where=[]

self.__limit=None

self.__orderby=[]

self.__execute_filter=execute_filter

def where(self,expression):

"""Specify an expression to append to the WHERE clause"""

self.__where.append(expression)

def limit(self,value=None):

"""Specify the limit to the query"""

self.__limit=value

@property

def sql(self):

query=""

args=[]

iflen(self.__where)&gt;0:

where=self.__where[0]

forclause inself.__where[1:]:

where=where&amp;clause

args=where.arguments

query+=" WHERE "+str(where)

ifself.__limit isnotNone:

query+=" LIMIT "+self.__limit

returnquery,args

def execute(self,cur):

"""

Executes this query on the passed cursor and returns either the result

of the filter function or the cursor if there is no filter function.

"""

query=self.sql

cur.execute(query[0],query[1])

ifself.__execute_filter:

returnself.__execute_filter(self,cur)

else:

returncur

classDbSelectQuery(DbQuery):

"""

Creates a select query to a database based upon DbObjects

"""

def __init__(self,execute_filter=None):

DbQuery.__init__(self,execute_filter)

self.__select=[]

self.__froms=[]

self.__joins=[]

self.__orderby=[]

def select(self,*columns):

"""Specify one or more columns to select"""

self.__select+=columns

def from_table(self,dbo_type,prefix):

"""Specify a table to select from"""

self.__froms.append((dbo_type,prefix))

def join(self,dbo_type,prefix,on):

"""Specify a table to join to"""

self.__joins.append((dbo_type,prefix,on))

def orderby(self,*columns):

"""Specify one or more columns to order by"""

self.__orderby+=columns

@property

def sql(self):

query="SELECT "

args=[]

iflen(self.__select)==0:

raise DbQueryError("No selection in DbSelectQuery")

query+=','.join([col.prefix+"."+

col.name forcol inself.__select])

iflen(self.__froms)==0:

raise DbQueryError("No FROM clause in DbSelectQuery")

fortable inself.__froms:

query+=" FROM "+table[0].dbo_tablename+" "+table[1]

iflen(self.__joins)&gt;0:

forjoin inself.__joins:

query+=" JOIN "+join[0].dbo_tablename+" "+join[1]+

" ON "+str(join[2])

query_parent=super(DbSelectQuery,self).sql

query+=query_parent[0]

args+=query_parent[1]

iflen(self.__orderby)&gt;0:

query+=" ORDER BY "+

','.join([col.prefix+"."+

col.name forcol inself.__orderby])

returnquery,args

classDbInsertQuery(DbQuery):

"""

Creates an insert query to a database based upon DbObjects. This does not

include any where or limit expressions

"""

def __init__(self,dbo_type,prefix,execute_filter=None):

DbQuery.__init__(self,execute_filter)

self.table=(dbo_type,prefix)

self.__values=[]

def value(self,column,value):

self.__values.append((column,value))

@property

def sql(self):

iflen(self.__values)==0:

raise DbQueryError("No values in insert")

tablename=self.table[0].dbo_tablename

query="INSERT INTO {table} (".format(table=tablename)

args=[val[1]forval inself.__values

ifval[0].prefix==self.table[1]]

query+=",".join([val[0].name forval inself.__values

ifval[0].prefix==self.table[1]])

query+=") VALUES ("

query+=",".join(["%s"forxinargs])

query+=")"

returnquery,args

classDbUpdateQuery(DbQuery):

"""

Creates an update query to a database based upon DbObjects

"""

def __init__(self,dbo_type,prefix,execute_filter=None):

"""

Initialize the update query

dbo_type: table type to be updating

prefix: Prefix the columns are known under

"""

DbQuery.__init__(self,execute_filter)

self.table=(dbo_type,prefix)

self.__updates=[]

def update(self,left,right):

self.__updates.append((left,right))

@property

def sql(self):

iflen(self.__updates)==0:

raise DbQueryError("No update in DbUpdateQuery")

query="UPDATE "+self.table[0].dbo_tablename+" "+self.table[1]

args=[]

query+=" SET "

forupdate inself.__updates:

ifisinstance(update[0],DbQueryColumn):

query+=update[0].prefix+"."+update[0].name

else:

query+="%s"

args.append(update[0])

query+="="

ifisinstance(update[1],DbQueryColumn):

query+=update[1].prefix+"."+update[1].name

else:

query+="%s"

args.append(update[1])

query_parent=super(DbUpdateQuery,self).sql

query+=query_parent[0]

args+=query_parent[1]

returnquery,args

classDbDeleteQuery(DbQuery):

"""

Creates a delete query for a database based on a DbObject

"""

def __init__(self,dbo_type,prefix,execute_filter=None):

DbQuery.__init__(self,execute_filter)

self.table=(dbo_type,prefix)

@property

def sql(self):

query="DELETE FROM "+self.table[0].dbo_tablename+" "+

self.table[1]

args=[]

query_parent=super(DbDeleteQuery,self).sql

query+=query_parent[0]

args+=query_parent[1]

returnquery,args

Each of the SELECT, INSERT, UPDATE, and DELETE query types inherits from a base DbQuery which does execution and such. I decided to make the DbQuery object take a PEP 249-style cursor object and execute the query itself. My hope is that this will make this a little more portable since, to my knowledge, I didn’t make the queries have any MySQL-specific constructions.

The different query types each implement a variety of statements corresponding to different parts of an SQL query: where(), limit(), orderby(), select(), from_table(), etc. These each take in either a DbQueryColumn (such as is the case with where(), orderby(), select(), etc) or a string to be appended to the query, such as is the case with limit(). I could easily have made limit take in two integers as well, but I was kind of rushing through because I wanted to see if this would even work. The query is built by creating the query object for the basic query type that is desired and then calling its member functions to add things on to the query.

Executing the queries can cause a callback “filter” function to be called which takes in the query and the cursor as arguments. I use this function to create new objects from the data or to update an object. It could probably be used for more clever things as well, but those two cases were my original intent in creating it. If no filter is specified, then the cursor is returned.

Table and row objects

At the highest level of this hierarchy is the DbObject. The DbObject definition actually represents a table in the database with a name and a single primary key column. Each instance represents a row. DbObjects also implement the methods for selecting records of their type and also updating themselves when they are changed. They inherit change tracking from the ColumnSet and use DbQueries to accomplish their querying goals. The code is as follows:

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

88

89

90

91

92

93

94

95

96

97

98

99

100

101

102

classDbObject(ColumnSet):

"""

A DbObject is a set of columns linked to a table in the database. This is

synonomous to a row. The following class attributes must be set:

dbo_tablename : string table name

primary_key : Column for the primary key

"""

def __init__(self,**cols):

ColumnSet.__init__(self)

forname incols:

c=self.get_column(name)

c.update(cols[name])

@classmethod

def get_query_columns(self,prefix):

returnDbQueryColumnSet(self,prefix)

@classmethod

def select(self,prefix):

"""

Returns a DbSelectQuery set up for this DbObject

"""

columns=self.get_query_columns(prefix)

def execute(query,cur):

output=[]

block=cur.fetchmany()

whilelen(block)&gt;0:

forrow inblock:

values={}

i=0

forname incolumns:

values[name]=row[i]

i+=1

output.append(self(**values))

block=cur.fetchmany()

returnoutput

query=DbSelectQuery(execute)

query.select(*[columns[name]forname incolumns])

query.from_table(self,prefix)

returnquery,columns

def get_primary_key_name(self):

returntype(self).__dict__['primary_key'].name

def save(self,cur):

"""

Saves any changes to this object to the database

"""

ifself.primary_key isNone:

# we need to be saved

columns=self.get_query_columns('x')

def execute(query,cur):

self.get_column(self.get_primary_key_name()

).update(cur.lastrowid)

selection=[]

forname incolumns:

ifname==self.get_primary_key_name():

continue#we have no need to update the primary key

column_instance=self.get_column(name)

ifnotcolumn_instance.column.mutable:

selection.append(columns[name])

iflen(selection)!=0:

# we get to select to get additional computed values

def execute2(query,cur):

row=cur.fetchone()

index=0

forsinselection:

self.get_column(s.name).update(row[index])

index+=1

returnTrue

query=DbSelectQuery(execute2)

query.select(*selection)

query.from_table(type(self),'x')

query.where(columns[self.get_primary_key_name()]==

self.get_column(self.get_primary_key_name()

).value)

returnquery.execute(cur)

returnTrue

query=DbInsertQuery(type(self),'x',execute)

forname incolumns:

column_instance=self.get_column(name)

ifnotcolumn_instance.column.mutable:

continue

query.value(columns[name],column_instance.value)

print query.sql

returnquery.execute(cur)

else:

# we have been modified

modified=self.mutated

iflen(modified)==0:

returnTrue

columns=self.get_query_columns('x')

def execute(query,cur):

formod inmodified:

mod.update(mod.value)

returnTrue

query=DbUpdateQuery(type(self),'x',execute)

formod inmodified:

query.update(columns[mod.column.name],mod.value)

query.where(columns[self.get_primary_key_name()]==self.primary_key)

returnquery.execute(cur)

DbObjects require that the inheriting classes define two properties: dbo_tablename and primary_key. dbo_tablename is just a string giving the name of the table in the database and primary_key is a Column that will be used as the primary key.

To select records from the database, the select() function can be called from the class. This sets up a DbSelectQuery which will return an array of the DbObject that it is called for when the query is executed.

One fallacy of this structure is that at the moment it assumes that the primary key won’t be None if it has been set. In other words, the way I did it right now does not allow for null primary keys. The reason it does this is because it says that if the primary key hasn’t been set, it needs to generate a DbInsertQuery for the object when save() is called instead of a DbUpdateQuery. Both insert and update queries do not include every field. Immutable fields are always excluded and then later selected or inferred from the cursor object.

So, this last week I got an email from rackspace saying that my server was thrashing the hard drives and lowering performance of everyone else on that same machine. In consequence, they had rebooted my server for me.

I made a few mistakes in the setup of this first iteration of my server: I didn’t restart it after kernel updates, I ran folding@home on it while running nodejs, and I didn’t have backups turned on. I had it running for well over 200 days without a reboot while there had been a dozen or so kernel updates. When they hard rebooted my server, it wouldn’t respond at all to pings, ssh, or otherwise. In fact, it behaved like the firewalls were shut (hanging on the “waiting” step rather than saying “connection refused”). I ended up having to go into their handy rescue mode and copy out all the files. I only copied my www directory and the mysql binary table files, but as you can see, I was able to restore the server from those.

This gave me an excellent opportunity to actually set up my server correctly. I no longer have to be root to edit my website files (yay!), I have virtual hosts set up in a fashion that makes sense and actually works, and overall performance seems to be improved. From now on, I will be doing the updates less frequently and when I do I will be rebooting the machine. That should fix the problem with breaking everything if a hard reboot happens.

I do pay for the hosting for this, 1.5 cents per hour per 256Mb of RAM with extra for bandwidth. I only have 256Mb and since I don’t make any profit off this server whatsoever at the moment, I plan on keeping it that way for now. Considering that back in the day, 256Mb was a ton of memory, it clearly no longer suffices for running too much on my server (httpd + mysql + nodejs + folding@home = crash and burn).