Interfacing with the Arduino's bootloader directly from C# (to upload a HEX file)

Sending a program to a microcontroller (which is most often called “programming” the device) can be done in multiple ways. For example, one can do it
through the use of ICSP headers and dedicated hardware programmers.
In the Arduino world the usage of bootloaders is extremely common, since it is what allows you to program your device directly through the Serial (USB) connection.
This blog post will explore how we can upload a compiled program (in its Intel HEX form) to an Arduino UNO directly, by interacting
with the device’s bootloader from C#.

Arduino Bootloaders

A bootloader is a small, specialized piece of software on your microcontroller that runs first whenever your board powers on. It provides at
least the following functions:

provide a way to upload new programs to the device

start (“bootstrap”) the main program

Once the bootloader has given control to the main program, the latter will keep on running indefinitely.

Specific types of Arduino’s come preinstalled with a certain “flavour” of bootloader. You can change your bootloader or perhaps you don’t even have one installed (for example, when you ordered ATMega chips directly).
When in doubt, one can always use Nick Gammon’s Atmega_Board_Detector (requires 2 Arduinos) in order to identify
the currently installed bootloader.

The most common Arduino bootloaders implement (a subset of) upload protocols that originate with
Atmel (e.g. STK500v1 & STK500v2). These (binary) protocols are quite well documented.

This blog post will focus exclusively on the following combination of elements (as we find them in a modern Arduino UNO): ATMega328P (MCU) - Optiboot
(bootloader) - STK500v1 (upload protocol).

How the Arduino IDE uploads a program

Toggle “Show verbose output” for both compilation and upload in the IDE’s preferences.

As of today (Arduino IDE version 1.6.11) the process is two-pronged:

Compilation: the IDE calls avr-gcc in order to compile your source code (Arduino sketches are really nothing more than regular C++ code with some
domain specific libraries). The end result of this step is a binary file in Intel HEX format. At the end of the
verbose output, you will be able to see exactly where these files end up on your filesystem.

You can open up the .HEX file in a regular text editor (since it stores information in an ASCII form):

Note: For those interested in how to parse the Intel Hex file structure, have a look at the following .NET project on GitHub:
IntelHexFormatReader.

Upload: the IDE calls a tool called avrdude (homepage) which originated on FreeBSD and has been the de facto standard (for
many years) for uploading programs to all kinds of microcontrollers. In the output you can see how it will “flash” the .HEX file generate in the previous step (
to COM port 4, at a baud rate of 115200 bps, with an “arduino” type programmer, targetting the ATMega328P chip on the UNO):

1. Configure and open the serial port

Instantiate a serial port instance (of type System.IO.Ports.SerialPort) first, with the port name pointing to the actual
port where the Arduino is attached, at a baud rate of 115200 bps (as that is the speed used to communicate with the Optiboot bootloader).

We will reset the Arduino first, as that is the only way to get the bootloader code to run again. There is only a short
interval available after powering up the board (where the bootloader listens for a potential upload). After that, it will
call the entrypoint to the main program. Resetting an Arduino UNO can be done by toggling DTR/RTS:

5. Enable programming mode

We will send a command to enter “program mode”.

The relevant documentation entry:

The implementation:

publicoverridevoidEnableProgrammingMode(){SendWithSyncRetry(newEnableProgrammingModeRequest());varnextByte=ReceiveNext();if(nextByte==Constants.RESP_STK_OK)return;if(nextByte==Constants.RESP_STK_NODEVICE||nextByte==Constants.RESP_STK_Failed)UploaderLogger.LogAndThrowError<IOException>("Unable to enable programming mode on the device!");}

6. Program the device

This method takes a MemoryBlock (a type from IntelHexFormatReader that has
the “memory” representation of the HEX file after interpreting the records in it) and
iterates over its contents (per page).

If a page has any byte set to “modified” (as per the HEX file), we will mark it as needsWrite. This way, we will only
write to the pages required. Even if a page is needsWrite, we will first read the contents of the page and compare it
to the write payload. That way (e.g. if you are flashing the same program over and over again) we can conserve
(finite and thus precious) write cycles. After the read, we will read the page again (in order to verify that the
contents have been written perfectly).

7. Leave programming mode

We will send a last command to leave programming mode on the device (both the documentation and the implementation
should look fairly predictable by now):

publicoverridevoidLeaveProgrammingMode(){SendWithSyncRetry(newLeaveProgrammingModeRequest());varnextByte=ReceiveNext();if(nextByte==Constants.RESP_STK_OK)return;if(nextByte==Constants.RESP_STK_NODEVICE||nextByte==Constants.RESP_STK_Failed)UploaderLogger.LogAndThrowError<IOException>("Unable to leave programming mode on the device!");}

At the very end, it’s important to reset the Arduino again (by toggling DTR/RTS) if we want to immediately
start running the freshly uploaded code.

The code

As mentioned at the top of this article, the full .NET library ArduinoSketchUploader (which also
supports other architectures, bootloaders than the one dissected above)
is available to download on GitHub.. It has a nuget package
for those seeking to integrate this feature into their own .NET projects.

Also, a command line utility (at the moment Windows only) is supplied.