Author
Topic: Porting a Canon firmware update (Read 21093 times)

Pretty much every week someone posts something like, "When is Magic Lantern going to be working with X.X.X version firmware?" The answer is usually, "Downgrade to X.X.X -1." Why doesn't ML work with the latest Canon firmware? There are reasons including:

There's nothing in the firmware update that you really need.

If ML works on X.X.X why take a chance on breaking it?

Developers shouldn't waste their time on trivial updates.

Because you need a degree in computer science, know ARM processing and spend hundreds of hours reverse engineering to port Magic Lantern.

However, there are also reasons for porting to the latest Canon firmware:

Canon recommends using their latest firmware update.

The old firmware that ML needs is no longer on the Canon support site so you'll have to get it from a non-official source.

Canon may have actually fixed a bug that affects your camera.

Because you can.

Let's put the emphasis on you can meaning anyone reading this article should be able to port ML to a Canon firmware update, though it probably shouldn't the first project you attempt. You don't need to know anything about programming or reverse engineering and quite frankly most of what is required to do is grunt work that the main developers aren't really interested in doing. The first thing you need is to set up a development system in order to compile Magic Lantern. There are several ways to do this and there are plenty of tutorials for compiling ML so we won't go into it here.

Note that this article is about doing a relatively minor firmware update like the type Canon released around October of 2016 to fix a problem that affected a couple of lenses. Major updates that add new features or porting to a new camera isn't covered here.

OK--first step is getting ML working on your camera with the supported firmware. This is necessary in order to get firmware dumps, set the camera boot flag and to set the boot flag on the memory cards that you'll be using. You'll find the ROM dumps in the ML/LOGS directory of the card. Save those files. We'll get back to them soon.

You should install ML on a few cards if for nothing else so you'll have cards with the boot flag set. You can also set the card boot flag using a utility like EOScard for Windows and for Mac try MacBoot. I recommend having various different cards with the boot flag set because soon we'll be running something that might not work on certain cards or might corrupt the file system. @a1ex gives a good explanation of what is happening and how to work around it in Reply #1 below. This explains why my trusty 2GB Patriot SD card works like a champ while the much higher performing SanDisk Extreme PRO cards I tried usually failed.

Keep one card without the boot flag in order to do the actual Canon firmware update. Copy both the Canon firmware that is currently working with ML and the version you are upgrading to. That way with one card you can upgrade and downgrade the Canon firmware. If you have several different camera models you can keep all the firmware versions on that one card because the camera will pick out only updates that are valid for that model.

Note that on the 5D3 you can't downgrade from 1.3.X in camera, you need to use EOS Utility. However, what EOS Utility does is to copy the firmware to the card and the camera takes over from there. This process is to discourage users from downgrading. There seems to be a good reason for this because 1.3.X has some added features that might corrupt user settings when downgrading. Simply clearing the Canon settings when downgrading seems to clear up those corruption issues.

Updating the Canon firmware

Here we go--Canon firmware update on card without the boot flag set, check. Update the firmware. Note that this is an article that attempts to cover all ML enabled cameras so I'll be mixing up screenshots from various models.

Dumping the ROM

When you install ML it sets a boot flag on the camera and a boot flag on the card. The camera will always check first to see if the card has the boot flag set and if it is set, attempt to boot off an autoexec.bin file on the card. There are some special autoexec.bin files available on the download page under the heading of ROM dumpers, scroll down the page to find them. You are most like going to need the one labeled Portable. Put it on one of your cards that has the boot flag set and insert the card in your camera. It will probably start working without even turn on the power. You should see something like this:

Or this:

Didn't happen on your first try? Some times it is stubborn so try it again or switch to another card. Once again, check out the comment on Reply #1 if you are having trouble getting it to work. The point of this is to get a ROM dump of the new firmware.

If you want to be meticulous about it you should run an MD5 check on your file. When I did it both attempts had the same MD5 value and the dumps matched each other in size so I was confident that I had a good dump to work with. Lazy me. Open up a terminal (Powershell on Windows), navigate to where your ROM dumps and MD5 files are saved and:

Now you've got several sets of verified ROM dumps, make sure you know which one is which. In most cases you'll only need to work with the ROM1.BIN for the Canon firmware that you started from and the one for the Canon firmware you're porting to.

Congratulations, you just completed the most essential step towards porting Magic Lantern!

If you're in a hurry, curious, stupid or all of the above you'll be tempted to try your current version of ML on the new firmware. Guess which description I fit into?

Disassembly

There are various methods of disassembling the ROM dumps in order to work with them. Everyone seems to have their favorite method. ARMu is quick but only works on Windows--ok it also works on Mac under wine but I haven't figured it out yet. ARM-console is very difficult to set up and use but it is very powerful--this currently my favorite disassembler. There is also a commercial product called IDA but in order to disassemble ARM code you need to buy a license. By far the easiest for beginners is disassemble.pl which is a perl script that needs just a few tweaks to get running properly as explained in this excellent tutorial on finding stubs.

What about QEMU? That could be very useful, especially when porting major firmware updates and new cameras but we're doing just a minor firmware update that can be done with just a few basic tools and a lot of perseverance.

Now it might be tempting to upload and share these ROM dumps and disassemblies but DON'T DO IT! You're holding copyrighted material that is owned by a very large and powerful corporation. You bought the camera so you can look at what is inside it but you aren't supposed to publish "intellectual property" in an open forum like this one. I'll be editing the bits in this post so they don't perfectly match the real disassembly but it should be good enough for instructional purposes.

So let's do this. Assuming disassemble.pl is in the same directory as your ROM1A.BIN file:

The file you are interested in is ROM1A.BIN.dis. You'll be spending a lot of time together.

Preparing your local repository for the update.

There are lots of ways to work with your local repository. It really depends how you are most comfortable. If you plan to eventually do a pull request you should consider creating a "scratch" or development branch where you can experiment and post all your notes and then when everything is working, copy over your changes to a fresh branch to avoid cluttering up the revision history with trivial commits.

Take a look at some recent firmware updates so you'll know what you are getting yourself into:

The first thing you'll notice is that it seems that we're throwing out the old and creating new files. That's not really the case, we're renaming the old files but don't do it with the operating system because it defeats the point of keeping everything under version control. Generally I like to work with the SourceTree application but copying directories is one of those things that is probably best done on the command line. There are two directories and one file that needs to be renamed. For example, when going from 1.1.4 to 1.1.5 on the 700D:

Now for the good news. This step has registered a lot of changes but you'll only need to modify a few of these files. Most of the changes will be happening in the stubs.S file. While it is challenging looking for new stubs or finding stubs for a new camera model, a minor firmware update is rather easy.

Let's start with a very easy change. In magic-lantern/installer/Makefile you'll see this line:

Pretty easy to figure out what to change on that line. However, there's something to watch out for when editing a Makefile, you got to make sure that your text editor doesn't change the indents from tabs to spaces. A Makefile needs those tabs.

Now check the pull requests on bitbucket and you'll see several other places where you need to change from the old to the new firmware version. Don't forget to make changes in the modules: adtg_gui, dual_iso, mlv_rec and mlv_lite.

Another rather easy change to make is to remove the old ML-SETUP.FIR file. It won't work with the new firmware and a developer will have to create a new one for you. In the case of a minor firmware update if you did the Canon update with the boot flag set on the camera, you won't have any problems later when you test out your new port. Just don't rush it! There are a few things that need to be done in the right order before attempting to boot into your newly ported ML.

In fact if you want to be extra careful you could take the advice that is in property.c and disable the prop_request_change function. If you're a suspenders and belt kind of guy also go into your platform's internals.h and comment out '#define CONFIG_PROP_REQUEST_CHANGE'. This is highly recommended for new ports because you can do some serious damage. I didn't take that precaution because mine were very minor updates.

Finding where the stubs moved

Now we get into the nitty gritty of finding stubs. We aren't porting a new camera, just a minor firmware update so all the stubs that you should be able to easily look up in the old disassembled ROM1.BIN disassembly should also be in the new disassembly--only in a different location.

If you're working with a DIGIC V the first thing you will notice in the stubs.S file is something like this (700D.114):

#define RAM_OFFSET (0xFFA5E8B0-0x1900) // Nanomad: some functions are copied to RAM at around ff0c0098; they have to be called from RAM...

There is a very good explanation of what this RAM_OFFSET is all about in a1ex's Tutorial: finding stubs. In this case Nanomad left a clue so let's check out what line 0xff0c0098 says in the 700D.114 disassembly. Depending which disassembler you used it might look like this:

Now start going through the rest of the stubs.S file. One trick I discovered is that if you're tackling one of those minor updates Canon pushed in October of 2016, any stub that doesn't use RAM_OFFSET remains unchanged and those with the offset move the same amount in the same direction that RAM_OFFSET changed--for the most part. A few will move. Why? Because there was a few lines of code that were changed somewhere that shifted things around slightly. In any case you should check the address to every stub to make sure. In some cases there's a descriptive string around the stub you're checking into, other times there is nothing to go by except if several lines "look" the same. It becomes sort of a pattern matching game like those cartoons that you try to find the difference between two drawings that look almost identical.

You can take a chance that it didn't change but it is best to look at the hints, search the forum, read the wiki or post a question how to find these values. Leave those challenging ones until the end because by the time you've gone through all of the other stubs you'll become more of an expert at sleuthing out those stubs.

There are some helpful scripts in the magic-lantern/contrib directory. Try running check-stubs.py to get a report on the differences between the old and new stubs.S files. I posted my results on the 700D.115 pull request.

Make sure you got all of the changed addresses

Not all of the addresses you need to check are in the stubs.S file. The source code for some of the modules also have references to firmware versions and/or addresses that need checking and probably changing. For example, adtg_gui.c, dual_iso.c, mlv_rec.c and mlv_lite.c. By now your pattern matching skills are probably pretty good so good luck hunting them down.

Getting the firmware signature

The next step involves something a little more exciting that pouring over endless lines of disassembly code. You'll need to find the new firmware's signature and for that you'll need to build Magic Lantern, but this isn't a working build--not yet.

Take a look at fw-signature.h. Your updated firmware version is probably not in the list. Now check out boot-hack.c and you'll find this near the beginning of the file:

So it took a bit more effort to find those addresses but magic-lantern/contrib/indy/find_fnt.py found them automatically.

On the 5D3.134 there were some other things that needed attention in consts.h. You would think that constants by definition don't change but like the old saying goes, "the only thing constant is change."

Oh, one last thing--the ML-SETUP.FIR file. Usually that's one of the first things that you need if you're working on a new port but since we're only doing a minor firmware update we can wait until the end. Only one of the main developers can prepare this file for you so you'll have to ask.

Found this list of the recent Canon firmware updates that were released in October 2016. I added which version is currently supported in order to help you decide if you would like to attempt to port a simple firmware update.

Didn't happen on your first try? Some times it is stubborn so try it again or switch to another card. The point of this is to get a ROM dump of the new firmware. If you want to be meticulous about it you should run an MD5 check on your file.

Here, checking the MD5 is not about being meticulous; the bootloader file I/O routines *are* really stubborn. They may corrupt the filesystem, insert a few or a lot of wrong bytes, lock up the camera, refuse to run on any or all of your cards and so on.

So far, what worked every single time was formatting the card at a much lower capacity (I've used 256MB successfully in many cases). To do this, you can write the SD image that comes with QEMU to your SD or CF card. This guide is helpful; just don't forget to unzip the SD image. This image contains the portable display test and is bootable (so, you can test it in the camera straight away).

Even in this case, checking the MD5 is mandatory. On Linux, this is as simple as:

It is possible to include both Canon firmware versions but it is easier to support one version at a time. The 5D3 is an exception. The developers didn't want to remove 1.1.3 because 1.2.3 and up has some issues with ML. Minor Canon firmware updates like the one you're doing shouldn't introduce any new bugs, at least not in theory.

Got my hands on a 7D so I thought hey, what the heck. No big deal, right? This one turned out the be quite a challenge because the portable dumper doesn't work on it. Well it works in QEMU but not on the camera.

My timing was pretty good because g3gg0 added a display dumper to the portable dumper in order to copy the ROM of some of the newer cameras. Turns out that it worked on the 7D. You can follow along the adventures of learning how to use the display dumper on the Portable ROM dumper topic starting here.

I should mention that porting a firmware update almost never works the first time. First time I ran it on the camera it looked like this:

I was sure the font address was fine because I used contrib/indy/find_fnt.py to verify that it didn't change:

Next, how about running some tests? The stub check test in the selftest module passed with flying colors:

However, the lua api test caused all sorts of issues. The camera would hang requiring a battery pull and I seldom was able to get a test to run much less to complete. The most complete test I got was with an STM lens that apparently wasn't controllable via lua on the 7D. I'll publish these tests on separate posts because of the per post character limit on this forum.

Rather disappointing so I regressed to the old 2.0.3 firmware and found out that was also happening. Good to know I didn't break anything but strange nonetheless.

Speaking of strange, the 7D has two processors, a "master" and a "slave" and ML only runs on the slave. So what does the 7D_MASTER platform do? I could only get this out of it:

Just for good form I updated the 7D_MASTER but since I couldn't figure out any of it, all the addresses are still for the 2.0.3 firmware.

This was an interesting exercise so I thought I'd share the experience and post a pull request.

And this is the most complete lua test I was able to pull off. Note that the other lua scripts seem to working fine, it is just that the 7D is having problems with the test using either the 2.0.3 or 2.0.6 firmware.

qqluqq started doing a 550D.109 to 110 update but ran into some problems. Let's see what's possible. First of all since he published his work in progress it looked like it was only a matter of checking what wasn't working. He tried to keep both 550D.109 and 550D.110 working at the same time but that's a bit advanced and not really needed on this camera. A quick check with check-stubs.py turned up a few problems that were very easily resolved. Finally, a few constants changed -- remember, the only thing constant is change.

No, just make a clean break from 109 to 110. It is easy to go back and forth between the two if you feel like testing but don't want to commit until it is accepted into the main repository. I posted both of the firmware updates on my Bitbucket downloads page so testers don't have to search for them.

The only case that we've got more than one firmware version on a camera platform is with the 5D3 because 1.1.3 has fewer issues but 1.2.3 has some new features like full resolution HDMI output. The 1.3.4 update adds some improved AF performance in LiveView with wide angle lenses and a fix for lens aberration correction on the EF 70-300mm lens but it also makes it more difficult to downgrade the firmware.

Ok--just discovered that the Developers have thought of everything, including the possibility of Pelican's site going down. If you go to the downloads page and click on the Useful Links button you'll get a link for the Canon firmware updater:

This links to web.archive.org instead of directly to pel.hu so even though the website is down you can still get the old Canon firmware updater.

Just thought I'd bump this topic because it looks like all of the cameras that are currently behind Canon's latest firmware are now in progress. I've been updating Reply #4 with the status of each of these cameras.

Beyond figuring out where all the function entry points are in the base firmware, what else is typically required when bringing up new hardware? (I currently have a 5D Mark IV and a 6D Mark II, hence my interest.) Does it require anything invasive like soldering JTAG headers to the main board, or is the bootloader sufficiently standardized to make it possible to do a software-only board bringup?