While I've been in Greece, Nikolaus has been busy as well, working on the CPU boards.

There was one nasty issue, that was hard to trace, but which has finally been found this weekend!
Which means: We're now 99,99% sure that the CPU Board is working perfectly fine!
And the mainboard looks great as well, but this will get more extensive testing during the next week.

So, what was that issue?

Well, while the CPU boards were booting, there were some CRC errors when booting from the first MMC slot on the mainboard.

Interestingly, this never happened when using the debug MMC slot which can be soldered directly onto the CPU board.

So... the question to solve was: Why?

Want to know why?
Read the Spoiler!

Was that a reflection from the bus from that debug MMC slot?
Were the traces to the SD Card slot on the mainboard too long and caused issues?

Putting a finger onto the open pads of the debug MMC slot caused these issues to improve... adding a capacitor there didn't do any fixes (and we couldn't deliver the Pyra with a finger included )

Also, some PCBs were not impacted by that issue at all.
And to put some sugar on top: It only happened with the first four tries. After that, the errors were gone and everything worked as it should.

Taking all that into account, it sounded like some tolerance issues - something which is annoying to debug... VERY hard and annoying.

So, based on what we knew, we suspected the following:

1. Open pads causing a reflection / receiving some other distortion signals
2. Timing issues in the MMC driver (software setup). Could also be timing issues with the DRAM, as CRC errors could also be the DRAM and not the MMC.
3. Too long traces which cause a bit of a power loss (would also explain why some work and others didn't, as all PCBs have a bit of tolerances).
4. Some other software programming of the power chip (the PALMAS) that causes the voltage to drop.
5. Soldering process issues causing bad contacts (which could explain why the work after a few seconds, as it makes good contacts when the chip is heated up a bit).

That was quite a bit of possible causes, and finding the real one was a VERY time consuming task.

Nikolaus spent a lot of time studying datasheets of the battery charger chip and the PALMAS power management chip. As the EVM doesn't have a battery charger, the setup is different in our case, so some software configuration for the PALMAS could be wrong.
And while he found some things that needed to be changed in order for the charger to work properly together with the PALMAS, it wasn't the cause of the MMC CRC issues.
Well, at least the work was not useless

As we use a different DRAM than the EVM, Nikolaus rechecked the timing settings as well and made a small program to test the memory.
This was time consuming, as every single change in the timing settings needed a recompile of the bootloader and another long run with the memory tester.
After one day, we knew that our DRAM is working fine - but still the MMC issue remained.

Testing with different temperatures (a cooled PCB, etc.) showed us that the temperature doesn't affect it. There were only read errors for the first few tries, then everything worked fine.
So no temperature problem (and therefore no solder process issue, which would've been bad).

So, next up: Figuring out whether the longer traces could be the issue.
A good idea is testing whether the second SD Card slot also has CRC errors, as the trace length is similar.
However, as the EVM only has one SD Card slot, the second one was not yet included in the bootloader.
Another task for Nikolaus: Adding the second and third SD Card slot to the device tree.

Once that was done, testings showed that no other SD Card slot had any CRC issues, so it's unlikely that the trace length itself was the issue.

So... what COULD be the issue?
What's different using the SD Card slot on the mainboard compared to the one on the CPU board?

Finally, Nikolaus found ONE thing that's different:
There's one condensator on the MMC bus which had been chosen for optimum GPS reception.

And yes: After experimenting a bit, Nikolaus found out that it was IN FACT the condesator!
A tiny change: Using 15pf causes CRC errors, using 12pf already fixes everything!
We'll use 10pf (to give it a bit more tolerance), and that's still good enough to not worsen the GPS reception.

That reminds me a bit of the wrong resistor Michael had to replace on the first few hundred Pandora PCBs, that caused Wifi to be horrible!

I hope I explained everything correctly - I might have not understood everything exactly and maybe have forgotten something, but it should be mostly correct

Well, while it took a while resolving that issue, the good thing is that now all the other MMC buses are included in the device tree as well - that work had to be done anyways.

So, what's next?

Now that the CPU board is up and running, the next step is booting a real Linux system and testing all the remaining hardware components on the mainboard as well.

We don't expect many issues there, as a most of them had already been tested with the debug board and the remaining ones have already appeared on the bus they use, so that should be pretty straightforward.

After all, connecting and using sensors, Wifi modules or other chips to an existing I2C or USB bus is nothing hard (if Linux drivers exist).

Thanks for the update
Would be great to see some videos of the final stages of getting the Pyra up and running. The whole development has been fascinating so far and I've even learnt a few things along the way. Thanks

Thanks for the update
Would be great to see some videos of the final stages of getting the Pyra up and running. The whole development has been fascinating so far and I've even learnt a few things along the way. Thanks

Click to expand...

You can be sure that happens once Nikolaus has fixed all boards and sends me a working set

Great news! I'm so exited. But you changed the capacitor, not a condensator ;-)

Click to expand...

There is no difference between a condenser and a capacitor. Condenser used to be a more traditional term for a Capacitor years ago among English speaking countries. It's still used much in Germany if I'm not mistaken.

Good call on not including a finger. Since I won't be getting one of the first couple handfuls of Pyras by the time you got around to mine you probably wouldn't have any fingers left to send out (or assemble, package and ship) my Pyra.

I am glad we have someone willing to put in the hours to figure things out the right way. Thanks, guys, and keep up the awesome work.

This is indeat a huge milestone, 99,99 % working CPU Board sounds great,
And there allways the tiny parts who cause issues..
So hopefully,when the greece ship the Case Prototyphe to you, whe will mybe see the first working prototyphes whit all parts..

Sounds like you are working really well with Nikolaus, to get that level of trial and error reporting. Many engineers like to keep that stuff to themselves, and only report on the successes, but understanding some of the steps needed to resolve the problems helps us all appreciate the successes more.

FWIW, I'd recommend you translate 'Kondensator' as 'Capacitor' in English, if you can remember. 'Condensator' may be valid historical english usage, but it makes me do a double take as a modern english user. Of course, I know what you mean mind you.

That's it? Such a small change and such a huge effect? Wow, when I think how many components such an PCB has, it really can be searching for the needle in the hay stack. Good to have people that know where to search for.

That's it? Such a small change and such a huge effect? Wow, when I think how many components such an PCB has, it really can be searching for the needle in the hay stack. Good to have people that know where to search for.

Click to expand...

Yes... In fact the change is not huge. It is from 100.00% reliable to 99,95%. Isn't huge, but very significant for digital systems.

The detailled background is:
a) MMC/SD cards typically run at 25.000 MHz clock
b) the 63rd harmonic is therefore at 1575 MHz
c) this is the frequency where the GPS receiver is looking for very very weak signals
d) and the wire from a CPU to an SD slot is a short antenna
e) just some cm away

So running the SD/MMC with rectangular clock signals disturbs the GPS receiver.
We know this effect from both, the Openmoko Freerunner (GTA02) and the GTA04.

The solution is simple: make sure that the signals have weaker harmonics.
I.e. add a low pass filter that significantly attenuates 1575 MHz but not 25 MHz.

This is just this capacitor.

If its value is too big, a nasty effect occurs: the 25 MHz clock becomes a pure
sine wave. With not the full voltage swing. This deteriorates the timing with the
data lines.

So if the capacitor is too big, we get errors on the MMC interface. If it is too
small, we have too much noise for GPS.

Unfortunately we did choose 15pF - which is too big for the OMAP5 MMC interface.
And it was indeed finding the needle in the haystack. But that is the fun if we have it