Posts Tagged ‘gongkai’

About a year and a half ago, I wrote about a $12 “Gongkai” cell phone (pictured above) that I stumbled across in the markets of Shenzhen, China. My most striking impression was that Chinese entrepreneurs had relatively unfettered access to cutting-edge technology, enabling start-ups to innovate while bootstrapping. Meanwhile, Western entrepreneurs often find themselves trapped in a spiderweb of IP frameworks, spending more money on lawyers than on tooling. Further investigation taught me that the Chinese have a parallel system of traditions and ethics around sharing IP, which lead me to coin the term “gongkai”. This is deliberately not the Chinese word for “Open Source”, because that word (kaiyuan) refers to openness in a Western-style IP framework, which this not. Gongkai is more a reference to the fact that copyrighted documents, sometimes labeled “confidential” and “proprietary”, are made known to the public and shared overtly, but not necessarily according to the letter of the law. However, this copying isn’t a one-way flow of value, as it would be in the case of copied movies or music. Rather, these documents are the knowledge base needed to build a phone using the copyright owner’s chips, and as such, this sharing of documents helps to promote the sales of their chips. There is ultimately, if you will, a quid-pro-quo between the copyright holders and the copiers.

This fuzzy, gray relationship between companies and entrepreneurs is just one manifestation of a much broader cultural gap between the East and the West. The West has a “broadcast” view of IP and ownership: good ideas and innovation are credited to a clearly specified set of authors or inventors, and society pays them a royalty for their initiative and good works. China has a “network” view of IP and ownership: the far-sight necessary to create good ideas and innovations is attained by standing on the shoulders of others, and as such there is a network of people who trade these ideas as favors among each other. In a system with such a loose attitude toward IP, sharing with the network is necessary as tomorrow it could be your friend standing on your shoulders, and you’ll be looking to them for favors. This is unlike the West, where rule of law enables IP to be amassed over a long period of time, creating impenetrable monopoly positions. It’s good for the guys on top, but tough for the upstarts.

This brings us to the situation we have today: Apple and Google are building amazing phones of outstanding quality, and start-ups can only hope to build an appcessory for their ecosystem. I’ve reviewed business plans of over a hundred hardware startups by now, and most of them are using overpriced chipsets built using antiquated process technologies as their foundation. I’m no exception to this rule – we use the Freescale i.MX6 for Novena, which is neither the cheapest nor the fastest chip on the market, but it is the one chip where anyone can freely download almost complete documentation and anyone can buy it on Digikey. This parallel constraint of scarce documentation and scarce supply for cutting edge technology forces Western hardware entrepreneurs to look primarily at Arduino, Beaglebone and Raspberry Pi as starting points for their good ideas.

Chinese entrepreneurs, on the other hand, churn out new phones at an almost alarming pace. Phone models change on a seasonal basis. Entrepreneurs experiment all the time, integrating whacky features into phones, such as cigarette lighters, extra-large battery packs (that can be used to charge another phone), huge buttons (for the visually impaired), reduced buttons (to give to children as emergency-call phones), watch form factors, and so forth. This is enabled because very small teams of engineers can obtain complete design packages for working phones – case, board, and firmware – allowing them to fork the design and focus only on the pieces they really care about.

As a hardware engineer, I want that. I want to be able to fork existing cell phone designs. I want to be able to use a 364 MHz 32-bit microcontroller with megabytes of integrated RAM and dozens of peripherals costing $3 in single quantities, instead of a 16 MHz 8-bit microcontroller with a few kilobytes of RAM and a smattering of peripherals costing $6 in single quantities. Unfortunately, queries into getting a Western-licensed EDK for the chips used in the Chinese phones were met with a cold shoulder – our volumes are too small, or we have to enter minimum purchase agreements backed by hundreds of thousands of dollars in a cash deposit; and even then, these EDKs don’t include all the reference material the Chinese get to play with. The datasheets are incomplete and as a result you’re forced to use their proprietary OS ports. It feels like a case of the nice guys finishing last. Can we find a way to still get ahead, yet still play nice?

We did some research into the legal frameworks and challenges around absorbing Gongkai IP into the Western ecosystem, and we believe we’ve found a path to repatriate some of the IP from Gongkai into proper Open Source. However, I must interject with a standard disclaimer: we’re not lawyers, so we’ll tell you our beliefs but don’t construe them as legal advice. Our intention is to exercise our right to reverse engineer in a careful, educated fashion to increase the likelihood that, if push comes to shove, the courts will agree with our actions. However, we also feel that shying away from reverse engineering simply because it’s controversial is a slippery slope: you must exercise your rights to have them. If women didn’t vote and black people sat in the back of the bus because they were afraid of controversy, the US would still be segregated and without universal suffrage.

Sometimes, you just have to stand up and assert your rights.

There are two broad categories of issues we have to deal with, patents and copyrights. For patents, the issues are complex, yet it seems the most practical approach is to essentially punt on the issue. This is what the majority of the open source community does, and in fact many corporations have similar policies at the engineering level. Nobody, as far as we know, checks their Linux commits for patent infringement before upstreaming them. Why? Among other reasons, it takes a huge amount of resources to determine which patents apply, and if one could be infringing; and even after expending those resources, one cannot be 100% sure. Furthermore, if one becomes very familiar with the body of patents, it amplifies the possibility that an infringement, should it be found, is willful and thus triple damages. Finally, it’s not even clear where the liability lies, particularly in an open source context. Thus, we do our best not to infringe, but cannot be 100% sure that no one will allege infringement. However, we do apply a license to our work which has a “poison pill” clause for patent holders that do attempt to litigate.

For copyrights, the issue is also extremely complex. The EFF’s Coders’ Rights Project has a Reverse Engineering FAQ that’s a good read if you really want to dig into the issues. The tl;dr is that courts have found that reverse engineering to understand the ideas embedded in code and to achieve interoperability is fair use. As a result, we have the right to study the Gongkai-style IP, understand it, and produce a new work to which we can apply a Western-style Open IP license. Also, none of the files or binaries were encrypted or had access controlled by any technological measure – no circumvention, no DMCA problem.

Furthermore, all the files were obtained from searches linking to public servers – so no CFAA problem, and none of the devices we used in the work came with shrink-wraps, click-throughs, or other end-user license agreements, terms of use, or other agreements that could waive our rights.

Thus empowered by our fair use rights, we decided to embark on a journey to reverse engineer the Mediatek MT6260. It’s a 364 MHz, ARM7EJ-S, backed by 8MiB of RAM and dozens of peripherals, from the routine I2C, SPI, PWM and UART to tantalizing extras like an LCD + touchscreen controller, audio codec with speaker amplifier, battery charger, USB, Bluetooth, and of course, GSM. The gray market prices it around $3/unit in single quantities. You do have to read or speak Chinese to get it, and supply has been a bit spotty lately due to high Q4 demand, but we’re hoping the market will open up a bit as things slow down for Chinese New Year.

For a chip of such complexity, we don’t expect our two-man team to be able to unravel its entirety working on it as a part-time hobby project over the period of a year. Rather, we’d be happy if we got enough functionality so that the next time we reach for an ATMega or STM32, we’d also seriously consider the MT6260 as an alternative. Thus, we set out as our goal to port NuttX, a BSD-licensed RTOS, to the chip, and to create a solid framework for incrementally porting drivers for the various peripherals into NuttX. Accompanying this code base would be original hardware schematics, libraries and board layouts that are licensed using CC BY-SA-3.0 plus an Apache 2.0 rider for patent issues.

And thus, the Fernvale project was born.

Fernvale Hardware

Compared to the firmware, the hardware reverse engineering task was fairly straightforward. The documents we could scavenge gave us a notion of the ball-out for the chip, and the naming scheme for the pins was sufficiently descriptive that I could apply common sense and experience to guess the correct method for connecting the chip. For areas that were ambiguous, we had some stripped down phones I could buzz out with a multimeter or stare at under a microscope to determine connectivity; and in the worst case I could also probe a live phone with an oscilloscope just to make sure my understanding was correct.

The more difficult question was how to architect the hardware. We weren’t gunning to build a phone – rather, we wanted to build something a bit closer to the Spark Core, a generic SoM that can be used in various IoT-type applications. In fact, our original renderings and pin-outs were designed to be compatible with the Spark ecosystem of hardware extensions, until we realized there were just too many interesting peripherals in the MT6260 to fit into such a small footprint.

Above: early sketches of the Fernvale hardware

We settled eventually upon a single-sided core PCB that we call the “Fernvale Frond” which embeds the microUSB, microSD, battery, camera, speaker, and Bluetooth functionality (as well as the obligatory buttons and LED). It’s slim, at 3.5mm thick, and at 57x35mm it’s also on the small side. We included holes to mount a partial set of pin headers, spaced to be compatible with an Arduino, although it can only be plugged into 3.3V-compatible Arduino devices.

The remaining peripherals are broken out to a pair of connectors. One connector is dedicated to GSM-related signals; the other to UI-related peripherals. Splitting GSM into a module with many choices for the RF front end is important, because it makes GSM a bona-fide user-installed feature, thus pushing the regulatory and emissions issue down to the user level. Also, splitting the UI-related features out to another board costs down the core module, so it can fit into numerous scenarios without locking users into a particular LCD or button arrangement.

Above: Fernvale system diagram, showing the features of each of the three boards

As an interesting side-note, I had some X-rays taken of the MT6260. We did this to help us identify fake components, just in case we encountered units being sold as empty epoxy blocks, or as remarked versions of other chips (the MT6260 has variants, such as the -DA and the -A, the difference being how much on-chip FLASH is included).

X-ray of the MT6260 chip. A sharp eye can pick out the outline of multiple ICs among the wirebonds. Image credit: Nadya Peek

To our surprise, this $3 chip didn’t contain a single IC, but rather, it’s a set of at least 4 chips, possibly 5, integrated into a single multi-chip module (MCM) containing hundreds of wire bonds. I remember back when the Pentium Pro’s dual-die package came out. That sparked arguments over yielded costs of MCMs versus using a single bigger die; generally, multi-chip modules were considered exotic and expensive. I also remember at the time, Krste Asanović, then a professor at the MIT AI Lab now at Berkeley, told me that the future wouldn’t be system on a chip, but rather “system mostly on a chip”. The root of his claim is that the economics of adding in mask layers to merge DRAM, FLASH, Analog, RF, and Digital into a single process wasn’t favorable, and instead it would be cheaper and easier to bond multiple die together into a single package. It’s a race between the yield and cost impact (both per-unit and NRE) of adding more process steps in the semiconductor fab, vs. the yield impact (and relative reworkability and lower NRE cost) of assembling modules. Single-chip SoCs was the zeitgeist at the time (and still kind of is), so it’s interesting to see a significant datapoint validating Krste’s insight.

Reversing the Boot Structure

The amount of documentation made available to Shanzhai engineers in China seems to be just enough to enable them to assemble a phone and customize its UI, but not enough to do a full OS port. You eventually come to recognize that all the phones based on a particular chipset have the same backdoor codes, and often times the UI is inconsistent with the implemented hardware. For example, the $12 phone mentioned at the top of the post will prompt you to plug headphones into the headphone jack for the FM radio to work, yet there is no headphone jack provided in the hardware. In order to make Fernvale accessible to engineers in the West, we had to reconstruct everything from scratch, from the toolchain, to the firmware flashing tool, to the OS, to the applications. Given that all the Chinese phone implementations simply rely upon Mediatek’s proprietary toolchain, we had to do some reverse engineering work to figure out the boot process and firmware upload protocol.

My first step is always to dump the ROM, if possible. We found exactly one phone model which featured an external ROM that we could desolder (it uses the -D ROMless variant of the chip), and we read its contents using a conventional ROM reader. The good news is that we saw very little ciphertext in the ROM; the bad news is there’s a lot of compressed data. Below is a page from our notes after doing a static analysis on the ROM image.

One concern about reverse engineering SoCs is that they have an internal boot ROM that is always run before code is loaded from an external device. This internal ROM can also have signature and security checks that prevent tampering with the external code, and so to determine the effort level required we wanted to quickly figure out how much code was running inside the CPU before jumping to external boot code. This task was made super-quick, done in a couple hours, using a Tek MDO4104B-6. It has the uncanny ability to take deep, high-resolution analog traces and do post-capture analysis as digital data. For example, we could simply probe around while cycling power until we saw something that looked like RS-232, and then run a post-capture analysis to extract any ASCII text that could be coded in the analog traces. Likewise, we could capture SPI traces and the oscilloscope could extract ROM access patterns through a similar method. By looking at the timing of text emissions versus SPI ROM address patterns, we were able to quickly determine that if the internal boot ROM did any verification, it was minimal and nothing approaching the computational complexity of RSA.

Above: Screenshot from the Tek MDO4104B-6, showing the analog trace in yellow, and the ASCII data extracted in cyan. The top quarter shows a zoomed-out view of the entire capture; one can clearly see how SPI ROM accesses in gray are punctuated with console output in cyan.

From here, we needed to speed up our measure-modify-test loop; desoldering the ROM, sticking it in a burner, and resoldering it onto the board was going to get old really fast. Given that we had previously implemented a NAND FLASH ROMulator on Novena, it made sense to re-use that code base and implement a SPI ROMulator. We hacked up a GPBB board and its corresponding FPGA code, and implemented the ability to swap between the original boot SPI ROM and a dual-ported 64kiB emulator region that is also memory-mapped into the Novena Linux host’s address space.

Block diagram of the SPI ROMulator FPGA

There’s a phone in my Novena! What’s that doing there?

A combination of these tools – the address stream determined by the Tek oscilloscope, rapid ROM patching by the ROMulator, and static code analysis using IDA (we found a SHA-1 implementation) – enabled us to determine that the initial bootloader, which we refer to as the 1bl, was hash-checked using a SHA-1 appendix.

Building a Beachhead

The next step was to create a small interactive shell which we could use as a beachhead for running experiments on the target hardware. Xobs created a compact REPL environment called Fernly which supports commands like peeking and poking to memory, and dumping CPU registers.

Because we designed the ROMulator to make the emulated ROM appear as a 64k memory-mapped window on a Linux host, it enables the use a variety of POSIX abstractions, such as mmap(), open() (via /dev/mem), read() and write(), to access the emulated ROM. xobs used these abstractions to create an I/O target for radare2. The I/O target automatically updates the SHA-1 hash every time we made changes in the 1bl code space, enabling us to do cute things like interactively patch and disassemble code within the emulated ROM space.

We also wired up the power switch of the phone to an FPGA I/O, so we could write automated scripts that toggle the power on the phone while updating the ROM contents, allowing us to do automated fuzzing of unknown hardware blocks.

Attaching a Debugger

Because of the difficulty in trying to locate critical blocks, and because JTAG is multiplexed with critical functions on the target device, an unconventional approach was taken to attach a debugger: xobs emulates the ARM core, and uses his fernly shell to reflect virtual loads and stores to the live target. This allows us to attach a remote debugger to the emulated core, bypassing the need for JTAG and allowing us to use cross-platform tools such as IDA on x86 for the reversing UI.

At the heart of this technique is Qemu, a multi-platform system emulator. It supports emulating ARM targets, specifically the ARMv5 used in the target device. A new machine type was created called “fernvale” that implements part of the observed hardware on the target, and simply passes unknown memory accesses directly to the device.

The Fernly shell was stripped down to only support three commands: write, read, and zero-memory. The write command pokes a byte, word, or dword into RAM on the live target, and a read command reads a byte, word, or dword from the live target. The zero-memory command is an optimization, as the operating system writes large quantities of zeroes across a large memory area.

In addition, the serial port registers are hooked and emulated, allowing a host system to display serial data as if it were printed on the target device. Finally, SPI, IRAM, and PSRAM are all emulated as they would appear on the real device. Other areas of memory are either trapped and funneled to the actual device, or are left unmapped and are reported as errors by Qemu.

The diagram above illustrates the architecture of the debugger.

Invoking the debugger is a multi-stage process. First, the actual MT6260 target is primed with the Fernly shell environment. Then, the Qemu virtual ARM CPU is “booted” using the original vendor image – or rather, primed with a known register state at a convenient point in the boot process. At this point, code execution proceeds on the virtual machine until a load or store is performed to an unknown address. Virtual machine execution is paused while a query is sent to the real MT6260 via the Fernly shell interface, and the load or store is executed on the real machine. The results of this load or store is then relayed to the virtual machine and execution is resumed. Of course, Fernly will crash if a store happens to land somewhere inside its memory footprint. Thus, we had to hide the Fernly shell code in a region of IRAM that’s trapped and emulated, so loads and stores don’t overwrite the shell code. Running Fernly directly out of the SPI ROM also doesn’t work as part of the initialization routine of the vendor binary modifies SPI ROM timings, causing SPI emulation to fail.

Emulating the target CPU allows us to attach a remote debugger (such as IDA) via GDB over TCP without needing to bother with JTAG. The debugger has complete control over the emulated CPU, and can access its emulated RAM. Furthermore, due to the architecture of qemu, if the debugger attempts to access any memory-mapped IO that is redirected to the real target, the debugger will be able to display live values in memory. In this way, the real target hardware is mostly idle, and is left running in the Fernly shell, while the virtual CPU performs all the work. The tight integration of this package with IDA-over-GDB also allows us to very quickly and dynamically execute subroutines and functions to confirm their purpose.

Below is an example of the output of the hybrid Qemu/live-target debug harness. You can see the trapped serial writes appearing on the console, plus a log of the writes and reads executed by the emulated ARM CPU, as they are relayed to the live target running the reduced Fernly shell.

From this beachhead, we were able to discover the offsets of a few IP blocks that were re-used from previous known Mediatek chips (such as the MT6235 in the osmocomBB http://bb.osmocom.org/trac/wiki/MT6235) by searching for their “signature”. The signature ranged from things as simple as the power-on default register values, to changes in bit patterns due to the side effects of bit set/clear registers located at offsets within the IP block’s address space. Using this technique, we were able to find the register offsets of several peripherals.

Booting an OS

From here we were able to progress rapidly on many fronts, but our goal of a port of NuttX remained elusive because there was no documentation on the interrupt controller within the canon of Shanzhai datasheets. Although we were able to find the routines that installed the interrupt handlers through static analysis of the binaries, we were unable to determine the address offsets of the interrupt controller itself.

At this point, we had to open the Mediatek codebase and refer to the include file that contained the register offsets and bit definitions of the interrupt controller. We believe this is acceptable because facts are not copyrightable. Justice O’Connor wrote in Feist v. Rural (449 U.S. 340, 345, 349 (1991). See also Sony Computer Entm’t v. Connectix Corp., 203 F. 3d 596, 606 (9th Cir. 2000); Sega Enterprises Ltd. v. Accolade, Inc., 977 F.2d 1510, 1522-23 (9th Cir. 1992)) that

“Common sense tells us that 100 uncopyrightable facts do not magically change their status when gathered together in one place. … The key to resolving the tension lies in understanding why facts are not copyrightable: The sine qua non of copyright is originality”

and

“Notwithstanding a valid copyright, a subsequent compiler remains free to use the facts contained in another’s publication to aid in preparing a competing work, so long as the competing work does not feature the same selection and arrangement”.

And so here, we must tread carefully: we must extract facts, and express them in our own selection and arrangement. Just as the facts that “John Doe’s phone number is 555-1212” and “John Doe’s address is 10 Main St.” is not copyrightable, we need to extract facts such as “The interrupt controller’s base address in 0xA0060000”, and “Bit 1 controls status reporting of the LCD” from the include files, and re-express them in our own header files.

The situation is further complicated by blocks for which we have absolutely no documentation, not even an explanation of what the registers mean or how the blocks function. For these blocks, we reduce their initialization into a list of address and data pairs, and express this in a custom scripting language called “scriptic”. We invented our own language to avoid subconscious plagiarism – it is too easy to read one piece of code and, from memory, code something almost exactly the same. By transforming the code into a new language, we’re forced to consider the facts presented and express them in an original arrangement.

Scriptic is basically a set of assembler macros, and the syntax is very simple. Here is an example of a scriptic script:

The original code actually goes on for pages and pages, and even this snippet is surrounded by conditional statements which we culled as they were not relevant facts to initializing the PLL correctly.

With this tool added to our armory, we were finally able to code sufficient functionality to boot NuttX on our own Fernvale hardware.

Toolchain

Requiring users to own a Novena ROMulator to hack on Fernvale isn't a scalable solution, and thus in order to round out the story, we had to create a complete developer toolchain. Fortunately, the compiler is fairly cut-and-dry – there are many compilers that support ARM as a target, including clang and gcc. However, flashing tools for the MT6260 are much more tricky, as all the existing ones that we know of are proprietary Windows programs, and Osmocom's loader doesn't support the protocol version required by the MT6260. Thus, we had to reverse engineer the Mediatek flashing protocol and write our own open-source tool.

Fortunately, a blank, unfused MT6260 shows up as /dev/ttyUSB0 when you plug it into a Linux host – in other words, it shows up as an emulated serial device over USB. This at least takes care of the lower-level details of sending and receiving bytes to the device, leaving us with the task of reverse engineering the protocol layer. xobs located the internal boot ROM of the MT6260 and performed static code analysis, which provided a lot of insight into the protocol. He also did some static analysis on Mediatek's Flashing tool and captured live traces using a USB protocol analyzer to clarify the remaining details. Below is a summary of the commands he extracted, as used in our open version of the USB flashing tool.

After about a year of on-and-off effort between work on the Novena and Chibitronics campaigns, we were able to boot a port of NuttX on the MT6260. A minimal set of hardware peripherals are currently supported; it’s enough for us to roughly reproduce the functionality of an AVR used in an Arduino-like context, but not much more. We’ve presented our results this year at 31C3 (slides).

The story takes an unexpected twist right around the time we were writing our CFP proposal for 31C3. The week before submission, we became aware that Mediatek released the LinkIT ONE, based on the MT2502A, in conjunction with Seeed Studios. The LinkIT ONE is directly aimed at providing an Internet of Things platform to entrepreneurs and individuals. It’s integrated into the Arduino framework, featuring an open API that enables the full functionality of the chip, including GSM functions. However, the core OS that boots on the MT2502A in the LinkIT ONE is still the proprietary Nucleus OS and one cannot gain direct access to the hardware; they must go through the API calls provided by the Arduino shim.

Realistically, it’s going to be a while before we can port a reasonable fraction of the MT6260’s features into the open source domain, and it’s quite possible we will never be able to do a blob-free implementation of the GSM call functions, as those are controlled by a DSP unit that’s even more obscure and undocumented. Thus, given the robust functionality of the LinkIT ONE compared to Fernvale, we’ve decided to leave it as an open question to the open source community as to whether or not there is value in continuing the effort to reverse engineer the MT6260: How important is it, in practice, to have a blob-free firmware?

Regardless of the answer, we released Fernvale because we think it’s imperative to exercise our fair use rights to reverse engineer and create interoperable, open source solutions. Rights tend to atrophy and get squeezed out by competing interests if they are not vigorously exercised; for decades engineers have sat on the sidelines and seen ever more expansive patent and copyright laws shrink their latitude to learn freely and to innovate. I am saddened that the formative tinkering I did as a child is no longer a legal option for the next generation of engineers. The rise of the Shanzhai and their amazing capabilities is a wake-up call. I see it as evidence that a permissive IP environment spurs innovation, especially at the grass-roots level. If more engineers become aware of their fair use rights, and exercise them vigorously and deliberately, perhaps this can catalyze a larger and much-needed reform of the patent and copyright system.

I recently did an interview for CSDN.net, which stands for “China Software Developer Network”, or more colloquially, “Programmer Magazine”, about open source hardware, and what it means to be a hacker.

[biographical details and photo omitted from this translation. Caption to the bio photo reads: “My PhD advisor in college, Dr. Tom Knight, had a profound impact upon me. I often asking myself “what would Tom do?” when I am stuck or lost, and the answer I come back with is usually the right one.”]

About Open-source Hardware and the Maker movement

CSDN: The Maker and Open hardware movements attract a lot of attention recent years, Chris Anderson wrote a book Makers, Paul Graham called it The Hardware Renaissance. From your perspective and observation, how do you think about this movement will affect ordinary people, developers and our IT industry?

bunnie: This movement, as it may be, is more of a symptom than a cause, in my opinion. Let’s review how we got here today.

In 1960, for all practical purposes there was only hardware, and it was all open. When you bought a transistor radio, it had its schematic printed in the back. If it broke, you had to fix it yourself. It was popular to buy kits to make your own radios.

Around 1980-1990, the personal computer revolution began. Computers started to become powerful enough to run software that was interesting and enabling.

From 1990-2005, Moore’s Law drove computers to be twice as fast, and have twice as much memory every 1.5-2 years. All that mattered in this regime is the software; unless you could afford to fab a chip in the latest technology, it wasn’t worth it to make hardware, because by the time you got the components together a new chip was out that made your design look slow. It also wasn’t worth it to optimize software. By the time you made your software optimized, you could have bought a faster computer and run the old software even faster. What mattered was features, convenience, creativity. “Making” fell out of fashion: you had to ship code or die, there was no time to make.

From 2005-2010, computers didn’t get much faster in terms of MHz, but they got smaller. Smartphones were born. Everything became an app, and everything is becoming connected.

From about 2010-now, we find that Moore’s Law is slowing down. This slowdown is rippling through the innovation chain. PCs are not getting faster, better, or cheaper in a meaningful way; basically we buy a PC just to replace a broken one, not because it’s so much better. It’s a little bit too early to tell, but smartphones may also be solidifying as a platform: the iPhone5 is quite similar to the iPhone4; the Samsung phones are also looking pretty similar across revisions.

How to innovate? how to create market differentiation? With Moore’s Law slowing down, it’s now possible to innovate in hardware and not have your innovation look slow because a new chip came out. You have steady platforms (PCs, smartphones, tablets) which you can target your hardware ideas toward. You don’t have to necessarily be fabbing chips just to have an advantage. We are now sifting through our past, looking for niches that were overlooked during the initial fast-paced growth of technology. Even an outdated smartphone motherboard can look amazing when you put it in quadcopters, satellites, HVAC systems, automobiles, energy monitoring systems, health monitoring systems, etc.

Furthermore, as humans we fundamentally feel differently toward “real” things as opposed to “virtual” things. As wonderful as apps are, a human home consists of more than a smartphone, a food tray, a bed and a toilet. We still surround ourselves with knick-knacks, photos of friends, physical gifts we give each other on special occasions. I don’t think we will ever get to the point where getting a “virtual” teddy bear app will be as valued as getting a “real” teddy bear.

As a result, there will always be a place for people to make hardware that fills this need for tangible goods. This hardware will merge more technology and run more software, but in the end, there is a space for Makers and hardware startups, and that space is just getting bigger now that hardware technology is stabilizing.

CSDN: Compared with the past, Arduino and Raspberry Pi’s appearance seems to reduce the threshold of doing hardware design, what do you think this will effect hardware product industry? Do you think these platforms will bring real leaps and bounds in the industry? Or to make really innovative hardware product, what do we need?

bunnie: Arduino and RPi serve specific market niches.

Arduino’s key contribution is the reduction of computation to a easy-to-use physical form. It was made first and foremost by designers and artists, and less so by technologists. This unique perspective on technology turned out to be very powerful: it turns out people who aren’t programmers or hardware designers also want to access hardware technology.

Some very moving and deep interactive art pieces have been made using the Arduino, allowing hardware to transcend menial control applications into something that changes your mood or makes you think about life differently. I think Arduino is just the first step toward taking the “tech” out of technology and letting every day people not just use technology, but create with it. There will be other platforms, for sure.

Raspberry Pi is a very inexpensive embedded hardware reference module; it is cheap enough that for many applications, it’s just fine to buy the RPi as-is and you don’t have to design your own hardware. I think there will also be other followers in their footsteps. The nice thing about RPi for hardware professionals is that instead of buying a reference design and then having to spin your own board, you can just buy the RPi and ship it in the product. For people who have relatively low-volume products, this makes sense.

As an ongoing trend, I see product design becoming more feasible at low volumes. There will still be the million-unit blockbusters for things like smartphones and coffee makers, but there will also be a new market for 1k-10k of something, but with a much higher margin. These small run products will be developed and sold by teams of just one or two people, so that the profit on such a small run of product is still a good living for the individuals. The key to the success of these products is that they are highly customized and help solve the exact problem a small group of users have, and because of this the users are willing to pay
more for it.

CSDN: When new concept or technologies first appear, they always cause a lot of optimism discussion, but most of them, only take a long time development, will really affect our lives. When we are talking about Marker or Open hardware movement, are we too optimistic? For the average person, if there are common misunderstandings about this field?

bunnie: Yes, it does take a long time for technology to really change our lives.

The Maker movement, I think, is less about developing products, and more about developing people. It’s about helping people realize that technology is something man-made, and because of this, every person has the power to control it: it just takes some knowledge. There is no magic in technology. Another way to look at it is, we can all be magicians with a little training.

Open Hardware is more of a philosophy. The success or failure of a product is largely disconnected with whether the hardware is open or closed. Closing hardware doesn’t stop people from cloning or copying, and opening hardware doesn’t mean that bad ideas will be copied simply because they are open. Unlike software, hardware requires a supply chain, distribution, and a network of relationships to build it at a low cost. Because of this, hardware being open or closed is only a small part of the equation for hardware, and mostly it’s a question of how much you want to involve end users or third parties to modify or interoperate with your product.

CSDN: When we look at the future of open source hardware, could we analogy it with the open source software industry that we have already seen (many commercial companies also support open source software, many software companies to provide support for the open source software for living)? What are the difference between them?

bunnie: I don’t think the analogy is quite the same. In software, the cost to copy, modify, and distribute is basically zero. I can download a copy of linux, and run “make” and have the same high-quality kernel running on my desktop as runs on top-end servers and supercomputers.

In hardware, there is a real cost to copy hardware. And the cost of the parts, the factories and skilled workers used to build them, the quality control procedures, and the process to build are all important factors in how the final hardware cost, looks, feels, and performs. Simply giving someone a copy of my schematics and drawings doesn’t mean they can make exactly my product. Even injection molding has art to it: if I give the same CAD drawing to two tooling makers, the outcome can be very different depending on where the mold maker decides to place the gates, the ejector pins, the cooling for the mold, the mold cycle time, temperature, etc.

And then there is the distribution channel, reverse logistics, financing, etc.; even as the world becomes more efficient at logistics, you will never be able to buy a TV as easily as you can download the movies that you watch on the same TV.

CSDN: What kind of business model do you think is ideal for open source hardware company? Could you give an example?

bunnie: One of my key theories behind open source hardware is that hardware, at least at the level of schematics and PCB layout, is “essentially open”. This is because for a relatively small amount of money you can pay services to extract the detail required to copy a PCB design. Therefore an asymptotic assumption is that once you have shipped hardware, it can be copied.

If you can accept this assumption, then not releasing schematics and PCB layouts will not stop people from copying your goods. It will be copied if someone wants to copy it. So it makes no difference to copying whether or not you have shared your design files.

However, if you do share your design files, it does make a difference to a different and important group of people. There are other businesses and individual innovators who could use your design files to design accessories, upgrades, or third party enhancements that rely upon your product.

Thus, in a “limiting case”, sharing your design files doesn’t change the situation on copying, but does improve your opportunity for new business relationships. Therefore, the practical suggestion is to just share the design files, using the guidelines of the open source hardware licenses to help reserve a few basic rights and protections.

Clearly, there are some hardware strategies that are not compatible with the idea of open hardware. If your sole value to the consumer is your ability to make stand-alone hardware, and you have no strategic advantage in terms of cost, then you would like to keep your plans secret to try to delay the low-cost copiers for as long as you can.

However, we are finding today that the most innovative products are not just a piece of hardware, but they also involve software and services. Open hardware business models work better when they are in such hybrid products. In many cases, consumers are willing to pay annuity revenue in some form (e.g. subscriptions, advertising, upsells, accessories, royalties and upgrades) for many types of products. In fact, it is most profitable to just collect these fees and not involve yourself in the hardware manufacturing portion; and it is much easier to control access to an ongoing service than it is to the plans of a piece of hardware.

Thus, if you are running an on-line service that is coupled to your hardware, open hardware makes a lot of sense: if your on-line service is profitable, letting other people copy the hardware, sell it, and then add more users to your on-line service simply means you get more revenue without more risk in the production of hardware.

CSDN: We know you often come to China and know a lot about this country. China’s software technology is not advanced, do you think its current position — the world factory center will help it improve its overall level of technology? How can this country change to a design, research and development focus place but not just a manufacturing center? What is China missing?

bunnie: I wouldn’t say I know much about China. I know a little bit about one small corner of China in one specific area — hardware manufacturing. If there is one thing I do know about China, it’s that it is a very big country, with many different kind of people, and a long history that I am only beginning to understand.

However, the history of high technology is almost entirely contained within my life span, so I can comment on the relationship between high technology and people, from which we can derive some perspective about China.

The first observation is that every technology power-house today started with manufacturing. The US started as simple colonies of Britain, mining ores, trapping furs and farming cotton and tobacco. Over time, the US had steel mills and linen production. It wasn’t until the early 1900’s before the US really started to rise in developing original technology, and not until the mid 1900’s before things really took off.

The same goes for Japan. They started in manufacturing, copying many of the US made goods. In fact the first cars and radios they made were, if you believe the historical accounts, not so good. It took the US and Japan decades to go from a manufacturing-based economy to a service-based economy.

If you compare that to China, the electronics manufacturing industry started there maybe only 20 years ago, at most, and China is just turning the corner from being a manufacturing-oriented economy to one that can do more design and software technology. I believe this is a natural series of events: some portion of entry-level workers will eventually become technicians, then some technicians become designers, then some designers become successful entrepreneurs.

In terms of concrete numbers, if you have 10 million factories workers, maybe 1% of them will learn enough from their job such that after a few years they can become technicians. This gives you 100,000 technicians. After a few years of technician work, maybe 1% will gain enough skill to become original designers. This gives you 1,000 designers. These experienced, grass-roots designers become the core of your entrepreneurial economy, and from there the economy begins to transform.

From a thousand companies, you will eventually winnow down to just a handful of global brand companies. This whole process takes a decade or two, and I believe we are currently witnessing China going through the final phase of the transformation, where we now have a lot of people in Shenzhen who have the experience of manufacturing, the wisdom to do design, and now are just starting to apply their talent to innovation and original product design. The next decade will be an exciting one for China’s technology industry, if the current policies on economic and intellectual development stay roughly on course.

This pattern applies primarily to hardware or hardware-dominated products. Software products have a similar pattern, but I believe there are unique cultural aspects that can give the West an advantage in software design. In hardware, if a process is not efficient or producing low yield, one can easily identify the root cause and produce direct physical evidence of the problem. Hardware problems, in essence, are indisputable.

In software, if code is not efficient or poorly written, it’s very hard to identify the exact problem that causes it. One can see the evidence of programs crashing or running slowly, but there is no broken wire or missing screw you can simply hold up and show everyone to show why the software is broken. Instead, you need to sit down with your collaborators and review a complex design, consider many opinions, and ultimately you need to identify a problem which is ultimately due to a bad decision made by an individual, and nothing more than that. All software APIs are simply constructs of human opinions: nothing more.

Asian cultures have a strong focus on guanxi, reputation, and respect for the elders. The West tends to be more rebellious and willing to accept outsiders as champions, and they have less respect for the advice of elders. As a result, I think it’s very culturally difficult in an Asian context to discuss code quality and architectural decisions. The field of software itself is only 30 years old, and older, more experienced engineers are also the most out of date in terms of methodology and knowledge. In fact, the young engineers often have the best ideas. However, if it’s culturally difficult for the young engineers to challenge the decisions of the elder engineers, you end up with poorly architected code, and you have no hope to be competitive.

It’s not hopeless to overcome these obstacles, but it requires a very strong management philosophy to enforce the correct incentives and culture. The workers should be rewarded fairly for making correct decisions, and there can be no favorites based upon friendship, relationship, or seniority. Senior engineers and managers must see a real financial reward for accepting their mistakes, instead of saving face by forcing junior engineers to code patches around their bad high level decisions. Usually, this alignment is achieved in US contexts by sharing equity in a company among the engineers, so that the big payout only comes if the company as a whole survives, regardless of the ego of the individual.

CSDN: What do you think the role (or relationship) of individual maker and commercial company in the future? And as individual maker may not only compete with commercial companies, but also will compete with other makers in the future, what are the factors critical to the success of a product?

bunnie: As a general trend, MOQs are reducing and innovation is getting closer to the edge. So I think commercial companies will see more competition from Makers, especially as the logistics industry transforms itself into a API that can plug directly into websites.

At the end of the day, the most critical factors to success will still be how much value the consumer will perceive from the product. A part of this is related to superior features and good product quality, but also an important part of this is in the presentation to the consumer and how clearly the benefits are explained. As a result, it’s important to make sure the product is visually appealing, easy to use, and to create marketing material that clearly explains the benefit of the product to the customer. This is often a challenge for individual makers, because their talent is normally in the making of the product’s technical value, but less so on the marketing and sales value. Makers who can master both will have an edge over makers who focus specifically on offering just a technical value.

About Hardware Hackers

CSDN: You have participated in the development process of many products, what is your personal goal? In this long period, what is the greatest pleasure?

bunnie: I would like to make people happy by building things that improve their life in some way. The greatest pleasure is to see someone enjoying something you have made because you have improved their life in some small way. Sometimes your product is solving a big problem for them. Other times it is more whimsical and happiness comes from fun or beauty. But either way, you are helping another person.

That is what is important to me. One thing I have learned in the past few years is that money beyond a certain level doesn’t make me any happier. This makes me difficult to work with, because it’s hard for people to just hire my service by offering me a lot of money. Instead, they need to convince me that the activity will somehow also make people happy.

Another important goal for me is to just understand how the world works. I have a natural curiosity and I want to learn and understand all kinds of things. The universe has a lot of patterns to it, and sometimes you will find seemingly unrelated pieces fitting together just like magic. Discovering these links and seeing the world fit together like a big jigsaw puzzle is very profound and satisfying.

CSDN: Which project you focus on recently? Why you choose them?

bunnie: I haven’t really been focused, actually. Recently, I have intentionally been *not* focused. If you focus on one idea for too long, many good innovative ideas just pass you by.

One of the hardest parts of being an entrepreneur or innovator is to have the patience to review many ideas, know when to say no to bad ideas, and then cultivate a few good ideas at the same time, and to accept many changes to your concept along the way. This process can take months, if not years.

I have a project to build my own laptop, but it’s not meant to be a business. It’s a subject meant to encourage personal growth, to challenge my abilities and learn things that I don’t know about the entire computer design process.

I have a project to reverse engineer the firmware in SD cards. It’s also not a business, it’s meant to satisfy my curiosity.

I have a project to learn how companies in Shenzhen build such amazing phones for such a low price. If I can learn some of their techniques, then hopefully I can apply this knowledge to create new and compelling products that make people happy.

I have a project to build flexible circuit stickers. It will hopefully help introduce more people into using technology in their everyday life. It will also challenge my understanding of manufacturing processes, as I must develop a few new processes that don’t exist today to make the stickers robust and inexpensive enough for every day use.

I have a project to help advise new entrepreneurs and innovators, and help get them started on their adventures. I may not have the best ideas, or all the talent needed to make a business, but if I can help teach others to fly, then maybe that can make up for my inability to find success in business.

So, I have many projects, and no focus. Maybe someday I will find one to focus on — sometimes it is also important to focus — but as of today, I haven’t found what I’m looking for.

CSDN: Failure tends to give people more experience, could you talk about the not-so-successful projects you have participated, or if you’ve ever seen other failed projects that inspired you.

bunnie: My life is a story of failures. The only thing I have done repeatedly and reliably is fail. However, I have two rules when handling failure: (1) don’t give up and (2) don’t make the same mistake twice. If you follow these rules, eventually, you will find a success after many failures.

Actually, I have an interview that focuses on one of my recent failures. You can read it here.

CSDN: Your book Hacking the Xbox has been published ten years, for people who want to learn reverse engineering, or want to become a hardware hacker today, whether these experiences and skills has still apply?

bunnie: I’d like to think that the core principles covered in the book are still relevant today. The Xbox is simply an example of how to do things, but the approach described in the book and the techniques are applicable to a broad range of problems.

For the Chinese audience, I have found the mobile phone repair manuals to be quite interesting to read. I have bought a couple of them, and even though I can’t read Chinese well, I still find them pretty interesting. Sometimes their descriptions on the theory of electronics are not completely accurate, but practically speaking they are good enough, and it’s a quick way to get started while learning immediately useful skills in repairing phones.

There’s also a Chinese magazine, 无线电, which I have found to be quite good. If you get started building the projects in there, I think you will learn very quickly.

CSDN: For users, the new Xbox One has more stringent restrictions, what do you think about this? Do you have interesting to explore this black box and upgrade your book?

bunnie: I haven’t done much work on video game consoles in a while — there is a whole new generation of console hackers who are excited to explore them, and I’m happy for that. As for the Xbox One security, I’m sure it is one of the most secure systems built. They did a very good job on the Xbox360, and I know some of the Xbox One security team members and they have a very solid understanding of the principles needed to build secure hardware. It should be very hard to crack.

That being said, I’m glad I have no desire to buy or use one. I think it would become very frustrated with their use policies and restrictions very quickly.

CSDN: There are a lot of controversy about if electronic devices should have a lock to prevent user rooting, what do you think about this? Whether there is a contradiction between ensure the safety of user and make user get complete control of their device?

bunnie: I believe users should “own” their hardware, and “owning” means having the right to modify, change, etc. including root access rights. If the company has a concern about users being unsafe, then it’s easy enough to include an “opt-out” where users can simply select an electronic waiver form, and give up their support and warranty right to gain access to their own machine. Most people who care to root their machine are already smarter than the phone support they would be calling inside the company, so anyways it’s not a problem.

The laws have changed to make it illegal to do some rooting activities, even to hardware that you bought and own. I think this reduction in our natural rights of ownership is dangerous and can lead to consumers being put in an unfair situation, and it also discourages consumers to explore and learn more about the technology they have become so dependent upon.

CSDN: The integration of hardware is increasing, do you think doing hardware hacking is getting more and more difficult or do you worry about the hardware hackers to extinct? How could we change this situation?

bunnie: The increasing integration has been true for a long time — from the TX-0 which just used transistors, to the Apple II which used TTL ICs, to the PCs which used controller chipsets, to the mobile phones which have just a single SoC now. It does make it harder to hack some parts, but there is always opportunities at the system integration level.

In other words, I still think there is art in hardware, just the level at which hardware hackers have to work gets higher every day, and this is a good thing, because it means our hacks are also getting more powerful with time as well.

CSDN: Your book is dedicated to Aaron Swartz, could you talk about why do you think the hacker spirit is important in our era today.

bunnie: The hacker spirit is the ultimate expression of human problem solving ability. It’s about the ability to see the world for what it is, and not the constructs and conventions that society puts in place. A brick is not just used to make buildings, it can be a doorstop, a weapon, a paperweight, a heating ballast, or it can be ground up and used for soil. Hackers question convention through the lens of doing what’s most practical and correct for the situation at hand: they see things for what they are, and not by the labels put on them. Sometimes their methods are not always harmonious, as hackers often times prioritize doing the right thing over being nice or playing by the rules.

I find the more difficult situations become, the more pervasive and stronger the hacker spirit becomes among common people. I see evidence of this around the world. It is linked to the human will to survive and to thrive. I think it’s important for a society to culture and tolerate the hacker spirit. Not everyone has it, but the few who do have it help make society more resilient and survivable in hard times.

CSDN: Do you have other words you would like to share with Chinese readers?

bunnie: Recently, I was reading some comments on a Chinese web forum, and it seems that many Chinese regard the term “Shanzhai” as a negative term. This was surprising to me, because as an outsider, I feel that the Shanzhai have done a lot of very interesting and useful innovation. I think in English, we have a similar problem. The term “hacker” in English started as a good term, but over time became associated with many kinds of negative acts. Recently the term “Maker” was coined to distinguish between the positive and negative aspects of hackers (I still call myself a hacker because I still adhere to the traditional definition of the word).

It may be easier to explain the innovation happening in China, if in Chinese a similar linguistic bifurcation could happen. I had recently proposed referring to the innovative, open aspects of what the Shanzhai do as “gongkai” (公开), referring to their method of sharing design files. Significantly, the term 开放 as used in 开放源代码 I feel doesn’t quite apply, because it refers to a specific Western-centric legal aspect of being open, which is not applicable to the methods engaged in the Chinese ecosystem.

However, the fact that China has found its own way to share IP, unique from the Western system, doesn’t mean it’s bad. I think in fact, it’s quite interesting and I’m very curious to see where it goes. Since I see positive value in some of the methods that the Shanzhai use, I’d propose using the more positive/generic term “gongkai” to describe the style of IP sharing commonly engaged in China.

But then again, who am I to say — I’m not a native Chinese speaker, and maybe there is a much better way to address the situation.

Recently, I paid $12 at Mingtong Digital Mall for a complete phone, featuring quad-band GSM, Bluetooth, MP3 playback, and an OLED display plus keypad for the UI. Simple, but functional; nothing compared to a smartphone, but useful if you’re going out and worried about getting your primary phone wet or stolen.

Also, it would certainly find an appreciative audience in impoverished and developing nations.

$12 is the price paid for a single quantity retail, contract-free, non-promotional, unlocked phone — in a box with charger, protective silicone sleeve, and cable. In other words, the production cost of this phone is somewhere below the retail price of $12. Rumors place it below $10.

This is a really amazing price point. That’s about the price of a large Domino’s cheese pizza, or a decent glass of wine in a restaurant. Or, compared to an Arduino Uno (admittedly a little unfair, but humor me):

How is this possible? I don’t have the answers, but it’s something I’m trying to learn. A teardown yields a few hints.

First, there are no screws. The whole case snaps together.

Also, there are (almost) no connectors on the inside. Everything from the display to the battery is soldered directly to the board; for shipping and storage, you get to flip a switch to hard-disconnect the battery. And, as best as I can tell, the battery also has no secondary protection circuit.

The Bluetooth antenna is nothing more than a small length of wire, seen on the lower left below.

Still, the phone features accoutrements such as a back-lit keypad and decorative lights around the edge.

The electronics consists of just two major ICs: the Mediatek MT6250DA, and a Vanchip VC5276. Of course, with price competition like this, Western firms are suing to protect ground: Vanchip is in a bit of a legal tussle with RF Micro, and Mediatek has also been subject to a few lawsuits of its own.

The MT6250 is rumored to sell in volume for under $2. I was able to anecdotally confirm the price by buying a couple of pieces on cut-tape from a retail broker for about $2.10 each. [No, I will not broker these chips or this phone for you…]

That beats the best price I’ve ever been able to get on an ATMega of the types used in an Arduino.

Of course, you can’t just call up Mediatek and buy these; and it’s extremely difficult to engage with them “going through the front door” to do a design. Don’t even bother; they won’t return your calls.

However, if you know a bit of Chinese, and know the right websites to go to, you can download schematics, board layouts, and software utilities for something rather similar to this phone…”for free”. I could, in theory, at this point attempt to build a version of this phone for myself, with minimal cash investment. It feels like open-source, but it’s not: it’s a different kind of open ecosystem.

Introducing Gongkai

Welcome to the Galapagos of Chinese “open” source. I call it “gongkai” (公开). Gongkai is the transliteration of “open” as applied to “open source”. I feel it deserves a term of its own, as the phenomenon has grown beyond the so-called “shanzhai” (山寨) and is becoming a self-sustaining innovation ecosystem of its own.

Just as the Galapagos Islands is a unique biological ecosystem evolved in the absence of continental species, gongkai is a unique innovation ecosystem evolved with little western influence, thanks to political, language, and cultural isolation.

Of course, just as the Galapagos was seeded by hardy species that found their way to the islands, gongkai was also seeded by hardy ideas that came from the west. These ideas fell on the fertile minds of the Pearl River delta, took root, and are evolving. Significantly, gongkai isn’t a totally lawless free-for-all. It’s a network of ideas, spread peer-to-peer, with certain rules to enforce sharing and to prevent leeching. It’s very different from Western IP concepts, but I’m trying to have an open mind about it.

I’m curious to study this new gongkai ecosystem. For sure, there will be critics who adhere to the tenets of Western IP law that will summarily reject the notion of alternate systems that can nourish innovation and entrepreneurship. On the other hand, it’s these tenets that lock open hardware into technology several generations old, as we wait for patents to expire and NDAs to lift before gaining access to the latest greatest technology. After all, 20 years is an eternity in high tech.

I hope there will be a few open-minded individuals who can accept an exploration of the gongkai Galapagos. Perhaps someday we can understand — and maybe even learn from — the ecosystem that produced the miracle of the $12 gongkai phone.