Archive for the ‘Ponderings’ Category

There is a lot of righteous anger directed toward Intel over CPU bugs that were revealed by Spectre/Meltdown. I agree that things could have been handled better, particularly with regards to transparency and the sharing of information among the relevant user communities that could have worked together to deploy effective patches in a timely fashion. People also aren’t wrong that consumer protection laws obligate manufacturers to honor warranties, particularly when a product is not fit for use as represented, if it contains defective material or workmanship, or fails to meet regulatory compliance.

However, as an open source hardware optimist, and someone who someday aspires to see more open source silicon on the market, I want to highlight that demanding Intel return, exchange, or offer rebates on CPUs purchased within a reasonable warranty period is entirely at odds with demands that Intel act with greater transparency in sharing bugs and source code.

Transparency is Easy When There’s No Penalty for Bugs

It’s taken as motherhood and apple pie in the open source software community that transparency leads to better products. The more eyes staring at a code base, the more bugs that can be found and patched. However, a crucial difference between open source software and hardware is that open source software carries absolutely no warranty. Even the most minimal, stripped down OSS licenses stipulate that contributors carry no liability. For example, the BSD 2-clause license has 189 words, of which 116 (60%) are dedicated to a “no warranty” clause – and all in caps, in case you weren’t paying attention. The no-warranty clause is so core to any open source license it doesn’t even count as a clause in the 2-clause license.

Of course contributors have no liability: this lack of liability is fundamental to open source. If people could sue you for some crappy code you you pushed to github years ago, why would you share anything? Github would be a ticking time bomb of financial ruin for every developer.

It’s also not about code being easier to patch than hardware. The point is that you don’t have to patch your code, even if you could. Someone can file a bug against you, and you have the legal right to ignore it. And if your code library happens to contain an overflow bug that results in a house catching fire, you walk away scot-free because your code came with no warranty of fitness for any purpose whatsoever.

Oohh, Shiny and New!

Presented a bin of apples, most will pick a blemish-free fruit from the bushel before heading to the check-out counter. Despite the knowing the reality of nature – that every fruit must grow from a blossom under varying conditions and hardships – we believe our hard-earned money should only go toward the most perfect of the lot. This feeling is so common sense that it’s codified in the form of consumer protection laws and compulsory warranties.

This psychology extends beyond obvious blemishes, to defects that have no impact on function. Suppose you’re on the market to buy a one-slot toaster. You’re offered two options: a one-slot toaster, and a two-slot toaster but with the left slot permanently and safely disabled. Both are exactly the same price. Which one do you buy?

Most people would buy the toaster with one slot, even though the net function might be identical to the two-slot version where one slot is disabled. In fact, you’d probably be infuriated and demand your money back if you bought the one-slot toaster, but opened the box to find a two-slot toaster with one slot disabled. We don’t like the idea of being sold goods that have anything wrong with them, even if the broken piece is irrelevant to performance of the device. It’s perceived as evidence of shoddy workmanship and quality control issues.

News Flash: Complex Systems are Buggy!

Hold your breath – I’d wager that every computer you’ve bought in the past decade has broken parts inside of them, almost exactly like the two-slot toaster with one slot permanently disabled. There’s the set of features that were intended to be in your chips – and there’s the subset of series of features that finally shipped. What happened to the features that weren’t shipped? Surely, they did a final pass on the chip to remove all that “dead silicon”.

Nope – most of the time those partially or non-functional units are simply disabled. This ranges from blocks of cache RAM, to whole CPU cores, to various hardware peripherals. Patching a complex chip design can cost millions of dollars and takes weeks or even months, so no company can afford to do a final “clean-up” pass to create a “perfect” design. To wit, manufacturers never misrepresent the product to consumers – if half the cache was available, the spec sheet would simply report the cache size as 128kB instead of 256kB. But surely some customers would have complained bitterly if they knew of the defect sold to them.

Despite being chock full of bugs, vendors of desktop CPUs or mobile phone System on Chips (SoCs) rarely disclose these bugs to users – and those that do disclose almost always disclose a limited list of public bugs, backed by an NDA-only list of all the bugs. The top two reasons cited for keeping chip specs secret are competitive advantage and liability, and I suspect in reality, it’s the latter that drives the secrecy, because the crappier the chipset, the more likely the specs are under NDA. Chip vendors are deathly afraid users will find inconsistencies between the chip’s actual performance and the published specs, thus triggering a recall event. This fear may seem more rational if you consider the magnitude of Intel’s FDIV bug recall ($475 million in 1994).

This is a pretty typical list of SoC bugs, known as “errata”. If your SoC’s errata is much shorter than this, it’s more likely due to bugs not being disclosed than there actually being less bugs.

If you Want Messages, Stop Shooting the Messengers

Highly esteemed and enlightened colleagues of mine are strongly of the opinion that Intel should reimburse end users for bugs found in their silicon; yet in the same breath, they complain that Intel has not been transparent enough. The point that has become clear to me is that consumers, even open-source activists, are very sensitive to imperfections, however minor. They demand a “perfect” machine; if they spend $500 on a computer, every part inside better damn well be perfect. And so starts the vicious cycle of hardware manufacturers hiding all sorts of blemishes and shortcomings behind various NDAs, enabling them to bill their goods as perfect for use.

You can’t have it both ways: the whole point of transparency is to enable peer review, so you can find and fix bugs more quickly. But if every time a bug is found, a manufacturer had to hand $50 to every user of their product as a concession for the bug, they would quickly go out of business. This partially answers the question why we don’t see open hardware much beyond simple breakout boards and embedded controllers: it’s far too risky from a liability standpoint to openly share the documentation for complex systems under these circumstances.

To simply say, “but hardware manufacturers should ship perfect products because they are taking my money, and my code can be buggy because it’s free of charge” – is naïve. A modern OS has tens of millions of lines of code, yet it benefits from the fact that every line of code can be replicated perfectly. Contrast to a modern CPU with billions of transistors, each with slightly different electrical characteristics. We should all be more surprised that it took so long for a major hardware bug to be found, than the fact that one was ever found.

Complex systems have bugs. Any system with primitives measured in the millions or billions – be it lines of code, rivets, or transistors – is going to have subtle, if not blatant, flaws. Systems simple enough to formally verify are typically too simple to handle real-world tasks, so engineers must rely on heuristics like design rules and lots and lots of hand-written tests.

There will be bugs.

Realities of the Open Hardware Business

About a year ago, I had a heated debate with a SiFive founder about how open they can get about their documentation. SiFive markets the RISC-V CPU, billed as an “open source CPU”, and many open source enthusiasts got excited about the prospect of a fully-open SoC that could finally eliminate proprietary blobs from the boot chain and ultimately through the same process of peer review found in the open source software world, yield a more secure, trustable hardware environment.

However, even one of their most ardent open-source advocates pushed back quite hard when I suggested they should share their pre-boot code. By pre-boot code, I’m not talking about the little ROM blob that gets run after reset to set up your peripherals so you can pull your bootloader from SD card or SSD. That part was a no-brainer to share. I’m talking about the code that gets run before the architecturally guaranteed “reset vector”. A number of software developers (and alarmingly, some security experts) believe that the life of a CPU begins at the reset vector. In fact, there’s often a significant body of code that gets executed on a CPU to set things up to meet the architectural guarantees of a hard reset – bringing all the registers to their reset state, tuning clock generators, gating peripherals, and so forth. Critically, chip makers heavily rely upon this pre-boot code to also patch all kinds of embarrassing silicon bugs, and to enforce binning rules.

The gentleman with whom I was debating the disclosure of pre-boot code adamantly held that it was not commercially viable to share the pre-boot code. I didn’t understand his point until I witnessed open-source activists en masse demanding their pound of flesh for Intel’s mistakes.

As engineers, we should know better: no complex system is perfect. We’ve all shipped bugs, yet when it comes to buying our own hardware, we individually convince ourselves that perfection is a reasonable standard.

The Choice: Truthful Mistakes or Fake Perfection?

The open source community could use the Spectre/Meltdown crisis as an opportunity to reform the status quo. Instead of suing Intel for money, what if we sue Intel for documentation? If documentation and transparency have real value, then this is a chance to finally put that value in economic terms that Intel shareholders can understand. I propose a bargain somewhere along these lines: if Intel releases comprehensive microarchitectural hardware design specifications, microcode, firmware, and all software source code (e.g. for AMT/ME) so that the community can band together to hammer out any other security bugs hiding in their hardware, then Intel is absolved of any payouts related to the Spectre/Meltdown exploits.

This also sets a healthy precedent for open hardware. In broader terms, my proposed open hardware bargain is thus: Here’s the design source for my hardware product. By purchasing my product, you’ve warranted that you’ve reviewed the available design source and decided the open source elements, as-is, are fit for your application. So long as I deliver a product consistent with the design source, I’ve met my hardware warranty obligation on the open source elements.

In other words, the open-source bargain for hardware needs to be a two-way street. The bargain I set forth above:

Rewards transparency with indemnity against yet-to-be-discovered bugs in the design source

Burdens any residual proprietary elements with the full liability of fitness for purpose

Simultaneously conserves a guarantee that a product is free from defects in materials and workmanship in either case

The beauty of this bargain is it gives a real economic benefit to transparency, which is exactly the kind of wedge needed to drive closed-source silicon vendors to finally share their full design documentation, with little reduction of consumer protection.

So, if we really desire a more transparent, open world in hardware: give hardware makers big and small the option to settle warranty disputes for documentation instead of cash.

The first is whether hardware makers will accept the offer to provide documentation in lieu of liability.

The second, and perhaps more significant, is whether you would make the offer for design documentation in lieu of design liability in the first place. It’s important that companies who choose transparency be given a measurable economic advantage over those who choose obscurity. In order for the vicious cycle of proprietary hardware to be broken, both consumer and producer have to express a willingness to value openness.

There’s an Internet controversy going on between Dale Dougherty, the CEO of Maker Media and Naomi Wu (@realsexycyborg), a Chinese Maker and Internet personality. Briefly, Dale Doughtery tweeted a single line questioning Naomi Wu’s authenticity, which is destroying Naomi’s reputation and livelihood in China.

In short, I am in support of Naomi Wu. Rather than let the Internet speculate on why, I am sharing my perspectives on the situation preemptively.

As with most Internet controversies, it’s messy and emotional. I will try my best to outline the biases and issues I have observed. Of course, everyone has their perspective; you don’t have to agree with mine. And I suspect many of my core audience will dislike and disagree with this post. However, the beginning of healing starts with sharing and listening. I will share, and I respectfully request that readers read the entire content of this post before attacking any individual point out of context.

The key forces I see at play are:

Prototype Bias – how assumptions based on stereotypes influence the way we think and feel

Power Asymmetry – those with more power have more influence, and should be held to a higher standard of accountability

Guanxi Bias – the tendency to give foreign faces more credibility than local faces in China

All these forces came together in a perfect storm this past week.

1. Prototype Bias

If someone asked you to draw a picture of an engineer, who would you draw? As you draw the figure, the gender assigned is a reflection of your mental prototype of an engineer – your own prototype bias. Most will draw a male figure. Society is biased to assign high-level intellectual ability to males, and this bias starts at a young age. Situations that don’t fit into your prototypes can feel threatening; studies have shown that men defend their standing by undermining the success of women in STEM initiatives.

The bias is real and pervasive. For example, my co-founder in Chibitronics, Jie Qi, is female. The company is founded on technology that is a direct result of her MIT Media Lab PhD dissertation. She is the inventor of paper electronics. I am a supporting actor in her show. Despite laying this fact out repeatedly, she still receives comments and innuendo implying that I am the inventor or more influential than I really am in the development process.

Any engineer who observes a bias in a system and chooses not to pro-actively correct for it is either a bad engineer or they stand to benefit from the bias. So much of engineering is about compensating, trimming, and equalizing imperfections out of real systems: wrap a feedback loop around it, and force the error function to zero.

So when Jie and I stand on stage together, prototype bias causes people to assume I’m the one who invented the technology. Given that I’m aware of the bias, does it make sense to give us equal time on the stage? No – that would be like knowing there is uneven loss in a channel and then being surprised when certain frequency bands are suppressed by the time it hits the receivers. So, I make a conscious and deliberate effort to showcase her contributions and to ensure her voice is the first and last voice you hear.

Naomi Wu (pictured below) likely challenges your prototypical ideal of an engineer. I imagine many people feel a cognitive dissonance juxtaposing the label “engineer” or “Maker” with her appearance. The strength of that dissonant feeling is proportional to the amount of prototype bias you have.

I’ve been fortunate to experience breaking my own prototypical notions that associate certain dress norms with intelligence. I’m a regular at Burning Man, and my theme camp is dominated by scientists and engineers. I’ve discussed injection molding with men in pink tutus and learned about plasmonics from half-naked women. It’s not a big leap for me to accept Naomi as a Maker. I’m glad she’s challenging these biases. I do my best engineering when sitting half-naked at my desk. I find shirts and pants to be uncomfortable. I don’t have the strength to challenge these social norms, and secretly, I’m glad someone is.

Unfortunately, prototype bias is only the first challenge confronted in this situation.

2. Idol Effect

The Idol Effect is the tendency to assign exaggerated capabilities to public figures and celebrities. The adage “never meet your childhood hero” is a corollary of the Idol Effect – people have inflated expectations about what celebrities can do, so it’s often disappointing when you find out they are humans just like us.

One result of the Idol Effect is that people feel justified taking pot shots at public figures for their shortcomings. For example, I have had the great privilege of working with Edward Snowden. One of my favorite things about working with him is that he is humble and quick to correct misconceptions about his personal abilities. Because of his self-awareness of his limitations, it’s easier for me to trust his assertions, and he’s also a fast learner because he’s not afraid to ask questions. Notably, he’s never claimed to be a genius, so I’m always taken aback when intelligent people pull me aside and whisper in my ear, “You know, I hear Ed’s a n00b. He’s just using you.” Somehow, because of Ed’s worldwide level of fame that’s strongly associated with security technology, people assume he should be a genius level crypto-hacker and are quick to point out that he’s not. Really? Ed is risking his life because he believes in something. I admire his dedication to the cause, and I enjoy working with him because he’s got good ideas, a good heart, and he’s fun to be with.

Because I also have a public profile, the Idol Effect impacts me too. I’m bad at math, can’t tie knots, a mediocre programmer…the list goes on. If there’s firmware in a product I’ve touched, it’s likely to have been written by Sean ‘xobs’ Cross, not me. If there’s analytics or informatics involved, it’s likely my partner wrote the analysis scripts. She also edits all my blog posts (including this one) and has helped me craft my most viral tweets – because she’s a genius at informatics, she can run analyses on how to target key words and pick times of day to get maximum impact. The fact that I have a team of people helping me polish my work makes me look better than I really am, and people tend to assign capabilities to me that I don’t really have. Does this mean I am a front, fraud or a persona?

I imagine Naomi is a victim of Idol Effect too. Similar to Snowden, one of the reasons I’ve enjoyed interacting with Naomi is that she’s been quick to correct misconceptions about her abilities, she’s not afraid to ask for help, and she’s a quick learner. Though many may disapprove of her rhetoric on Twitter, please keep in mind English is her second language — her sole cultural context in which she learned English was via the Internet by reading social media and chat rooms.

Based on the rumors I’ve read, it seems fans and observers have inflated expectations for her abilities, and because of uncorrected prototype bias, she faces extra scrutiny to prove her abilities. Somehow the fact that she almost cuts her finger using a scraper to remove a 3D print is “evidence” that she’s not a Maker. If that’s true, I’m not a Maker either. I always have trouble releasing 3D prints from print stages. They’ve routinely popped off and flown across the room, and I’ve almost cut my fingers plenty of times with the scraper. But I still keep on trying and learning – that’s the point. And then there’s the suggestion that because a man holds the camera, he’s feeding her lines.

When a man harnesses the efforts of a team, they call him a CEO and give him a bonus. But when a woman harnesses the efforts of a team, she gets accused of being a persona and a front. This is uncorrected Prototype Bias meeting unrealistic expectations due to the Idol Effect.

The story might end there, but things recently got a whole lot worse…

3. Power Asymmetry

“With great power comes great responsibilities.”
-from Spider Man

Power is not distributed evenly in the world. That’s a fact of life. Not acknowledging the role power plays leads to systemic abuse, like those documented in the Caldbeck or Weinstein scandals.

Editors and journalists – those with direct control over what gets circulated in the media – have a lot of power. Their thoughts and opinions can reach and influence a massive population very quickly. Rumors are just rumors until media outlets breathe life into them, at which point they become an incurable cancer on someone’s career. Editors and journalists must be mindful of the power they wield and held accountable for when it is mis-used.

As CEO of Maker Media and head of an influential media outlet, especially among the DIY community, Dale Dougherty wields substantial power. So a tweet promulgating the idea that Naomi might be a persona or a fake does not land lightly. In the post-truth era, it’s especially incumbent upon traditional media to double-check rumors before citing them in any context.

What is personally disappointing is that Dale reached out to me on November 2nd with an email asking what I thought about an anonymous post that accused Naomi of being a fake. I vouched for Naomi as a real person and as a budding Maker; I wrote back to Dale that “I take the approach of interacting with her like any other enthusiastic, curious Maker and the resulting interactions have been positive. She’s a fast learner.”

Yet Dale decided to take an anonymous poster’s opinion over mine (despite a long working relationship with Make), and a few days later on November 5th he tweeted a link to the post suggesting Naomi could be a fake or a fraud, despite having evidence of the contrary.

So now Naomi, already facing prototype bias and idol-effect expectations, gets a big media personality with substantial power propagating rumors that she is a fake and a fraud.

But wait, it gets worse because Naomi is in China!

4. Guanxi Bias

In China, guanxi (关系) is everything. Public reputation is extremely hard to build, and quick to lose. Faking and cloning is a real problem, but it’s important to not lose sight of the fact that there are good, hard-working people in China as well. So how do the Chinese locals figure out who to trust? Guanxi is a major mechanism used inside China to sort the good from the bad – it’s a social network of credible people vouching for each other.

For better or for worse, the Chinese feel that Western faces and brands are more credible. The endorsement of a famous Western brand carries a lot of weight; for example Leonardo DiCaprio is the brand ambassador for BYD (a large Chinese car maker).

Maker Media has a massive reputation in China. From glitzy Maker Faires to the Communist party’s endorsement of Maker-ed and Maker spaces as a national objective, an association or the lack thereof with Maker Media can make or break a reputation. This is no exception for Naomi. Her uniqueness as a Maker combined with her talent at marketing has enabled her to do product reviews and endorsements as source of income.

However, for several years she’s been excluded from the Shenzhen Maker Faire lineup, even in events that she should have been a shoo-in for her: wearables, Maker fashion shows, 3D printing. Despite this lack of endorsement, she’s built her own social media follower base both inside and outside of China, and built a brand around herself.

Unfortunately, when the CEO of Maker Media, a white male leader of an established American brand, suggested Naomi was a potential fake, the Internet inside China exploded on her. Sponsors cancelled engagements with her. Followers turned into trolls. She can’t be seen publicly with men (because others will say the males are the real Maker, see “prototype bias”), and as a result faces a greater threat of physical violence.

A single innuendo, amplified by Power Asymmetry and Guanxi Bias, on top of Idol Effect meshed against Prototype Bias, has destroyed everything a Maker has worked so hard to build over the past few years.

If someone spread lies about you and destroyed your livelihood – what would you do? Everyone would react a little differently, but make no mistake: at this point she’s got nothing left to lose, and she’s very angry.

Reflection

Although Dale had issued a public apology about the rumors, the apology fixes her reputation as much as saying “sorry” repairs a vase smashed on the floor.

Image: Mindy Georges CC BY-NC

At this point you might ask — why would Dale want to slander Naomi?

I don’t know the background, but prior to Dale’s tweet, Naomi had aggressively dogged Dale and Make about Make’s lack of representation of women. Others have noted that Maker Media has a prototype bias toward white males. Watch this analysis by Leah Buechley, a former MIT Media Lab Professor:

Dale could have recognized and addressed this core issue of a lack of diversity. Instead, Dale elected to endorse unsubstantiated claims and destroy a young female Maker’s reputation and career.

Naomi has a long, uphill road ahead of her. On the other hand, I’m sure Dale will do fine – he’s charismatic, affable, and powerful.

When I sit and think, how would I feel if this happened to the women closest to me? I get goosebumps – the effect would be chilling; the combination of pervasive social biases would overwhelm logic and fact. So even though I may not agree with everything Naomi says or does, I have decided that in the bigger picture, hiding in complicit silence on the sidelines is not acceptable.

We need to acknowledge that prototype bias is real; if equality is the goal, we need to be proactive in correcting it. Just because someone is famous doesn’t mean they are perfect. People with power need to be held accountable in how they wield it. And finally, cross-cultural issues are complicated and delicate. All sides need to open their eyes, ears, and hearts and realize we’re all human. Tweets may seem like harmless pricks to the skin, but we all bleed when pricked. For humanity to survive, we need to stop pricking each other lest we all bleed to death.

I’ve often said that there are no secrets in hardware — you just need a bigger, better microscope.

I think I’ve found the limit to that statement. To give you an idea, here’s the “lightbulb” that powers the microscope:

It’s the size of a building, and it’s the Swiss Light Source. Actually, not all of that building is dedicated to this microscope, just one beamline of an X-ray synchrotron capable of producing photons at an energy of 6.5keV (X-rays) at a flux of close to a billion coherent photons per second — but still, it’s a big light bulb. It might be a while before you see one of these popping up in a hacker’s garage…err, hangar…somewhere.

The result? One can image, in 3-D and “non-destructively” (e.g., without having to delayer or etch away dielectrics), chips down to a resolution of 14.6nm.

That’s a pretty neat trick if you’re trying to reverse engineer modern silicon.

You can read the full article at Nature (“High Resolution non-destructive three-dimensional imaging of integrated circuits” by Mirko Holler et al). I’m a paying subscriber to Nature so I’m supposed to have access to the article, but at the moment, their paywall is throwing a null pointer exception. Once the paywall is fixed you can buy a copy of the article to read, but in the meantime, SciHub seems more reliable.

Today I filed a lawsuit against the US government, challenging Section 1201 of the Digital Millennium Copyright Act. Section 1201 means that you can be sued or prosecuted for accessing, speaking about, and tinkering with digital media and technologies that you have paid for. This violates our First Amendment rights, and I am asking the court to order the federal government to stop enforcing Section 1201.

Before Section 1201, the ownership of ideas was tempered by constitutional protections. Under this law, we had the right to tinker with gadgets that we bought, we had the right to record TV shows on our VCRs, and we had the right to remix songs. Section 1201 built an extra barrier around copyrightable works, restricting our prior ability to explore and create. In order to repair a gadget, we may have to decrypt its firmware; in order to remix a video, we may have to strip HDCP. Whereas we once readily expressed feelings and new ideas through remixes and hardware modifications, now we must first pause and ask: does this violate Section 1201? Especially now that cryptography pervades every aspect of modern life, every creative spark is likewise dampened by the chill of Section 1201.

The act of creation is no longer spontaneous.

Our recent generation of Makers, hackers, and entrepreneurs have developed under the shadow of Section 1201. Like the parable of the frog in the well, their creativity has been confined to a small patch, not realizing how big and blue the sky could be if they could step outside that well. Nascent 1201-free ecosystems outside the US are leading indicators of how far behind the next generation of Americans will be if we keep with the status quo.

Our children deserve better.

I can no longer stand by as a passive witness to this situation. I was born into a 1201-free world, and our future generations deserve that same freedom of thought and expression. I am but one instrument in a large orchestra performing the symphony for freedom, but I hope my small part can remind us that once upon a time, there was a world free of such artificial barriers, and that creativity and expression go hand in hand with the ability to share without fear.

Front-line journalists are high-value targets, and their enemies will spare no expense to silence them. Unfortunately, journalists can be betrayed by their own tools. Their smartphones are also the perfect tracking device. Because of the precedent set by the US’s “third-party doctrine,” which holds that metadata on such signals enjoys no meaningful legal protection, governments and powerful political institutions are gaining access to comprehensive records of phone emissions unwittingly broadcast by device owners. This leaves journalists, activists, and rights workers in a position of vulnerability. This work aims to give journalists the tools to know when their smart phones are tracking or disclosing their location when the devices are supposed to be in airplane mode. We propose to accomplish this via direct introspection of signals controlling the phone’s radio hardware. The introspection engine will be an open source, user-inspectable and field-verifiable module attached to an existing smart phone that makes no assumptions about the trustability of the phone’s operating system.