Parallel

Welcome to the Parallel Jungle!

By Herb Sutter, January 29, 2012

Herb Sutter dives into the repercussions of parallel's reach from mobile devices, to the desktop, to clusters, and  at the highest level of granularity  to the cloud. This welter of different parallel implementations presents significant challenges for programming. The free lunch of sequential programming is well and truly over.

Exit Moore, Pursued by a Dark Silicon Bear

Finally, let's return one more time to the end of Moore's Law to see what awaits us in our near future (Figure 14), and why we will likely pass through three distinct stages as we navigate Moore's End.

Figure 14.

Eventually, the tired miners will reach the point where it's no longer economically feasible to operate the mine. There's still gold left, but it's no longer commercially exploitable. Recall that Moore's Law has been interesting only because we have the ability to transform its raw resource of "more transistors" into one of two useful forms:

Exploit #1: Greater throughput. Moore's Law lets processors deliver more transistors, and therefore more complex chips, at the same cost. That's what will let us continue to deliver more computational performance per chip  as long as we can find ways to harness the extra transistors for computation.

Exploit #2: Lower cost/power/size. Alternatively, Moore's Law lets us deliver the same number of transistors at a lower cost, including in a smaller area and at lower power. That's what will let us continue to deliver powerful experiences in increasingly compact and mobile and embedded form factors.

The key thing to note is that we can expect these two ways of exploiting Moore's Law to end, not at the same time, but one after the other and in that order.
Why? Because Exploit #2 only relies on the basic Moore's Law effect, whereas the first relies on Moore's Law and the ability to use all the transistors at the same time.

Which brings me to one last problem down in our mine…

The Power Problem: Dark Silicon

Sometimes you can be hard at work in a mine, still productive, when a small disaster happens: a cave-in, or striking water. Such disasters can render entire sections of the mine unreachable. We are now starting to hit exactly those kinds of problems.

This "dark silicon" effect is like a Shakespearian bear chasing our doomed character offstage. Even though we can continue to pack more cores on a chip, if we cannot use them at the same time, we have failed to exploit Moore's Law to deliver more computational throughput (Exploit #1). When we enter the phase where Moore's Law continues to give us more transistors per die area, but we are no longer able to power them all, we will find ourselves in a transitional period where Exploit #1 has ended while Exploit #2 continues and outlives it for a time.

This means that we will likely see the following major phases in the "scale-in" growth of mainstream machines. (Note that these apply to individual machines only, such as your personal notebook and smartphone or an individual compute node; they do not apply to a compute cloud, which we saw belongs to a different "scale-out" mine.)

Exploit #1 + Exploit #2: Increasing performance (compute throughput) in all form factors (1975  mid-2010s?). For a few years yet, we will see continuing increases in mainstream computer performance in all form factors from desktop to smartphone. As of today, the bigger form factors still have more parallelism, just as today's desktop CPUs and GPUs are routinely more capable than those in tablets and smartphones  as long as Exploit #1 lives, and then…

Exploit #2 only: Flat performance (compute throughput) at the top end, and mid and lower segments catching up (late 2010s  early 2020s?). Next, if problems like dark silicon are not solved, we will enter a period where mainstream computer performance levels out, starting at the top end with desktops and game consoles and working its way down through tablets and smartphones. During this period we will continue to use Moore's Law to lower cost, power, and/or size  delivering the same complexity and performance already available in bigger form factors also in smaller devices. Assuming Moore's Law continues long enough beyond the end of Exploit #1, we can estimate how long it will take for Exploit #2 to equalize personal devices by observing the difference in transistor counts between current mainstream desktop machines and smartphones; it's roughly a factor of 20, which will take Moore's Law about eight years to cover.

Democratization (early 2020s?  onward). Finally, this democratization will reach the point where a desktop computer and smartphone have roughly the same computational performance. In that case, why buy a desktop ever again? Just dock your tablet or smartphone. You might think that there are still two important differences between the desktop and the mobile device: power, because the desktop is plugged in, and peripherals, because the desktop has easier access to a bigger screen and a real keyboard/mouse  but once you dock the smaller device, it has the same access to power and peripherals.

Speaking of Smartphones Pocket Tablets and Democratization

Note that the word "smartphone" is already a major misnomer, because a pocket device that can run apps is not primarily a phone at all. It's primarily a general-purpose personal computer that happens to have a couple of built-in radios for cell and WiFi service  making the "traditional cell phone" capability just an app that happens to use the cell radio, and the Skype "IP phone" capability on the same device just another similar app that happens to use the WiFi radio instead.

The right way to think about even today's mobile landscape is that there are not really "tablets" and "smartphones"; there are just page-sized tablets and pocket-sized tablets, both already available with or without cellular radios. That they run different operating systems today is just a point-in-time effect.

This is why those people who said an iPad is just a big iPhone without the cellular radio had it exactly backwards  the iPhone (3G or later, which allows apps) is a small iPad that fits in your pocket and happens to have a cellular radio in order to obsolete another pocket-sized device. Both devices are primarily tablets  they minimize hardware chrome and "turn into" the full-screen immersive app, and that's the closest thing you can get today to a morphing device that turns into a special-purpose device on demand. (Aside: It'll be great when we figure out how to get past the flat-glass-pane model to let the hardware morph, too, initially just raised bumps so we can feel where the keys and controls are, and then eventually more; but hardware morphing is a separate topic and flat glass is plenty fine for now.) Many of us routinely use our "phones" mostly as a small tablet  spending most of our time on the device running apps to read books, browse news, watch movies, play games, update social networks, and surf the net. I already use my phone as a small tablet far more often than I use it as a phone, and if you have an app-capable phone then I'll bet you already do that, too.

Well before the end of this decade, I expect the most likely dominant mainstream form factor to be "page-sized and pocket-sized tablets, plus docking"  where "docking" means any means of attaching peripherals like keyboards and big screens on demand, which today already encompasses physical docks and Bluetooth and "Play To" connections, and will only continue to get more wireless and more seamless.

This future shouldn't be too hard to imagine, because many of us have already been working that way for a while now: For the past decade I've routinely worked from my notebook as my primary and only environment. Usually, I'm in my home office or work office where I use a real keyboard and big screens by docking the notebook and/or using it via a remote-desktop client, and when I'm mobile I use it as a notebook. In 2012, I expect to replace my notebook with an x86-based modern tablet and use it exactly the same way.

We've seen it play out many times:

Many of us used to carry around both a PalmPilot and a cell phone, but then the smartphone took over the job of the dedicated PalmPilot and eliminated a device with the same form factor.

Similarly, today many of us carry around a notebook and a dedicated tablet, and convergence will again let us eliminate a device with the same form factor.

Computing loves convergence. In general-purpose personal computing (like notebooks and tablets, not special-purpose appliances like microwaves and automobiles that may happen to use microprocessors), convergence always happily dooms special-purpose devices in the long run, as each device either evolves to take over the others' job or gets taken over. We will continue to have distinct pocket-sized tablets and page-sized tablets for a time because they are different form factors with different mobile uses, but even that may last only until we find a way to unify the form factors (fold them?) so that they too can converge.

Dr. Dobb's encourages readers to engage in spirited, healthy debate, including taking us to task.
However, Dr. Dobb's moderates all comments posted to our site, and reserves the right to modify or remove any content that it determines to be derogatory, offensive, inflammatory, vulgar, irrelevant/off-topic, racist or obvious marketing or spam. Dr. Dobb's further reserves the right to disable the profile of any commenter participating in said activities.

Video

This month's Dr. Dobb's Journal

This month,
Dr. Dobb's Journal is devoted to mobile programming. We introduce you to Apple's new Swift programming language, discuss the perils of being the third-most-popular mobile platform, revisit SQLite on Android
, and much more!