Oracle Blog

CMT musings

Wednesday Jun 11, 2008

It is time to wrap this CMT blog journey with a paradox, because CMT has done the unthinkable. CMT started as a radical exercise in putting all processing in one socket next to one big fat memory to provide a nice "symmetrical" SMP system. Well CMT has gone multi-socket with the UltraSPARC T2 Plus. I relish the paradox of CMT born as a single socket architecture going multi-socket and conventional multi-socket processors going multi-core. Not quite architectural convergence, but some evidence of imitation being the best form of flattery.

Relishing a paradox is the refuge of the perplexed, a way of finding order when reality yields the opposite of what we expect. Anti-correlation is a form of order and causality after all, more palatable than the chaos of no correlation.

Free markets and level playing fields offer paradoxes to relish each time reality does not live up to free markets just emanating from basic freedoms, or when playing fields show steep gradients. If competitors like IBM, HP, and Sun are legally prohibited from colluding to fix the prices of the computers they sell for, say, oil exploration, yet said oil producers collude as a cartel to sell their product, is that strange or what. Sit back and enjoy that paradox.

Or when Sun hardware runs into bogus import regulations overseas meant to protect local competitors, but shoes coming the other way enjoy some most favored nation moniker and displace perfectly comfortable shoes from Brazil, for example, the field ain't level. It may be hard for engineers to sit back and enjoy this paradox with gusto while their product turns obsolete awaiting import approval. Shoes have longer shelf lives than computers.

Yet fortunately other aspects of our high tech professions can be lived "paradox free". Take employment, for example, an important part of our lives. Jobs are so important in this valley of ours that they settle the fuss about marriage being exclusively between a man and a woman. Plenty of marriages between a man and a job in California.

A friend who recently went through the hiring process, suggests that even job seeking can be a "paradox free" private affair to the benefit of both parties. That was until my brother spoiled it all. See, my friend had the lofty goals of maximizing his benefit while eliminating haggling. Haggling leaves a bad taste and is no way to start a relationship, right? So he informed the interested employers that on a set date he would decide where to go based on a sealed bid auction. Best offer takes him, no reserve. And that is precisely what happened, he got all job offers, lined them up, and accepted the best one. No negotiation, no recruiter calls, no re-bids.

Enter my brother. First, the Revenue Equivalence Theorem shows that the various auction methods are equivalent, and none of them is capable of extracting for my friend the maximal valuation unless he gets offers from an infinite number of employers. An infinite number of interviews is not practical, shaking that many hands surely leads to arthritis.

Second the Winner's Curse shows that the winning participant in an auction will typically pay an overvalued price. The moment my friend accepted the offer his market value immediately declined in the eyes of his new boss. Bad way to start a new job... So according to game theory my friend cannot elude paradoxes, but I visited him yesterday at his new job and both parties are happy, what gives?

Paradoxes being useful refuges doesn't mean that behind every paradox there is something unpleasant to take refuge from. A harmless paradox appears to be the case with my friend.

A harmless paradox lies behind multi-socket CMT systems. The proof points are plentiful, in the form of official white papers like this one, or the unfiltered voices of bloggers that looked into details like whether any multi-socket NUMA effects get in the way, the scalability of the OS, its mutexes and locks going to 128 threads, how to get 10G network rates, and how well Java runs and scales on these systems. Allan Packer's entry CMT_goes_multi is a great repository for all these perspectives on the multi-socket T2 Plus.

Since CMT it is hard to agree on what a processor is anymore (a chip, a core, a thread) but a socket remains a clear entity even when the component is not socketed at all. Increasingly a processor socket is a unit of memory interface, that is because by now most processors have done the flattering imitation and incorporated their memory controllers on chip. And that unit of memory interface gives a multi-socket CMT a larger memory footprint, which is where we started our journey.

The two socket UltraSPARC T2 Plus enterprise systems are out since April, taking the thread count to 128 per system. The Telco variants of these systems come out next to augment the scope of what CMT can do for the network infrastructure we care about. And when they arrive you can always find them here .

Friday Jan 25, 2008

Apropos second generations, our second generation ATCA lineup was announced and it has 10G all over. This Eweek article
was printed back in November, but may still be news to readers of my sparse blog.

10G and ATCA are a great fit because no optics are needed to run 10G between blades and fabric. The server 10G cost barrier is gone. Gone is another 10G adoption barrier, the need to upgrade both the server and the switch at the same time. Our ATCA refresh is quite comprehensive, with UltraSPARC T2 and x86 processors, with 10G in the blades (Neptune), in the fabric, through RTMs, and with server and packet processing on the same blade (see previous blog entry).

As all-things-ATCA get interesting some observers ask about the one or two competitors that quit. As a Telco and Carrier platform supplier we definitely like the technical, market, and standard attributes of ATCA. Though some may argue that our perspective carries "Survivor Bias".

The Survivor Bias artifact, in finance and in statistics, is the hazard of excluding population samples that did not survive. Like measuring the growth of all public companies yet overlook the companies that went under. Or worse, the ones that went under and survived (ask me how I know).

Recently I tasted Survivor Bias when our instructor wrapped up a leadership training course. The closing, the Grand Finale, was an attempt to put our professional travails in perspective by hearing what famous people thought really mattered in life, in retrospective, mind you. I silently cried foul. I felt that the success of these quotable luminaries tainted their collective wisdom with Survivor Bias. Their perspective devalued the priorities that consume most of my daily energies. And sure enough they no longer worry about the daily grind. Their job is done.

It took me a while to recognize two distinct objections at play, one has to do with judgment calls a posteriori, once the outcome is known, and the other with the effect of survival itself on the validity of the data.

The temporal objection is against the natural diminution of bridges already crossed. If our preoccupation is to be a strong link in a much longer chain, once the chain goes on, our opinions and priorities are of much lesser value. My objection against judging past priorities stands.

Outside of hindsight the value of the survivor perspective may depend on the survival process. The Car Talk radio show had a puzzler where a mathematician recommends armor plating WWII airplane wings and fuselage in spots where the returning airplanes showed no bullet holes. Absence of holes does not mean the area is not exposed, but possible vulnerability, if there is a correlation between planes hit in those spots and planes not returning.

This is just to illustrate how survivor data validity depends on whether we think survival stochastics are random or correlated. The example is neat because given a large sample, the mathematician can determine both the existence of the correlation, and the areas to protect.

Back to ATCA, our perspective is timely rather than "a posteriori", so we are safe there. And Sun's Netra platform success in the Telco market was not random, so on the correlation front we are peachy. Finally, we vote our perspective with a new generation of ATCA products. Every generation reinforces wings and fuselage looking at its bullet holes. And at the bullet holes of competitors that did not return from the sortie.

Tuesday Oct 09, 2007

A special post indeed in this series of amorphous blogs initially inspired by CMT topics. The arrival of second generation CMT systems (the T5120 and family) is a critical evolutionary step, and a good excuse to contemplate the evolution theme.

Interpreting Darwin's evolution as the natural selection of "good" heritable traits, my beef is that bad traits must “express themselves” before they can propagate, for selection to work. If we have children in our twenties but get really sick in our seventies, it may be too late for Darwin to kick in. We could even postulate an evolution path towards species that are perfectly healthy until reproduction age, but not a day more. NiCd battery species, kind of.

It is not fair to mess up with Darwin when he isn't around to respond. We can taunt contemporary scientists instead by stating that thanks to evolution the path to longevity is through stretching the reproduction age rather than expensive medicine. Let's postpone marriage by 40 years and watch life expectancy soar! There. Taunted the health sciences community plus many other innocent bystanders, let's now run to technology topics for cover.

Darwinian world or not, computer systems don't randomly mutate traits over infinite product generations, traits are introduced deliberately and rather frequently. Before creationsists celebrate the “deliberate” aspect, let me debunk intelligent creation with a single stroke, a three-finger stroke to be precise. The infamous CTRL-ALT-DEL single handedly debunks both intelligent creation and evolution theories for computer systems. OK. I take back "single handedly"...

Products and technologies are vulnerable in their long term survival, much more so than dinosaurs were in their day. Second generations, like the T5120, are then as much about product family continuity as about constant technology improvement. Improvements that are as natural as the desire for our children to go beyond our own reach.

CMT waltzes to the cadence of Moore's Law, and T2 boxes are here less than two years after the T1 boxes. True to the spirit of the Law, T2 systems doubled the number of physical threads per socket. The thread bump has replaced the speed bump. The bump is a nice integer 2x factor, actually more than 2x, after all the T2 processor also has a faster pipeline, a larger cache with higher associativity, crypto acceleration, more memory bandwidth, a no compromises floating point unit per core, and built-in 10G networking.

The links below have plenty of data on T2 systems across different workloads, so I will stick to the networking angles for now.

High Speed Networking evolved in multiples of ten. 100 Mbps around 1995, and two 10x factors since then took us to 10Gbps. Processors doing 2x every two years would be a factor of 64, modulo sampling noise. As processors and networks evolve, the question is how to jump across these moving trains. When should the next network speed be adopted? The answer is easy if we agree on who is at the center, the host or the network.

If your religion is host centric then, for their own sake, put servers on 10G as soon as they can do more than 1G. If network infrastructure is supreme, then build its temple only with processors and servers that can do more than 10G. Copernicus isn't around either to settle centricity questions, so we built these T2 based systems to satisfy both cases.

Just like a deep analysis of Darwin takes you to his actual writings, the proof points around Sun's Unified Network Platforms, consolidation,
packet processing, and the movement towards data plane on general purpose platforms may require going through white papers and kicking the tires with reference apps. Here is a start: