Friday, July 13, 2018

Solaris 11.4: KSplice Arriving!

Abstract:

The concept of patching an OS system has existed since the beginning of computers. At one time, source code needed to be compiled and a system rebooted with the new OS. Pre-compiled patches were later shipped by OS vendors. Patches were often allowed to be applied while the OS in Single User Mode. Sun had created the concept of Live Upgrade, where patching could be done to an alternate boot environment & booted at later convenient time, to reduce downtime to a simple reboot. In 2007, another feature referred to as Deferred Activation Patching allowed for the current running Sun Solaris 10 OS to be patched, while deferring some of the patches until the next boot, leveraging loopback file systems. In 2008, KSplice concept was birthed from MIT for LINUX kernels, to perform some degree of patching, while severely limiting reboots. Oracle purchased Sun and Solaris in 2009. The MIT students won a $100K award, to start up a company in 2009. KSplice was purchased by Oracle in 2011. The promise of KSplice and rebootless patching for Solaris was understood in concept, but not executed, until recently.

The engineering task of merging DTrace & KSplice technologies between Linux & Solaris was seemingly on. In 2015, NetMgt reported discussion of KSplice for Platinum Support Linux with DTrace became a major differentiator for Oracle, but KSplice discussion regarding Solaris was still dark... until 2018.

Solaris 11.4 (aka Solaris 12)

Since KSplice was a significant feature, those of us in the industry expected it to arrive in Solaris 12. To casual observers, it became clear that "big bang" approach to OS's was becoming obsolete. Apple started delivering MacOSX, continually, with no major interruptions. Windows started delivering Windows 10 continually, with no major future interruptions planned. Solaris announced they would be entering a similar Continuous Delivery model in 2017. Some reporters spread #FakeNews about Solaris being dead, but those who understood industry trends were watching for features from Solaris 12 to be integrated. The first & second Open Beta release of Solaris 11.4 was done... and it was clear that Solaris 12 features were being bundled into the Solaris 11 stream, but KSplice was conspicuously missing.

KSplice Hints

We started seeing hints of KSplice in 2017. The test cases for spliceadm were now being bundled. Many thanks to Tim Foster, to providing a hint into the future!

KSplice Infrastructure

It seems the infrastructure for KSplice was bundled in Solaris 11.4, but live splices have not been made publicly available. A sample of what we can observe from the manual.

SpliceAdm

The use of the "adm" suffix in commands for administration has become quite well known in Solaris. A new such command magically appeared in the 11.4 - "spliceadm". The manual page says "Splices are fixes for specific bugs that can be applied on a live system" - KSplice arrived! The Interface Stability is declared as "Committed" - it is here to stay!

Freeze and Unfreeze

A new concept was introduced to KSplice in Solaris 10 - freezing. This seems to be a way to allow for automatic updates, limit automatic updates to a particular splice version, or even facilitate rollbacks to a lower numerical splice. This facilitates the downloading of splices to a system, yet restriction of splice activation/rollback until a time of low utilization.

The Back Story

A former engineer, Enrico, from Sun/Oracle published the back-story on KSplice for Oracle Solaris. It appears he had worked with Tim [Foster, perhaps?] on the project. He spoke of resolving scaling problems, not experienced under Linux, due to huge stack sizes supported by SPARC Solaris systems while running latency sensitive Oracle Cluster. He also spoke of KSplice & DTrace compatibility - which we discussed earlier in this article with Linux.

Conclusions:

As an Application Architect in Remote Network & Systems Management, Configuration Management, and Incident Management - I had personally developed & used techniques to deploy code live in arenas where continual uptime was required. It is good to see similar techniques reach into the OS arena.

Sun had previously set a very high bar with Solaris, offering the ability to patch a live system with complete patch application on a reboot, or applying many parts of a system (until eventual reboot.) Some of this flexibility was rolled back with Solaris 11, to see KSplice finally start arriving on the scene with Oracle Solaris 11.4.

While Oracle Solaris is not rolling splices yet, we look forward to 100% uptime with KSplice, in conjunction with using LDom's on SPARC hardware.

Wednesday, May 23, 2018

Spectre - SPARC Solaris: The Safe Choice

Abstract:

As the industry continues to struggle with Meltdown, a second vulnerability family appeared referred to as Spectre. As of this article publication, there are 4 variants of Spectre, the latter two variants referred to as Spectre-NG. All SPARC systems are safe, if the most recent systems are on the most current firmware & OS releases. As of this publishing, the latest application/OS & firmware patches fixes the first two. The later 2 does not affect SPARC, as the rest of the Intel and other CPU communities are struggling with their cloud and local server infrastructures.

Spectre v2

the attacker tricks the speculative execution to predictively execute
instructions erroneously. In a nutshell, the predictor is coerced to
predict a specific branch result (if -> true), that results in asking
for an out-of-bound memory access that the victim process would not
normally have requested, resulting in incorrect speculative execution.
Then by the side-channel, retrieves the value of this memory. In this
way, memory belonging to the victim process is leaked to the malicious
process.

SPARC

Sun UltraSPARC

Older Sun UltraSPARC 64 bit Servers do not have the CPU feature which could possibly be exploited and were not vulnerable... they did not issue speculative instructions. Oracle had purchased Sun, so their support channel can provide a definitive explanation. Performance was mostly driven on these servers leveraging SMP chassis, Multi-Core sockets, and large memory footprints.

[Oracle Logo, courtesy Oracle Corporation]

Oracle SPARC

Newer Oracle SPARC Solaris servers are possibly vulnerable, if you are running a modern CPU which initiates speculative instructions (i.e. T4 or newer) while older 64 bit CPU's are not vulnerable. It has been reported on Solaris WTF that "Spectre (CVE-2017-5753 and CVE-2017-5715)" has been fixed in firmware (i.e. T4: 8.9.10 or greater; T5, M5, M6: 9.6.22a or greater; M7, S7, M8: 9.8.5c or greater.)

The short story, a firmware patch for CPU's newer than T4 are required and the impact is very minor in performance, according to the previous blog. Stock Firefox as shipped with Solaris 10 is vulnerable to Spectre v1, Solaris 11 fixed Firefox vulnerability early 2018, so users should migrate to Solaris 11.

[Fujitsu Logo, courtesy Fujitsu corporation]

Fujitsu SPARC

Sun and Oracle are not the only 2 vendors, who have produced 64 bit SPARC platforms. Newer Fujitsu SPARC Servers are also super-scalar, possibly vulnerable to Spectre v2 (CVE-2017-5753), and have been been fixed in firmware (i.e. M10: XCP2351; M12: XCP3051.)

Conclusions:

If you are using an older Sun UltraSPARC server, you are OK. If you are running a newer Oracle SPARC (i.e. T4 or newer) server, you should update Firefox on Solaris 10 or get on the latest Solaris 11 release to be protected from Spectre v1. For the same class of hardware, apply firmware patches available today to protect from Spectre v2. SPARC is immune to Spectre v3 & v4. Get with your Oracle support for the first 2 variants (doc id 2349278.1) and second 2 variants.

Friday, May 18, 2018

Meltdown - SPARC Solaris: The Only Safe Choice

Abstract:

As the rest of the industry has been struggling with security vulnerabilities, SPARC Solaris platforms have been relatively quiet. Meltdown, otherwise known as CVE-2017-5754, has taken the world by storm. Operating Systems have long relied on Memory Management Units to isolate user application programs from the OS kernel. This had come to a screeching halt, leaving lesser secure systems in a world of hurt.

Meltdown Vulnerability:

Some OS's will keep the Kernel Pages mapped into the same context as User Application Pages. This is often done for speed (i.e. linux) but places extra dependencies upon the MMU for isolation. Nearly all OS's had ceded this security concern to the CPU vendor, instead of applying the most secure practice in the OS architecture.

Meltdown:

As one vendor noted, SPARC Solaris is immune from Meltdown and about the only platform not subject to this critical vulnerability in the data center. This was accomplished by OS designers placing Kernel and User pages into different contexts, a design which added additional security, but at a performance cost that other OS designers in the industry were not willing to cede.

Conclusion:

Some Solaris systems, decades ago, may be affected, but nothing modern. Secure by Design is a typical decision for Solaris architects, a decision that has served them well for the decades they served a 64 bit OS to the user community, as other OS vendors played "catch up" in performance or features or functionality.

Tuesday, May 8, 2018

The Future - SPARC M8+ & Solaris

Abstract:

Solaris has been the heart of Sun Microsystems since 1982, with 32 bit SPARC RISC CPU since 1987. Fujitsu joined the SPARC/Solaris community in 1992, with others to follow. In 1995, SPARC went 64 bit, and has been ever since. In 2009, Oracle Corporation purchased Sun and all of it's SPARC & Solaris assets. Oracle & Fujitsu had been releasing processors & Solaris releases in rapid succession, ever since.

Fujitsu SPARC Roadmap

In 2017, Fujitsu released one of the fastest processors on the market, which happened to be a SPARC. Since they, they had been very clear
about it's roadmap for SPARC & Solaris, reaching out to a new SPARC
release in 2020. Their roadmap has not changed in close to a year, so they appear to be on-track.

[SPARC & Solaris Roadmap, courtesy Oracle]

Oracle SPARC Roadmap

Shortly after Fujitsu's SPARC release, Oracle also released their SPARC M8 processor, which became the fastest socket in the world, once again. Also in September, Oracle release a roadmap where there was no future SPARC socket. This has been remedied in Spring 2018, where a snapshot of the official Oracle SPARC roadmap mirrors Fujitsu.

The spread between M6, M7, M8, and M8+ all seem to be spaced about 2.5 years apart, indicating not much of a change in the silicon release schedule from Oracle. Both Fujitsu & Oracle are now both using the same TSC fab for their SPARC silicon, which hints at a degree of consolidation outside of their roadmaps.

As with the way Embedded 10 Gig Ethernet was consolidated on the old T2+ socket, Network Management is wondering if Oracle will finally release the embedded Infiniband on the M8+ sockets. Embedded Infiniband was conspicuously missing from the "Sonoma" S7 release. (This would be a huge bonus to the Engineered Systems, to add a high degree of consolation & increase reliability & increased performance - TODAY.)

More interesting, there is a new hardware category called "Next Generation Storage", linked with the M8+ servers. Oracle had never been a true hardware company before purchasing Sun - they mostly re-sold hardware from third party vendors in their engineered systems, where Oracle felt they could add services value. Seeing SPARC in the roadmap, for engineered systems, is interesting, but will Solaris SPARC assume the role (with Solaris 11.4's new Storage features) or Intel Linux (with new Block volumes in Linux Storage Appliance)?

[Solaris Logo, Courtesy Sun Microsystems]

Oracle Solaris Roadmap

Oracle announced in early 2017 that SPARC & Solaris will move to an Agile Continuous Improvement cycle. Solaris 12 was erased from the roadmap, which was expected with Continuous Delivery, but created an odd amount of uncertainty in the media. Human resources alignments occurred in 2 batches, creating a large amount of uncertainty, but a new M8 processor was released. Assurances that Premier Support would continue to at least 2038!

January 2018, Solaris 10 went on Extended Support, meaning an uplift for patches are required, and 3 more years before patches cease. The final Premier Support patches were released, called 2018-01. The push to Solaris 11 is on!

Solaris 11.3 continued to get monthly SRU's (i.e. Solaris 11.3 SRU 30, Solaris 11.3 SRU 31, etc.) Open Beta of Solaris 11.4 was released, followed by a refresh, and finally an announcement that there will be a major point release of Solaris ever summer moving forward. The latest Solaris 11.4 refresh is nearly all 64 bit with buffer overflow protections, which is quit an accomplishment! The roadmap shows the next release, Solaris Next, which will likely be Solaris 11.5

Conclusions:

As the market uncertainty over SPARC & Solaris continues, new silicon and new operating system releases continue to occur. Long term industry players continue to release hardware. Solaris continues is march forward with aggressive new features, even pushing all components in the operating system to 64 bit. The only promise still outstanding is rebootless patching, possibly with KSplice. Wim, where is Solaris KSplice? You're in charge now, right?

Wednesday, April 25, 2018

State of The Art - SPARC M8 & Solaris

Abstract:

The SPARC processor was developed by Sun Microsystems and had existed since the exit of major systems manufacturers from the Motorola 68K environment. Multiple manufacturers had always existed in this environment, to provide to consumers multiple supply chains in this commodity hardware market. The migration from 32 bit to 64 bit computing in SPARC occurred decades ago, as current computing systems still wrestle with the complexities. The next increment of hardware from Oracle was released September 17, 2017 - once again maintaining the fastest CPU sockets and SMP systems in the world. It is past 7 months since the release, it is long overdue for an article.

Block Diagram Layout

A logical
block diagram can help the reader gain a better understanding of the
heart of the system. That author of the Next Platform publication speculates,
how four S7 processors may have been merged onto the same die or
silicon, using a switched interconnect encapsulated in silicon.

Overall, it does appear to look like a fully switched quad-socket 32 core 256 thread SMP system, burned onto a single piece of silicon. This is a pretty astounding effort, any way one chooses to examine the diagram!

Architecture Changes

In general, processors have undergone many changes, mostly regarding consolidation of functions. As time progressed, the pattern of consolidating resulted in lower part counts, higher performance, higher reliability. Components were designed, sometimes combined, sometimes re-separated in various architectures:

Integer Processing Units

Floating Point Processing Units

Memory Management Units

Processor Socket Cross-Bar Switch

1st level cache

2nd level cache

3rd level cache

Network Interfaces

Video Processing

Encryption Units

De-encryption Units

Decompression Units

[DAX] Database Analytics Acceleration Units

The M8 is no exception... being a server & cloud oriented, video has been de-emphasized for some time & no longer developed actively by Oracle. Previous processors had integrated 10 Gig Ethernet (i.e. Sun Microsystems SPARC T2+) or attempted to integrate Infiniband (i.e. Oracle SPARC "Sonoma" S7) - but Oracle had decided to de-emphasize these commodity features. Oracle, being primarily a software company, concentrated on the most "bang for the buck" with faster cores, encryption, decryption, decompression, and database acceleration units.

Socket Availability Architecture

The SPARC M8 Processor continues to play a vital role in the Availability Architecture of Solaris systems. With the innermost 4 layers of the onion, features like: parity, error correction, reissue on error in hardware, internal power management, voltage scaling, frequency scaling. Decades of design continue to exhibit robust features unavailable in commodity CPU architectures.

Memory Availability Architecture

Normally, error correction exists on DIMM's chosen for the SPARC Servers. If one of the DIMM's repeatedly experiences errors on a SPARC M8, Solaris will initiate the process of retiring the faulty DIMM. The hardware can interleave memory across all 16 DIMM's, for maximum throughput, and can also facilitate interleaving memory requests across 15 DIMM's, in order to facilitate a DIMM retirement. During retirement, there is a slight decrease in the amount of available memory.

Virtualization Architecture

The SPARC M8 processor works in combination with OpenFirmware in order to implement the lowest levels of virtualization.

Physical Domains, which allows for electrically disconnected domains, is facilitated by the LOM (Lights Out Management) card, allowing for an OpenBoot instance to operate on each physical domain (note: this feature is only available on higher end systems.) There are no particular features within Physical Domains, which specifically benefit from the M8 processor.

Within the Physical Domain, a Hypervisor program is run in the OpenFirmware, allowing for virtual instances of OpenBoot to run for different groups of devices (i.e. console, memory, pci-cards slots, network cards, disk devces.) Different OS's may run inside of these logically separated OpenBoot instances. A Logical Domain can be live-migrated to a different Physical domain or even a different SPARC chassis, taking advantage of M8 enbedded engines at wire-speed such as Encryption, Decryption, and Decompression.

Zone virtualization is also available under Solaris, regardless of underlying architecture. The SPARC M8 brings hardware accelerated encryption, decryption, and uncompression for activities like live migration, similar to the benefits to Logical Domains.

Recent Processor Comparisons

The SPARC M8 processor has certainly advanced in performance over the past few generations of silicon, both from Oracle and Fujitsu. The Register was kind enough to put together a simple chart, illustrating the differences between each of the recent processors, each processor being the fastest socket of it's kind during it's release.

Performance Comparisons

As The Register had pointed out, the new core's "wider instruction pipeline... coupled with the increase in clock frequency and the larger L1 code cache" sees to wring out the most recent performance gains from this silicon. These features help to define the 5th generation S5 core, located in the former [1/4 sized] S7 and now M8 processors. Boosting the clock rate nearly 25% from slightly over 4GHz M7 to an industry "unheard of" 5 GHz clearly is beneficial.

Security Comparisons

The Register also reported important enhancements in security, "it can perform
hardware-accelerated cryptographic functions – such as AES, RSA and
SHA-512 – at twice the speed of the M7." In addition, the SPARC M8 is the only processor in the industry which supports SHA-3 in silicon. With functions like Random Number Generators and [En|De]Crypto operations in hardware, there are no excuses for lack of security in a data center since there is virtually no cost or processing overhead. Crypto functions operate fast enough to occur at wire-speed for network with virtually unnoticable CPU overhead, reserving the rest of the system resources for processing business/military data.

Database & Analytics Comparisons

The massive thread count with 32x cores, 8x threads per core, 32x floating point units are not unusual for the SPARC M8, although these are already unusually large in comparison to other slower commodity & proprietary processors on the market. These significantly boost application & database performance to non-competitive proprietary Intel and other processors.

The SPARC M8 advances on the former "State of the Art" SPARC M7 with larger caches & faster processing for all workloads, including Database & Analytics.

The real secret of this processor lies in the Next-Generation Data Analytics Accelerator (DAX) engines. In-line decompression enables 200%+ increase in data held in memory for processing without performance impact at 120+ GB/second. Performance improvements of up to 700% for in-memory query acceleration is demonstrated! Oracle number processing is now managed through accelerators,
short-cutting software routines to directly pass processing of numbers
larger than 16 bytes to the hardware for all database workloads,
resulting in 1000% increases in performance! Java 8 Streams API can be used to work with data-sets directly via the DAX, eliminating iteration operations [since they are implied] providing 2000% increase in performance on some operation types!

Conclusion

There are no competing processors in the same Integer or Floating Point Processing Class as the Oracle SPARC M8 processor. Full security, with nearly any cryptography library of choice, is no longer a burden with the SPARC M8... there are no security compromises with this processor, outrunning any workload on any similarly configured socket-to-socket configuration, even with full encryption enabled under SPARC. Once one tries to compare database processing or java application processing, it is quickly observed that many industry players may be a decade or more away in performance of similar workloads. All of this, with the ability to virtualize, with no virtualization performance or monetary tax, makes this an exceptional core to any system in any data center. It will be difficult for any hardware vendor to build any kind of superior next generation processor in the first quarter of the 21st century... the SPARC M8 is the apex of modern day computing.

Tuesday, April 24, 2018

State of The Art - SPARC S7 & Solaris

Abstract:

The SPARC processor was developed by Sun Microsystems and had existed
since the exit of major systems manufacturers from the Motorola 68K
environment. Multiple manufacturers had always existed in this
environment, to provide to consumers multiple supply chains in this
commodity hardware market. The migration from 32 bit to 64 bit computing
in SPARC occurred decades ago, as current computing systems still
wrestle with the complexities. Oracle ceased producing horizontal scaling CPU sockets once purchasing Sun Microsystems. In June 2016, Oracle decided to re-enter the commodity market with the SPARC S7. NetMgt had published an article, but it was lost, and it was decided it was time to re-publish it again.

S7 "Sonoma" Floor Plan:

It can be clearly seen that the new die photo shows the use of 2x 4 Core Clusters. The cores are nearly the same 4th generation S4 cores, bundled in it's 32 core sister M7 processor. Glueless Coherency links were bundled, to scale an S7 system from 8 to 16 cores, with little external circuitry. With DDR4 memory interfaces on-chip, latency is cut down by eliminating external chips. Database Analytics Accelerators have been included, although not as many per-core as with the larger M7.

Close to 20% of the space used by an Infiniband network interface. The
last time integrating network on-board silicon occurred was with the
UltraSPARC T2+ (which integrated 10 Gig Ethernet) - but it was not released to customer facing production system.This was a significant disappointment to NetMgt, since an new Blade system with an Infiniband backplane allowing scalability to thousands of sockets would have been a welcome addition for cloud computing.

Architectural Changes:

The S7 is a smaller processor, returns to 8 cores on a die, similar to the T4 from back in 2011, but even outperforms processors from 2013 with higher cache, clock speed, and 4th generation core. It was designed to be a competitive socket (in price/performance) to commodity proprietary CPU's (instead of being the fastest performing socket in the market.)

Dual Socket Configuration:

The glueless dual-socket configuration allows for outstanding communication speeds between sockets. It is apparent that more bandwidth may be available over PCIe links than over the un-exposed Infiniband. Without the Infiniband exposed, the S7 looks more like an UltraSPARC IIIi.

Unexposed Infiniband Performance:

The unexposed infiniband offered the possibility of significant packet performance improvement, over a PCIe card, even under high load. Note, the red Sonoma IB line mostly maintaining between 20-60 Millions of Packets per second.

A blade chassis connecting all minimal Sonoma "S7" blades (holding memory & 2 sockets) connecting back to infiniband storage in an MPP environment scaling across thousands of nodes could have been an amazing entry into Cloud Computing or Super Computing environments.

Conclusions:

While NetMgt welcomes the new low-cost "Sonoma" S7 options, we see the form-factor produced as a "game changer" unrealized, by not placing the chip in a socket, exposing Infiniband, and into a chassis form factor which could most leverage it's strength. Such a socket could have easily replaced & out-performed the Intel based Storage subsystem Oracle's Infiniband native Engineered Systems. A such a socket in a blade chassis would have filled-out a gaping hole in Oracle's systems portfolio. Furthermore, the cost of memory is so high in the chassis that the cost difference between SPARC "Sonoma" S7 and SPARC M7 are marginal if selecting a chassis to perform LDom virtualization. It is a beautiful chip, for what it is, but a great opportunity lost.