Demartek provides real-world, hands-on research & analysis by focusing on industry analysis
and lab validation testing
of server, network, storage and security technologies, for the
small and medium business (SMB) environment and the large enterprise.

Dennis Martin often shares test results from the Demartek lab when he speaks at conferences,
user group meetings, etc. View his speaking schedule at the
Events and Speaking page.

Demartek Comments on IDF2012 and NVMe

I spent part of this week at the Intel Developer Forum (IDF) 2012 in San Francisco, primarily to learn more about
NVM Express (NVMe) and take a first look at the prototype solutions on display. I spent time with representatives
from several companies and discussed some of the technical details with a few of the architects.

NVM Express is a scalable host controller interface designed for enterprise and client systems that use PCI Express SSDs. NVMe
was developed by an industry consortium of more than 80 member companies and is directed by a 13-company promoter group that
includes Cisco, Dell, EMC, IDT, Intel, LSI, Marvell, Micron, NetApp, Oracle, Samsung, SanDisk and STEC. The original NVMe 1.0
specification was published in March 2011. The 1.0c specification was published in February 2012. A version 1.1 is underway that
has added enterprise features that support multi-host usage models including the use of reservations.

The basic idea is to streamline storage access to SSDs (NAND flash and future technologies) that are connected directly to
the PCIe bus. NVMe is compatible with the SCSI command set, but underneath has implemented a smaller command set of six I/O
commands that aims for efficiency, scalability and performance. It operates with 4KB commands that can be obtained with a
single 64 byte DMA fetch. NVMe supports very deep queues, with 64K commands per queue and up to 64K queues. This parallel
architecture is designed for memory-type devices that are used for current and future SSDs and is not limited to the smaller
queue depths and numbers of queues that are typically used for spinning hard drives.

For enterprise applications, NVMe fully supports end-to-end data protection features such as DIF and DIX. It also supports
MSI-X and interrupt steering, and is designed to operate appropriately in multi-core environments.

One of the goals is to reduce latency in the storage stack. Intel claims that the NVMe Linux storage stack reduces latency
overhead by more than 50% when compared to the SCSI/SAS storage stack in Linux.

NVMe reference drivers for Linux, Windows and VMware are available, and ongoing work is expected in this area. There is also
a prototype driver available for Solaris. There is development with NVMe for the UEFI ecosystem as well. The NVMe working
group is collaborating with the University of New Hampshire Interoperability lab (UNH-IOL) to develop an interoperability
program, and a plug-fest is expected in 1H 2013.

I saw an impressive display of NVMe in the NVMe pavilion at IDF2012. Intel had setup an FCoE SAN with twelve servers (initiators)
communicating via FCoE to a single storage server with six prototype NVMe PCIe SSDs. They were achieving 3.1 million IOPS at 4KB and
serving this over the FCoE SAN. These prototype NVMe SSDs are not publicly available yet, but this is clearly an indicator
of things to come. The storage server was running the Intel S2600IP4 server board with dual Intel Xeon E5-2690 processors and four
Intel X520 10GbE adapters. Two of these adapters were quad-port and two were dual-port for a total of twelve 10GbE ports on this
storage server, providing 120 Gbps of combined bandwidth to the client servers.

There were 18 industry booths in the NVMe pavilion showing controllers, tools, silicon IP, software and SSDs. The Storage Networking
Industry Association (SNIA) has an NVM programming model technical working group (TWG). EMC was showing that they are “NVMe-Ready” by
showing a version of their “Thunder” storage system with one of the same prototype NVMe SSDs that Intel was showing in their display.
OCZ and SanDisk showed prototype NVMe SSDs. Dell was showing servers that can accept NVMe drive form-factor SSDs. IDT showed its native
NVMe flash controllers. IBM was showing a modified version of the DB2 database application that uses the NVM programming model proposed
by the SNIA NVM TWG. The full list of organizations in the NVMe Community exhibitors include: Agilent Technologies, Cadence Design
Systems, Dell, Inc., EMC Corporation, Integrated Device Technology (IDT), IBM Corporation, Intel Corporation, LSI Corporation, Micron,
NetApp, Inc., OCZ Technology, SanDisk, STEC Inc., Storage Networking Industry Association, Teledyne LeCroy and Virident Systems, Inc.

The first NVMe products are expected to become generally available later in 2012 and these are expected to be enterprise products.

NVMe is an exciting and relatively new technology that aims to improve storage performance using PCIe as a carrier. However, this
does not mean that other technologies such as the Advanced Host Controller Interface (AHCI) or SCSI/SAS are going away anytime soon.
We expect both AHCI and SAS to continue to develop and gain faster speeds and other improvements. The enterprise storage market tends
to be very conservative and adopts changes slowly, primarily because storage is the place where our data lives and storage administrators
generally can’t afford to take risks with losing data. Having said that, however, NVMe can be a great solution for high-frequency storage
and ultimately making storage platforms faster and less expensive.