12 key SSD ideas which changed in 2014

This is about closing important gaps in the
intelligence of message passing and the speed of data access between
application processors and SSD controllers.

(SSD controllers which -
in the vast majority of SSDs - come attached with their own offload processors
or associated data movement engines.)

Traditionally SSDs have been
designed to reduce the access times to data - but within the framework of
commands, APIs and data structures which have been designed for applications
agnostic data storage drives.

And in order to make SSDs easier to
use - classic SSD controllers also perform a lot of house-keeping and
data integrity
related functions - in a way which is the apps processor doesn't need to know
about (although it can collect stats related to endurance etc).

There
is much evidence to support the idea that both applications performance and
data storage efficiency
can be greatly improved if the flash management and apps management processor
are either the same CPU - or if they can talk to each other in a more
effective way.

These improvements (which enable data handling
responsibilities to be partitioned downwards to the flash or upwards to the
apps host - depending on which has the best view of what is needed) have
already been implemented by various SSD vendors in experimental or custom
APIs.

These concepts - which first took root in large web scale server
farms - have also delivered useful results in some industrial SSDs when
scaled down to a single SSD.

80% of fast enterprise data IO is transient - but legacy storage software
often sucks this up as part of its reliability functions and then attaches a
similar weight of metadata and overhead as if it were archive data. This is some
of what I learned in various conversations with David Flynn
cofounder Primary Data

key SSD idea #2

re
PCIe SSDs

This year - as part of a continuing trend - we've seen an
upswing in the number of companies who offer
PCIe compatible SSDs
in form factors like M.2 and 2.5".

The barriers to market have
been reduced by standards such as NVMe and SATA express - which by creating
frameworks of software and hardware interchangeability - have minimized the
risks for oems who incoprorate such SSDs into their storage and computing
systems.

An important new factor for the PCIe SSD market this year
was the materialization of product announcements centered around the core
concept of using PCIe as an interconnection fabric between racks.

The idea of using
flash as a new memory tier isn't new. And neither is the idea of using flash in
DRAM memory slots. But in 2014 there were several developments which added
weight to the usefulness of these ideas.

Applicable to any kind of standard flash SSD -
SanDisk's ZetaScale
software (described by StorageSearch.com as "one of the most significant
SSD software products launched in 2014") is an API toolset which gives
software designers the freedom to treat flash in a similar way to DRAM - thereby
being able to rely on much higher capacities within any given monetary budget
ceiling.

Although the performance characteristics of such memory
won't suit all applications - the ability to experiment and invest in a
technology platform which promises to avoid lock-in to any particular SSD form
factor - will encourage the development of new types of data repurposing
platforms.

Those who may have been disappointed by the low aspirations of
Diablo's 1st generation
memory channel SSDs - were given a glimpse of something more akin to what they
might have been wishing for - in the unveiling of an ambitious 2nd
generation architecture which promised to go much further in 2015.

The
key ingredient here is a new software framework (Carbon2) with features like
NanoCommit technology.

The new software is being offered as part of
developer packages which anticipate 2nd generation MCS hardware which will be
fast flash DIMMs compatible with DDR4.

One of the trends in computer
architecture in recent years is that new software architectural concepts which
deliver sustainable efficiency or management efficiencies have found it easier
to get their benefits established and recognized at a large scale - as part
of big web entities or cloud infrastructure.

But the lessons learned
have been duly noted and reapplied to other use cases and are now finding
their way into individual rack scale products too.

3 companies which
stand out for their different approaches in this respect are:-

Because looking ahead from the perspective
of 2012 they and their licensees or acquirers were going to be among the first
vendors who could leverage the economics of next generation flash.

They
did this by moving away from classical flash controller technologies - which
relied on anonymous industry wide characterization statistics for key flash
parameters - and moving towards an adaptive model - which was able to
recognize and grade different qualities of individual flash blocks (even
within the same SSD).

The new adaptive DSP technology was able to
choose from a wide bandolero of timing and ECC techniques instead of being
dependent on a single caliber flash manage bullet.

By the middle of
2014 - adaptive R/W had become a mainstream technology - deployed by most
leading enterprise SSD systems (in applicable products) - so its strategic
advantage as a competitive differentiator has diminished.

Instead it
has become the new "standard technology" for handling all sub 20nm
planar MLC flash devices.

But it would be wrong to think of it as a
uniform technology. There are significant differences in the scope,
granularity and associated controller and power footprints of the many
different adaptive DSP flash IP sets used in the SSD market.

key SSD idea #6

3D
nand flash -may be tough enough for industrial markets

Although 3D
nand flash SSDs have been shipping in the market - the current technology
doesn't deliver enough efficiency and cost advantages to replace 2D in the
short term. Many manufacturability and design problems remain to be solved
before that is likely to happen in mainstream SSD markets.

On the other
hand the raw
endurance of
1st generation 3D flash seems to be 3x to 4x better than 2D at
the same line geometries - according to early work done by an
industrial SSD
company FMJ Storage.

If
these early impressions are confirmed in later volume production - this could
open up the possibility of alternative markets for this type of flash.

Acquisitions
reported in 2014 seemed to indicate that SSD companies aren't worth as much as
they were before.

Although there are special factors which complicate
any particular analysis - as I discussed in the cases of
Seagate acquiring
LSI's SSD business,
and SanDisk acquiring
Fusion-io - it's
clear that from the viewpoint of the people who matter (those with the money)
an SSD company with a rich set of IP and strong market recognition in 2014
isn't generally worth as much as you might have thought if you had extrapolated
from SSD company values in 2013.

Why is that?

In one way it
seems perverse - given that the overall market opportunity for SSDs is now
generally assumed to be much larger than it was before.

I think the
key factor at work here is evidence (as reported in financial reports of some
leading SSD companies) that competition is much tougher than before (due to
the growing number of competitors and also the rise in the quality of such
competitors).

But another key risk factor (for any encumbent SSD
vendor) is vulnerability to future technology shocks - which can disrupt
their business prospects.

These technology shocks don't just stem
from new startup SSD companies - but can also occur as a result of macro changes
in the market as users
change the way they use and deploy the same type of SSDs when using
different software.

key SSD idea #8

SSD
pricing and business models

How much should you pay for an
enterprise SSD array?

And what exactly is it that you're getting?

Although
SSD vendors had always been enthusiastic about what their products and
technologies could do in the first decade of enterprise flash - the language
with which they bundled their pricing offers did not show the same leaps of
creative imagination which they were expecting their customers to make.

But
in 2014 - a small number of SSD pricing pioneers designed new enticing
pricing models for their flagship flash arrays which broke away from the
formulas of the past.

Behind these new pricing models was the explicit
recognition that there is always a high degree of uncertainty involved in
such purchases for various technical and business reasons.

Surprisingly - given its already substantial size
and gravitational business pull for SSD drive makers - there are still
significant parts of the enterprise SSD market which remain uncharted and
unsatisfied.

For investors and SSD startups the opportunities to grow
business in under exploited high value user territories may be a source of
comfort - given the potential upside.

However, for users who are still
waiting for vendors to offer them the kind of products and services they
really need - it's a source of frustration.

In some embedded markets -
which use a lot of SSDs - the rackmount SSD is simply viewed as a dumb component
- like a 2.5" drive. It's a component with larger capacity and more
performance - but is simply one of many. And although it's a box - it's not "the
system". In fact its internal cleverness and associated software are
sometimes regarded as a nuisance by true applications aware systems software
which has a much better idea of what's going on then the little SSD box
designers could ever have imagined.

key SSD idea #10

the
importance of SSD software

One of the key ideas which permeates
everything now in the SSD market is the importance of software to the SSD
market.

"the SSD software market is getting
ready for a world in which all enterprise data touches SSDs"

And
elsewhere in the same article I also asserted

"the winners in
SSD software could be as important for data infrastructure as Microsoft was for
PCs, or Oracle was for databases, or Google was for search."

Apart
from any confirmatory events in 2014 - when I think about to what might
happen in the next few years - the overwhelming importance of SSD-centric
software seems like a no brainer.

I think we will see not only new
predictable generations of SSD software coming to market (which will be
designed to work with currently known computer architecture models) but also
entirely new data architectures and ecosystems whose very existence has
been predicated on the assumption of a widely deployed SSD enhanced base
infrastucture.

key SSD idea #11

re
industrial SSDs - designers have refocused and chosen the viable reality of
excellence in selected niches above the unfeasible goal of having the best
technology roadmap for all applications

I have talked to many
leading
industrial SSD
companies this year - and there is definitely a different mood in the air
about this market and some confidence that vendors can carve sustainable
business niches - having found (differing) rational strategies to cope with the
chaotic changes in the general SSD market.

Those companies which still
have industrial SSDs as their main product lines - when many other companies
have exited this market - due to the siren pull of bigger markets (such as the
call to consumer SSDs in 2006, and the big pull towards enterprise flash
which was hard to resist by 2008) have survived several waves of turbulent
change in their own market in the past 5 years.

In 2011 to 2012 - the long held assumption that MLC
flash would never be good enough for industrial applications was called into
question by the apparent disproof of that very notion in many enterprise
products. That caused customers to question - why can't we use MLC in
industrial SSDs?

This was a difficult balancing act for
industrial business owners - because even if they solved the problem of
sourcing and managing
reliable enough MLC for some applications - with new controllers - the added
cost of other factors in the design - due to the increased hold up time needed
to clean up block management operations in MLC compared to SLC (new firmware)
- and the lower capacities used in many industrial systems - meant that the cost
benefits of making the transition to MLC - were not always clear cut or
immediate.

Customers told me they experienced a distinct lag in the
market of about 2 years - during the transition in the top 20 or so industrial
suppliers from talking about the availability of MLC in their products -
while others were actually doing it.

Part of that was due to
competitive market differences (some companies do things faster than others) but
another factor was a fundamental difference in views about whether that was the
right solution for all products. (It still isn't.)

What has become
clear in 2014 - is that there is now a greater degree of specialization
within the industrial SSD market.

This has come about because no
single company has a single set of IP which is most competitive for all form
factors and interfaces.

when it comes to controllers - industrial SSD makers have different
approaches even within their own product lines.

The diveristy of
industrial controller solutions goes beyond the simple filter of performance /
form factor / power consumption and memory type - and standard versus in-house
design - to encompass firmware adaptions of standard controllers, and stretches
to customized firmware which can optimize system performance for known
configuartions and software environments.

The industrial market represents a bigger total available market than ever
before.

But set against that is the need for greater specialization -
and application specific optimizations.

The result is greater market
fragmentation - and more niches - rather than a small set of big broadly
overlapping markets.

And industrial SSD companies are also finding new
markets in the enterprise too - in hot spots in blades and small solo SSDs
which are used in managing services rather than as primary storage.

enterprise
SSD designers will adopt any kind of naughty flash - once they've figured out
what to do with it - and have validated the memory in less intense consumer
markets

Continuing this
10 year trend
- in 2014 - 3D MLC indisputably joined the roster of flash types deemed good
enough to ship in enterprise SSDs - notably confirmed - if there were any
doubts - by the announcement in
September 2014 -
that Samsung was using
3D nand in a new PCIe SSD - rated at 10
DWPD for 5 years.

Currently there is no type of mainstream nand flash which isn't
being used in some type of enterprise SSD systems.

And if you hear
vendors say - that their array is better because it uses so called enterprise
MLC (eMLC) it really means that they don't know how to manage the flash with
their own IP and have passed the buck to their
memory suppliers and to
their customers (who have to pay more).

In some high end enterprise
market applications - there are valid reasons you might choose to
pay more for your
flash and have your flash array delivered in a bigger box - but in most
applications - that choice is a
customer
preference.

Maybe you like the software which comes with the box -
or it will cost you more to validate alternative suppliers. But eMLC is not -
and has not been for many years - a necessity in most enterprise flash arrays.

On
the other hand - if you are a worrier - rest assured that the reliability of 3D
nand will need to be reassessed in future generations as the stack layers
progress upwards in number. (Bad things might still happen.)

23 years later... and still
counting - as in 12 key SSD ideas, Top 10 SSD Companies etc

Where does - 23 years later come in?

Earlier this month
(December 2014) LinkedIn's software picked up the fact that it was the 23 year
anniversary of my having founded the enterprise
publisher which
publishes StorageSearch.com.

Although 23 is a prime number - and in
that way is - I suppose - interesting for some - it wasn't a milestone I had
planned to write about or mention on these pages. But as some of you picked up
the bot generated posting and said nice kind things I thought that deserved
some kind of human generated response from me. So here's what I said...

"Thanks
for your kind and motivational comments. In the new era of SSD guides the
balance of effort has moved away from - what are the new products and
technologies? - To inferring - where they are - on known and unknowable
intersecting roadmaps of evolutionary and disruptive change - with destinations
which don't yet have words in the jargon of computer architecture."

re
publishing and social networks

Everyone has their own preferences. In
my case - I've become accustomed to my "social informative network"
being my readers rather than any of the new fangled channels such as LinkedIn
or Twitter. So this web site is where I invest most of my efforts.

But
it's not always obvious what I've been working on - and a reader asked my
about that recently. He said he hadn't seen anything from me in recent months
about the SSD market - and asked if everything was OK.

In his case it
was simply that he had recently joined a Top 10 SSD Company - whose corporate
servers deny its employees access to StorageSearch.com - because their firewall
filters out sites which have cartoon-like content. I won't name the company -
they've had this internal problem for over a year. I guess my readership would
be bigger otherwise. The marketers in that company like the mice and they like
the content BTW.

After our email exchange - we spoke for about an hour
about strategic changes in the enterprise SSD market. Very interesting - but
for background - rather than a new article.

Apart from corporate
firewall walls which block out StorageSearch.com from some desktops - another
reason you might not see what I've been writing about recently is that I know
you won't be interested in it yet - so I don't flag the links loudly.

When
I choose to write about a new SSD topic - it's because it's interesting or
important to me - and my gut tells me that it will be of interest to seriously
minded readers like you at some time in the future. Sometimes it can take 5 to
10 years for new SSD related ideas to get into the mainstream market. But
by writing about them early - I'm able to begin a conversation with the blue sky
architects and business visionaries who will do the difficult part - which is
making these things happen - and fit into a world which wasn't created for their
kind.

The truth is - there isn't enough time to complete half the SSD
articles I get started on. Too many interesting things going on.

From
time to time - it's useful to distill the essence of all that raw random SSD
chatter into something simpler.

Editor:- December 2, 2014 - What next - when
PCIe SSDs are already
everywhere? You know you need them. But you need more too. Signs of interesting
times ahead in
2015 are revealed in
the new edition of the
Top SSD Companies -
based on market metrics in
Q3 2014 -
researched and published by StorageSearch.com

For
over 5 years - Fusion-io
had occupied the top #1 spot in this list as the SSD company which
was most researched by our readers. That was inevitably going to change. And it
did. But not in the way you might have expected.

The new #1 SSD
company is Diablo
Technologies - creator of the Memory Channel Storage platform.

The roadmap vision
I'm seeing emerge from enterprise SSD developments in 2014 - is that while oems
and users are being offered more choices in form factors and flash memory types
- each of which adds to the raw confusion of which one is best to use - the
mission statement for the software developers and fabric enablers - will be to
create SSDcentric platforms which enable these disparate pieces to be seen as
interoperable subsets of a bigger continuum architecture...

This acquisition demonstrates a new wisdom:- to succeed in the
enterprise SSD market today - and to achieve the ultimate efficiencies at the
manufacturing level - vendors have to think like systems companies.

For 7 years the market for
SSD data recovery
- which is largely aimed at
consumer style SSDs
in notebooks - had been a sterile zone when it came to hard facts and
statistics - but some data finally emerged in 2014.

As previoulsy
expected - SSDs are more reliable than hard drives when used in notebooks - and
less likely to need data recovery.

How much better are SSDs? About
5x.

The main problem found in SSDs which were being sent for
data recovery was data corruption due to
endurance
mechanisms.

Endurance problems were 20x more likely to lead to
SSD data recovery than all other component failures combined.

historic perspective and
context of enterprise SSD market and data architecture in 2014

In recent years hundreds of companies
have been attracted by the idea of playing in the SSD market
casino /
bubble due to the
absence of sustainable market and technical
leadership positions.

The changeability of leadership positions was partly due to
changing
priorities in memory management due to smaller
cell geometries but also
due to the huge
waste inherent in data
architecture which
had been layered on the magnetic storage server model during the first decade of
the dotcom ecosystem.

There was a point in the 1990s when users were getting good enough
results from the distributed workloads of
RAIDHDD and
multiple core CPUs
so that system architects didnt consider serious alternatives and didnt worry
about the wastefulness of those systems.

Therefore SSD based accelerators were viewed
initially by the
enterprise mainstream as
exotically
expensive cures for rarely articulated customer headaches rather than as part
of the latency based architectural hierarchy which
should have been
part of the software to begin with.

When solutions started to appear they were from small outsider
companies which didnt have the financial resources to change the messy world
they wanted to live in.

Partly what weve seen in the past 7 years of
enterprise flash
as the SSD market
has grown in size is that software entanglement problems which were uneconomic
to solve when the market was smaller have been creeping in as the useful part
of some bundled SSD solution.

But changing the course of the whole data processing industry while
enabling it to keep chugging on without disrupting previously embarked user
missions is a fascinating story.

This owes nearly as much to the
long standing inertia of innovation averse old style (pre SSD era
storage / server vendors) who created the fertile ground for the disruptive
market which followed as it does to the bold SSD adventurers who planted their
random little seeds and amazed the world and themselves by seeing what a huge
reaction they got.