September 21, 2007

Putting the Database Where It Belongs

One of my earlier blog posts (on the GigaSpaces Blog) was "Persistence and the reliability myth". I've been talking about alternative approaches to reliability and scalability in a stateful environments, other than the traditional database-centric approach, since the inception of GigaSpaces.
It is very encouraging to see that what used to be said behind closed doors is now becoming common knowledge.

"What I see happening is that the large database vendors, whom I’ll call the elephants, are selling a one-size-fits-all, 30-year-old architecture that dates from somewhere in the late 1970s.
Way back then the technological requirements were very different; the machines and hardware architectures that we were running on were very different. Also, the only application was business data processing.
For example, there was no embedded database market to speak of. And there was no data warehouse market. Today, there are a variety of different markets with very different computing requirements, and the vendors are still selling the same one-size-fits-all architecture from 25 years ago.
There are at least half a dozen or so vertical markets in which the one-size-fits-all technology can be beaten by one to two orders of magnitude, which is enough to make it interesting for a startup. So I think the aging legacy code lines that the major elephants have are presenting a great opportunity, as are the substantial number of new markets that are becoming available."

RDBMS succeeded to become the de-facto standard to building system because they offer some very compelling attributes - ACID brings a lot of piece of mind. Large scale systems,low-latency system and fault tolerant systems opt for another set of compelling attributes (BASE). The point is that when you design your next solution maybe the conventional database thinking is something that you should at least give another thought to and instead of just following dogma.

As you can imagine I'm in violent agreement with both Arnon and Stonebraker. Although the database is not going to disappear, it is obviously not the best fit with the scalability requirements of many of today's applications. Having said that, the question is where and how it will fit in the architecture of this new class of applications?

To answer that, we need to look at what the database brings to the table in this context:

The database manages our ACID transactions

Durability – the database serves as a transaction-safe persistent storage on top of a file-system

The challenge is how to combine the database and the Data Grid in a way that will fit a large-scale environment.
I suggest the concept of Persistence as a Service (PaaS), in which the database will play a similar role to a Data Warehouse. In other words, it will move a step back to play more of a background role in the data processing food chain.

Comments

Putting the Database Where It Belongs

One of my earlier blog posts (on the GigaSpaces Blog) was "Persistence and the reliability myth". I've been talking about alternative approaches to reliability and scalability in a stateful environments, other than the traditional database-centric approach, since the inception of GigaSpaces.
It is very encouraging to see that what used to be said behind closed doors is now becoming common knowledge.

"What I see happening is that the large database vendors, whom I’ll call the elephants, are selling a one-size-fits-all, 30-year-old architecture that dates from somewhere in the late 1970s.
Way back then the technological requirements were very different; the machines and hardware architectures that we were running on were very different. Also, the only application was business data processing.
For example, there was no embedded database market to speak of. And there was no data warehouse market. Today, there are a variety of different markets with very different computing requirements, and the vendors are still selling the same one-size-fits-all architecture from 25 years ago.
There are at least half a dozen or so vertical markets in which the one-size-fits-all technology can be beaten by one to two orders of magnitude, which is enough to make it interesting for a startup. So I think the aging legacy code lines that the major elephants have are presenting a great opportunity, as are the substantial number of new markets that are becoming available."

RDBMS succeeded to become the de-facto standard to building system because they offer some very compelling attributes - ACID brings a lot of piece of mind. Large scale systems,low-latency system and fault tolerant systems opt for another set of compelling attributes (BASE). The point is that when you design your next solution maybe the conventional database thinking is something that you should at least give another thought to and instead of just following dogma.

As you can imagine I'm in violent agreement with both Arnon and Stonebraker. Although the database is not going to disappear, it is obviously not the best fit with the scalability requirements of many of today's applications. Having said that, the question is where and how it will fit in the architecture of this new class of applications?

To answer that, we need to look at what the database brings to the table in this context:

The database manages our ACID transactions

Durability – the database serves as a transaction-safe persistent storage on top of a file-system

The challenge is how to combine the database and the Data Grid in a way that will fit a large-scale environment.
I suggest the concept of Persistence as a Service (PaaS), in which the database will play a similar role to a Data Warehouse. In other words, it will move a step back to play more of a background role in the data processing food chain.