In Memory Data Grid

In-Memory Data Grid (IMDG)

In my current project I am using SOSS (Scale out state server) which internally uses IMDG concept. And I found this as one of the good technique to save data in out of proc memory like controllers/actions permissions of various Role Groups in MVC applications, users Sessions’, not so frequent changing data of database, other application level data. So, the data base hit gets reduced and we read data from RAM only which is very fast. The application performance increases tremendously. Also, we can store as much data as we want; we need to have that much RAM only which is easily available and cheap. Below I am adding some more details. Also, I am aware of a demo in which lakhs (~8) of filtered records were displayed in console within few (~5) seconds through SOSS.

IMDGs has an small disadvantage, when the application starts and if at that time data is not loaded in IMDGs then the data gets loaded in there and hence small time is spent in loading the data in IMDGs.

Table of Contents:

Scale out shortcomings

What is an In-Memory Data Grid

Top Benefits of IMDGs

Why IMDG?

What is IMDG?

Factors for Accelerating IMDGs

IMDGs Products (some of them)

Architectural points

Scale out shortcomings:

In Scale out we have many more than two servers and client request are transferred to one server using load balancing. And these servers are not connected to each-other.

Scaling out:

Offers excellent scalability.

But is challenging to implement:

How to share data across servers –as servers are not co nnected to each other

How to maintain high Availability – If one serer fails how to read data present on that server.

Faster access time for business logic state or database data – Data storing in IMDGs from Database or through some business logic is quite fast.

Scalable throughput to match growing workload and keep response times low – You can load as much data as the RAM of your server farm can store and also data reading speed from these IMDGs are nearly constant because the data is stored in Key/value pair (Hashing) in the RAM and hence data reading speed is independent of Data stored in the RAM.

High availability to prevent data loss if a grid server (or network link) fails – IMDGs have Data Replication techniques so due to this same Data copy resides on many IMDGs nodes. So, when one node fails still we can read the data from other nodes.

Shared access to data across the server farm – Data is shared between the servers in the farm. So, all the servers are in connected mode always, we can hit any sever and can reach to any other server in no-time.

Fast data analysis for quickly and easily mining data using “map/reduce” – It uses “map/reduce” technique so data analysis times is reduced significantly (like in Hadoop).

Data Querying/ OQL: These solutions generally provide SQL-like querying language that allows you to access data stored. OQL syntax is very similar to SQL but they have other differences as well.

Continuous Queries: IMDB provides a feature that combines a query result with a continuous stream of related events in order to maintain an up-to-date query result in a real-time fashion. This capability is called Continuous Query, because it has the same effect as if the desired query had zero latency and the query were being executed several times every millisecond.

Failover and High Availability: When one server crashes, the client connections automatically connect to the other servers.

User Authentication: When applications aren’t using sensitive information, we can go by simple UserName/Password based basic authentication. But anything beyond basic authentication like LDAP, PKCS or Kerberos will depend upon the product in use. So, the product (like Coherence, VMWare Gemfire etc.) should be pluggable for respective authentication technique.