Slideshare uses cookies to improve functionality and performance, and to provide you with relevant advertising. If you continue browsing the site, you agree to the use of cookies on this website. See our User Agreement and Privacy Policy.

Slideshare uses cookies to improve functionality and performance, and to provide you with relevant advertising. If you continue browsing the site, you agree to the use of cookies on this website. See our Privacy Policy and User Agreement for details.

4.
Purpose/Goals
• CAP Theorem is at the base of
developing distributed systems
• Still - not everyone aware of it.
• We want to (goals)
o introduce it (theorem)
o understand what it means to us as
developers (implications and criticism)
o learn (CAP in practice):
 of what others are doing
 what can be done

5.
Brewer‟s CAP Theorem
• Presented as a conjuncture at PODC 2000
(Brewer's conjecture)
• Formalized and proved in 2002 by Nancy
Lynch and Seth Gilbert (MIT)
• Consistency, Availability and Partition-
Tolerance cannot be achieved all at the
same time in a distributed system
• There is a tradeoff between these 3
properties

6.
CAP - Definition
In simple terms:
in an asynchronous network that performs as
expected, where messages may be lost
(partition-tolerance), it is impossible to
implement a service that provides consistent
data and responds eventually to every request
(availability) under every pattern of message
loss.

7.
CAP - Definitions
• Consistency
• Data is consistent and the same for all nodes.
• All the nodes in the system see the same state of the
data
vi
• Availability
• Partition-tolerance
vi vi vi

23.
Large Scale RDBMS
• We saw that is expensive to scale
• We know what it gives us :
• Guarantees Atomicity, Consistency, Isolation
and Durability
• E.g. Support transactions
• “You know what you will get”
• But how well it works in case of:
• largely distributed environment?
• Very large volumes of data?

24.
Reminder – Consistency Model
• We've talked about it before:
• Linearization, quasi-linearization
• Transactional memory
• Etc.
• When does change take effect
• What is the "contract" between the system
and the developer
• Will discuss more later...

25.
Consistency Model in RDBMS
• We are very used to RDBMS
• Clear contract
o Transactions
• Ever thought about the consistency
model?
• Consistency Model in RDBMS is
ACID

26.
• Atomicity
• of an operation(transaction) - "All or nothing“ – If part fails, the
entire transaction fails.
• Consistency*
• Database will remain in a valid state after the transaction.
• Means adhering to the database rules (key, uniqueness, etc.)
• Different from CAP‟s consistency definition.
• Isolation
• 2 Simultaneous transactions cannot interfere one with the other.
(Executed as if executed sequentially)
• Durability
• Once a transaction is commited, it remains so indefinitely, even
after power loss or crash. (no caching)
Definition – ACID

29.
Back to CAP
• Vendors therefore came up with their own storage
solutions, e.g.
• Google BigTable (over Google File System)
• Amazon DynamoDB
• Facebook – hybrid (Cassandra, Hadoop)
• Twitter – move from MySQL to Cassandra
• These solutions, as a group, is dubbed as "NoSQL"; more
on this later...
• Common approach is to relax the consistency
requirements for higher availability (or latency)
• Sacrifice ACID-compliance in order to achieve higher
performance – this is according to CAP.

30.
Relaxed Consistency?
Why do we need to give up consistency for
availability?
Can't we have both (at the same time)?
Let‟s look more deeply into CAP Theorem..

31.
CAP - Model
• Atomic Data Object
• There must exist a total order of operations s.t. each operation
looks as if it where completed at a single instant – equivalent
to as if they were executed on a single node.
• Available Data Object
• Every request receive by non-failing node must get a
response (any alg. used in service must eventually terminate)
• Partition Tolerance
• Both above should tolerate partitions
• Model partitions as messages delayed/lost in network

32.
CAP – Theorem 1
It is impossible in the asynchronous network model to
implement a read/write data object that guarantees the
following properties:
• Availability
• Atomic Consistency
in all fair executions (including those in which messages
are lost)
Asynchronous, i.e. there is no clock, nodes make decisions based
only on the messages received and local computation.

33.
CAP – Corollary 1.1
It is impossible in the asynchronous network model to
implement a read/write data object that guarantees the
following properties:
• Availability, in all fair executions
• Atomic Consistency, in fair executions in which no
messages are lost.
Intuition
• In asynchronous model the alg. doesn„t know if messages are lost
• Thus, there is no difference between this definition to theorem 1 and
if such alg. exist it contradicts theorem 1.

34.
CAP – Theorem 2
It is impossible in the partially asynchronous network
model to implement a read/write data object that
guarantees the following properties:
• Availability
• Atomic Consistency
in all fair executions (including those in which messages
are lost)
Partially synchronous, i.e. every node has a clock, and all clocks
increase at the same rate. However, they are not synchronized.

37.
Side note - context
CAP ~ Impossibility of guaranteeing both
safety and liveness in an unreliable
distributed system:
Consistency ~ safety - every response sent
to a client is correct
Availability ~ liveness - every request
eventually receives a response

38.
Side note - context
Actually - this is similar to (a case of)
consensus in an asynchronous system with
faulty processes ( impossible).
Consensus is more difficult to meet than the
requirements of CAP - achieving agreement
is (provably) harder than achieving CAP‟s
consistency requirement.
CAP also implies that it is impossible to
achieve consensus in a system subject to
partitions.

39.
Side note - context
Criticism: The reduction is not one-to-one
Requirement/definition in CAP for availability is
slightly different from the fail-stop assumption
in FLP - Failed nodes still participate.
Read more here:
http://the-paper-trail.org/blog/flp-and-cap-arent-the-same-thing/
(It is important and practical to know these "theoretical" problems - Take the distributed algorithms
course next year!)

40.
Eventual Consistency - BASE
Along with the CAP conjuncture, Brewer suggested a new
consistency model - BASE (Basically Available, Soft state, Eventual consistency)
• BASE model gives up on Consistency from the CAP
Theorem.
• This model is optimistic and accepts eventual
consistency, in contrast to ACID.
o Given enough time, all nodes will be consistent and
every request will result with same responses.
• Brewer points out that ACID and BASE are two
extremes and one can have a range of options in
choosing the balance between consistency and
availability. (consistency models).

41.
Eventual Consistency & BASE
• Basically Available - the system does guarantee
availability, in terms of the CAP theorem. It is always
available, but subsets of data may become unavailable
for short periods of time.
• Soft state - State of system may change over time, even
without input. Data does not have to be consistent.
• Eventual Consistency - System will become consistent
eventually in the future. ACID, on the contrary, enforces
consistency immediately after any operation.

42.
CAP Implications/Perspectives
CAP is very prominent in discussion over
the development of large, distributed
systems.
A new “eco-system” for “CAP aware”
solutions is available and made common
by the increase in massive web services.
Over the years since introduced there is
some criticism regarding the theorem.

43.
Cannot omit Partition-Tolerance
You can‟t really choose 2 out of 3:
For a distributed (i.e., multi-node) system to
not require partition-tolerance it would
have to run on a network which is
guaranteed to never drop messages (or
even deliver them late) and whose nodes
are guaranteed to never die. We do not
work with these types of systems - simply
because they don't exist.

44.
CAP - Revisited
CAP was devised and proved relatively early to the
prevalence of systems that are affected by it
Brewer - "CAP Twelve Years Later: How the “Rules” Have
Changed" (2012)(reference)
o Discusses misconceptions, suggests different models
of consistency [3]
Gilbert, Lynch - "Perspectives on the CAP Theorem"
(2012)(reference)
o Revisit proof concepts and discuss practical
implications [4]

46.
CAP in Practice
• CAP implications changes way of
thinking/developing distributed systems
• When designing/developing such a
system one should be aware of CAP‟s
considerations
• We will now explore practical examples
and techniques

47.
Give up Scale
Develop as usual with ACID
Restricts the growth options
This is more a business/design decision
But some really don't need it (Small
businesses that with relatively small data
or limited number of users)
Not so interesting for this discussion so we
will continue.

48.
NoSQL - Give up Consistency
• Coined in 1998 by Carlo Strozzi for his RDBMS that
does not use the standard SQL interface
• Usually, gives up consistency to achieve availability
o Does not support joins (are expensive)
o No constraints (PK-FK) (related to joins)
o Denormalization
• Re-coined by a Rackspace employee in 2009 to label
all data stores that do not provide ACID.
• Started by internet giants (Google, Amazon etc) and
later released as open source
• Many, many variants...
o see http://nosql-database.org/ for a list.

49.
NoSQL
• Not Only SQL (initially strictly NO but...)
• Sacrifices ACID-Compliance in order to achieve
higher performance (Use BASE)
o Maintains eventual consistency instead.
• Distributed and fault tolerant
• Scalable, redundancy on other servers (If one fails
we can recover)
• Usually scales horizontally and manages big
amounts of data.
• Used when performance and real-time-ness is
more important than consistency of data.

50.
NoSQL
• Does not use SQL - Data does not necessarily
follow a schema - it is partitioned among many
nodes. We cannot do join operations.
• Optimized for retrieval and appending. Usually
works in key-value record storage.
• Useful when working with a lot of data, that does
not require following the ACID model.
o Maybe not the best idea for your next banking
application.

54.
NoSQL - Document Oriented
• Document-oriented systems store structured
"documents" such as JSON or XML but have no joins
(joins must be handled within your application). It's very
easy to map data from object-oriented software to these
systems.
• Query the document with relatively familiar syntax (same
as the document syntax)
• Isolation at document level but not between documents
o Can have a transaction on a document
o Not so easy for many documents
• Examples:
◦CouchDb
◦MongoDb
RavenDb (.NET)

57.
NoSQL - Summary
• New classes of databases that are based on example
work from internet giants
• Usually relaxes consistency
o but not always - read your manual
• Not as trivial as using SQL
o Transaction support may be limited and restricted in
scope
o No joins - need to care for it yourself
o Design around questions and not "model"
 E.g. aggregated keys
• Implement various consistency models (coming up)
• And be careful with it !

58.
Consistency Models
• Give up (some) consistency in response to CAP
• Several models that may fit different usage
scenarios
• Combine between models in application
o Catalog with eventual consistency
o Checkout/register with strong consistency
• Next, look at some of the variants
• We look from the client side (programmer)
based on discussion in http://www.allthingsdistributed.com/2008/12/eventually_consistent.html

59.
Consistency Models - Examples
Causal consistency.
• cause (and effect, not casual)
• If process A has communicated to process B that it has
updated a data item, a subsequent access by process B
will return the updated value, and a write is guaranteed
to supersede the earlier write.
o Processes A and B are causally related
• Access by process C that has no causal relationship to
process A is subject to the normal eventual consistency
rules.
o A and C have not relation

60.
Read-your-writes consistency.
This is an important model where process A, after it has
updated a data item, always accesses the updated value
and will never see an older value. This is a special case
of the causal consistency model. (A is causal to itself)
Session consistency.
Practical version of the previous model
Within a session the system guarantees read-your-writes
consistency (use cache on server)
When session terminates data is stored.
Guarantees do not overlap the sessions
Consistency Models - Examples

61.
Monotonic read consistency.
If a process has seen a particular value for the object, any subsequent
accesses will never return any previous values.
Good when data relatively static, even use local cache
Monotonic write consistency.
In this case the system guarantees to serialize the writes by the same
process. Systems that do not guarantee this level of consistency are
notoriously hard to program.
Consistency Models

62.
Segmentation - Smart
Partitioning
• No single uniform requirement
o some aspects require strong consistency
o others high availability.
• Segmentation to component an approach to
circumventing CAP
o each provide different types of guarantees.
• Overall guarantees neither consistency nor availability, yet
ultimately each part of the service provides exactly what is needed.
• Can be partitioned along various dimensions.
o guarantees not always clear
o specific to the given application and the particular partitioning scheme.
o Thus, difficult but maybe necessary

66.
Partitioning - Examples
Hierarchical partitioning
• Large global services with local "extensions"
• Different location in hierarchy may use different
consistency
• Example
o Local servers (better connected) guarantee more
consistency and availability
o Global servers has more partition and thus relax one
of the requiements

68.
Partitioning - Examples
Functional partitioning
• System consists of sub-services
• Different sub-services provide balance
according to requirements
• The composition (whole system) is not always
available and consistent but each part is
assured to work well.

69.
Best-effort availability
• This means sacrificing availability for consistency
• Still optimize to give as much availability as
possible
• Makes more sense when the network is more
reliable

70.
Summary
• Introduced CAP
• What it is, what it is made of and to where it applies
• Explore its properties and Implications
• You can‟t have it all
• You can‟t really give up on partition tolerance
• Saw some ways systems are designed around
this concept in order to achieve their (business)
goals

71.
Summary
• The last point is important – we have to understand
the limitations we face and see how to achieve the
requirements of the system while taking these
limitations into account
• Must be another consideration in the design phase
• Technical
• But also, functional/business decision

72.
So, what to use?
“And remember also, most people are not building
facebook, they are building reservation systems, tracking
systems, HR systems, finance systems, order entry
systems, banking systems, etc - things where transactions
are sort of important (lose my status update - no big deal,
lose my $100 transfer and I'm sort of mad). There is room
for a lot of things out there.”
(from AskTom answer)