Contents

Those days, I’ve been observing the emergence of radically innovative approaches to data persistence. This article starts with a quick summary of the – now a classic – NoSQL domain and then introduces the event based persistence approach that is emerging.

NoSQL

It’s now been several years that the NoSQL databases are being talked about and are maturing with the experience feedback, sometimes with some difficulties. The concept of the “polyglot persistence” is spreading out. This term, first coined by Martin Fowler, means that several types of databases are used in the same software, each one fitting particular needs.

I also use that approach myself with ZenModeler (neo4j graph database, MongoDB document database, classical relational DB and I look with a strong interest towards event based databases). Each database type fulfills a precise need. The encapsulation of the data access behind a Repository object is essential according to me because it strongly helps to separate concerns between business logic and persistence, and then to encapsulate the persistence logic behind a clean business interface.

Key-Value databases : needs more efffort to query, but very easy to scale. Typical databases for parallel processing like “Map-Reduce”. Lots of people here, like Cassandra, Riak, Dynamo and the french guys at bureau14 with wrpme.

Document oriented databases : store document and access them with a key, the hype being to store document as JSON like with MongoDB (if you plan to use MongoDB, you have to check the »MongoDB gotchas »). The DDD Aggregate pattern is very useful with that kind of storage.

Object databases or distributed caches : I’m feeling nostalgic about that one as I work with the ancestor Versant in 1999 for one of my internship :-). Nowadays, most of the object oriented databases switched to distributed caches in front of relational databases.

That‘s it for the technical aspects. There are big differences within the data structures used in each of them ranging from the narrower to the wider : maps, trees (as JSON or XML) or graphs. Keeping in mind that the more the better, a graph database can represent a tree and each node or relation can also store a map. For the substance I don’t see any differences because these tools don’t integrate any paradigm shift toward persistence approach…

Data Persistence Fundamentals

Let’s get back to the fundamental of what we store : data that describe reality !

A data is the relationship formed by a concept and its measure. A measure is characterized by a type : either quantitative (weight, amount, age) or qualitative (name, city, date). The composition of several data in a structure leads to the description of a fact.

For example, a user is browsing on an e-commerce website and is interacting with it : she adds some products to her cart and removes some, when she has finalized her purchases, she validates her order and then pays. Each of the user’s action is processed by the system. Overall, the later guarantees the ACID properties for her actions : atomicity, consistency, integrity and durability. The last point, durability, is essential in case a link of that extremely complex chain would fail. That durability is guaranteed by redundancy mechanisms and by writing on supports that “persist” in the event of a lack of electricity (sorry I need to get back to some basics before moving on :-)).

There is now two ways to analyze the user’s actions :

From the point of view in the cart: it changes along with the user’s actions (add a product, remove one, modify its quantity, recalculate the balance, etc.). The cart is always in a coherent state characterized by its identifier, the product list with its total quantity and the total balance. It is more or less the vision of “objects and states” of the OO paradigm.

From the user’s point of view: user is doing a sequence of actions (add, remove, change quantity, etc.). Of course, each action will be received by the cart which will process the necessary state transitions. The vision is more functionally oriented, where events trigger functions that make the state transitions. The whole thing being deterministic and, by definition, the event immutable…so it’s more or less the vision of « function » of the FP paradigm.

Event Oriented Persistence

An event describes a change that occurred in the past. Each event triggers an action that modifies the state of the object which receives it through a state transition. The receiving object becomes known in a specific state, following the transitions that occurred in the triggered functions associated with each event. Arriving in a specific state can also triggers another internal event.

For example : the addition of 3 products of $100 each results to a cart balance of $300.

If these events are replayed from the starting state in the same order, then the final state is identical in a deterministic manner. Moreover, if every event can be associated to its inverse that compensate the original one, the starting state can then be reversed forward from the final state.

Events are an implicit way of measuring the time that is passing by. The order of the events are a relative measure of time. The instant when the event occurred is the absolute one measured by our clocks, that’s to say an absolute measure of time.

The mutable objects, those performing state transitions, are the ones we are interested in their life cycles. The DDD approach names that kind of objects “Entities” rather than “Value Objects” (see that post about identity and state).

The following schema describes the relationship between the identity of the entity, its states and the events it received.

The events are the « atoms » that can be stored in order to retrieve our object states at any point in time instead of the traditional way of persisting only the « head » or the last known state of our object.

Conclusion

In conclusion, a fundamental paradigm shift is at work with the new kids on the block databases like Datomic as they force us to think about persistence in a completely new way: to wrap up, thinking about storing the state transition represented by the event and not only about storing the last known data. The approaches related to the « event » way of storing data are named Event sourcing, CQRS, etc.

My next article on that persistence topic will be about the blurred differences between fast memory and mass memory with the introduction of the memristor and the consequences in our software design.

The ‘design’ word is one of the strangest beast that lives in the IT world. I hear that word very often as people talk a lot about it. Yet it has a very broad meaning and hence a misunderstood signifier among people, with inaccuracy and positive and negative myths about it… In this article I try to answer to the most common questions about Software Design. You can also replace the word "Design" with the word "Architecture", both are often poorly defined and understood within IT. I personaly prefer using "Design" as a verb and "Architecture" as a deliverable.

What Design refers to?

‘Design’ defines both the activity with the verb ‘to design’ and the result of that activity with the noun ‘a design’.

Can a system have no design?

I had sometimes heard: “this system has no design”. That sentence refers to the lack of a proper design documentation about the system and also sometimes to the lack of attention from the developers to get some insight and structure the system wisely. But a system always has a particular design, sometimes poorly documented and not carefully crafted, but it has one because a source code exists and it runs.

What is a design made of?

A design is about all the decisions made to build the system, both the very large ones that impact the entire system and the small ones with an impact scoped to a small area (a function, a class, a sub-component of the system).

How to best make the design decisions?

First question: do I really need to make this decision now?

If not, how can I revert this decision later? that question is about flexibility and stuffs that are hard to change after they are done.

Or else, what can I do to defer this decision?

Also, always remember to question the validity of the user requirement with a simple « Why? ». One of the most famous experience feedback came from the F16 aircraft design, I advise you to look at that great experience feedback from the father of the F16, Harry Hillaker. In short, US Air Force (the client) ask for a plane capable of a speed over Mach2, but Harry Hillaker was the one who asked « Why? » and the response was « to be able to escape from combat” which was the real requirement. It allows Hillaker and his team to enlarge their perimeter of thinking for the solution and indeed provide a lot of innovations.

If the decision cannot be avoided, check the following three items:

Which user’s requirement this decision helps to fulfill?

What are the advantages/inconveniences of that decision?

Are the responsibilities still clearly separated among components after that decision?

How to properly document a software design?

You’ll have to talk about the “why” of the large decisions that impact the way the system will be structured or will behave. Because if we understand the “Why?”, the “How?” will be easier to grasp. The source code represents the ultimate design documentation of the softwares, but with a huge amount of details. Then we have to abstract that detailed representation, that is to say select only the meaningful information from that ocean of details. But we only get the ‘What?’ (the structure) and sometimes the ‘How?’ (the behavior), which is not very useful if not properly chosen.

What is a good design process?

A good design needs feedback from the user of the system, so the system has to be in front of the user (getting real) as soon as possible, with frequent releases of new versions once the users’ feedback has been received. Hence, a good design process is an iterative and incremental one. There is an acronym for the lack of an iterating process : BUFD for Big Up Front Design. You can also look at that great article from Jeff Atwood about the iterating process in a cat product.

How can a design emerge?

This is a concept coming from the agile world. In short, a design can emerge if we get feedback continuously. Hence TDD and BDD practices help the design to emerge, making all the decisions we need with good insights, because we have a direct and real usage of the system coming from the scenarios. I have had some project experiences where writing the scenarios right before coding – and only with the domain (no persistence, services, web layer, etc.) – saved me hours of refactoring after discovering mistakes I had made in my implementation. Neal Ford has good slides about the Emergent Design concept.

What is the goal of the design activity?

Deliver the best solution to the users given the 4 fundamental constraints of a building project: scope, resources (money is a good shortcut for resources…), time and the most important one, quality.

What are the 4 basics designer’s talents?

Understand user needs

Find an adequate solution

Abstract details

Separate responsabilities / concerns

Each one of these « talents » needs a dedicated article because I have a lot to say about those talents.

In the end, what makes a good design?

I would say a design is good when there is a good balance between the different constraints at work, particulary those that are immediate to the end user. Fundamentally, a good user experience is what makes a design compelling. But the others internal quality attributes have their importance, like those defined by the Software Quality ISO norm.

Feedback is an essential notion in usability. This is the notion which struck me the most as I read the book « Design of Everyday Things » by Donald Norman and of which I am now the most attentive to in my every-day life. This article explores the way we can apply the feedback concept in other field than usability.

But first *how can we define what Feedback is? *

Feedback is data or results which are given in return for a completed action.
me

Every action should have a reaction
Arnie Lund

Feedback is also a central concept in system theory, especially cybernetic system theory, every action performed by a system on another one needs a feedback to measure its effect and so adjust its action to the reaction it perceives. You can replace « system » in the previous sentence with « human ». Also the feedback can be asynchronous with a response delay that can vary in time.

Examples Of Feedbacks In The Physical World

Donald Norman, in his book, uses as an example telephone keys which produce a sound when they are correctly pressed (a different sound to every key too). This allows the user to know that his action has been taken into account.

On my part, I remember for example having rented a car of a recent model a while ago and during the ignition not hearing the engine (‘feedback’) and spending 30 seconds just looking for a feedback to my action of ignition, actually the dashboard lights were on but unfortunately they were flooded in sunlight. Until I finally put it into gear and the car started going in reverse – the engine was on and the car had been operational the entire time!

Another example is the knobs which control the volume on my old Dell had a horrible feedback: a bar would appear on the screen but gave no indication of what the real level of volume was. Now on my Mac, as well as the appearance of a visual bar, the volume buttons emit a beep of the volume which has been modified by my action – a feedback I personally find much more useful!

In short, every action must have a feedback worthy of the name, especially in software!

Agile Practices Are Built With Feedback Loops

I find that the Feedback notion is a fundamental one and can be applied to our organizations. It is also a good way to explain the Agile practices as they are part of a feedback loop with just a different time laps which depends on the action (I stick to the number 3 to emphasize the unit change for each practice, a kind of Rule of Three ! :-) ). Hence:

Pair programming: feedback within 3 seconds

Unit tests: Feedback within 3 minutes

Refactoring and continuous integration: feedback within 3 hours

Standup meeting and planning: feedback within one to 3 days

Iteration: feedback within 3 weeks

The Lean Startup movement and the Minimum Viable Product (MVP) notion also try to integrate a feedback loop with startup’s clients during the product or service building process.

This feedback can sometimes be quantitative and very rational like with practice of A/B testing which consists of providing two different version of the same website/application and collect metrics about the different behaviors from users according to each version.

The faster the return, the better, which is why real-time BI systems are popular. Because, like in real life, feedback makes it possible to adapt the action in a continuous manner according to the returns.

In short, every activity can be questioned according to the feedback it returns. And how about you, what are your feedback loops for your activities?