unclouding the cloud

In my series of DocumentDB I will discuss more advanced features of DocumentDB. This post covers concurrency options for DocumentDB but starts with an overview of the problems with concurrency in general.

Concurrency options in DocumentDB

What is concurrency?

Consider the scenario that you have a hotel room booking application. You have a web site that allows people to book rooms. The key scenario here is that there may be several users looking at the same available (at the time of loading) hotel room for the same night. You typically want to avoid that more than one person books the same room. So when the two users push the “Book room” button on the web page you can only allow one of them (typically the first one to win and get the reservation confirmed). So for a system allowing more than one source/actor/system/user to edit the same data you need a concurrency strategy.

General Concurrency Methods

Three of the most common strategies to handle these are:

Optimistic Concurrency: This strategy knows there could be other actors updating the same data as you are working with. This approach does however assume that this is likely not the normal case – but rather has the optimistic approach that it will hopefully succeed. So you would typically update the data in your process and when you try to commit your changes (like a save to the database) you will get a green light or not. To determine whether you get the green light or not you would have an (implicit or explicit) Version Number,TimeStamp or similar construction and only allow update if the version/timestamp is the same as when you read the data. If not you will get a concurrency exception and your save will fail. It is then up to you to load the latest data and merge your changes with the refreshed data or just to cancel your changes and start over. This approach suits read-intensive applications best where the chance of concurrency exceptions is a low percentage and the drawback of having to say to the actor that his save failed is acceptable.

Pessimistic Concurrency: Just as the words suggests this is the opposite of optimistic concurrency. This approach is targeting to limit the number of times we press the save button and get the result that someone else has updated your data. It is done by setting some kind of lock on the data so that only you can update the data. Once the lock is in place you can be assured that nobody else “steals” your record/data/document. The more likely a concurrency situation is to occur the more efficient this strategy is. This can be combined with setting the lock for a certain period to avoid that something gets locked forever. Consider when you visit the cinema website and you first reserve some seats before you pay. The actual booking is not complete until you pay but you don’t want people to go through a long process just to discover that the seats you selected was booked by someone else.

Last-Write Wins: This is a strategy that you should only use for certain scenarios where it is OK that the last write is ok. Maybe you get fresh stats from a sensor and you are only interested in the last reading. In this case you could just overwrite the old data no matter what. It is not suitable for the hotel room or movie seat reservation scenarios described above. The benefit of this is that it allows a higher throughput and no errors and will take fewer resources than the previous strategies. This is the default concurrency update in DocumentDB. This may sound strange but if you think about it this is the default in SQL databases as well (though on a more granular level – two update statements after each other will have the second overwrite the first one).

DocumentDB Optimistic Concurrency

The examples to update documents presented in the last part of the post series used a last-write-wins concurrency philosophy. This means I cannot use it for the two more complex scenarios above.

DocumentDB is designed to read/write with extremely high throughput so be aware that adding support for optimistic or pessimistic concurrency will affect throughput. DocumentDB has built-in support for Optimistic Concurrency but we will have to adjust the DocumentDB repository that we created last time.

Understand the ETag

Each DocumentDB document has an ETag property that is updated each time the document is updated and you can enforce that the ETag must be identical to what you opened before when you replace the your document.

The problem with the ETag is that it is attached to the document not the stored JSON object. So all methods we have for fetching a typed Order and write typed Linq statements against the order is effectively made unusable. You shouldn’t (in my opinion) manually add the ETag to the orders object as JSONProperty either. In theory you could add a property in your objects like

But that would serialize the old (before update) Etag into your document when you Replace the document so I would recommend using the other approach that I will explain further down.

This means that we need to work with other methods if we plan to update the Documents with Optimistic Concurrency. Note that if you only want to read data the GetItem with LINQ/Lambda syntax is still the preferred method. But for this scenario we cannot work directly with the Order object but we rather need to fetch the Document instance instead or work directly with dynamic (untyped) objects. The two following methods should do the trick.

Not that these the second method just differs that it allows you to define the return type in the method instead of creating a new DocumentDBRepository if you have already declared it as DocumentDBReportitory<Order>.

Now that we can return the data we can work with the object in some different ways.

The first alternative

fetches a DocumentDB document (not an Order) through the SQL like syntax. Note the <Document> declaration that forces the result to be a DocumentDB object.

To simplify working with the Order in the Document we cast it through dynamic to our Order object with var theOrder = (Order)(dynamic)orderDoc;.

We change the FirstName

We replace the document (no optimistic concurrency yet). Notice the Etag changes before and after the save.

Alternative. This is very similar to the code above but we get the dynamic return value instead it and then cast it into one Order object Order order = (Order)orderquery5 and one DocumentDB document Document doc = (Document)orderquery5;.

In both the methods above we get access to the ETag of the document (see the debug.log to track the ETag) so we can start implementing the optimistic concurrency.

Configure DocumentDB for optimistic concurrency using the ETag

So now we have prepared for usage of Optimistic Concurrency. Now it is just to make sure that we use it. The way DocumentDB implements Optimistic Concurrency is by using AccessConditions. These are added as RequestOptions in the ReplaceDocument API call.

I added the following two overloads in the DocumentDBRepository. One accepts dynamic and one that accepts a Document (I recommend using the Document overload)

The important part is the AccessCondition where I add the loaded ETag as a condition and set the AccessCondition to IfMatch (enforcing that the ETag must be like the one supplied.

The call to the ReplaceDocumentAsync is similar to before except that it adds a RequestOptions parameter using the AccessCondition just declared.

What will happen is that the ReplaceDocumentAsync will generate and Exception.To know that is is a ConcurrencyException we need to catch a DocumentClientException exception and thereafter check the StatusCode which should be PreconditionFailed (rather cryptic name but this is what you need to check for). I post the whole test scenario below.

Load two instances of the same Document

Update the FirstName on both (it does not matter if I set it to the same value or not)

Replace the Document with the first instance

Try to update the Document with the second instance (which will fail)

The errorhandler will check the StatusCode to determine whether it is a concurrency exception or not.

Note that what you do to resolve this is up to you. In some cases you may say to the caller that the update failed but in others you may want to merge the data.

DocumentDB Pessimistic concurrency

There is no built-in support for handling pessimistic concurrency in DocumentDB so you would have to implement it yourself. I would not recommend this because you have to handle deadlocks, lock expiry and advanced database topics manually. I haven’t read up on permissions on documents yet and maybe that could be a start to limit access to a document temporary (have to return on that issue). Doing it the hard way and implementing your own pessimistic locking mechanism is to a series of posts itself and frankly it will most likely be a bad idea as one of DocumentDBs major benefits isthe fast throughput/access times which you may very likely kill if you add pessimistic concurrency to this component.

Partial Updates

Partial updates are currently not supported in DocumentDB. This means that you will save the entire document upon time of writing. So a scenario of having two order lines being modified by separate workers is not possible and you should use optimistic concurrency and reload the object if it has been updated elsewhere.

I just re-ran my sample and still get only the DocumentClientException. My sample used “Microsoft.Azure.DocumentDB” version=”1.5.2″ with targetFramework=”net452″. Which version do you run so I can try to emulate your conditions? And also – which InnerExceptions do you get in the aggregate one?

Could you please clarify why is it problematic to have ETag on your strongly typed object? If you consider that you want to send document to client and get it back updated by client (with old etag) then you have to send it some way together with your document, dont you? So if we consider the case:

1) Read from documentdb to strongly typed object with etag property
2) Send it to client
3) Client updates it and sends back
4) Now you have new version of document with old etag on it (lets say it was changed in db by someone else in the meantime)
5) You send that document for replace to documentdb with AccessCondition set to etag as gotten from client
6) ETag is get only, set exclusively by server so cannot be overwritten in db by old value (thats the whole point ain’t it?)

So, i have a hard time understanding – why bother with dynamic/Document in the first place?

I actually thought and tried the same way in the beginning (and frankly – it would have been much nicer code than using dynamic or returning the Document itself as in the examples in the post) and I was pretty upset that it did not work the way I wanted. The problem is that if you unpack to a strongly typed object it is only using the JSON content/body of the document. The Etag is kind of a system metadata field on the document, and does not get saved in the JSON content/body itself. This means that if you add a ETag field in your typed object – the internal system Etag will not be unpacked into your own typed object Etag property. Likewise when you save – the system will not interpret your typed objects ETag property as the system one but will rather save a JSON property named Etag into the JSON content/body itself.

This means that to work with the system Etag you have to return the Document (which uses the system Etag value) to enable proper use of the Etag.
This response is however based on my findings from back then and I haven’t revisited this problem since so I have to leave a small opening for if something has been updated to enable this since this post. After all CosmosDB has undergone a lot of nice updates since the version of DocumentDB that I based this post on. Maybe time to reverify this and if you do it before I do…please post your finding here.

Now that I think about it, I recall that it was easy to get the system ETag info your own typed object but that was more thanks to the JSON deserializer. So I was able to get the system ETag into my typed object, but as I mentioned in my last comment. When I save that property was just treated like any JSON property (saved in the content/body) and could therefor not be used for the system to compare the Etags as a part of optimistic concurrrency. So I ended up saving an old ETag in my document but failed to setup optimistic concurrency.

But if you look closely what you are trying to implement here is actually your own Document class. I.e. a class that contains meta data such as etag as well as the json object data itself. So either use the Document class instead or try to subclass it (inherit). When you post your object and need optimistic concurrency I believe it must it must be a Document or a class that atleast have inherited from it for the Etag to work.

Yes, you are right and thats one thing i still just can’t make my mind around yet.

What i want to do (at almost all cost) is to keep DocumentDB related implementation within boundaries of my repository and dont want Document (or any other documentdb related) class go through my upper layers: service, business, presentation.

Thats why i think that generic one might work.
Of course, its a kind of DAL DTO and DTOs often look like pure code duplication although they serve additional purposes.

it absolutely makes sense and believe me I was trying to do exactly the same thing and I was very frustrated that I could not get it to work in a disconnected multi-tier solution. If you throw away the document you can never re-establish the Document again so you can’t use a custom etag solution. One alternative may be to continue with your latest example but before you save you 1) reopen the document in the business logic or data layer 2) Verify that the etag is still the same or throw your own concurrency error 3) If same etag – serialize your own updated strongly type object to the reopened document content/body 4) try to save the re-opened and updated document with the re-opened tag with the AccessCondition (as in my code example). This should be full proof (bold statement) but will hit performance because it will require an additional fetch and manual etag comparison.

I am glad we are on the same line that its a problem if it leaves repository.

With the disclaimer that i havent started yet with my project so i am speaking just based on my reading docs and theoreticaly.

I have a feeling that that generic VersionedStronglyTyped version might work because in that case i actually dont need Document anymore. I just read it first time, get entity (strongly typed) and etag. On replace, i feed in strongly typed entity and before that i put etag in AccessCondition.