One theory proposed by Martin Fowler is that NoSQL arose as a way to get around the constraint of the DBA bottleneck. When developers want to change the schema or get copies of the source database the DBAs are just seen as an obstacle. The obstacle of provisioning environment is told in nail biting prose in Gene Kim’s “The Phoenix Project”

Long before NoSQL and The Phoenix project, about 13 years ago, I was the DBA at a startup, back in the first dot com bubble. As a startup we were trying to get our project to market ASAP. The development team was working crazy hours and the schema was changing all the time. As the DBA, it was my responsibility to validate all the schema changes and apply the schema changes to the development database. Such validation and application could take a couple hours considering all the other work that was on my plate. This couple hour delay was considered such a hinderance to developers that they unilaterly decided to dump the structured schema and go to an EAV data store. EAV stands for “Entity Attribute Values” model. From wikipedia

There are certain cases where an EAV schematic is an optimal approach to data modelling for a problem domain. However, in many cases where data can be modelled in statically relational terms an EAV based approach is an anti-pattern which can lead to longer development times, poor use of database resources and more complex queries when compared to a relationally-modelled data schema.

The EAV “schema” (more like schema-less) meant that perfromance was dreadful and the SQL was impossible to debug as it was completely obfuscated. An example of an EAV query might look ike

Eventually as development stabilized and performance was abominable we were able to get everyone on board and move the EAV back into a structured schema. Once we went to a structured relational schema the load on the database fell from completely saturated to almost idle.

Moral of the story

The ironic thing is that many of the NoSQL setups are basically EAVs and/or databases that allow users to make “schema” type changes with out the intervention of a DBA. For example in Mongo or Couchdb, since they are JSON based, you can just add new fields to the JSON and the database deals with it. If you are using a database like Riak, it’s just a key value store, which is basically an EAV.

EAV can be good for quick development but for production it’s completely unreadable thus impossible to debug and the performance is atrocious when it comes to relational type database operations of selecting all the fields in a row and joining rows from different “tables” together.

Instead of using EAV one could give each developer a virtual relational database, like using Delphix with Oracle, and let them work as fast as they can and then let them test out merging there changes in virtual databases before merging them in to trunk which is itself a virtual database that can be branched from and merged into.

Virtual databases take up almost no storage and can be made in minutes with a developer self service interface.

I’ve run into the wait “kksfbc child completion” a few times over the past but found very little written about it. I don’t have an explanation, but I have something that might be as good – a way to reproduce it. By being able to reproduce I at least test theories about it.
I ran a tight loop of

insert into t value(1);

See anything strange? I put “value” instead of “values” so the insert errored out. Thus causing parsing every execute. So this wait seems to be related to parsing. Could run further tests by correcting and adding a different litteral value to the insert to see what kind of waits might result.

Another tell tale sign that there was an error was that I also recieved “SQL*Net break/reset to client” which is a sign of an error in the sql statements that the application is attempting to execute.