[ https://issues.apache.org/jira/browse/COUCHDB-1092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13008210#comment-13008210
]
Robert Newson commented on COUCHDB-1092:
----------------------------------------
At the risk of adding more confusion, my suggestion for {_id:foo}{"foo":"bar"} was not an
API change, merely the on-disk format of {_id:foo,"foo":"bar"} but avoiding any fiddly encoding
issues.
I'm basically agreeing with Paul, though he's followed up with working code (I plead hospitalization
in my defence).
I love the performance enhancements, but share Paul's concern on "correctness first". I can
comment more cogently tomorrow when I'm a bit more settled at home and have re-read the code
some more.
> Storing documents bodies as raw JSON binaries instead of serialized JSON terms
> ------------------------------------------------------------------------------
>
> Key: COUCHDB-1092
> URL: https://issues.apache.org/jira/browse/COUCHDB-1092
> Project: CouchDB
> Issue Type: Improvement
> Components: Database Core
> Reporter: Filipe Manana
> Assignee: Filipe Manana
>
> Currently we store documents as Erlang serialized (via the term_to_binary/1 BIF) EJSON.
> The proposed patch changes the database file format so that instead of storing serialized
> EJSON document bodies, it stores raw JSON binaries.
> The github branch is at: https://github.com/fdmanana/couchdb/tree/raw_json_docs
> Advantages:
> * what we write to disk is much smaller - a raw JSON binary can easily get up to 50%
smaller
> (at least according to the tests I did)
> * when serving documents to a client we no longer need to JSON encode the document body
> read from the disk - this applies to individual document requests, view queries with
> ?include_docs=true, pull and push replications, and possibly other use cases.
> We just grab its body and prepend the _id, _rev and all the necessary metadata fields
> (this is via simple Erlang binary operations)
> * we avoid the EJSON term copying between request handlers and the db updater processes,
> between the work queues and the view updater process, between replicator processes,
etc
> * before sending a document to the JavaScript view server, we no longer need to convert
it
> from EJSON to JSON
> The changes done to the document write workflow are minimalist - after JSON decoding
the
> document's JSON into EJSON and removing the metadata top level fields (_id, _rev, etc),
it
> JSON encodes the resulting EJSON body into a binary - this consumes CPU of course but
it
> brings 2 advantages:
> 1) we avoid the EJSON copy between the request process and the database updater process
-
> for any realistic document size (4kb or more) this can be very expensive, specially
> when there are many nested structures (lists inside objects inside lists, etc)
> 2) before writing anything to the file, we do a term_to_binary([Len, Md5, TheThingToWrite])
> and then write the result to the file. A term_to_binary call with a binary as the
input
> is very fast compared to a term_to_binary call with EJSON as input (or some other
nested
> structure)
> I think both compensate the JSON encoding after the separation of meta data fields and
non-meta data fields.
> The following relaximation graph, for documents with sizes of 4Kb, shows a significant
> performance increase both for writes and reads - especially reads.
> http://graphs.mikeal.couchone.com/#/graph/698bf36b6c64dbd19aa2bef63400b94f
> I've also made a few tests to see how much the improvement is when querying a view, for
the
> first time, without ?stale=ok. The size difference of the databases (after compaction)
is
> also very significant - this change can reduce the size at least 50% in common cases.
> The test databases were created in an instance built from that experimental branch.
> Then they were replicated into a CouchDB instance built from the current trunk.
> At the end both databases were compacted (to fairly compare their final sizes).
> The databases contain the following view:
> {
> "_id": "_design/test",
> "language": "javascript",
> "views": {
> "simple": {
> "map": "function(doc) { emit(doc.float1, doc.strings[1]); }"
> }
> }
> }
> ## Database with 500 000 docs of 2.5Kb each
> Document template is at: https://github.com/fdmanana/couchdb/blob/raw_json_docs/doc_2_5k.json
> Sizes (branch vs trunk):
> $ du -m couchdb/tmp/lib/disk_json_test.couch
> 1996 couchdb/tmp/lib/disk_json_test.couch
> $ du -m couchdb-trunk/tmp/lib/disk_ejson_test.couch
> 2693 couchdb-trunk/tmp/lib/disk_ejson_test.couch
> Time, from a user's perpective, to build the view index from scratch:
> $ time curl http://localhost:5984/disk_json_test/_design/test/_view/simple?limit=1
> {"total_rows":500000,"offset":0,"rows":[
> {"id":"0000076a-c1ae-4999-b508-c03f4d0620c5","key":null,"value":"wfxuF3N8XEK6"}
> ]}
> real 6m6.740s
> user 0m0.016s
> sys 0m0.008s
> $ time curl http://localhost:5985/disk_ejson_test/_design/test/_view/simple?limit=1
> {"total_rows":500000,"offset":0,"rows":[
> {"id":"0000076a-c1ae-4999-b508-c03f4d0620c5","key":null,"value":"wfxuF3N8XEK6"}
> ]}
> real 15m41.439s
> user 0m0.012s
> sys 0m0.012s
> ## Database with 100 000 docs of 11Kb each
> Document template is at: https://github.com/fdmanana/couchdb/blob/raw_json_docs/doc_11k.json
> Sizes (branch vs trunk):
> $ du -m couchdb/tmp/lib/disk_json_test_11kb.couch
> 1185 couchdb/tmp/lib/disk_json_test_11kb.couch
> $ du -m couchdb-trunk/tmp/lib/disk_ejson_test_11kb.couch
> 2202 couchdb-trunk/tmp/lib/disk_ejson_test_11kb.couch
> Time, from a user's perpective, to build the view index from scratch:
> $ time curl http://localhost:5984/disk_json_test_11kb/_design/test/_view/simple?limit=1
> {"total_rows":100000,"offset":0,"rows":[
> {"id":"00001511-831c-41ff-9753-02861bff73b3","key":null,"value":"2fQUbzRUax4A"}
> ]}
> real 4m19.306s
> user 0m0.008s
> sys 0m0.004s
> $ time curl http://localhost:5985/disk_ejson_test_11kb/_design/test/_view/simple?limit=1
> {"total_rows":100000,"offset":0,"rows":[
> {"id":"00001511-831c-41ff-9753-02861bff73b3","key":null,"value":"2fQUbzRUax4A"}
> ]}
> real 18m46.051s
> user 0m0.008s
> sys 0m0.016s
> All in all, I haven't seen yet any disadvantage with this approach. Also, the code changes
> don't bring additional complexity. I say the performance and disk space gains it gives
are
> very positive.
> This branch still needs to be polished in a few places. But I think it isn't far from
getting mature.
> Other experiments that can be done are to store view values as raw JSON binaries as well
(instead of EJSON)
> and optional compression of the stored JSON binaries (since it's pure text, the compression
ratio is very high).
> However, I would prefer to do these other 2 suggestions in separate branches/patches
- I haven't actually tested
> any of them yet, so maybe they not bring significant gains.
> Thoughts? :)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira