JSON::Stream is a finite state machine based JSON parser that generates events
for each state change. This allows us to stream both the JSON document into
memory and the parsed object graph out of memory to some other process. This
is much like an XML SAX parser that generates events during parsing.

There is no requirement for the document nor the object graph to be fully buffered in
memory. This is best suited for huge JSON documents that won't fit in memory.
For example, streaming and processing large map/reduce views from Apache CouchDB.

Dependencies

JSON::Stream is a pure Ruby library that depends on no external libraries. It will
run on Ruby 1.9.2 or better, including JRuby.

Install

$ gem install json-stream

Usage

The simplest way to parse is to read the full JSON document into memory
and then parse it into a full object graph. This is fine for small documents
because we have room for both the document and parsed object in memory.

While it's possible to do this with JSON::Stream, we really want to use the json
gem for documents like this. JSON.parse() is much faster than this parser
because it can rely on having the entire document in memory to analyze.

For larger documents we can use an IO object to stream it into the parser.
We still need room for the parsed object, but the document itself is never
fully read into memory.

Notice how the parser accepts chunks of the JSON document and parses up
to the end of the available buffer. Passing in more data resumes the
parse from the prior state. When an interesting state change happens, the
parser notifies all registered callback procs of the event.

The event callback is where we can do interesting data filtering and passing
to other processes. The above example simply prints state changes, but
imagine the callbacks looking for an array named "rows" and processing sets
of these row objects in small batches. We can process millions of rows streaming
over the network in constant memory space this way.

Authors

David Graham (david.malcom.graham@gmail.com)

License

JSON::Stream is open source software released under the terms of the
MIT license.
If you have improvements to the code you'd like to share, send us a
GitHub pull request
with the changes and unit tests.

Download

You can download this project in either
zip or
tar formats.
You can also clone the project with Git by running: