Using jq with nested objects

19 Nov 2014

jq is a tool that we use at Development Seed when processing data at the command line for quick bash scripts. It has really powerful features for transforming, extracting and filtering JSON. It’s good as a pipe into json2csv, or another bash or python script. Handling nested data is however not very well documented. The following assumes that you’ve played around with jq or are at least familiar with it, because we’ll get into some of its more obtuse syntax.

The manual for jq is here and I encourage you to try out the examples in the jq playground with some of your own data!

Record lists vs Nested objects

This is useful for iteration in the case of for (key in country_list) type of iteration, but might be less suited for different types of groupBy or aggregation functions. A record list keeps the key within the record itself:

Transforming data

As an example of a transformation, we want to remove the score and rank attributes from this JSON object and keep gni and gini. If the data were in the form of a record list, this would be a simple task in jq.

Conclusion

jq is a pretty powerful tool as part of a data toolchain, and can handle a ton of JSON operations, but it’s sometimes hard to figure out due to its complexity. Hopefully this small guide will help you with nested to flat record conversions in the future.