Decide the Output Shape First
Renames, type mismatches and structural mismatches all happen in the same pass. Decide the output shape first, and the tool follows:
| Output shape | Typical need | First choice |
|---|---|---|
| One record | Rename, unwrap nesting | JS destructuring / jq {...} |
| A list of records | Filter, map | jq map / JS filter + map |
| A flat table | Tree expansion, export | recursive flatten + CSV |
Type mismatches hide better than names: 0/1 standing in for booleans, money in minor units, timestamps in seconds versus milliseconds, numbers serialized as strings. Normalise to what downstream expects and write the conversion explicitly (price: Number(it.price)), never leave it to implicit coercion.
jq: A Processing Language That Reshapes
jq chains pipes: the left side is one result, the right side the next operation. What it adds over JSONPath is construction—building new objects, renaming keys, merging arrays.
Basics: Select Fields, Build Objects
# extract a single field
jq '.name' data.json
# build a new object (renamed keys + nested lookups)
jq '{id: .user_id, name: .profile.name, active: (.status == "active")}' data.json
Values inside {...} may be any expression, so renaming, re-nesting and deriving in one pass costs a single traversal.
Arrays: Map and Filter
jq '[.items[] | {sku: .sku, price: .price}]' data.json
jq '[.items[] | select(.price >= 100)]' data.json
The two forms produce different shapes, which is the usual confusion: .[] | select(...) emits a stream of independent results (good for further piping), while map(select(...)) still emits an array (good as final output). Continue treating it as an array → use the latter.
Advanced: Merge, Group, Recurse
# merge objects from two files
jq -s '.[0] * .[1]' a.json b.json
# group by status
jq 'group_by(.status) | map({status: .[0].status, count: length})' data.json
# recursively collect all prices and sum them
jq '[.. | .price? // empty] | add' data.json
// empty drops absent fields so null never enters the sum; .. is recursive descent, paid for with a full tree walk.
JSONPath: Query Only, No Reshaping
JSONPath projects and nothing more: right for "keep the hierarchy, pick some nodes", wrong for building new objects.
| Need | Expression |
|---|---|
| All array elements | $.items[*] |
| Conditional filter | $.items[?(@.price>100)] |
| A field at any depth | $..price |
| Array slice | $.items[0:3] |
Renaming or constructing requires jq or code. Validate expressions in the JSONPath finder before putting them in a script.
map / filter / reduce: Choose by Output Shape
The three main lines divide cleanly: jq for the command line, logs and one-off batches; JS for browsers and Node with zero extra dependencies; Python for analysis and scripts. All three must first answer "array or scalar"—array output means map/filter, scalar output means reduce.
// JS: array in, array out
const rows = data.items
.filter((it) => it.status === 'paid')
.map((it) => ({ id: it.user_id, amount: it.total }))
# Python: aggregate, scalar out
total = sum(item["total"] for item in data["items"] if item["status"] == "paid")
Note that JS map returns a new array without touching the source; but if the callback mutates it, upstream data is still polluted. Build new objects instead of assigning back through the reference. Two quieter side effects: undefined properties are dropped by JSON.stringify, so fields "vanish" from the output, and mixing numbers with strings makes sorting and summing behave unexpectedly. Write ?? fallback defaults in the transform and compare Object.keys once before emitting.
A Real Case: Paginated Responses Into One Table
An API paginates as { records: [...], page, pages } while downstream wants one table:
const flat = pages.flatMap((p) => p.records); // concatenate all pages
const rows = flat.map(({ id, user: { name }, total }) => ({
id, customer: name, amount: total, // rename + destructure
}));
flatMap does "take arrays + flatten one level" in a step; when fields align, map to new objects before concatenating so raw structures never leak downstream. Pagination keys differ by vendor (records/data/list)—confirm the key before concatenating instead of assuming.
Flattening: Tree Into a Table
function flatten(node, prefix = '', out = {}) {
for (const [k, v] of Object.entries(node)) {
const key = prefix ? `${prefix}.${k}` : k;
if (v && typeof v === 'object' && !Array.isArray(v)) flatten(v, key, out);
else out[key] = v;
}
return out;
}
Arrays stay whole as a single value—expanding them into rows makes the row count unbounded, so handle that separately before loading. A dotted key can collide with a literal dotted key already in the document; check target columns for duplicates before exporting. The output drops straight into the JSON to CSV converter.
Performance Notes
- Stream large files:
jq --stream, or process JSONL line by line; neverJSON.parsethe whole tree before transforming. - Avoid intermediate arrays: chained map/filter allocates several temporaries; use one loop or a generator at scale.
- Project first, restructure second: narrowing to the needed keys cuts every later traversal.
- Validate the shape once before writing results; if a transform runs repeatedly, freeze it into a script instead of rebuilding a command each time.
Reproducible Results
One payload, one jq line, three Python lines—the results must match, and mismatches usually come from types:
import json
data = {"items": [{"sku": "A-01", "price": 39.9}, {"sku": "A-02", "price": 120}]}
rows = [{"sku": it["sku"], "price": it["price"]} for it in data["items"] if it["price"] >= 100]
print(json.dumps(rows, ensure_ascii=False))
# → [{"sku": "A-02", "price": 120}]
The jq equivalent is [.items[] | select(.price >= 100) | {sku, price}]. Running both over the same file and diffing the output with JSON Diff is the cheapest regression test for a transform.