JSON Data Transformation Techniques: jq, JSONPath, Mapping, Filtering & Reducing in Practice

Decide the Output Shape First

Renames, type mismatches and structural mismatches all happen in the same pass. Decide the output shape first, and the tool follows:

Output shape Typical need First choice
One record Rename, unwrap nesting JS destructuring / jq {...}
A list of records Filter, map jq map / JS filter + map
A flat table Tree expansion, export recursive flatten + CSV

Type mismatches hide better than names: 0/1 standing in for booleans, money in minor units, timestamps in seconds versus milliseconds, numbers serialized as strings. Normalise to what downstream expects and write the conversion explicitly (price: Number(it.price)), never leave it to implicit coercion.

jq: A Processing Language That Reshapes

jq chains pipes: the left side is one result, the right side the next operation. What it adds over JSONPath is construction—building new objects, renaming keys, merging arrays.

Basics: Select Fields, Build Objects

# extract a single field
jq '.name' data.json

# build a new object (renamed keys + nested lookups)
jq '{id: .user_id, name: .profile.name, active: (.status == "active")}' data.json

Values inside {...} may be any expression, so renaming, re-nesting and deriving in one pass costs a single traversal.

Arrays: Map and Filter

jq '[.items[] | {sku: .sku, price: .price}]' data.json
jq '[.items[] | select(.price >= 100)]' data.json

The two forms produce different shapes, which is the usual confusion: .[] | select(...) emits a stream of independent results (good for further piping), while map(select(...)) still emits an array (good as final output). Continue treating it as an array → use the latter.

Advanced: Merge, Group, Recurse

# merge objects from two files
jq -s '.[0] * .[1]' a.json b.json

# group by status
jq 'group_by(.status) | map({status: .[0].status, count: length})' data.json

# recursively collect all prices and sum them
jq '[.. | .price? // empty] | add' data.json

// empty drops absent fields so null never enters the sum; .. is recursive descent, paid for with a full tree walk.

JSONPath: Query Only, No Reshaping

JSONPath projects and nothing more: right for "keep the hierarchy, pick some nodes", wrong for building new objects.

Need Expression
All array elements $.items[*]
Conditional filter $.items[?(@.price>100)]
A field at any depth $..price
Array slice $.items[0:3]

Renaming or constructing requires jq or code. Validate expressions in the JSONPath finder before putting them in a script.

map / filter / reduce: Choose by Output Shape

The three main lines divide cleanly: jq for the command line, logs and one-off batches; JS for browsers and Node with zero extra dependencies; Python for analysis and scripts. All three must first answer "array or scalar"—array output means map/filter, scalar output means reduce.

// JS: array in, array out
const rows = data.items
  .filter((it) => it.status === 'paid')
  .map((it) => ({ id: it.user_id, amount: it.total }))
# Python: aggregate, scalar out
total = sum(item["total"] for item in data["items"] if item["status"] == "paid")

Note that JS map returns a new array without touching the source; but if the callback mutates it, upstream data is still polluted. Build new objects instead of assigning back through the reference. Two quieter side effects: undefined properties are dropped by JSON.stringify, so fields "vanish" from the output, and mixing numbers with strings makes sorting and summing behave unexpectedly. Write ?? fallback defaults in the transform and compare Object.keys once before emitting.

A Real Case: Paginated Responses Into One Table

An API paginates as { records: [...], page, pages } while downstream wants one table:

const flat = pages.flatMap((p) => p.records);                 // concatenate all pages
const rows = flat.map(({ id, user: { name }, total }) => ({
  id, customer: name, amount: total,                           // rename + destructure
}));

flatMap does "take arrays + flatten one level" in a step; when fields align, map to new objects before concatenating so raw structures never leak downstream. Pagination keys differ by vendor (records/data/list)—confirm the key before concatenating instead of assuming.

Flattening: Tree Into a Table

function flatten(node, prefix = '', out = {}) {
  for (const [k, v] of Object.entries(node)) {
    const key = prefix ? `${prefix}.${k}` : k;
    if (v && typeof v === 'object' && !Array.isArray(v)) flatten(v, key, out);
    else out[key] = v;
  }
  return out;
}

Arrays stay whole as a single value—expanding them into rows makes the row count unbounded, so handle that separately before loading. A dotted key can collide with a literal dotted key already in the document; check target columns for duplicates before exporting. The output drops straight into the JSON to CSV converter.

Performance Notes

  • Stream large files: jq --stream, or process JSONL line by line; never JSON.parse the whole tree before transforming.
  • Avoid intermediate arrays: chained map/filter allocates several temporaries; use one loop or a generator at scale.
  • Project first, restructure second: narrowing to the needed keys cuts every later traversal.
  • Validate the shape once before writing results; if a transform runs repeatedly, freeze it into a script instead of rebuilding a command each time.

Reproducible Results

One payload, one jq line, three Python lines—the results must match, and mismatches usually come from types:

import json
data = {"items": [{"sku": "A-01", "price": 39.9}, {"sku": "A-02", "price": 120}]}
rows = [{"sku": it["sku"], "price": it["price"]} for it in data["items"] if it["price"] >= 100]
print(json.dumps(rows, ensure_ascii=False))
# → [{"sku": "A-02", "price": 120}]

The jq equivalent is [.items[] | select(.price >= 100) | {sku, price}]. Running both over the same file and diffing the output with JSON Diff is the cheapest regression test for a transform.

Frequently Asked Questions

When should I use jq versus JSONPath?

jq is a full processing language: filtering, mapping, reducing, conditionals, functions and formatted output — ideal for shell pipelines and scripts. JSONPath is only a query syntax: an XPath-like expression that pulls values out of a document, ideal for config-driven extraction inside application code (for example $.data.items[*].id). Use jq to reshape, JSONPath to select. The [JSONPath finder](/jsonpath-finder.html) lets you click through a document to build the expression before you commit it to code.

How do I choose between map, filter and reduce?

Choose by the shape of the output you want: map is array → array of the same length (transform each element, one-to-one); filter is array → shorter array (keep elements matching a predicate); reduce is array → any value (aggregate into a number, an object, or another structure). A quick rule: use map when you still want a list, filter when you want part of the list, reduce when you want a single result.

Fields disappear after transformation — how do I debug it?

Four frequent causes: (1) key case or naming style mismatch (backend sends createdAt, frontend reads created_at); (2) null treated as missing — null, [] and '' mean different things in most languages; (3) a wrong path that silently returns nothing (many JSONPath implementations return undefined rather than throwing); (4) number precision (large integers rewritten as approximations). Debug in order: confirm the field really exists in the source, then validate the path expression one level at a time.

How do I keep large transformations responsive?

Three options: (1) stream it — do not JSON.parse the whole file at once; use a streaming parser and transform as you read; (2) trim before transforming — pull out only the fields you need with JSONPath or jq, then map over that subset; (3) stay off the browser main thread — run large array operations in a Web Worker, or the page freezes. Also watch intermediate arrays: a map().filter().map() chain allocates several temporary arrays, and collapsing it into a single reduce is cheaper at scale.

← Back to Blog