JSON Data Transformation Techniques: jq, JSONPath, Mapping, Filtering & Reducing in Practice

Why JSON Transformation?

In modern development, JSON data comes from various sources — API responses, config files, databases, logs. Raw data is rarely directly usable and needs transformation to meet business requirements. Common scenarios include:

• Extracting specific fields from nested structures
• Filtering records that match conditions
• Mapping data from one structure to another
• Aggregating statistics (sum, group, count)
• Merging multiple data sources

jq: The Command-Line JSON Powerhouse

jq is the Swiss Army knife for JSON processing, supporting filtering, mapping, and transformation with concise yet powerful syntax.

Basic Syntax

# Pretty-print
echo '{"name":"John","age":30}' | jq '.'

# Extract a field
echo '{"name":"John","age":30}' | jq '.name'

# Extract nested field
echo '{"user":{"name":"John"}}' | jq '.user.name'

Array Operations

# All array elements
echo '[1,2,3,4,5]' | jq '.[]'

# Array slice
echo '[1,2,3,4,5]' | jq '.[1:3]'

# Map
echo '[{"name":"John","age":30},{"name":"Jane","age":25}]' | jq 'map(.name)'

# Select (filter)
echo '[{"name":"John","age":30},{"name":"Jane","age":25}]' | jq '.[] | select(.age > 28)'

Advanced Transformations

# Construct new objects
echo '[{"name":"John","age":30}]' | jq 'map({username: .name, years: .age})'

# Group and aggregate
echo '[{"dept":"Eng","salary":10000},{"dept":"Eng","salary":12000},{"dept":"Sales","salary":8000}]' \
  | jq 'group_by(.dept) | map({dept: .[0].dept, total: (map(.salary) | add)})'

# Key-value transformation
echo '{"a":1,"b":2,"c":3}' | jq 'to_entries | map({(.key): (.value * 10)}) | add'

JSONPath Query Language

JSONPath is an XPath-like query expression for JSON, widely used in programming libraries and API queries.

Basic Syntax

Expression	Description
$	Root object
.property	Access property
['property']	Access property (for special chars)
[index]	Array index
[start:end]	Array slice
[*]	All elements
..	Recursive descent
?(filter)	Filter expression

Query Examples

Given data with a store.books array:

// All book titles
$.store.books[*].title

// Books with price > 50
$.store.books[?(@.price > 50)]

// Engineering category books
$.store.books[?(@.category == 'engineering')]

// Recursively find all prices
$..price

Language Implementations

• JavaScript: jsonpath-plus, jsonpath
• Python: jsonpath-ng
• Java: Jayway JsonPath
• Go: github.com/PaesslerAG/jsonpath

Mapping

Mapping transforms each element in a collection into a new form.

const users = [
  { id: 1, name: 'John', email: 'john@example.com' },
  { id: 2, name: 'Jane', email: 'jane@example.com' }
];

// Extract names
const names = users.map(u => u.name);

// Convert to ID-keyed object
const userMap = Object.fromEntries(users.map(u => [u.id, u]));

// Reshape structure
const dtos = users.map(({ id, name }) => ({ userId: id, displayName: name }));

Filtering

Filtering retains elements that satisfy a condition.

const products = [
  { name: 'Keyboard', price: 200, inStock: true },
  { name: 'Mouse', price: 80, inStock: false },
  { name: 'Monitor', price: 1500, inStock: true }
];

const affordable = products.filter(p => p.inStock && p.price < 1000);

Reducing

Reducing accumulates a collection into a single value — useful for summing, grouping, and flattening.

const orders = [
  { product: 'Keyboard', qty: 2, price: 200 },
  { product: 'Mouse', qty: 3, price: 80 },
  { product: 'Keyboard', qty: 1, price: 200 }
];

// Total amount
const total = orders.reduce((sum, o) => sum + o.qty * o.price, 0);

// Group by product
const grouped = orders.reduce((acc, o) => {
  acc[o.product] = (acc[o.product] || 0) + o.qty;
  return acc;
}, {});

// Flatten nested arrays
const flat = [[1, 2], [3, 4], [5]].reduce((acc, arr) => acc.concat(arr), []);

Practical Case: API Data Transformation

Transforming a paginated API response into frontend table data — extracting user_id → id, user_name → name, and formatting the timestamp, all via map.

Flattening a Tree Structure

function flattenTree(nodes, parentId = null) {
  return nodes.reduce((acc, node) => {
    acc.push({ id: node.id, name: node.name, parentId });
    if (node.children) {
      acc.push(...flattenTree(node.children, node.id));
    }
    return acc;
  }, []);
}

Performance Tips

• For large JSON (>10MB), use streaming parsers to avoid loading everything into memory
• Use jq --stream or --slurp for large file processing
• Avoid complex transformations inside render loops on the frontend
• Consider JSON Schema + code generation for fixed-structure transformations

Recommended Tools

Tool	Type	Use Case
jq	CLI	Shell scripts, quick debugging
JSONPath Plus	JS library	Complex query expressions
Jayway JsonPath	Java library	Spring project integration
JMESPath	Query language	AWS CLI, general queries
JSONata	Expression language	Complex transformation rules