Guide to jq Command | Learn The Shell

Introduction

The jq command line tool is a powerful and flexible utility for working with JSON data in Linux. It allows you to parse, filter, and transform JSON data easily. This guide covers the basic usage of jq, including accessing JSON properties, working with arrays, slicing data, using functions, and converting between JSON and JSONL. Each section includes detailed explanations and examples to help you understand and utilize jq effectively.

TL;DR

You can find a shorter cheat sheet version of this article here.

Open Table of contents

Basic Usage
- Reading JSON Input
Accessing JSON Properties
Working with Arrays
Slicing Arrays
Using Functions
Using Multiple Filters
Converting Between JSON and JSONL
- JSONL to JSON
- JSON to JSONL
Conclusion

Basic Usage

Reading JSON Input

To read JSON input from a URL and format it:

curl https://json-api-url | jq '.'

This command fetches JSON data from the specified URL and formats it for easier reading. The . filter simply outputs the entire JSON input in a more readable format.

To read JSON input from a file:

jq '.' file.json

This reads the JSON data from file.json and formats it. The . filter outputs the entire JSON content, making it easier to read and analyze.

Accessing JSON Properties

To access a specific JSON property:

jq '.url' file.json

This extracts the value of the url property from file.json. If the JSON object is:

{
  "url": "https://example.com",
  "name": "example"
}

The output will be:

"https://example.com"

For nested objects:

jq '.committer.login' file.json

This accesses the login property within the committer object. For the JSON object:

{
  "committer": {
    "login": "user123",
    "id": 1
  }
}

The output will be:

"user123"

To retrieve multiple keys, separate them with commas:

jq '.committer.login, .committer.type, .committer.url' file.json

This extracts the login, type, and url properties from the committer object, producing:

"user123"
"User"
"https://api.example.com/user123"

To wrap all retrieved properties in a new object:

jq '{login: .committer.login, type: .committer.type, url: .committer.url}' file.json

This creates a new JSON object with the specified properties:

{
  "login": "user123",
  "type": "User",
  "url": "https://api.example.com/user123"
}

Working with Arrays

To iterate over a JSON array:

jq '.[]' file.json

This processes each element in the array. For the JSON array:

[
  {"name": "Alice", "age": 30},
  {"name": "Bob", "age": 25}
]

The output will be:

{"name": "Alice", "age": 30}
{"name": "Bob", "age": 25}

To access an element by its index:

jq '.[0]' file.json

This retrieves the first element of the array. For the same array, the output will be:

{"name": "Alice", "age": 30}

To display a specific element property:

jq '.[].name' file.json

This extracts the value of name from each object in the array, producing:

"Alice"
"Bob"

Slicing Arrays

To get a range of elements:

echo '[1,2,3,4,5,6,7,8,9,10]' | jq '.[3:6]'

This retrieves elements from index 3 to 5 (inclusive of the start, exclusive of the end). The output will be:

[4, 5, 6]

To get the first 5 elements:

echo '[1,2,3,4,5,6,7,8,9,10]' | jq '.[:5]'

This retrieves the first 5 elements of the array:

[1, 2, 3, 4, 5]

To get the last 2 elements:

echo '[1,2,3,4,5,6,7,8,9,10]' | jq '.[-2:]'

This retrieves the last 2 elements of the array:

[9, 10]

To get the 3rd element from the end:

echo '[1,2,3,4,5,6,7,8,9,10]' | jq '.[-3]'

This retrieves the third element from the end:

Using Functions

To get the length of an array:

jq 'length' file.json

This returns the number of elements in the array. For example, for the array:

[1, 2, 3, 4, 5]

The output will be:

To get the keys of a JSON object:

jq 'keys' file.json

This returns an array of the object’s keys. For the JSON object:

{
  "name": "Alice",
  "age": 30
}

The output will be:

["age", "name"]

To find the minimum value in an array:

jq '[.[].price] | min' file.json

This returns the minimum price value from an array of objects. For the JSON array:

[
  {"item": "A", "price": 30},
  {"item": "B", "price": 20}
]

The output will be:

To find the maximum value in an array:

jq '[.[].price] | max' file.json

This returns the maximum price value from an array of objects. For the same array, the output will be:

To select elements based on a condition:

jq 'select(.Verified == true)' file.json

This filters the elements where Verified is true. For the JSON array:

[
  {"name": "Alice", "Verified": true},
  {"name": "Bob", "Verified": false}
]

The output will be:

{"name": "Alice", "Verified": true}

For multiple conditions:

jq 'select((.status_code == 200) and (.content_type == "application/json"))' file.json

This selects elements where status_code is 200 and content_type is "application/json". For the JSON array:

[
  {"status_code": 200, "content_type": "application/json"},
  {"status_code": 404, "content_type": "text/html"}
]

The output will be:

{"status_code": 200, "content_type": "application/json"}

Using Multiple Filters

To pass the output of one filter to another and build an object from those fields:

jq '.[] | {message: .commit.message, name: .commit.committer.name}' file.json

This creates a new object with message and name fields from each element in the array. For the JSON array:

[
  {"commit": {"message": "Initial commit", "committer": {"name": "Alice"}}},
  {"commit": {"message": "Added new feature", "committer": {"name": "Bob"}}}
]

The output will be:

{"message": "Initial commit", "name": "Alice"}
{"message": "Added new feature", "name": "Bob"}

To build an array of objects from the filter:

jq '[.[] | {message: .commit.message, name: .commit.committer.name}]' file.json

This creates an array of objects with message and name fields. For the same input, the output will be:

[
  {"message": "Initial commit", "name": "Alice"},
  {"message": "Added new feature", "name": "Bob"}
]

Converting Between JSON and JSONL

JSONL to JSON

To convert JSONL (JSON Lines) to JSON:

jq -s '.' input.jsonl > output.json

This reads the JSONL file and converts it to a single JSON array. If input.jsonl contains:

{"name": "Alice"}
{"name": "Bob"}

The output in output.json will be:

[
  {"name": "Alice"},
  {"name": "Bob"}
]

JSON to JSONL

To convert JSON to JSONL:

jq -c '.[]' input.json > output.jsonl

This converts each element of a JSON array to a separate line in a JSONL file. If input.json contains:

[
  {"name": "Alice"},
  {"name": "Bob"}
]

The output in output.jsonl will be:

{"name": "Alice"}
{"name": "Bob"}

Conclusion

The jq command line tool is an indispensable utility for anyone working with JSON data on Linux. Its powerful and flexible features make it easy to parse, filter, and transform JSON data, whether you are a developer, system administrator, or data analyst. By mastering the basic usage, accessing properties, manipulating arrays, slicing data, using built-in functions, and converting between JSON and JSONL, you can harness the full potential of jq to streamline your workflow and efficiently handle JSON data.

This guide provided a comprehensive overview of essential jq commands with detailed explanations and practical examples. With practice, you’ll be able to use jq to tackle more complex JSON processing tasks, enabling you to manipulate and analyze data with precision and ease.

For further learning and advanced usage, refer to the official jq documentation and the jq cookbook. These resources offer a wealth of information and examples to deepen your understanding and proficiency with jq.