esc
  • WorkflowsPages
  • Case studiesPages
  • Machine Learning FoundationsLearn
  • AI EngineeringLearn
  • AI Guardrails & Red TeamingLearn
  • UI/UXLearn
  • Design SystemsLearn
  • Production System DesignLearn
  • Book a callContact
  • Download ResumeContact
Let’s talk

I’ll build the agent for it.

Machine Learning Foundations / Beginner / 01.49

JSON files

JSON is the plain-text format almost every service speaks. Learn to read it, flatten it into a table, store big datasets line by line, and dodge the traps.

14 min readFree

Why JSON matters for machine learning

In the last lesson you read a spreadsheet. A spreadsheet is a grid: rows and columns, one shape for everything. Real data is often messier. A user has a name, but also a list of orders, and each order has a list of items. That does not fit one grid cell. JSON is the format that holds data like this.

The APIs lesson parsed one JSON response. This lesson goes deeper. We look at every piece of the syntax, see how it maps to Python, flatten nested data into a table, store a big dataset one record per line, and learn the traps that quietly break datasets.

The official standard, RFC 8259, begins: 'JavaScript Object Notation (JSON) is a lightweight, text-based, language-independent data interchange format.' JSON.org says: 'It is easy for humans to read and write. It is easy for machines to parse and generate.' That is why it is everywhere: config files, web APIs and exported datasets.

Key words

JSON
Short for JavaScript Object Notation. A plain-text way to write data with names and values.
Object
A group of name/value pairs inside curly brackets. Think of one record, such as one person.
Array
An ordered list of values inside square brackets.
Parse
Read text and turn it into data your program can use. The reverse, turning data into text, is called serializing.
Nested
Data inside data, such as a list inside an object inside a list.
Flatten
Turn nested data into plain columns, so each example becomes one row.

The syntax: six kinds of value

RFC 8259 says 'A JSON value MUST be an object, array, number, or string, or one of the following three literal names': false, null and true. That is the whole list. Four simple kinds and two that hold other values.

  • Strings and numbers

    A string is text in double quotes. The RFC says 'A string begins and ends with quotation marks.' A number is written plainly, such as 31 or 4.5. Numbers are never in quotes.

  • true, false and null

    Three words, always in lowercase. The RFC says 'The literal names MUST be lowercase. No other literal names are allowed.' So True or None are not JSON.

  • Objects and arrays

    An object is curly brackets with name/value pairs. An array is square brackets with values. The RFC says for objects 'A single colon comes after each name, separating the name from the value. A single comma separates a value from a following name.'

Here is one record in JSON. Read it slowly. The name of each pair is a string in quotes, then a colon, then the value.

Text
{
  "name": "Mia",
  "age": 31,
  "score": 4.5,
  "active": true,
  "tags": ["ml", "ux"],
  "boss": null
}

Tip

null means 'there is no value here'. It is not zero and not an empty string. In a dataset, a null is a missing value, and you must decide what to do with it.

How JSON maps to Python

Python reads and writes JSON with its built-in json module, so there is nothing to install. The Python docs list how each JSON kind becomes a Python kind: an object becomes a dict, an array a list, a string a str, a whole number an int, a real number a float, true and false become True and False, and null becomes None.

Python
import json

text = '{"name": "Mia", "age": 31, "score": 4.5, "active": true, "tags": ["ml", "ux"], "boss": null}'
person = json.loads(text)
for key, value in person.items():
    print(key, "->", repr(value), type(value).__name__)

Output

name -> 'Mia' str
age -> 31 int
score -> 4.5 float
active -> True bool
tags -> ['ml', 'ux'] list
boss -> None NoneType

Notice that 31 became an int and 4.5 became a float. In a CSV file everything starts as text. JSON keeps the difference between the number 31 and the text "31", which saves you a cleaning step.

Quick check

A JSON file contains the word True with a capital T. What does the RFC say?

Nested data and flattening

A model wants a table: one row per example, one column per feature. JSON often comes nested. So the work is to flatten it. A common way is to join the names with an underscore, so user inside the record becomes user_name and user_city.

  1. 1

    1. Look at the shape

    Print one record. Find which values are plain and which hold other objects or lists.

  2. 2

    2. Flatten the objects

    Walk each pair. If the value is an object, go inside it and add its name as a prefix.

  3. 3

    3. Decide about lists

    A list does not fit one cell. Either summarise it (a mean, a count) or make one row per item. Which one depends on what the model should learn.

  4. 4

    4. Check the result

    Count the rows and columns. Make sure no value was lost by accident.

Python
record = {
    "id": 7,
    "user": {"name": "Mia", "city": "Pune"},
    "ratings": [4, 5, 3],
}

def flatten(data, prefix=""):
    flat = {}
    for key, value in data.items():
        name = prefix + key
        if isinstance(value, dict):
            flat.update(flatten(value, name + "_"))
        else:
            flat[name] = value
    return flat

row = flatten(record)
print(row)
row["ratings_mean"] = sum(row["ratings"]) / len(row["ratings"])
print(row["ratings_mean"])

Output

{'id': 7, 'user_name': 'Mia', 'user_city': 'Pune', 'ratings': [4, 5, 3]}
4.0

The object user became two columns. The list ratings stayed a list, so we turned it into one number, the mean. That is a choice, not a rule. If each rating mattered on its own, one row per rating would be better.

Another thing to know: the RFC says 'The names within an object SHOULD be unique.' We come back to what happens when they are not.

Reading and writing JSON files

The json module has four main functions, and two letters tell them apart. The ones ending in s work on strings. The ones without it work on files.

  • loads and load

    Read. The Python docs say loads will 'deserialize s (a str, bytes or bytearray instance containing a JSON document) to a Python object'. load does the same from an open file.

  • dumps and dump

    Write. dumps gives you a string of JSON text. dump writes it straight into an open file.

  • indent and ensure_ascii

    indent adds line breaks and spaces so people can read it. ensure_ascii is on by default and turns letters like u-umlaut into \u escapes. Pass ensure_ascii=False to keep the real letters.

The Python docs explain the default: the serializer sets 'ensure_ascii=True by default, thus escaping the output so that the resulting strings only contain printable ASCII characters.' The data is the same either way. Below we write the same record three ways, then save it and read it back.

Python
import json

data = {"city": "Zürich", "temps": [3, 5.5], "ok": True, "note": None}

print(json.dumps(data))
print(json.dumps(data, ensure_ascii=False))
print(json.dumps(data, indent=2, ensure_ascii=False))

with open("data.json", "w", encoding="utf-8") as f:
    json.dump(data, f, indent=2, ensure_ascii=False)
with open("data.json", encoding="utf-8") as f:
    back = json.load(f)
print(back == data)

Output

{"city": "Z\u00fcrich", "temps": [3, 5.5], "ok": true, "note": null}
{"city": "Zürich", "temps": [3, 5.5], "ok": true, "note": null}
{
  "city": "Zürich",
  "temps": [
    3,
    5.5
  ],
  "ok": true,
  "note": null
}
True

See how True became true and None became null on the way out, then came back unchanged. We also passed encoding="utf-8" when opening the file. Python's docs say the RFC requires JSON to be in UTF-8, UTF-16 or UTF-32, 'with UTF-8 being the recommended default for maximum interoperability'.

Tip

Use indent=2 while you look at a file yourself. Leave it out when you save a big file, because the extra spaces make it larger.

JSON Lines: one record per line

A normal JSON file is one big value. To read it you must load all of it into memory first. For a dataset with millions of records, that is a problem. Python's docs even warn that 'JSON is not a framed protocol, so trying to serialize multiple objects with repeated calls to dump() using the same fp will result in an invalid JSON file.' You cannot just write one record after another.

JSON Lines fixes this. The jsonlines.org site calls it 'a convenient format for storing structured data that may be processed one record at a time.' It has three requirements: UTF-8 encoding, 'Each Line is a Valid JSON Value', and a line terminator of '\n'. It adds that 'The most common values will be objects or arrays, but any JSON value is permitted. e.g. null is a valid value but a blank line is not.' Files may use the extension .jsonl.

Python
import json

records = [
    {"id": 1, "text": "great phone", "label": "positive"},
    {"id": 2, "text": "battery died in a day", "label": "negative"},
    {"id": 3, "text": "caf\u00e9 was nice", "label": "positive"},
]

with open("reviews.jsonl", "w", encoding="utf-8") as f:
    for r in records:
        f.write(json.dumps(r, ensure_ascii=False) + "\n")

with open("reviews.jsonl", encoding="utf-8") as f:
    for line in f:
        r = json.loads(line)
        print(r["id"], r["label"], r["text"])

Output

1 positive great phone
2 negative battery died in a day
3 positive café was nice

That script wrote this file. Each line stands alone, so you can open it in any text editor, count the records with a line count, or read just the first ten.

reviews.jsonl
{"id": 1, "text": "great phone", "label": "positive"}
{"id": 2, "text": "battery died in a day", "label": "negative"}
{"id": 3, "text": "café was nice", "label": "positive"}

The loop reads one line at a time, so memory use stays small however long the file is. Adding a record later is just adding a line at the end. The site says it is also 'a great format for log files'.

Quick check

Why is JSON Lines a good fit for a very large dataset?

Traps that break datasets

JSON looks easy, and that is the danger. These traps cost people hours. We take them one at a time.

Trap 1: JSON is stricter than it looks

The RFC says 'A single comma separates a value from a following name.' So a comma after the last pair is an error. The grammar also has no place for comments, and strings need double quotes. Python's own habits (single quotes, # or // notes) do not carry over. Python's parser tells you what went wrong.

Python
import json

bad = {
    "trailing comma": '{"a": 1, "b": 2,}',
    "comment": '{"a": 1 // note\n}',
    "single quotes": "{'a': 1}",
}
for name, text in bad.items():
    try:
        json.loads(text)
    except json.JSONDecodeError as e:
        print(name, "->", e)

Output

trailing comma -> Illegal trailing comma before end of object: line 1 column 16 (char 15)
comment -> Expecting ',' delimiter: line 1 column 9 (char 8)
single quotes -> Expecting property name enclosed in double quotes: line 1 column 2 (char 1)

Read the line and column in the message. It tells you where the parser stopped.

Trap 2: NaN, repeated names and keys

NaN means 'not a number', a value that appears when a calculation fails. The Python docs are clear: 'The RFC does not permit the representation of infinite or NaN number values. Despite that, by default, this module accepts and outputs Infinity, -Infinity, and NaN as if they were valid JSON number literal values'. So Python will happily write a file that other programs cannot read. You can ask it to refuse with allow_nan=False.

Two more surprises. First, the docs say 'Keys in key/value pairs of JSON are always of the type str.' So a dictionary with number keys comes back with text keys. Second, when a name is repeated, the docs say that by default the module 'ignores all but the last name-value pair for a given name'. The RFC says only that the behaviour is 'unpredictable' for software that receives such an object, so another tool may pick the first one instead.

Python
import json

print(json.loads('{"x": NaN}'))
print(json.dumps({"x": float("nan")}))
try:
    json.dumps({"x": float("nan")}, allow_nan=False)
except ValueError as e:
    print("ValueError:", e)
print(json.dumps({1: "a", 2: "b"}))
print(json.loads(json.dumps({1: "a"})) == {1: "a"})
print(json.loads('{"x": 1, "x": 2}'))

Output

{'x': nan}
{"x": NaN}
ValueError: Out of range float values are not JSON compliant: nan
{"1": "a", "2": "b"}
False
{'x': 2}

Tip

For a missing number, write null, not NaN. null is valid JSON and every tool reads it. You can turn it into a NaN later, inside your own code.

Trap 3: big numbers and decimals

Python integers can be as large as you like. Many other tools cannot do that. The Python docs warn: 'it is common for JSON numbers to be deserialized into IEEE 754 double precision numbers and thus subject to that representation's range and precision limitations.' The RFC gives the safe zone: integers 'in the range [-(2**53)+1, (2**53)-1] are interoperable in the sense that implementations will agree exactly on their numeric values.' That number is about nine quadrillion. If you need a bigger whole number to survive other tools, store it as a string.

The RFC also says a number such as 3.141592653589793238462643383279 'may indicate potential interoperability problems'. Python reads it as a float and keeps only these digits.

Python
import json

big = 12345678901234567890
print(json.loads(json.dumps(big)) == big)
print(float(2**53 + 1) == float(2**53))
print(json.loads("3.141592653589793238462643383279"))

Output

True
True
3.141592653589793

Python kept the big integer exactly, so the first line says True. The second line shows what a double does: 2 to the power 53, plus 1, is the same float as 2 to the power 53. A tool that uses doubles would lose that last digit.

Trap 4: keys that are not there

In a CSV file every row has every column. JSON has no such promise. One object may have an age and the next may leave it out. Reading a missing key with square brackets stops your program. The get method gives None instead, which you can count and handle.

Python
rows = [{"id": 1, "age": 30}, {"id": 2}]
for r in rows:
    print(r["id"], r.get("age"))

try:
    print(rows[1]["age"])
except KeyError as e:
    print("KeyError:", e)

Output

1 30
2 None
KeyError: 'age'

Silent gaps are a bigger problem than loud errors. If half your records lack a field and you never check, the model learns from a hidden half.

Schemas: agreeing on the shape

JSON lets you build the same record in many ways. The JSON Schema site shows two ways to write a person: one with a single name and address text, another with first_name, last_name and an address object. It says 'Both representations are equally valid'. So how does your program know which one it will receive?

The site answers: 'we need to know what fields are expected, and how the values are represented. That's where JSON Schema comes in.' A schema is a JSON document that describes the shape of other JSON: which fields exist, and whether each one is a string, a number or an object. A tool can then check a file against it and fail early.

JSON Schema needs an extra tool to run, and this course uses only what ships with Python. So here is the same idea by hand: a tiny check that says which fields are missing or have the wrong type.

Python
def check(record):
    problems = []
    for name, kind in {"id": int, "text": str, "label": str}.items():
        if name not in record:
            problems.append("missing " + name)
        elif not isinstance(record[name], kind):
            problems.append(name + " should be " + kind.__name__)
    return problems

print(check({"id": 1, "text": "ok", "label": "positive"}))
print(check({"id": "1", "text": "ok"}))

Output

[]
['id should be int', 'missing label']

An empty list means the record is fine. A real schema does this job with a standard vocabulary that many tools understand, so you write it once and use it everywhere.

Putting it together

The path for JSON in a project: read it with loads or load, check the shape, flatten what is nested, count what is missing, keep big IDs as text, write missing numbers as null, and use JSON Lines when the file gets large.

Tip

When a file will not load, read the line and column in the error message first. Common causes are a trailing comma, a comment or a single quote.

Next: Parquet.

Sources

How is this lesson?

Last updated on 16 November 2026

Learn with me

Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.