esc
  • WorkflowsPages
  • Case studiesPages
  • Machine Learning FoundationsLearn
  • AI EngineeringLearn
  • AI Guardrails & Red TeamingLearn
  • UI/UXLearn
  • Design SystemsLearn
  • Production System DesignLearn
  • Book a callContact
  • Download ResumeContact
Let’s talk

I’ll build the agent for it.

Machine Learning Foundations / Beginner / 01.46

Data formats: how data is written down

Before a computer can learn from data, the data has to sit in a file. The format of that file decides how big it is, how fast it loads, and who can open it.

14 min readFree

What a file format is

In the last lesson we looked at where data comes from: databases, APIs, devices. This lesson asks a plainer question. Once you have data, how do you write it into a file so that it can be saved, sent to a friend and read again later?

Think of a recipe. You could write it as a paragraph, as a numbered list, or as a table of ingredients. The food is the same, but the way you write it down changes how easy it is to follow. Data works the same way.

Wikipedia puts it like this: 'A file format is the way that information is encoded for storage in a computer file.' It adds that a format can be 'standardized (which can be proprietary or open) or it can be an ad hoc convention.' In plain words, some formats are written down in public rules anyone can read, some belong to one company, and some are just habits people share.

Key words

File format
The way information is written into a file, from the low-level bytes up to the high-level layout.
Byte
A small chunk of computer storage that holds a number from 0 to 255. Files are made of bytes.
Text file
A file whose bytes are letters, digits and symbols that a person can read in a plain text editor.
Binary file
Wikipedia: 'a computer file that is not a text file.' Its bytes are meant for programs, not people.
Encoding
The rules for turning characters into bytes, so that the letter 'A' or the symbol 'é' can be stored.
Schema
A description of what a dataset contains: the names of its fields and the type of each one.
Compression
Packing a file so it takes fewer bytes, in a way that can be undone.

This is an overview. We meet each major format briefly here, and later lessons take them one at a time: CSV, Excel, JSON, Parquet and more.

Text files and binary files

Every file is bytes. The split that matters most is whether those bytes spell out readable text or not.

A text file can be opened in any plain text editor and you can read it. Wikipedia describes the other kind this way: 'A binary file is a computer file that is not a text file.' If you open one in a text editor, Wikipedia says you see 'a (probably unintelligible) display of textual characters'. Another program might 'interpret the numbers in the bytes as colors and display the corresponding picture.'

Here is the same number written both ways. As text, 8.5 takes three characters. As a binary number, it takes a fixed block of bytes. The code below also uses a type called float, which is a number with a decimal point.

Python
import csv, io, json, struct

for row in csv.DictReader(io.StringIO("id,score\n1,8.5\n")):
    print(row, type(row["score"]).__name__)

print(json.loads('{"id": 1, "score": 8.5}'))
print(type(json.loads('{"score": 8.5}')["score"]).__name__)

print(len("8.5"), "bytes as text")
packed = struct.pack("<d", 8.5)
print(len(packed), "bytes as a binary number:", packed.hex())

Output

{'id': '1', 'score': '8.5'} str
{'id': 1, 'score': 8.5}
float
3 bytes as text
8 bytes as a binary number: 0000000000002140

Two things happened. First, the text version is shorter here and a person can read it. The binary version is longer for this one tiny number, but a program can read those eight bytes back as the number 8.5. Second, look at the first two lines. The CSV reader gave back '8.5' as a string, which is just text. The JSON reader gave back a real number. That difference matters, and we return to it in the section on types.

Formats also often start with a small label. Wikipedia says the information at the start of a file 'is often called a file header', and a short mark that names the format is called a 'magic number'. Its example: 'GIF images, for instance, always begin with the ASCII representation of either GIF87a or GIF89a'. Programs read that start to check that a file is what it claims to be.

Tip

Check what is inside a file. The first bytes often name the format, and the first lines of a text file show its shape.

Tables: CSV

Most data for beginners is a table: rows of records, columns of facts. The simplest way to store a table is CSV, short for Comma-Separated Values. It is a text file. Each line is a row, and commas separate the columns.

The official note on CSV, RFC 4180, lists a few rules. 'Each record is located on a separate line, delimited by a line break (CRLF).' Then: 'There maybe an optional header line appearing as the first line of the file', holding the names of the fields. And: 'Fields containing line breaks (CRLF), double quotes, and commas should be enclosed in double-quotes.' So a city called 'Paris, Texas' goes inside quote marks, or its comma would split it into two columns.

One warning. Python's own documentation says CSV was used 'for many years prior to attempts to describe the format in a standardized way in RFC 4180', and that this 'lack of a well-defined standard means that subtle differences often exist in the data produced and consumed by different applications.' Always look at a CSV file before you trust it.

Here are three records written as CSV, and then the same three records as JSON. We write both from Python, then count the bytes.

Python
import csv, io, json

rows = [
    {"id": 1, "name": "Asha", "city": "Pune", "score": 8.5},
    {"id": 2, "name": "Ben", "city": "Leeds", "score": 7.0},
    {"id": 3, "name": "Chloé", "city": "Paris", "score": 9.2},
]

buf = io.StringIO()
writer = csv.DictWriter(buf, fieldnames=["id", "name", "city", "score"], lineterminator="\n")
writer.writeheader()
writer.writerows(rows)
as_csv = buf.getvalue()
as_json = json.dumps(rows, ensure_ascii=False)

print(as_csv, end="")
print(as_json)
print("csv bytes: ", len(as_csv.encode("utf-8")))
print("json bytes:", len(as_json.encode("utf-8")))

Output

id,name,city,score
1,Asha,Pune,8.5
2,Ben,Leeds,7.0
3,Chloé,Paris,9.2
[{"id": 1, "name": "Asha", "city": "Pune", "score": 8.5}, {"id": 2, "name": "Ben", "city": "Leeds", "score": 7.0}, {"id": 3, "name": "Chloé", "city": "Paris", "score": 9.2}]
csv bytes:  70
json bytes: 174

CSV writes the column names once, at the top. JSON repeats every name in every record. That is why the JSON is more than twice as big here. The cost buys something, which we see next. A later lesson is about CSV up close, and another covers spreadsheet files.

Nested data: JSON

Some data is not a flat table. A person has several phone numbers. An order has a list of items, and each item has its own details. Forcing that into rows and columns is awkward.

JSON, short for JavaScript Object Notation, handles it. RFC 8259 calls it 'a lightweight, text-based, language-independent data interchange format.' It has few building blocks: 'JSON can represent four primitive types (strings, numbers, booleans, and null) and two structured types (objects and arrays).'

  • Object

    The standard says 'An object is an unordered collection of zero or more name/value pairs.' Think of a form: field names on the left, answers on the right.

  • Array

    'An array is an ordered sequence of zero or more values.' Think of a list where the order is kept.

  • Nesting

    A value can itself be an object or an array, so an order can hold a list of items, each item holding its own fields. CSV has no way to do this: Wikipedia notes that 'CSV has no provision for structures with more than one level.'

Because JSON is text and 'language-independent', programs written in different languages can swap it. Python's json module describes it as 'a lightweight data interchange format'. A later lesson goes deep on JSON.

Big tables: Parquet

CSV and JSON are text. For small data that is fine. For millions of rows, a format designed for bulk data can do better.

Parquet is a format built for that job. The Apache Parquet site describes it as 'an open source, column-oriented data file format designed for efficient data storage and retrieval.' It 'provides high performance compression and encoding schemes to handle complex data in bulk.'

Column-oriented means the file keeps all the values of one column together, instead of all the values of one row together. Picture a school register. A row-style file is the register read one pupil at a time. A column-style file is every pupil's age listed together, then every pupil's city together. If you only need ages, you can read just that list.

The Parquet site lists libraries and tools in many programming languages that read and write it. This lesson uses only what ships with Python, so there is no Parquet code here. A later lesson covers it.

Pictures and other files

Not all ML data is a table. Images are data too, and they are stored in binary formats made for pictures. Wikipedia gives one example: 'PNG files, for example, store bitmapped images using lossless data compression.' Data compression, which we meet below, means packing a file into fewer bytes.

As the binary file page says, a program can read the numbers in a file's bytes as colours and show a picture. The file format is the agreement about what those numbers mean.

Tip

Some formats are wrappers. Wikipedia says Ogg 'can act as a container for different types of multimedia including any combination of audio and video'. A container holds several kinds of data in one file.

Encodings: from letters to bytes

A text file still has to turn letters into bytes. The rule for that is the encoding. The Python Unicode HOWTO says: 'The rules for translating a Unicode string into a sequence of bytes are called a character encoding, or just an encoding.'

Unicode is a list that gives every character a number, and the HOWTO says it 'aims to list every character used by human languages and give each character its own unique code.' UTF-8 is one way of turning those numbers into bytes. The HOWTO calls it 'one of the most commonly used encodings', and Python 'often defaults to using it.' Its rules: a code point below 128 is a single byte, and anything higher becomes 'a sequence of two, three, or four bytes'.

RFC 3629 says UTF-8 'has the characteristic of preserving the full US-ASCII range'. So plain English text is the same bytes in ASCII and in UTF-8. And RFC 8259 says JSON sent between systems 'MUST be encoded using UTF-8'. Let us see the letter 'é' in bytes.

Python
text = "é"
data = text.encode("utf-8")
print(data)
print(list(data))
print(len("Pune".encode("utf-8")), "bytes for Pune")
print(len("Chloé".encode("utf-8")), "bytes for Chloé")
print(data.decode("latin-1"))

Output

b'\xc3\xa9'
[195, 169]
4 bytes for Pune
6 bytes for Chloé
é

The letter 'é' needs two bytes, 195 and 169. 'Pune' has four letters and four bytes. 'Chloé' has five letters but six bytes. The last line shows the classic mistake. We read UTF-8 bytes using a different encoding, called latin-1, and got two odd characters instead of one. If your data shows strange symbols where accents should be, the file was written in one encoding and read in another.

Tip

When a tool asks for an encoding, choose UTF-8 unless you know the file was made another way.

Compression: smaller files

Files can be packed to take fewer bytes. Python's gzip module 'provides a simple interface to compress and decompress files just like the GNU programs gzip and gunzip would.' The compressed output is a gzip file.

Compression works well on repetitive data, and tables repeat a lot: the same city names, the same small numbers. Here we build a 1,000-row CSV in memory and pack it.

Python
import csv, io, gzip

buf = io.StringIO()
writer = csv.writer(buf, lineterminator="\n")
writer.writerow(["id", "city", "score"])
cities = ["Pune", "Leeds", "Paris", "Oslo"]
for i in range(1, 1001):
    writer.writerow([i, cities[i % 4], i % 10])
raw = buf.getvalue().encode("utf-8")
packed = gzip.compress(raw)

print("raw bytes:   ", len(raw))
print("gzip bytes:  ", len(packed))
print("smaller by:  ", round(100 * (1 - len(packed) / len(raw))), "%")
print("same after unpacking:", gzip.decompress(packed) == raw)

Output

raw bytes:    11407
gzip bytes:   2280
smaller by:   80 %
same after unpacking: True

About 80 percent smaller, and unpacking gives back the exact same bytes. This data is very regular, so it packs well. Real data packs by different amounts. The cost is the work of packing and unpacking, and a compressed text file is no longer readable in a text editor until you unpack it.

Types and schemas

Remember the first code block: the CSV reader returned the string '8.5', not a number. CSV is plain text, so it has no idea whether a column holds numbers, dates or words. Your code must decide, and must convert.

That matters because models work on numbers. Google's Machine Learning Crash Course says its numerical data module 'focuses on preparing numerical data, such as temperature or weight, for use in machine learning models.' A score read as text is not yet usable that way.

A schema is the fix. It lists each field and its type, for example 'id is a whole number, score is a decimal, city is text'. JSON keeps a few types for you: strings, numbers, booleans and null. With CSV the schema lives in your head or in a separate note, so write it down.

Quick check

You load a CSV file in Python and the column 'score' arrives as text such as '8.5'. What is the right next step?

Choosing a format

There is no best format, only a good fit. Ask four questions, in this order.

  1. 1

    1. Who needs to open it?

    If a person must read it in a text editor, use a text format such as CSV or JSON. A binary file, or a format such as Parquet, needs a program that supports it.

  2. 2

    2. Is the data flat or nested?

    A flat table suits CSV. Records with lists inside records suit JSON. CSV has no way to nest.

  3. 3

    3. How big is it?

    For small data, text is fine. For huge tables, a column-oriented format with compression, such as Parquet, is designed for bulk data.

  4. 4

    4. Is the format documented?

    Wikipedia notes that 'file formats with publicly available specifications tend to be supported by more programs.' Open, written-down formats are safer to share.

  • Small and shared

    A short table to email to a classmate. CSV, because anyone can open it.

  • Nested and sent between programs

    The reply from an API, or records with lists inside. JSON, because it is text and holds nesting.

  • Big and read again and again

    Millions of rows for training. Parquet, because it is column-oriented and packs well.

You can also mix them. You might receive data as JSON from an API and keep a copy in another format that suits your next step.

Tip

Whatever format you pick, write the schema and the encoding next to the file. It takes a minute and saves hours.

Quick check

A team has one million rows. They only need two of the columns for each experiment, and they re-read the data daily. Which fits best?

What to remember

A file format is the way data is written into a file. Text files can be read by people, binary files by programs. CSV is a flat table in text, JSON handles nesting, and Parquet stores big tables by column. An encoding such as UTF-8 turns letters into bytes, compression makes files smaller, and a schema says what each field is. Pick by who opens it, the shape of the data and its size.

Next: CSV, up close.

Sources

How is this lesson?

Last updated on 16 November 2026

Learn with me

Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.