Machine Learning Foundations / Beginner / 01.50
Parquet: files that store data by column
A CSV file stores a table one row at a time. Parquet stores it one column at a time, and that small change makes big data files smaller and faster to read.
15 min readFree
Why a new file format?
In the last lessons you met files like CSV and JSON. They are easy to open and easy to read. A person can look inside them with any text editor. That is a real strength, and it is why they are everywhere.
The Apache Parquet site describes it in one line: 'Apache Parquet is an open source, column-oriented data file format designed for efficient data storage and retrieval.' It adds: 'It provides high performance compression and encoding schemes to handle complex data in bulk and is supported in many programming languages and analytics tools.'
Key words
- Row
- One record in a table, such as one customer with a name, a country and an age.
- Column
- One field across all records, such as the age of every customer.
- Row-oriented
- A file that stores the table one row after another.
- Column-oriented
- A file that stores the table one column after another. Also called columnar.
- Compression
- Squeezing data so it takes fewer bytes on disk. You can undo it later and get the same data back.
- Encoding
- A smart way of writing values down, such as writing a repeated word once with a count. It also saves space.
- Schema
- The list of columns in a table, with the name and type of each one.
- Metadata
- Data about the data, such as how many rows a file holds and where each column starts.
Rows or columns?
A table has two directions, rows and columns. But a file on disk is one long line of bytes. So the computer must decide how to flatten the table into that line. Wikipedia calls this choice 'data orientation', and says there are two common ways: column-oriented and row-oriented.
Here is an analogy. Imagine a school with a big wall of student cards. Each card has a name, a country and an age.
Row-oriented: a stack of cards
Each card holds everything about one student. It is easy to pull out one student and read the whole card. This is how CSV works. Wikipedia lists CSV as a row-oriented format.
Column-oriented: one sheet per question
One sheet holds only the ages of all students. Another holds only the countries. To find the average age, you pick up one sheet and ignore the rest.
Which is better?
Neither always wins. Wikipedia says: 'The choice of data orientation is a trade-off'. Row-oriented formats are more common for online transaction processing, such as taking orders. Column-oriented formats are more common for online analytical processing, such as questions over huge tables.
Quick check
You have a table of 5 million customers with 40 columns. You only need the average of one column. Which layout lets a program skip most of the file?
See it in a few lines of Python
You do not need Parquet installed to feel the difference. This program uses only Python's own tools. It keeps four students as a list of rows, then builds the same data as a dictionary of columns. Then it finds the average age both ways and counts how many values it had to look at.
rows = [
{"name": "Asha", "country": "India", "age": 21},
{"name": "Ben", "country": "India", "age": 34},
{"name": "Cleo", "country": "Kenya", "age": 28},
{"name": "Dev", "country": "India", "age": 19},
]
# Column layout: one list per column.
columns = {
"name": [r["name"] for r in rows],
"country": [r["country"] for r in rows],
"age": [r["age"] for r in rows],
}
print(columns)
# Average age: the row layout visits every field of every row.
touched = 0
total = 0
for r in rows:
for key, value in r.items():
touched += 1
if key == "age":
total += value
print("row layout:", total / len(rows), "values touched:", touched)
# The column layout reads only the one list it needs.
ages = columns["age"]
print("column layout:", sum(ages) / len(ages), "values touched:", len(ages))Output
{'name': ['Asha', 'Ben', 'Cleo', 'Dev'], 'country': ['India', 'India', 'Kenya', 'India'], 'age': [21, 34, 28, 19]}
row layout: 25.5 values touched: 12
column layout: 25.5 values touched: 4Both ways give the same answer, 25.5. But the row layout looked at 12 values and the column layout looked at 4. With 3 columns that is a small saving. With 40 columns and millions of rows, the gap is huge. This toy runs in memory, not on disk, so it shows the idea and not real speed. Parquet applies the same idea to files.
Why columns help
Storing by column gives three gains. The first one you have just seen: a program reads only the columns it needs. The other two come from the fact that values in one column look alike.
Read less
A query that needs two columns of forty reads two columns. Wikipedia's page on Parquet puts it this way: 'Queries that fetch specific column values need not read the entire row, thus improving performance'.
Compress better
A column holds one kind of thing, such as all countries or all ages. Wikipedia says column-oriented storage gets a 'smaller compressed size', and gives the reason: 'a higher homogeneity within a column than within multiple rows'. Homogeneity means 'being alike'.
Encode smarter
Because a column holds one type, each column can use an encoding made for that type. Wikipedia says column-wise compression 'enables different encoding schemes to be used for text and integer data'.
There is a cost. The same page says column-oriented formats like Parquet have 'better compression and encoding with improved read performance at the cost of slower writes'. So Parquet shines for data you write once and read many times, which is how most training data is used.
Two encodings: dictionary and run-length
The Parquet format page lists several encodings. Two of them are easy to understand, and you can build both yourself.
Dictionary encoding is like a short list of nicknames. The spec says: 'The dictionary encoding builds a dictionary of values encountered in a given column.' Each distinct value goes in the dictionary once. The column then stores small numbers that point into it. Instead of writing 'India' a thousand times, you write it once and then write 0 a thousand times.
Run-length encoding (RLE) is like counting in a queue. The spec says the encoding is used 'to more efficiently store repeated values'. When the same value repeats, you store it once with the number of repeats. Wikipedia explains it as 'a single value is stored once along with the number of occurrences'.
country = ["India", "India", "India", "India", "Kenya", "Kenya", "Peru",
"Peru", "Peru", "Peru", "Peru", "India"]
# Dictionary encoding: keep each distinct value once, store small numbers.
dictionary = sorted(set(country))
ids = [dictionary.index(c) for c in country]
print("dictionary:", dictionary)
print("ids:", ids)
# Run-length encoding: store each value once with how many times it repeats.
runs = []
for i in ids:
if runs and runs[-1][0] == i:
runs[-1][1] += 1
else:
runs.append([i, 1])
print("runs:", runs)
# Decode to check nothing was lost.
decoded = [dictionary[i] for i, n in runs for _ in range(n)]
print("same as original:", decoded == country)Output
dictionary: ['India', 'Kenya', 'Peru'] ids: [0, 0, 0, 0, 1, 1, 2, 2, 2, 2, 2, 0] runs: [[0, 4], [1, 2], [2, 5], [0, 1]] same as original: True
Twelve words became a three-word dictionary and four pairs of numbers. The last line proves that nothing was lost. Notice that this only works well because the column holds a few values that repeat. That is exactly the kind of column you often find in real data, such as country, plan or yes-or-no flags.
Does column order really shrink data?
Let us test the claim about compression. Python ships with zlib, a general compressor. We build 1,000 rows with an id, a country and a plan. We write the same values two ways, row by row and column by column, and compress both with zlib.
import json, zlib
countries = ["India", "Kenya", "Peru", "Brazil"]
plans = ["free", "pro"]
rows = [
{"id": n, "country": countries[(n // 50) % 4], "plan": plans[(n // 120) % 2]}
for n in range(1000)
]
# Row layout: each record in turn. Column layout: each column in turn.
row_bytes = json.dumps([[r["id"], r["country"], r["plan"]] for r in rows]).encode()
col_bytes = json.dumps([
[r["id"] for r in rows],
[r["country"] for r in rows],
[r["plan"] for r in rows],
]).encode()
print("row layout, raw:", len(row_bytes), "bytes")
print("column layout, raw:", len(col_bytes), "bytes")
print("row layout, zlib:", len(zlib.compress(row_bytes)), "bytes")
print("column layout, zlib:", len(zlib.compress(col_bytes)), "bytes")Output
row layout, raw: 23410 bytes column layout, raw: 21416 bytes row layout, zlib: 2520 bytes column layout, zlib: 2155 bytes
The column layout came out smaller both before and after compression. The gain here is modest, about 15 percent after zlib, because zlib is already good at spotting repeats and our data is very regular. This is a toy: it uses JSON text and zlib, not Parquet's own encodings, so do not read the exact numbers as what Parquet will do. What it shows is the direction. Alike values side by side are easier to squeeze.
What is inside a Parquet file
A Parquet file is not just a long list of columns. A big table is cut into pieces, and each piece has a name. The format spec gives a short glossary, which we repeat here in plain words.
Row group
The spec calls it 'a logical horizontal partitioning of the data into rows'. Think of cutting a long table into slices of rows. Each slice is one row group.
Column chunk
'A chunk of the data for a particular column.' Inside one row group, every column gets its own chunk, and the spec says chunks are 'guaranteed to be contiguous in the file', which means all in one piece.
Page
'Column chunks are divided up into pages.' A page is the smallest piece: the spec calls it 'conceptually an indivisible unit (in terms of compression and encoding)'.
Tip
The spec lists the 'unit of parallelization' for different jobs: for IO it is the column chunk, and for encoding and compression it is the page.
The spec sums up the nesting: 'a file consists of one or more row groups. A row group contains exactly one column chunk per column. Column chunks contain one or more pages.' So the order is file, then row groups, then column chunks, then pages.
The file also begins and ends with a special marker. The format page shows the layout with '4-byte magic number "PAR1"' at the start. Then come the column chunks of the first row group, then those of the second, and so on. After the data comes the file metadata, then 'the 4-byte length in bytes of file metadata (little endian)', and one more 'PAR1' at the very end. A magic number is a fixed tag that tells a program what kind of file it is looking at.
- 1
The file starts with PAR1
Four bytes that say 'this is Parquet'.
- 2
Row group 1, column chunk by column chunk
Column 1 chunk, then column 2 chunk, and so on for every column.
- 3
More row groups
The same pattern repeats for row group 2, 3 and so on, up to the last one.
- 4
File metadata
Written after all the data. The spec says the file metadata 'contains the locations of all the column chunk start locations'.
- 5
Length of the metadata, then PAR1 again
Four bytes that say how long the metadata is, then the closing marker.
Why put the metadata at the end? The spec explains: 'File Metadata is written after the data to allow for single pass writing.' A writer can stream the data out once, then write the map last, when it knows where everything landed.
Readers do the reverse. 'Readers are expected to first read the file metadata to find all the column chunks they are interested in.' So a program jumps to the end, reads the map, and then reads only the column chunks it needs. This is how Parquet makes column reading fast. It never has to scan the whole file.
The schema travels with the data
Open a CSV file and you see text. The Python docs say that when the csv module reads a file, 'Each row read from the csv file is returned as a list of strings. No automatic data type conversion is performed unless the QUOTE_NONNUMERIC format option is specified'. So the age 21 arrives as the text '21', and the program has to guess or be told it is a number.
Parquet keeps this information inside the file. The thrift definition, which is the formal description of the metadata, says the file metadata holds a 'Parquet schema for this file. This schema contains metadata for all the columns.' It also holds the 'Number of rows in this file', the list of row groups, and an optional name of the program that wrote the file. So the file can tell a reader what columns it has, and what type each one is, before reading a single value.
The set of types is small on purpose. The format README says the types are 'intended to be as minimal as possible, with a focus on how the types affect disk storage'. The list is BOOLEAN, INT32, INT64, INT96, FLOAT, DOUBLE, BYTE_ARRAY and FIXED_LEN_BYTE_ARRAY. The Types page notes that 16-bit integers are not stored directly 'since they are covered by 32-bit ints with an efficient encoding'.
Richer types sit on top of these. The README calls them logical types, used 'to extend the types that parquet can be used to store, by specifying how the primitive types should be interpreted'. For example: 'strings are stored with the primitive type BYTE_ARRAY with a STRING annotation'. That annotation is saved in the file metadata too.
Quick check
Where does a Parquet file keep its schema and the location of its column chunks?
Parquet, CSV or JSON?
Parquet is not a replacement for everything. It is a tool for one job: big tables that you read many times. Here is how the choice looks in practice.
Pick Parquet when
The table is large, you often need only some of the columns, you read it many times, and you want to save space. The Parquet site says it is 'supported in many programming languages and analytics tools'.
Pick CSV when
The file is small, a person needs to open it in a text editor or a spreadsheet, or you are sharing it with someone who has no special tools. Wikipedia lists CSV as a row-oriented format, which suits reading or adding one row at a time.
Pick JSON when
The data is nested or each record has a different shape, and a person or a web service must read it. We cover JSON in its own lesson.
Reading Parquet from Python
Python's standard library cannot read Parquet. You need a library. The usual route is pandas, which you met earlier. The pandas docs say of read_parquet: 'This function requires the pyarrow library.' PyArrow is the Python library for Apache Arrow, which Wikipedia describes as 'an in-memory complement to on-disk columnar formats like Parquet'.
Neither library is installed in the environment where this lesson was written, so we do not show real output for them. The lines below are the usage the pandas docs show, copied for you. Run them yourself after installing pandas and pyarrow.
df.to_parquet("df.parquet.gzip", compression="gzip")
pd.read_parquet("df.parquet.gzip")The first line writes a table to a Parquet file. The to_parquet docs list the compression choices: 'snappy', 'gzip', 'brotli', 'lz4' and 'zstd', with 'snappy' as the default, and 'None' for no compression. The second line reads it back into a table. Both are from the pandas documentation.
Parquet lets you read only some columns. The read_parquet docs say of the columns option: 'If not None, only these columns will be read from the file.' PyArrow's read_table has the same option. This is the pandas docs example.
pd.read_parquet(BytesIO(df_parquet_bytes), columns=["bar"])Here 'bar' is one column of a small example table in the docs, and df_parquet_bytes is that table saved to Parquet bytes. The docs show that this gives back only the 'bar' column. This is the column idea at work: the other columns are never loaded.
Tip
The Parquet overview warns that 'not all implementations support the same features of the Parquet format'. If you use more than one tool on the same files, test that they can read each other's output.
Wrapping up
Parquet trades slower writes for smaller files and fast column reads. Next: other data formats.
Sources
- Apache Parquet: Overview
- Apache Parquet: File Format
- Apache Parquet: Metadata
- Apache Parquet: Types
- Apache Parquet: Encodings
- Apache Parquet: Compression
- apache/parquet-format on GitHub: README
- apache/parquet-format on GitHub: parquet.thrift
- pandas: read_parquet
- pandas: DataFrame.to_parquet
- Apache Arrow: pyarrow.parquet.read_table
- Wikipedia: Apache Parquet
- Wikipedia: Data orientation (the page 'Column-oriented DBMS' redirects here)
- Python: csv module
- Python: zlib module
Last updated on 16 November 2026
Learn with me
Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.