esc
  • WorkflowsPages
  • Case studiesPages
  • Machine Learning FoundationsLearn
  • AI EngineeringLearn
  • AI Guardrails & Red TeamingLearn
  • UI/UXLearn
  • Design SystemsLearn
  • Production System DesignLearn
  • Book a callContact
  • Download ResumeContact
Let’s talk

I’ll build the agent for it.

Machine Learning Foundations / Core / 02.12

Data loading: from a file to your program

Before a model can learn, the data has to get from the disk into your program. Learn to load it a piece at a time, in a fair order, and to check what you got.

13 min readFree

From the disk to your program

In the last lessons you learned where data comes from and how it is stored in files such as CSV and Parquet. A file on a disk does nothing by itself. A program has to read it first. That step is called data loading.

It sounds simple: open the file, read it, done. For ten rows it is. But real training data can hold millions of rows. Then a few choices decide whether your program runs smoothly, runs slowly, or crashes. This lesson covers those choices.

Key words

Loading
Reading data from storage, such as a disk, into your program's memory.
Memory (RAM)
The space a program uses while it runs. It is fast, but much smaller than a disk.
Streaming
Reading data a little at a time instead of all at once.
Chunk
A group of rows read together from a file.
Batch
A small group of examples handed to the model together for one step of learning.
Generator
A Python function that hands you one value at a time, and only when you ask for it.
Shuffling
Mixing the rows into a random order, like shuffling a deck of cards.
Epoch
One full pass through all the training data.
Seed
A starting number for a random shuffle. The same seed gives the same shuffle every time.

All at once, or a little at a time?

Think about reading a very long book. You could photocopy every page and carry the whole stack around. Or you could read one page at a time and put it down. The stack is easy to flip through, but it needs a big table. One page at a time needs almost no space.

  • Read it all

    Load the whole file into memory first. This is simple, and it is fast to use afterwards. It works well when the data fits comfortably in memory.

  • Stream it

    Read a few rows, use them, then read the next few. Memory stays small, no matter how big the file is. The price is a little more code.

  • Read it in chunks or batches

    A middle path. Read, say, 4 rows at a time, or 1,000. You keep the memory small and still give the model a group of rows at once.

A good rule: if the file is small, load it all. If you are not sure it fits, stream it.

Generators: load lazily with yield

Python has a neat tool for streaming. It is called a generator. A normal function runs, returns one answer and ends. A generator function uses the word yield instead of return. Each yield hands back one value and then pauses.

The Python glossary explains the pause like this: 'Each yield temporarily suspends processing, remembering the execution state (including local variables and pending try-statements). When the generator iterator resumes, it picks up where it left off (in contrast to functions which start fresh on every invocation).'

Here is a tiny generator. It prints a line each time it makes a number, so you can see when the work really happens.

Python
def count_up(limit):
    for n in range(1, limit + 1):
        print("  making", n)
        yield n


gen = count_up(3)
print("created the generator, nothing ran yet")
print("first:", next(gen))
print("second:", next(gen))

Output

created the generator, nothing ran yet
  making 1
first: 1
  making 2
second: 2

Look at the order. Creating the generator ran nothing. The number 1 was made only when we asked for it with next. Number 3 was never made, because we never asked. This is called lazy loading: do the work only when it is needed.

Does it really save memory? Compare a list of a million numbers with a generator for the same numbers. The size shown is only for the list or generator object itself, not for the numbers inside. Even so, the gap tells the story.

Python
import sys

as_list = [n for n in range(1_000_000)]
as_generator = (n for n in range(1_000_000))

print("list:", sys.getsizeof(as_list), "bytes")
print("generator:", sys.getsizeof(as_generator), "bytes")

Output

list: 8448728 bytes
generator: 216 bytes

The list holds a million slots at once. The generator holds only the recipe for making the next number. That is why a generator can walk through a file of any size.

Read only the columns you need

Rows are one way to cut a table down. Columns are the other. A table may have 40 columns while your model uses 5. Why carry the other 35?

In the Parquet lesson you saw that a column-oriented file lets a program read just the columns it asks for. A CSV file stores whole rows, so a program must still read every line. But once a line is read, you can keep only the fields you need and throw the rest away. The example later in this lesson does that: it keeps three fields as numbers and ignores everything else.

Libraries have a shortcut for this. The pandas function read_csv has an option called usecols. Its docs describe it as the 'Subset of columns to select', and they add: 'Using this parameter results in much faster parsing time and lower memory usage.'

Text
# From the pandas read_csv docs: usecols takes a list of names or positions.
# A valid list-like usecols parameter would be [0, 1, 2] or ['foo', 'bar', 'baz'].
# (Shown from the docs. Not run in this lesson, pandas is not installed.)

Shuffling: why the order matters

Files often come in a neat order. They may be sorted by date, by name, or by the answer you want the model to learn. That order is a trap. Imagine a teacher who spends the first half of the term on one topic and the second half on another.

A model trained on batches can fall into the same trap. If a file is sorted so that all the 'no' answers come first, the first batches show the model only 'no'. Shuffling mixes the rows so each batch holds a fair mix. The libraries agree it is worth building in: PyTorch's DataLoader has a shuffle option 'set to True to have the data reshuffled at every epoch', and the TensorFlow guide has a whole shuffle step for its pipelines.

Python's random module can shuffle a list. Its docs say, plainly: 'Shuffle the sequence x in place.' In place means the list itself is changed, not a copy.

Quick check

A CSV file is sorted so that all the rows with the answer 0 come first, then all the rows with the answer 1. You load it in batches of 4 without shuffling. What will the first batch look like?

Worked example: batches, then a shuffle

Now let us build it. We will write a 10-row CSV file in a temporary folder, read it lazily with a generator, group the rows in batches of 4, and then shuffle with a seed. Only Python's own tools are used.

  1. 1

    Write the file

    Ten students. Each row has an id, the study hours, and whether they passed (0 or 1). The file is sorted: the first five did not pass, the last five did. That is the trap from the last section.

  2. 2

    Read it with a generator

    One function reads the file row by row and yields each row as numbers. Only one line is in memory at a time.

  3. 3

    Group rows into batches

    A second generator takes any stream of rows and yields lists of 4. The last batch may be shorter, because 10 does not divide by 4.

  4. 4

    Shuffle with a seed

    Load the rows into a list, shuffle it, and batch again. Run it twice with the same seed, then once with another.

First, writing the file and looking inside it.

Python
import csv
import os
import tempfile

folder = tempfile.mkdtemp()
path = os.path.join(folder, "students.csv")

with open(path, "w", newline="") as f:
    writer = csv.writer(f)
    writer.writerow(["id", "hours", "passed"])
    for i in range(1, 11):
        writer.writerow([i, i * 2, 1 if i > 5 else 0])

with open(path) as f:
    print(f.read())

Output

id,hours,passed
1,2,0
2,4,0
3,6,0
4,8,0
5,10,0
6,12,1
7,14,1
8,16,1
9,18,1
10,20,1

Now the full program. It writes the same file again, then loads it. Read the two small generator functions first. They are the heart of it.

Python
import csv
import os
import tempfile

folder = tempfile.mkdtemp()
path = os.path.join(folder, "students.csv")
with open(path, "w", newline="") as f:
    writer = csv.writer(f)
    writer.writerow(["id", "hours", "passed"])
    for i in range(1, 11):
        writer.writerow([i, i * 2, 1 if i > 5 else 0])


def read_rows(path):
    """Yield one row at a time. Only one line is in memory."""
    with open(path, newline="") as f:
        for row in csv.DictReader(f):
            yield {"id": int(row["id"]), "hours": int(row["hours"]), "passed": int(row["passed"])}


def batches(rows, size):
    """Group any stream of rows into lists of `size`."""
    batch = []
    for row in rows:
        batch.append(row)
        if len(batch) == size:
            yield batch
            batch = []
    if batch:
        yield batch


def show(batch_stream):
    for n, batch in enumerate(batch_stream, start=1):
        print("batch", n, "ids", [r["id"] for r in batch], "passed", [r["passed"] for r in batch])


print("In file order:")
show(batches(read_rows(path), 4))

rows = list(read_rows(path))  # 10 rows is tiny, so keeping them all is fine
print("rows loaded:", len(rows), "| first row:", rows[0])
print("types:", {k: type(v).__name__ for k, v in rows[0].items()})

import random

for seed in (42, 42, 7):
    random.Random(seed).shuffle(rows)
    print("Shuffled with seed", seed)
    show(batches(rows, 4))
    rows = list(read_rows(path))

Output

In file order:
batch 1 ids [1, 2, 3, 4] passed [0, 0, 0, 0]
batch 2 ids [5, 6, 7, 8] passed [0, 1, 1, 1]
batch 3 ids [9, 10] passed [1, 1]
rows loaded: 10 | first row: {'id': 1, 'hours': 2, 'passed': 0}
types: {'id': 'int', 'hours': 'int', 'passed': 'int'}
Shuffled with seed 42
batch 1 ids [8, 4, 3, 9] passed [1, 0, 0, 1]
batch 2 ids [6, 7, 10, 5] passed [1, 1, 1, 0]
batch 3 ids [1, 2] passed [0, 0]
Shuffled with seed 42
batch 1 ids [8, 4, 3, 9] passed [1, 0, 0, 1]
batch 2 ids [6, 7, 10, 5] passed [1, 1, 1, 0]
batch 3 ids [1, 2] passed [0, 0]
Shuffled with seed 7
batch 1 ids [9, 4, 2, 5] passed [1, 0, 0, 0]
batch 2 ids [8, 1, 10, 7] passed [1, 0, 1, 1]
batch 3 ids [3, 6] passed [0, 1]

Read the output slowly. In file order, batch 1 is all zeros and batch 3 is all ones. Three batches, three very different views of the world. After a shuffle, the batches hold a mix, though a small sample can still be lopsided (batch 3 with seed 42 is all zeros). That is normal with only 10 rows.

Also see that seed 42 gave the same order both times, and seed 7 gave a different one. That is the next idea.

Tip

The shuffle above needs the rows in memory, which is fine for 10 rows. For a file too big for memory, TensorFlow's guide describes a shortcut: its shuffle 'maintains a fixed-size buffer and chooses the next element uniformly at random from that buffer.' It is a partial shuffle, because only rows inside the buffer can swap places.

Check what you loaded

Loading without looking is how bugs sneak in. A file can load without any error and still be wrong: half the rows missing, numbers read as text, a header row counted as data. Always take a quick look before you train.

  • Count the rows

    Does the number match what you expect? Our example printed 'rows loaded: 10', and the file has 10 data rows plus one header.

  • Check the types

    Is each column the type you think? In the example, we turned each field into a whole number with int, and printed the types to prove it.

  • Peek at the first row

    A glance at one real row catches most mistakes. The example printed the first row and it looked right.

Make it repeatable

Machine learning is full of experiments. You change one thing, run again, and compare. That only works if everything else stays the same. A random shuffle breaks this, unless you fix it with a seed.

The Python docs say: 'By reusing a seed value, the same sequence should be reproducible from run to run as long as multiple threads are not running.' Our worked example showed it: seed 42 twice gave the same batches, and seed 7 gave a different set.

One more detail from the docs. The functions in random share one hidden generator, but 'you can instantiate your own instances of Random to get generators that don't share state.' That is why the example used random.Random(seed). The shuffle belongs to this one step and nothing else can disturb it.

  1. 1

    Fix a seed

    Choose any number, such as 42, and write it in your code, not in your head.

  2. 2

    Record the data version

    Note which file you loaded: its name, the date, or a version number. If the file changes, the results will too.

  3. 3

    Write both down with your results

    A result with no seed and no data version is hard to trust, even for you, a month later.

How libraries do it

You will rarely write batching code by hand. Popular libraries have it built in. They are not installed here, so the lines below are copied from their docs and were not run. Notice that they use the same ideas you just built.

pandas can read a CSV in chunks. In the docs, chunksize is the 'Number of lines to read from the file per chunk.' Passing it makes read_csv return a TextFileReader object for iteration, so you loop over it chunk by chunk.

PyTorch's DataLoader is the main way to feed data to a model. The docs give its signature, and describe batch_size as 'how many samples per batch to load (default: 1)'.

Text
# From the PyTorch docs (torch.utils.data). Not run here.
DataLoader(dataset, batch_size=1, shuffle=False, sampler=None,
           batch_sampler=None, num_workers=0, collate_fn=None,
           pin_memory=False, drop_last=False, timeout=0,
           worker_init_fn=None, *, prefetch_factor=2,
           persistent_workers=False)

The PyTorch docs also split datasets into two types. A map-style dataset can fetch the item at any position. An iterable-style dataset hands over items one after another, like our generators.

TensorFlow's tf.data builds pipelines from simple steps. The guide says: 'The simplest form of batching stacks n consecutive elements of a dataset into a single element.' That is the Dataset.batch step, the same job as our batches function.

Quick check

You run the same shuffle twice with seed 42, and then once with seed 7. Which result is correct?

What to remember

  • Small data: load it all. Big data: stream it

    Use a generator with yield to read lazily, a row or a batch at a time. Memory stays small.

  • Take only what you need

    Pick the columns first (usecols in pandas, or columns in Parquet), then the rows. Less data loaded means a faster program.

  • Shuffle, seed, and check

    Shuffle so batches are a fair mix. Fix the seed and note the data version so you can repeat it. Count rows, check types and peek before you train.

Next: splitting data into training and test sets.

Sources

How is this lesson?

Last updated on 16 November 2026

Learn with me

Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.