esc
  • WorkflowsPages
  • Case studiesPages
  • Machine Learning FoundationsLearn
  • AI EngineeringLearn
  • AI Guardrails & Red TeamingLearn
  • UI/UXLearn
  • Design SystemsLearn
  • Production System DesignLearn
  • Book a callContact
  • Download ResumeContact
Let’s talk

I’ll build the agent for it.

Machine Learning Foundations / Beginner / 01.43

APIs as a data source

An API is a waiter between your program and a service's data. Learn how to ask politely, read the answer and turn it into a table.

14 min readFree

Why APIs matter for machine learning

In the last lesson we met an API in one short paragraph. IBM called it 'a contract that enables software to access data or invoke functionality exposed by other software.' This lesson shows how to ask an API for data, read what comes back, and turn it into a table a model can learn from.

Why bother? Think about a restaurant. You sit at a table and you cannot walk into the kitchen. You tell the waiter what you want. The waiter carries your order to the kitchen and brings the food back. You never see how it was cooked. An API is the waiter. Your program is the customer, the service that holds the data is the kitchen, and the menu is the list of things you are allowed to ask for.

MDN, the web developer documentation from Mozilla, puts it this way: an API is 'a set of features and rules that exist inside a software program (the application) enabling interaction with it through software - as opposed to a human user interface.'

For machine learning this is very handy. A weather service, a map service or a house-listing service can hand you fresh data on request, already tidy, in a form a program can read.

Key words

API
Short for application programming interface. A set of rules that lets one program ask another program for data or for an action.
Client
The program that asks. In this lesson, that is your Python script. IBM says 'the application submitting the request is the client, and the server provides the response.'
JSON
A plain-text way to write data, with names and values. Almost every web API answers in it.
API key
A secret code that tells the service who is asking.
Rate limit
A cap on how many requests you may send in a given time.

How a web API is asked: REST basics

Most of the APIs you will meet are web APIs. They work over HTTP, the same rules that carry web pages to your browser. Many of them follow a style called REST. IBM defines a REST API as one 'that conforms to the design principles of the representational state transfer (REST) architectural style', and says REST was 'defined in 2000 by computer scientist Dr. Roy Fielding in his doctoral dissertation'. You only need the parts of a request.

  • The endpoint (the address)

    Each kind of thing has its own web address. IBM says that in a REST API 'the only information that the client application should know is the URI of the requested resource'. A URI is just the address. For example, https://api.example.com/v1/houses might mean 'the list of houses'. This address is made up for this lesson.

  • The method (the verb)

    This says what you want to do. MDN says 'The GET method requests a representation of the specified resource', and that GET requests 'should only retrieve data'. POST 'submits an entity to the specified resource, often causing a change in state or side effects on the server.' For collecting data you will use GET almost all the time. POST is for sending something in, such as a search that is too big for the address.

  • The headers (the labels)

    Extra notes on the request, such as which format you want back or who you are. IBM says request headers and parameters carry 'important identifier information such as metadata, authorizations, uniform resource identifiers (URIs), caching, cookies and more.'

Here is the whole picture as one made-up request. The first line holds the method and the address. The lines below are headers. The Authorization line is where a key goes, which we come back to later.

Text
GET /v1/houses?city=San+Diego&min_bedrooms=2 HTTP/1.1
Host: api.example.com
Accept: application/json
Authorization: Bearer YOUR_KEY_HERE

IBM lists what a call contains: 'a request verb, headers and sometimes, a request body. Together, these specify what is being requested, and how.' A request body is extra content sent along with the request, which is mostly used with POST.

Quick check

You only want to read a list of houses from an API. Which method fits best?

Query parameters: narrowing the question

You rarely want everything. You want houses in one city, or with two bedrooms. The usual way to say so is a query string: a question mark at the end of the address, then name=value pairs joined by ampersands. The pairs are called query parameters.

Writing these by hand goes wrong easily. A space or an ampersand inside a value breaks the address. Python's standard library does the careful part for you. The urllib.parse docs say urlencode will 'Convert a mapping object or a sequence of two-element tuples' to 'a percent-encoded ASCII text string'. A mapping is a Python dictionary. Percent-encoded means unsafe characters are swapped for safe codes. No request is sent; it only builds the address.

Python
from urllib.parse import urlencode

base = "https://api.example.com/v1/houses"
params = {"city": "San Diego", "min_bedrooms": 2, "page_size": 50}
url = base + "?" + urlencode(params)
print(url)

Output

https://api.example.com/v1/houses?city=San+Diego&min_bedrooms=2&page_size=50

See how 'San Diego' became San+Diego. The space could not sit in an address as it was, so it was encoded.

Reading the answer: JSON

The server answers with a status code and, usually, a body. For data APIs the body is almost always JSON. RFC 8259, the official standard, calls JSON 'a text format for the serialization of structured data', and says it 'can represent four primitive types (strings, numbers, booleans, and null) and two structured types (objects and arrays).'

The two structured types matter most. In the RFC's words, 'An object is an unordered collection of zero or more name/value pairs' and 'An array is an ordered sequence of zero or more values.' An object is written in curly brackets. An array is written in square brackets, with commas between the values. Think of an object as one house, and an array as a list of houses.

Python reads JSON with its json module. The docs say json.loads will 'deserialize s (a str, bytes or bytearray instance containing a JSON document) to a Python object'. They also list how JSON turns into Python: an object becomes a dict, an array a list, a number an int or a float, and null becomes None. In the code below, the response is a string we typed in ourselves, so nothing leaves your computer.

Python
import json

body = '{"results": [{"id": 101, "size_m2": 80, "bedrooms": 2, "price_usd": 240000, "agent_note": null}, {"id": 102, "size_m2": 120, "bedrooms": 3, "price_usd": 360000, "agent_note": "garden"}], "next_page": "p2"}'

data = json.loads(body)
print(type(data).__name__, list(data.keys()))
print(type(data["results"]).__name__, len(data["results"]))
first = data["results"][0]
print(first["price_usd"], type(first["price_usd"]).__name__)
print(first["agent_note"])

Output

dict ['results', 'next_page']
list 2
240000 int
None

The prices came back as real numbers, not text, unlike a CSV file. The agent_note was null, so Python gave us None, which means 'nothing here'. Missing values are common in API data.

Tip

The Python docs also warn: 'Be cautious when parsing JSON data from untrusted sources.' Limiting the size of what you parse is recommended.

Status codes: did it work?

Every response starts with a three-digit status code. MDN groups them by the first digit: 'Successful responses (200 – 299)', 'Redirection messages (300 – 399)', 'Client error responses (400 – 499)' and 'Server error responses (500 – 599)'. A client error means the fault is in your request; a server error means it is on their side.

  • 200 OK and 201 Created

    MDN says 200 means 'The request succeeded'. 201 means 'The request succeeded, and a new resource was created as a result', which you typically see after a POST.

  • 401, 403 and 404

    MDN says 401 'semantically this response means "unauthenticated"', so the service does not know who you are. 403 means 'The client does not have access rights to the content', and the server knows who you are but refuses. 404 means 'The server cannot find the requested resource'.

  • 429 and 5xx

    MDN says 429 'indicates the client has sent too many requests in a given amount of time'. A 500 is 'a situation it does not know how to handle'. A 503 means 'The server is not ready to handle the request'.

Your code should look at the status code before it reads the body. A 404 page is not data.

Quick check

An API replies 401. What does MDN say this means?

API keys and keeping them secret

Many services want to know who is calling. They give you an API key, which is a long secret string. You send it with each request, often in a header like the Authorization line you saw earlier. Google's documentation says a standard API key lets you 'associate a request with a project for billing and quota purposes'. So a key is also how the service counts your use.

That is why a leaked key hurts. Google warns that 'Unrestricted API keys are insecure', and recommends limiting a key to certain APIs and to certain websites, IP addresses or applications. It says: 'Copy your key string and keep it secure.'

Here are plain habits that follow from that. Never write a key into a file that you share or upload, such as code on a public site. Keep it in an environment variable instead, which is a setting your computer holds outside your code. If a key leaks, cancel it at once and make a new one. The code below reads a key the safe way.

Python
import os

key = os.environ.get("HOUSES_API_KEY")
if key is None:
    print("No key set. Stop here, do not guess one.")
else:
    headers = {"Authorization": "Bearer " + key}
    print("header ready, key length:", len(key))

Output

No key set. Stop here, do not guess one.

With no key set, the script stops on purpose. With a key, it prints only the key's length. Printing the key would leave it in your screen history.

Tip

Treat a key like a house key. You would not tape it to your front door, and you would not hand copies to strangers.

Rate limits and retries

A service that answers everyone cannot let one program ask a million times a minute. So it sets a rate limit. MDN describes the 429 status code as a way of 'asking the client to slow down the rate of requests', commonly called 'rate limiting'. It adds that limits are 'typically ... based on a client's IP but can be specific to users or authorized applications if requests are authenticated'.

A well-made service tells you how long to wait. MDN says a Retry-After header 'may be included' with a 429 and 'indicate how long a client should wait before making the request again'. Its value is 'a non-negative decimal integer indicating the seconds to delay after the response is received', or a date. MDN also warns that 'Support for the Retry-After header on both clients and servers is still inconsistent', so do not count on it always being there.

When the header is missing, a common plan is backoff: wait a little, try again, and wait twice as long each time, up to a cap. Doubling is a programmers' habit, not a rule from the standards. The code below only works out the waiting times. It sends nothing.

Python
def backoff_delays(retries, base=1, cap=30):
    return [min(cap, base * 2 ** n) for n in range(retries)]

print(backoff_delays(7))
print(sum(backoff_delays(4)), "seconds of waiting in total for 4 retries")

Output

[1, 2, 4, 8, 16, 30, 30]
15 seconds of waiting in total for 4 retries
  1. 1

    1. Send the request

    Look at the status code first.

  2. 2

    2. On 200, use the data

    Read the JSON and move on.

  3. 3

    3. On 429 or 503, wait

    If a Retry-After value is there, wait that many seconds. If not, use the next number from the backoff list.

  4. 4

    4. Try again, but not forever

    Stop after a fixed number of tries, and report the error. Never loop without end.

  5. 5

    5. On 400, 401, 403 or 404, stop

    Retrying will not fix a wrong address or a missing key. Fix the request instead.

Keep your request count small: fetch once, save the result to a file, and experiment on the file.

Pagination: data that arrives in pages

A service might hold a million houses. It will not send them all at once. Google's API design guide explains: 'collections can often be arbitrarily sized, and also often grow over time, increasing lookup time as well as the size of the responses being sent over the wire. Therefore, it is important that collections be paginated.' Pagination means you get one page of results at a time.

In Google's style you ask for a page size, and each reply includes a token for the next page. The guide says: 'If this field is omitted, there are no subsequent pages.' Other services use other names, so read the docs of the API you use. Below, two made-up pages stand in for a real service.

Python
import json

pages = {
    "first": '{"results": [{"id": 1, "price_usd": 150000}, {"id": 2, "price_usd": 240000}], "next_page": "p2"}',
    "p2": '{"results": [{"id": 3, "price_usd": 360000}], "next_page": null}',
}

rows = []
token = "first"
while token is not None:
    page = json.loads(pages[token])
    rows.extend(page["results"])
    print("got", len(page["results"]), "rows, next:", page["next_page"])
    token = page["next_page"]

print("total rows:", len(rows))

Output

got 2 rows, next: p2
got 1 rows, next: None
total rows: 3

The loop collects every page into one list. Notice how it stops: when the next page is None, there is nothing more to fetch.

Turning API results into a table

A model learns from a table: one row per example, one column per feature, plus a label. API results are nested lists and dictionaries, so there is one more step. We pick the fields we want and make one flat row for each item.

  1. 1

    1. Choose the columns

    We want size_m2, bedrooms and price_usd. price_usd is the label.

  2. 2

    2. Leave out what you do not need

    The response also has an owner_email. That is personal data, and we do not need it to predict a price, so we never copy it. Collect less, and there is less to protect.

  3. 3

    3. Handle missing values

    One house has bedrooms set to null. We skip that row and count it, so we know how much we lost.

  4. 4

    4. Write out the rows

    Print or save the table in CSV form, which you read in the last lesson.

Python
import json

body = '{"results": [{"id": 101, "size_m2": 80, "bedrooms": 2, "price_usd": 240000, "owner_email": "sam@example.com"}, {"id": 102, "size_m2": 120, "bedrooms": null, "price_usd": 360000, "owner_email": "lee@example.com"}, {"id": 103, "size_m2": 50, "bedrooms": 1, "price_usd": 150000, "owner_email": "kim@example.com"}]}'

keep = ["size_m2", "bedrooms", "price_usd"]
table = []
skipped = 0
for item in json.loads(body)["results"]:
    row = {name: item.get(name) for name in keep}
    if None in row.values():
        skipped += 1
        continue
    table.append(row)

print(",".join(keep))
for row in table:
    print(",".join(str(row[name]) for name in keep))
print("skipped rows:", skipped)

Output

size_m2,bedrooms,price_usd
80,2,240000
50,1,150000
skipped rows: 1

We kept two rows and skipped one. Always count the skips. If you drop half your data, the model describes only the half that was complete.

Terms of use and personal data

A public API is not a free-for-all. Almost every service publishes terms of use, which say what you may do with its data. Some forbid storing it for long, some forbid using it to train models, and some ask you to give credit. Read the terms before you collect anything, and keep a note of where each dataset came from and which terms applied. This is a habit, not legal advice.

Personal data needs extra care. If a response holds names, emails, locations or anything else that points to a real person, ask whether you need it at all. If you do not, drop it as early as you can, as the last example did. An API only delivers data. It does not promise the data is fair or complete.

Quick check

Which step is the best way to protect people in your dataset?

Putting it together

The whole path: build the request, send it with your key, check the status code, retry politely, read the JSON, follow the pages, drop what you do not need, and write out a table.

Tip

When an API call fails, read the status code first. It is the quickest clue to whether the fault is in your request or on the server.

Next: mobile apps as a data source. Phones carry sensors, usage records and permissions, and they bring new questions about who owns the data and who agreed to share it.

Sources

How is this lesson?

Last updated on 16 November 2026

Learn with me

Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.