esc
  • WorkflowsPages
  • Case studiesPages
  • Machine Learning FoundationsLearn
  • AI EngineeringLearn
  • AI Guardrails & Red TeamingLearn
  • UI/UXLearn
  • Design SystemsLearn
  • Production System DesignLearn
  • Book a callContact
  • Download ResumeContact
Let’s talk

I’ll build the agent for it.

Machine Learning Foundations / Beginner / 01.42

The internet as a data source

The web holds more text, prices and pictures than any team could type by hand. Here is how a page reaches you, how a program reads it, and how to do that politely.

15 min readFree

Why the web matters to machine learning

In the last lesson we met the web as one data source among many. Now we slow down and look at it properly. Think of the web as a huge library with no librarian. The shelves are open, but nobody checks what is on them or whether you may copy it. You bring your own manners.

Wikipedia says web pages 'frequently contain a wealth of data in text form'. Uses include market research, price comparison and artificial intelligence. It adds a warning: 'most web pages are designed for human end-users and not for ease of automated use.' So we must learn how a page is delivered, find the data inside it, and do it with care.

Key words

Client
The device and browser that ask for a page, such as your phone.
Server
A computer that stores web pages and hands them out when asked.
URL
A web address. MDN says URLs 'define the locations of unique resources on the internet'.
HTTP
The rules a client and server use to talk: 'a protocol for fetching resources such as HTML documents' (MDN).
Web scraping
Using a program to pull data out of web pages.

How a page reaches you

Picture a restaurant. You are the client. The kitchen is the server. The waiter carries your order in and your meal out.

MDN describes it the same way. Clients and servers 'communicate by exchanging individual messages'. The messages the client sends are called requests, and the messages the server sends back are called responses. HTTP is a client-server protocol, so, in MDN's words, requests are 'initiated by the recipient, usually the Web browser'. The server never pushes a page on you. You ask first.

The address: a URL

Every request starts with an address. MDN shows https://developer.mozilla.org/en-US/ and splits it into parts. The protocol, https, is the language used to send the request. The domain name, developer.mozilla.org, is 'the top-level location of the server you are connecting to'. The part after it, the path, says which page you want.

Python can split a URL for you with urllib.parse, a module for 'parse URLs into components'. This sample uses a made-up shop that does not exist. Nothing is sent over the internet. We only take the address apart. The query is the part after the question mark and the fragment is the part after the hash sign.

Python
from urllib.parse import urlsplit

url = "https://shop.example.com/books/page2.html?sort=price#top"
parts = urlsplit(url)
print("scheme  :", parts.scheme)
print("host    :", parts.netloc)
print("path    :", parts.path)
print("query   :", parts.query)
print("fragment:", parts.fragment)

Output

scheme  : https
host    : shop.example.com
path    : /books/page2.html
query   : sort=price
fragment: top

The whole trip, step by step

MDN lists what happens when you type a web address and press Enter.

  1. 1

    1. Look up the address

    The browser asks a DNS server for the real address of the server. MDN says the Domain Name System 'uses special servers that match up a web address you type into your browser (like mozilla.org) to the website's real (IP) address'.

  2. 2

    2. Send a request

    The browser sends an HTTP request message, asking the server to send a copy of the website. All data between client and server travels over your internet connection using TCP/IP.

  3. 3

    3. Get a status and the files

    If the server approves, it sends back a '200 OK' message, then starts sending the website's files as a series of small chunks called packets.

  4. 4

    4. Build the page

    The browser assembles the chunks into a complete web page and shows it to you.

A request is short. MDN's example for the MDN home page is a single line, GET /en-US/ HTTP/2, followed by a Host line naming the site. GET is the verb that means 'give me this'. The response starts with a line such as HTTP/2 200, then headers, which are extra facts about the answer, and then the body.

One more point matters for scrapers. MDN says that a 'user-agent' can be anything: 'Most of the time the user-agent is a Web browser, but it can be anything, for example, a robot that crawls the Web to populate and maintain a search engine index.' To a server, your Python program is just another client.

Status codes: the server's answer

Every response carries a three-digit status code. MDN groups them in five classes: 100 to 199 are informational, 200 to 299 are successful, 300 to 399 are redirects, 400 to 499 are client errors, and 500 to 599 are server errors. A program must check this code before it trusts the body.

  • 200 OK

    MDN: 'The request succeeded.' For a GET request, the page you asked for is in the body. This is the answer you want.

  • 301 Moved Permanently, and 404 Not Found

    For 301, 'The URL of the requested resource has been changed permanently. The new URL is given in the response.' For 404, 'The server cannot find the requested resource.'

  • 403, 429 and 503

    403 Forbidden: the server is 'refusing to give the requested resource'. 429 Too Many Requests: the client has sent too many requests in a given amount of time. 503 Service Unavailable: the server is 'not ready to handle the request', often down for maintenance or overloaded.

Tip

A 429 is the server saying 'slow down'. MDN says it may come with a Retry-After header telling the client how long to wait before asking again. Treat it as a polite request, and obey it.

Quick check

Your program gets status 404 for a page. What should it do?

What is inside the page: HTML

The body of a successful response is usually HTML. MDN says HTML marks up text, images and other content for display in a browser, using elements. An element is set off from other text by tags, 'which consist of the element name surrounded by < and >'. Think of a label on a box that says what is inside.

Getting data out of a page

The simplest way is text pattern matching: using 'the UNIX grep command or regular expression-matching facilities of programming languages' to find text that matches a pattern. Another is to control a real browser with a tool such as Selenium or Playwright, which suits 'dynamic sites'. A third is to parse the HTML itself, and that is what we do next.

Python has a parser built in. The html.parser module defines a class called HTMLParser, 'the basis for parsing text files formatted in HTML'. You feed it HTML, and it calls your methods when it meets a start tag, an end tag or some text. Its docs say you subclass it and override those methods. For a start tag, 'The attrs argument is a list of (name, value) pairs'. Our program below uses that.

A first parser, step by step

We will take a small page from our pretend book shop, held in a string so nothing needs the network. The goal is to pull out two things: the prices and the link to the next page.

  1. 1

    1. Make a parser class

    Subclass HTMLParser. Keep a list for prices and a list for links, and a flag that says whether we are inside a price.

  2. 2

    2. Watch start tags

    In handle_starttag, turn attrs into a dictionary. If the tag is a span with class price, raise the flag. If it is an a tag with an href, save the link.

  3. 3

    3. Read text only when the flag is up

    In handle_data, if we are inside a price, turn the text into a number and store it. In handle_endtag, lower the flag.

  4. 4

    4. Feed it the page

    Call parser.feed(page) and print what was collected.

Python
from html.parser import HTMLParser

page = """<html><head><title>Book shop</title></head>
<body>
<h1>Bestsellers</h1>
<ul>
  <li class="book">Learning Python <span class="price">24.99</span></li>
  <li class="book">Data Basics <span class="price">18.50</span></li>
</ul>
<a href="/books/page2.html">Next page</a>
</body></html>"""

class PriceParser(HTMLParser):
    def __init__(self):
        super().__init__()
        self.in_price = False
        self.prices = []
        self.links = []

    def handle_starttag(self, tag, attrs):
        attrs = dict(attrs)
        if tag == "span" and attrs.get("class") == "price":
            self.in_price = True
        if tag == "a" and "href" in attrs:
            self.links.append(attrs["href"])

    def handle_endtag(self, tag):
        if tag == "span":
            self.in_price = False

    def handle_data(self, data):
        if self.in_price:
            self.prices.append(float(data))

parser = PriceParser()
parser.feed(page)
print("prices:", parser.prices)
print("links :", parser.links)

Output

prices: [24.99, 18.5]
links : ['/books/page2.html']

Tip

Real pages are messy, and the Python docs warn about this parser: it 'does not check that end tags match start tags'. Always print a few results and look at them with your own eyes before you trust a scraper on a thousand pages.

Doing it responsibly

A program can read a thousand pages in the time you read one. That power comes with duties. Think of a guest in someone's house: you ask first and take only what you need.

  • Read the rules first

    Check robots.txt and the site's terms of use before you copy anything. If the site has an API, prefer it.

  • Take only what is allowed and needed

    Copyright, personal data and the site's own terms all limit what you may keep and use.

  • Go slowly and be honest

    Pause between requests, stop when told to slow down, and do not pretend to be someone else.

robots.txt: the sign on the door

Sites publish their wishes for crawlers in a small file. RFC 9309 says the rules 'MUST be accessible in a file named "/robots.txt" (all lowercase) in the top-level path of the service'. So for https://shop.example.com the file lives at https://shop.example.com/robots.txt. RFC 9309 calls them rules 'that crawlers are requested to honor', and adds: 'These rules are not a form of access authorization.'

Google's guide says the same from the site owner's side: 'The instructions in robots.txt files cannot enforce crawler behavior to your site; it's up to the crawler to obey them.' And it says a robots.txt file is used 'mainly to avoid overloading your site with requests'. In short, it is a sign on the door, not a lock. A good visitor reads the sign.

The file is simple. A line User-agent names which crawlers a group of rules is for. Disallow and Allow lines then say which paths are off limits or open. Python's urllib.robotparser answers the question 'may this crawler fetch this URL?'. Its can_fetch method 'returns True if the useragent is allowed to fetch the url'. We give it a made-up file with parse(), so nothing goes over the network.

Python
from urllib.robotparser import RobotFileParser

rules = """User-agent: *
Disallow: /private/
Allow: /

User-agent: badbot
Disallow: /
""".splitlines()

rp = RobotFileParser()
rp.parse(rules)

site = "https://shop.example.com"
for agent, path in [("my-course-bot", "/books/"), ("my-course-bot", "/private/notes.html"), ("badbot", "/books/")]:
    print(agent, path, "->", rp.can_fetch(agent, site + path))

Output

my-course-bot /books/ -> True
my-course-bot /private/notes.html -> False
badbot /books/ -> False

RFC 9309 also says what a crawler should do when the file cannot be fetched. If the server answers with a status in the 400 to 499 range, the file is 'unavailable', and 'the crawler MAY access any resources on the server'. If the file is unreachable because of server or network errors (status 500 to 599), then 'the crawler MUST assume complete disallow'.

Quick check

A site's robots.txt says 'Disallow: /private/'. Your program can still download /private/notes.html without a password. Is that okay?

Terms, copyright and personal data

robots.txt is not the only rule. Wikipedia says: 'The legality of web scraping varies across the world.' It adds that, in general, web scraping 'may be against the terms of service of some websites, but the enforceability of these terms is unclear.' Read a site's terms, and for real projects ask a lawyer. This lesson is not legal advice.

Copyright is the second rule. In the United States, Wikipedia notes, 'while outright duplication of original expression will in many cases be illegal', courts ruled in a famous case 'that duplication of facts is allowable'. Other countries have other rules.

Personal data is the third. Names, emails and photos of real people are not yours just because they are visible. Europe's GDPR sets principles for handling personal data. Article 5 says it must be 'adequate, relevant and limited to what is necessary in relation to the purposes for which they are processed', and 'collected for specified, explicit and legitimate purposes'. So collect only what you need, and leave out anything that points to a real person.

Be polite: slow down

Many requests in a short time can look like an attack, so sites defend themselves. Wikipedia says administrators can use various measures 'to stop or slow a bot', such as blocking an IP address or 'monitoring excess traffic', and that a CAPTCHA test can be triggered when a bot is 'making too many requests in a short time'. A server may also answer 429, as we saw. The kind way is to pause.

Python's robotparser can read a site's Crawl-delay line with crawl_delay(). This sample uses a fake fetch, so no real site is touched. It shows the shape of a polite loop.

Python
import time

def fake_fetch(url):
    # Stands in for a real request, so this sample never touches the network.
    return 200

for url in ["/books/1", "/books/2", "/books/3"]:
    print(url, fake_fetch(url))
    time.sleep(0.5)  # be polite: pause between requests
print("done")

Output

/books/1 200
/books/2 200
/books/3 200
done

Tip

A short checklist before you scrape: 1) Is there an API? 2) What does robots.txt say? 3) What do the terms say? 4) Am I collecting personal data? 5) Do I pause between requests and stop on a 429?

Traps to watch for

Getting the page is half the work. Here are traps that catch beginners, even when they are polite.

The data is not in the HTML

MDN explains two ways a page gets its content. With server-side rendering, the server generates the HTML and sends it. With client-side rendering, 'the client generates the HTML content using JavaScript'. In the second case the plain download can be an almost empty shell. A scraper that reads only the HTML sees no data, because the data shows up later, when a browser runs the script. The simplest check is to look at what you actually downloaded.

The sample below is a page whose prices would be drawn by a script. The parser skips script text, as a plain reader would, and finds nothing visible.

Python
from html.parser import HTMLParser

# What a plain download of a JavaScript-built page might contain.
raw = """<html><body>
<div id="app"></div>
<script>renderPrices();</script>
</body></html>"""

class Text(HTMLParser):
    def __init__(self):
        super().__init__()
        self.skip = False
        self.words = []
    def handle_starttag(self, tag, attrs):
        self.skip = tag == "script"
    def handle_endtag(self, tag):
        self.skip = False
    def handle_data(self, data):
        if not self.skip and data.strip():
            self.words.append(data.strip())

t = Text()
t.feed(raw)
print("visible text found:", t.words)

Output

visible text found: []

Pages change and repeat

A web page is not a fixed file. Wikipedia notes that 'bots rely on consistency in the front-end code of a target website', and that sites sometimes add 'small variations' to the HTML around important data. If the site changes its layout, your parser quietly returns nothing, or the wrong thing. Save the date you collected each page.

The web also repeats itself. Google's course lists 'Duplicate examples' as a cause of unreliable data and gives the example 'a server mistakenly uploaded the same log entries twice.' If you train on duplicates, the model counts them twice. The sample removes repeated rows.

Python
rows = [
    ("Learning Python", 24.99),
    ("Data Basics", 18.50),
    ("Learning Python", 24.99),
    ("learning python ", 24.99),
]
seen = set()
unique = []
for title, price in rows:
    key = (title.strip().lower(), price)
    if key not in seen:
        seen.add(key)
        unique.append((title, price))
print("rows in :", len(rows))
print("rows out:", len(unique))
print(unique)

Output

rows in : 4
rows out: 2
[('Learning Python', 24.99), ('Data Basics', 18.5)]

Whose voices are on the web?

Ask one more question before you train: who wrote this? Google's course asks whether data is 'properly filtered for your problem', for example whether search queries from bots belong in a dataset meant for humans. The web holds what people chose to publish, on the sites a crawler happened to reach, so a model trained on it learns from that slice only. Write down where your data came from.

Quick check

You scrape 500 product reviews and the same review appears three times on different pages. What is the best next step?

What to remember

In short: the status code tells you if the answer is good, and the body is usually HTML that a parser can read. Check robots.txt and the terms first, go slowly, and watch for script-built pages, changes, repeats and gaps.

Next: APIs. Some sites offer a front door built for programs. It gives clean data, clear limits and no HTML to untangle. That is the next lesson.

Sources

How is this lesson?

Last updated on 16 November 2026

Learn with me

Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.