PythonWeb Scraping

Web Scraping

Web scraping means programmatically downloading a web page and extracting the data you need from its HTML, instead of copying it out by hand. It is useful whenever a site has information you want but does not expose through an API. The classic Python combo is requests to download the page and BeautifulSoup to parse the HTML and pull out the pieces you care about.

Installing BeautifulSoup

Install

Bash
pip install beautifulsoup4

The package is called beautifulsoup4 on PyPI but imported as bs4.

Downloading and parsing a page

The workflow has two steps: fetch the raw HTML with requests, then hand that HTML to BeautifulSoup to turn it into a navigable tree of elements.

fetch_and_parse.py

Python
import requests
from bs4 import BeautifulSoup

response = requests.get("https://example.com", timeout=5)
response.raise_for_status()

soup = BeautifulSoup(response.text, "html.parser")
print(soup.title.text)   # the page's <title> text
Selecting elements

Once you have a soup object, BeautifulSoup gives you two main ways to find things: .find() / .find_all() for tag-and-attribute lookups, and .select() for CSS selectors if you already think in terms of div.class or #id.

Method

Returns

Example

.find(tag, **attrs)

First matching element or None

soup.find("h1", class_="title")

.find_all(tag, **attrs)

List of all matching elements

soup.find_all("a")

.select(css_selector)

List of elements matching a CSS selector

soup.select("div.card > a.link")

.select_one(css_selector)

First element matching a CSS selector

soup.select_one("#main-title")

A full worked example

To keep this example self-contained and runnable without depending on a live website, the HTML is a hardcoded string representing a small product listing page.

scrape_products.py

Python
from bs4 import BeautifulSoup

html = """
<html>
  <body>
    <div class="product-list">
      <div class="product" data-id="101">
        <h2 class="name">Wireless Mouse</h2>
        <span class="price">$19.99</span>
      </div>
      <div class="product" data-id="102">
        <h2 class="name">Mechanical Keyboard</h2>
        <span class="price">$89.99</span>
      </div>
      <div class="product" data-id="103">
        <h2 class="name">USB-C Hub</h2>
        <span class="price">$34.50</span>
      </div>
    </div>
  </body>
</html>
"""

soup = BeautifulSoup(html, "html.parser")

products = []
for card in soup.select("div.product"):
    products.append({
        "id": card["data-id"],
        "name": card.find("h2", class_="name").text.strip(),
        "price": card.find("span", class_="price").text.strip(),
    })

for product in products:
    print(product)

# {'id': '101', 'name': 'Wireless Mouse', 'price': '$19.99'}
# {'id': '102', 'name': 'Mechanical Keyboard', 'price': '$89.99'}
# {'id': '103', 'name': 'USB-C Hub', 'price': '$34.50'}

card["data-id"] reads an HTML attribute like a dictionary key, .find() narrows to a specific child tag, and .text pulls out just the visible text, stripped of surrounding whitespace with .strip().

Inspect the real page first
Before writing selectors, open the page in your browser's DevTools and inspect the actual HTML structure. Class names and layout change between sites (and over time on the same site), so there is no substitute for looking at what you are actually parsing.
Respect the site you're scraping
Check robots.txt and the terms of service first
Not everything that is technically reachable is fair to scrape. Check the site's `robots.txt` file (e.g. `example.com/robots.txt`) for rules about which paths automated tools may access, and read the site's terms of service — many explicitly prohibit scraping. Scrape politely: identify your bot with a `User-Agent`, add delays between requests, and never hit a server harder than a human browsing casually would.
When requests + BeautifulSoup isn't enough

  • Scrapy — a full scraping framework for crawling many pages at scale, with built-in concurrency, retries, and pipelines for storing results.

  • Selenium / Playwright — browser automation tools that actually render JavaScript. Use these when the data you need is injected into the page by client-side JavaScript after the initial HTML loads, so plain requests never sees it.

requests.get() only sees the initial HTML
`requests` does not execute JavaScript. If you fetch a page and the content you want is missing from `response.text`, it is likely being rendered client-side — that's the signal to reach for Selenium or Playwright instead of trying to parse HTML that was never sent.