Lesson 30: Web Scraping in Python

🎯 Lesson Objective

To understand how to extract data from websites using Python and use it for data analysis, automation, and reporting.

🧩 1. What Is Web Scraping?

Web scraping is the process of automatically extracting information from websites.

Example Uses:

Collecting product prices from e-commerce sites
Extracting news headlines or articles
Monitoring stock prices or cryptocurrency rates
Gathering weather data

⚙️ 2. Python Libraries for Web Scraping

Library	Purpose
`requests`	Send HTTP requests to get website content
`BeautifulSoup (bs4)`	Parse HTML and XML content
`lxml`	Fast HTML and XML parser (optional)
`pandas`	Store and manipulate scraped data
`selenium`	Automate browser interaction for dynamic websites
`re`	Extract specific patterns using regular expressions

🔹 3. Making HTTP Requests

Before scraping, fetch the HTML content:

import requests

url = "https://example.com"
response = requests.get(url)

print(response.status_code)  # 200 means success
print(response.text[:500])   # Print first 500 characters of HTML

✅ Tip: Always check the website’s robots.txt and terms of service before scraping.

🔹 4. Parsing HTML with BeautifulSoup

from bs4 import BeautifulSoup

html_content = response.text
soup = BeautifulSoup(html_content, "html.parser")

# Print pretty formatted HTML
print(soup.prettify())

Example 1 — Extracting Titles and Links

for link in soup.find_all("a"):  # Find all <a> tags
    print(link.get("href"), "-", link.text)

Example 2 — Extract Specific Elements

headlines = soup.find_all("h2")  # Extract all headlines inside <h2> tags
for headline in headlines:
    print(headline.text.strip())

🔹 5. Using CSS Selectors

# Extract elements using class or id
articles = soup.select(".article-title")  # Class
for article in articles:
    print(article.get_text())

main_section = soup.select_one("#main-content")  # ID
print(main_section.text)

🔹 6. Handling Tables and Structured Data

import pandas as pd

table = soup.find("table", {"id": "data-table"})
rows = table.find_all("tr")

data = []
for row in rows:
    cols = row.find_all("td")
    cols = [col.text.strip() for col in cols]
    data.append(cols)

df = pd.DataFrame(data, columns=["Name", "Age", "City"])
print(df)

✅ Converts scraped HTML tables into a pandas DataFrame for analysis.

🔹 7. Scraping JSON Data

Some websites provide data via APIs in JSON format:

import requests

url = "https://api.example.com/data"
response = requests.get(url)
data = response.json()

for item in data["items"]:
    print(item["name"], "-", item["price"])

✅ Easier and cleaner than parsing HTML.

🔹 8. Scraping Dynamic Websites Using Selenium

Dynamic sites load content via JavaScript:

from selenium import webdriver
from selenium.webdriver.common.by import By

driver = webdriver.Chrome()  # Ensure ChromeDriver is installed
driver.get("https://example.com")

titles = driver.find_elements(By.CLASS_NAME, "product-title")
for title in titles:
    print(title.text)

driver.quit()

✅ Can handle login pages, infinite scrolling, and button clicks.

🔹 9. Example — Scraping Product Prices

import requests
from bs4 import BeautifulSoup

url = "https://books.toscrape.com/"
response = requests.get(url)
soup = BeautifulSoup(response.text, "html.parser")

books = soup.find_all("h3")
prices = soup.find_all("p", class_="price_color")

for book, price in zip(books, prices):
    print(f"{book.a['title']} - {price.text}")

✅ Outputs book titles and prices.

🔹 10. Exporting Scraped Data

df.to_csv("scraped_data.csv", index=False)
df.to_excel("scraped_data.xlsx", index=False)

✅ Save data for further analysis or reporting.

🔹 11. Best Practices

Respect the website: Avoid overloading with requests. Use time.sleep()
Check robots.txt: Ensure scraping is allowed
Use headers: Some sites block default Python requests

headers = {"User-Agent": "Mozilla/5.0"}
response = requests.get(url, headers=headers)

Handle errors: Use try-except for failed requests
Paginate: Automate scraping across multiple pages