{"id":108,"date":"2025-10-25T07:18:30","date_gmt":"2025-10-25T07:18:30","guid":{"rendered":"https:\/\/codetypingpro.com\/?p=108"},"modified":"2025-12-17T07:49:58","modified_gmt":"2025-12-17T07:49:58","slug":"lesson-30-web-scraping-in-python","status":"publish","type":"post","link":"https:\/\/codetypingpro.com\/?p=108","title":{"rendered":"Lesson 30: Web Scraping in Python"},"content":{"rendered":"\n

\ud83c\udfaf Lesson Objective<\/strong><\/h3>\n\n\n\n

To understand how to extract data from websites using Python and use it for data analysis, automation, and reporting<\/strong>.<\/p>\n\n\n\n

\n\n\n\n

\ud83e\udde9 1. What Is Web Scraping?<\/strong><\/h2>\n\n\n\n

Web scraping is the process of automatically extracting information from websites<\/strong>.<\/p>\n\n\n\n

Example Uses:<\/strong><\/p>\n\n\n\n

    \n
  • Collecting product prices from e-commerce sites<\/li>\n\n\n\n
  • Extracting news headlines or articles<\/li>\n\n\n\n
  • Monitoring stock prices or cryptocurrency rates<\/li>\n\n\n\n
  • Gathering weather data<\/li>\n<\/ul>\n\n\n\n
    \n\n\n\n

    \u2699\ufe0f 2. Python Libraries for Web Scraping<\/strong><\/h2>\n\n\n\n
    Library<\/th>Purpose<\/th><\/tr><\/thead>
    requests<\/code><\/td>Send HTTP requests to get website content<\/td><\/tr>
    BeautifulSoup (bs4)<\/code><\/td>Parse HTML and XML content<\/td><\/tr>
    lxml<\/code><\/td>Fast HTML and XML parser (optional)<\/td><\/tr>
    pandas<\/code><\/td>Store and manipulate scraped data<\/td><\/tr>
    selenium<\/code><\/td>Automate browser interaction for dynamic websites<\/td><\/tr>
    re<\/code><\/td>Extract specific patterns using regular expressions<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
    \n\n\n\n

    \ud83d\udd39 3. Making HTTP Requests<\/strong><\/h2>\n\n\n\n

    Before scraping, fetch the HTML content:<\/p>\n\n\n\n

    import requests\n\nurl = \"https:\/\/example.com\"\nresponse = requests.get(url)\n\nprint(response.status_code)  # 200 means success\nprint(response.text[:500])   # Print first 500 characters of HTML\n<\/code><\/pre>\n\n\n\n
    \u2705 Tip:<\/strong> Always check the website\u2019s robots.txt<\/strong> and terms of service before scraping.<\/p>\n\n\n\n
    \n\n\n\n
    \ud83d\udd39 4. Parsing HTML with BeautifulSoup<\/strong><\/h2>\n\n\n\n
    from bs4 import BeautifulSoup\n\nhtml_content = response.text\nsoup = BeautifulSoup(html_content, \"html.parser\")\n\n# Print pretty formatted HTML\nprint(soup.prettify())\n<\/code><\/pre>\n\n\n\n
    Example 1 \u2014 Extracting Titles and Links<\/h3>\n\n\n\n
    for link in soup.find_all(\"a\"):  # Find all <a> tags\n    print(link.get(\"href\"), \"-\", link.text)\n<\/code><\/pre>\n\n\n\n
    Example 2 \u2014 Extract Specific Elements<\/h3>\n\n\n\n
    headlines = soup.find_all(\"h2\")  # Extract all headlines inside <h2> tags\nfor headline in headlines:\n    print(headline.text.strip())\n<\/code><\/pre>\n\n\n\n
    \n\n\n\n
    \ud83d\udd39 5. Using CSS Selectors<\/strong><\/h2>\n\n\n\n
    # Extract elements using class or id\narticles = soup.select(\".article-title\")  # Class\nfor article in articles:\n    print(article.get_text())\n\nmain_section = soup.select_one(\"#main-content\")  # ID\nprint(main_section.text)\n<\/code><\/pre>\n\n\n\n
    \n\n\n\n
    \ud83d\udd39 6. Handling Tables and Structured Data<\/strong><\/h2>\n\n\n\n
    import pandas as pd\n\ntable = soup.find(\"table\", {\"id\": \"data-table\"})\nrows = table.find_all(\"tr\")\n\ndata = []\nfor row in rows:\n    cols = row.find_all(\"td\")\n    cols = [col.text.strip() for col in cols]\n    data.append(cols)\n\ndf = pd.DataFrame(data, columns=[\"Name\", \"Age\", \"City\"])\nprint(df)\n<\/code><\/pre>\n\n\n\n
    \u2705 Converts scraped HTML tables into a pandas DataFrame<\/strong> for analysis.<\/p>\n\n\n\n
    \n\n\n\n
    \ud83d\udd39 7. Scraping JSON Data<\/strong><\/h2>\n\n\n\n
    Some websites provide data via APIs in JSON format<\/strong>:<\/p>\n\n\n\n
    import requests\n\nurl = \"https:\/\/api.example.com\/data\"\nresponse = requests.get(url)\ndata = response.json()\n\nfor item in data[\"items\"]:\n    print(item[\"name\"], \"-\", item[\"price\"])\n<\/code><\/pre>\n\n\n\n
    \u2705 Easier and cleaner than parsing HTML.<\/p>\n\n\n\n
    \n\n\n\n
    \ud83d\udd39 8. Scraping Dynamic Websites Using Selenium<\/strong><\/h2>\n\n\n\n
    Dynamic sites load content via JavaScript:<\/p>\n\n\n\n
    from selenium import webdriver\nfrom selenium.webdriver.common.by import By\n\ndriver = webdriver.Chrome()  # Ensure ChromeDriver is installed\ndriver.get(\"https:\/\/example.com\")\n\ntitles = driver.find_elements(By.CLASS_NAME, \"product-title\")\nfor title in titles:\n    print(title.text)\n\ndriver.quit()\n<\/code><\/pre>\n\n\n\n
    \u2705 Can handle login pages, infinite scrolling, and button clicks<\/strong>.<\/p>\n\n\n\n
    \n\n\n\n
    \ud83d\udd39 9. Example \u2014 Scraping Product Prices<\/strong><\/h2>\n\n\n\n
    import requests\nfrom bs4 import BeautifulSoup\n\nurl = \"https:\/\/books.toscrape.com\/\"\nresponse = requests.get(url)\nsoup = BeautifulSoup(response.text, \"html.parser\")\n\nbooks = soup.find_all(\"h3\")\nprices = soup.find_all(\"p\", class_=\"price_color\")\n\nfor book, price in zip(books, prices):\n    print(f\"{book.a['title']} - {price.text}\")\n<\/code><\/pre>\n\n\n\n
    \u2705 Outputs book titles and prices<\/strong>.<\/p>\n\n\n\n
    \n\n\n\n
    \ud83d\udd39 10. Exporting Scraped Data<\/strong><\/h2>\n\n\n\n
    df.to_csv(\"scraped_data.csv\", index=False)\ndf.to_excel(\"scraped_data.xlsx\", index=False)\n<\/code><\/pre>\n\n\n\n
    \u2705 Save data for further analysis or reporting<\/strong>.<\/p>\n\n\n\n
    \n\n\n\n
    \ud83d\udd39 11. Best Practices<\/strong><\/h2>\n\n\n\n
      \n
    • Respect the website:<\/strong> Avoid overloading with requests. Use time.sleep()<\/code><\/li>\n\n\n\n
    • Check robots.txt:<\/strong> Ensure scraping is allowed<\/li>\n\n\n\n
    • Use headers:<\/strong> Some sites block default Python requests<\/li>\n<\/ul>\n\n\n\n
      headers = {\"User-Agent\": \"Mozilla\/5.0\"}\nresponse = requests.get(url, headers=headers)\n<\/code><\/pre>\n\n\n\n
        \n
      • Handle errors:<\/strong> Use try-except<\/code> for failed requests<\/li>\n\n\n\n
      • Paginate:<\/strong> Automate scraping across multiple pages<\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"
        \ud83c\udfaf Lesson Objective To understand how to extract data from websites using Python and use it for data analysis, automation, and reporting. \ud83e\udde9 1. What Is Web Scraping? Web scraping is the process of automatically extracting information from websites. Example Uses: \u2699\ufe0f 2. Python Libraries for Web Scraping Library Purpose requests Send HTTP requests to […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[6,1],"tags":[],"class_list":["post-108","post","type-post","status-publish","format-standard","hentry","category-python-easy-course-outline","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/codetypingpro.com\/index.php?rest_route=\/wp\/v2\/posts\/108","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/codetypingpro.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/codetypingpro.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/codetypingpro.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/codetypingpro.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=108"}],"version-history":[{"count":2,"href":"https:\/\/codetypingpro.com\/index.php?rest_route=\/wp\/v2\/posts\/108\/revisions"}],"predecessor-version":[{"id":111,"href":"https:\/\/codetypingpro.com\/index.php?rest_route=\/wp\/v2\/posts\/108\/revisions\/111"}],"wp:attachment":[{"href":"https:\/\/codetypingpro.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=108"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/codetypingpro.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=108"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/codetypingpro.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=108"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}