Books

Best Web Scraping Books for Python: Scrapy, Playwright, Selenium

This post contains affiliate links. If you buy through them, we may earn a small commission at no extra cost to you. Learn more.

Scrapy, Playwright and requests with BeautifulSoup cover nearly every Python scraping job in 2026. The best web scraping books teach those three with code that still runs, and that second test matters more than it used to. Scrapy 2.16 removed the start_requests() method that some books still use, and a spider that relies on it alone now finishes with zero items and no error.

Original content from computingforgeeks.com - post 182462

We picked seven books, from Ryan Mitchell’s O’Reilly title to a 2026 Packt cookbook, and compared each table of contents and companion repo with what pip installs today. We checked those code paths in November 2026 on Ubuntu 24.04 with Python 3.12, Scrapy 2.19, Selenium 4.52 and Playwright 1.64.

Which web scraping book should you buy?

Buy Web Scraping with Python, 3rd Edition unless one of the cases below fits you better. It is the only current book that teaches parsing, crawling, Scrapy, Selenium 4, APIs and the legal side in one place, and its 2024 code uses current Selenium locators. Pick a different book when one of these describes you:

  • New to Python: Automate the Boring Stuff, 3rd Edition. Its web scraping chapter teaches requests, Beautiful Soup, Selenium and Playwright, and it is the cheapest book here.
  • You want the newest code: Python Automation Cookbook, 3rd Edition. It shipped in June 2026, pins Playwright 1.56, and its chapter 1 project requires Python 3.14.
  • You want every parser compared: Hands-On Web Scraping with Python, 2nd Edition, which covers lxml, PyQuery, Beautiful Soup, Scrapy and Selenium.
  • You crawl millions of pages: Getting Structured Data from the Internet, which moves the crawl onto AWS and Common Crawl.
  • You have to defend the project to a manager or a lawyer: Practical Web Scraping for Data Science, with its chapter on managerial and legal concerns.

Whichever you buy, plan to adapt some code. None of the companion repos we searched uses httpx for scraping, the two Playwright books stick to its sync API, and the three older books use Selenium 3 calls that Selenium 4 removed. The fixes are short, and we show each one further down.

How did we pick and check these books?

We started from the Amazon catalog API, searched by topic and then by publisher (O’Reilly, Packt, No Starch, Apress, BPB), and dropped the dozens of short self-published titles that have no table of contents or code online. For each book that was left, we confirmed the following:

  • The ASIN is the paperback of the edition on the cover, with its page count and publication date from the Amazon catalog.
  • The chapter list comes from the publisher, Springer, the O’Reilly catalog or the free online edition from the author.
  • The companion repo was searched for find_element_by_, start_requests, playwright and pinned library versions.
  • The core patterns the books teach (requests, Beautiful Soup, Selenium, Playwright and Scrapy) were run on a fresh Ubuntu 24.04 VM against current libraries.

We didn’t read these books cover to cover, so we don’t rate writing quality here. What we can show is which code from each era still runs, and that is in the lab section near the end. Two books didn’t make the list. Web Automation with Playwright and Python using AI and MCP (BPB, 2026) has no data extraction chapter, because it’s a testing book. Learning Scrapy (Packt, 2016) predates Scrapy 2 and listed at about $105, which is too much for code that old.

How do the web scraping books compare?

The tool columns matter more than the page count, because they tell you which code you will have to adapt. Every column comes from the table of contents or the companion repo:

BookEdition, yearPagesScrapySelenium APIPlaywrightPrice band
Web Scraping with Python3rd, 2024352YesSelenium 4No$44-46
Automate the Boring Stuff3rd, 2025672NoSelenium 4Yes, sync$26-28
Python Automation Cookbook3rd, 2026676NoNot coveredYes, sync$44-46
Hands-On Web Scraping with Python2nd, 2023324YesSelenium 4No$45-47
Getting Structured Data from the Internet1st, 2020397YesSelenium 3No$41-43
Practical Web Scraping for Data Science1st, 2018322NoSelenium 3No$55-56
Website Scraping with Python1st, 2018241YesSelenium 3No$28-29

Read across the Selenium and Playwright columns first. Every book from before 2021 needs Selenium edits before its browser code runs, and only two books teach Playwright. Prices are bands from the Amazon catalog on November 1, 2026, so check the live price before you buy.

1. Web Scraping with Python, 3rd Edition

Ryan Mitchell wrote the book we would hand to anyone who scrapes for a living, from the first parser to proxies.

Web Scraping with Python 3rd Edition by Ryan Mitchell book cover
Web Scraping with Python, 3rd Edition by Ryan Mitchell. Image: O’Reilly Media.

The 2024 edition runs 352 pages over 20 chapters. Part I covers how the web works, a legalities and ethics chapter, Beautiful Soup parsing, crawlers, crawling models, Scrapy and storing data. Part II covers documents, dirty data, natural language, forms and logins, JavaScript, APIs, OCR, scraping traps, testing, parallel scraping and proxies. The companion repo uses the Selenium 4 By locators throughout, so the JavaScript chapter runs on current Selenium.

One spider needs a fix. The first Scrapy example in the repo, article.py, builds its requests in start_requests(), so on Scrapy 2.16 or newer it crawls nothing. Replace it with an async def start(self) method that yields each request. A plain rename is not enough, because that method returns a list, and Scrapy 2.18 and newer raise TypeError when start() is not an async generator. The other chapter 8 spiders set start_urls, so the removal does not affect them. The book has no Playwright chapter either, which is the gap we would fill with Automate the Boring Stuff or the Playwright docs.

Who is it for? Developers who already write Python and want one book that covers the whole job. Look elsewhere if Python itself is new to you, because the book assumes you are comfortable with it. Amazon listed the paperback at about $44 to $46 with low stock when we checked.

2. Automate the Boring Stuff with Python, 3rd Edition

Al Sweigart wrote the cheapest way into scraping on this list, and one of only two picks that teach Playwright.

Automate the Boring Stuff with Python 3rd Edition by Al Sweigart book cover
Automate the Boring Stuff with Python, 3rd Edition by Al Sweigart. Image: No Starch Press.

Only one of its chapters is about the web, so think of it as a Python book with a strong scraping chapter. Chapter 13 covers downloading with requests, a weather API, HTML basics, Beautiful Soup, then browser control with both Selenium and Playwright. The Selenium code uses By locators, and the Playwright code uses the sync API with python -m playwright install. Sweigart publishes the full chapter free online, so you can try it before buying.

Scrapy, lxml and robots.txt do not appear in the chapter, so you’ll outgrow it as soon as you need a crawler. For a beginner that’s the right order anyway. The 672-page paperback listed at about $26 to $28, under half its list price and the lowest price of any pick.

3. Python Automation Cookbook, 3rd Edition

Jaime Buelta wrote the freshest scraping code on the list, published in June 2026.

Python Automation Cookbook 3rd Edition by Jaime Buelta book cover
Python Automation Cookbook, 3rd Edition by Jaime Buelta. Image: Packt Publishing.

Chapter 3, “Building Your First Web Scraping Application”, holds ten recipes. These are the ones you will use most on real sites:

  • Downloading pages, parsing HTML and crawling a whole site from its links.
  • Submitting forms, logging in with basic authentication and reusing your browser cookies.
  • Driving a page with Playwright when plain requests are not enough.
  • Reading feeds and web APIs instead of scraping HTML at all.

The tenth recipe covers speeding up a scraper. The companion repo pins requests==2.32.5, beautifulsoup4==4.14.3 and playwright==1.56.0, and its chapter 1 project sets requires-python = ">=3.14". The book uses uv for environments, and later chapters cover AI models, MCP servers and agents.

Scraping is one chapter out of 16, and there is no Scrapy and no Selenium, so it is a starter course rather than a reference. Buy it if you want the rest of the automation material too, from files and spreadsheets to AI agents. The 676-page paperback listed at about $44 to $46.

4. Hands-On Web Scraping with Python, 2nd Edition

Anish Chapagain compares more parsing libraries in this Packt book than any other pick here does.

Hands-On Web Scraping with Python 2nd Edition by Anish Chapagain book cover
Hands-On Web Scraping with Python, 2nd Edition by Anish Chapagain. Image: Packt Publishing.

The 2023 edition has 12 chapters across 324 pages. It moves from requests and urllib to XPath and CSS selectors with lxml, robots.txt and sitemaps, PyQuery, Beautiful Soup with a full Scrapy project, then forms, cookies, proxies, web APIs, Selenium, regex and PDFs. The last chapters turn the scraped data into analysis and a scikit-learn model. The repo lists Python 3.11, uses Selenium 4 locators, and its Scrapy spider starts from start_urls, so the start_requests() removal does not affect it.

It has no Playwright, no async code and no dedicated legal chapter. Choose Mitchell instead if you want one book, and this one if you want to see the same job done with four libraries before settling on one. It listed at about $45 to $47.

5. Getting Structured Data from the Internet

Jay M. Patel wrote the only book here about crawling at data-pipeline size rather than one site at a time.

Getting Structured Data from the Internet by Jay M. Patel book cover
Getting Structured Data from the Internet by Jay M. Patel. Image: Apress.

Its eight chapters treat scraping as one stage of a data pipeline, so most of the book is about what happens after the download:

  • Parsing with Beautiful Soup and lxml, then storing results in PostgreSQL or SQLite.
  • Running the crawl on AWS with EC2, S3, Athena, SQS and SNS.
  • Mining Common Crawl datasets and processing crawls at big-data scale.
  • Text analytics with scikit-learn, Gensim and spaCy.
  • A production Scrapy crawler with CAPTCHA handling and proxy rotation.

That makes it the book for teams who would rather query the Common Crawl archive than hit a site themselves.

The 2020 code shows its age. Two notebooks in the repo call find_element_by_*, which raises AttributeError on Selenium 4, and the notebooks mention Python 3.6. Its Scrapy spiders use start_urls, so those survive. The Amazon offer came from a third-party seller at about $41 to $43 when we checked, so look at the seller before you buy.

6. Practical Web Scraping for Data Science

Seppe vanden Broucke and Bart Baesens give the legal and managerial side a full chapter, more than most scraping books do.

Practical Web Scraping for Data Science by Seppe vanden Broucke and Bart Baesens book cover
Practical Web Scraping for Data Science by Seppe vanden Broucke and Bart Baesens. Image: Apress.

The 2018 book teaches HTTP, HTML and CSS parsing with requests and Beautiful Soup, a deeper HTTP chapter that covers cookies, JavaScript pages with Selenium, and crawling. Then it gives a whole chapter to managerial and legal concerns, and about a hundred pages to worked examples. The examples in the repo range from books.toscrape.com to IMDb, Hacker News and a CAPTCHA-cracking project. That chapter is the reason to buy it, since most scraping books give the law a few pages.

It doesn’t teach Scrapy, and its Selenium example in iata/with_selenium.py uses the removed find_element_by_* calls. At about $55 to $56 it is also the most expensive pick, so buy it for the legal chapter and the examples, and learn the tools from a newer book.

7. Website Scraping with Python

Gábor László Hajba builds one scraping project across the whole book, first with Beautiful Soup and then with Scrapy.

Website Scraping with Python Using BeautifulSoup and Scrapy by Gabor Laszlo Hajba book cover
Website Scraping with Python: Using BeautifulSoup and Scrapy by Gábor László Hajba. Image: Apress.

Six chapters take you from requirements to a Beautiful Soup scraper, then Scrapy, JavaScript pages with Selenium and Splash, and deployment to the Scrapinghub cloud. Building the same app twice is a good way to see what spiders, items and pipelines in Scrapy buy you over a hand-written loop.

It is a 2018 book, and the code inside the repo’s nested zip files shows it. The chapter 5 Splash spider and scrapy-selenium spider build their rendered requests in start_requests(), which Scrapy 2.16 and newer never call, and they lean on the scrapy-splash and scrapy-selenium plugins, which we did not run against current Scrapy. Its Selenium code also calls set_headless(), which Selenium 4 no longer has. Expect the JavaScript chapter not to run as written. Treat it as a cheap second Scrapy book at about $28 to $29, less than half its list price, and plan to rework that chapter with Playwright.

Does the code in older scraping books still run?

Most of it does. Three patterns broke on current libraries in our lab, and you should know them before chapter one:

  • Selenium 3 calls such as find_elements_by_css_selector raise AttributeError on Selenium 4.
  • A Scrapy spider that only defines start_requests() scrapes 0 items on Scrapy 2.16 or newer.
  • requests-html fails on import until you add the lxml_html_clean package.

We set up a clean Python environment on Ubuntu 24.04 and ran each pattern against quotes.toscrape.com, a sandbox site built for scraping practice, so you can repeat the core tests below. Create the project folder and a virtual environment first:

mkdir -p ~/scrape && cd ~/scrape
sudo apt-get update && sudo apt-get install -y python3-venv
python3 -m venv .venv
source .venv/bin/activate

Install the libraries the seven books use between them, plus httpx for the async test at the end:

pip install requests beautifulsoup4 lxml httpx scrapy playwright selenium

Playwright needs its own browser build and the system libraries that browser links against. The first command runs as root through the binary inside the venv, and the second downloads Chromium into your home directory:

sudo .venv/bin/playwright install-deps chromium
playwright install chromium

Check what pip gave you, because the versions decide which book code runs unchanged:

pip list 2>/dev/null | grep -iE '^(requests|beautifulsoup4|lxml|httpx|scrapy|playwright|selenium|parsel|twisted) '

Our VM got Scrapy 2.19.0, Selenium 4.52.0 and Playwright 1.64.1 on Python 3.12.3:

pip list showing requests 2.34.2, beautifulsoup4 4.15.0, Scrapy 2.19.0, playwright 1.64.1 and selenium 4.52.0 on Ubuntu 24.04

Every book above was written against older releases than these. The rest of this section shows where that matters.

Why does requests find nothing on some pages?

Because the page builds its content with JavaScript after it loads, and requests only downloads the HTML. We ran the same requests and Beautiful Soup script, the pattern every book here starts with, against the static page and the JavaScript version of the same site. Create the script:

vim static.py

Paste in the file we ran, which takes the URL as its only argument and counts the div.quote elements:

import sys
import requests
from bs4 import BeautifulSoup

url = sys.argv[1]
r = requests.get(url, timeout=30)
soup = BeautifulSoup(r.text, "lxml")
quotes = soup.select("div.quote")
print("status:", r.status_code)
print("div.quote count:", len(quotes))
if quotes:
    print("first author:", quotes[0].select_one("small.author").text)

Run it against both pages with python static.py https://quotes.toscrape.com/ and again with /js/ on the end. Both returned HTTP 200, but only the static page had quotes in it:

requests and BeautifulSoup finding 10 quotes on the static page and 0 on the JavaScript-rendered page

A 200 with zero results is the most common beginner trap, because nothing looks broken. For pages like that you need a real browser, and Playwright is the one Automate the Boring Stuff and the cookbook teach. Create the script:

vim pw.py

Our run used this file unchanged. It waits for the first div.quote to appear before reading the page:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    page.goto("https://quotes.toscrape.com/js/")
    page.wait_for_selector("div.quote")
    quotes = page.query_selector_all("div.quote")
    print("div.quote count:", len(quotes))
    for q in quotes[:3]:
        print(q.query_selector("small.author").inner_text())
    browser.close()

Run it with time python pw.py. Headless Chromium found all 10 quotes in under four seconds, browser start included:

Playwright Python sync API script scraping 10 quotes from a JavaScript-rendered page in 3.8 seconds

Before you reach for a browser, open the network tab in your browser and look for the JSON request the page makes. Calling that API directly with requests is faster than any browser, and the Mitchell chapter on crawling through APIs covers the technique. If you work in Node rather than Python, the same headless Chromium is a few lines of Puppeteer on Linux. For something lighter than Chromium, our guide to the Lightpanda headless browser installs and benchmarks it on Linux.

Error: “‘WebDriver’ object has no attribute ‘find_elements_by_css_selector'”

This error means the code was written for Selenium 3, and Selenium 4 removed the find_element_by_* helpers. Getting Structured Data from the Internet and Practical Web Scraping for Data Science both use them. We ran one script with the old call and one with the Selenium 4 form, and Selenium Manager downloaded a matching Chrome for Testing and driver by itself on the first run:

Selenium 4.52 AttributeError WebDriver object has no attribute find_elements_by_css_selector, then find_elements with By.CSS_SELECTOR returning 10

Updating the code is mechanical. Import By and pass the locator type as the first argument, so find_elements_by_css_selector("div.quote") becomes find_elements(By.CSS_SELECTOR, "div.quote"):

from selenium.webdriver.common.by import By

quotes = driver.find_elements(By.CSS_SELECTOR, "div.quote")

Old code that passes a driver path such as webdriver.Chrome(chromedriver_path) needs that argument removed too. On our headless VM we passed --no-sandbox next to --headless=new.

Why does a Scrapy spider scrape 0 items with no error?

On Scrapy 2.16 and newer, a spider that builds its first requests in start_requests() crawls nothing. The Scrapy release notes say the method was deprecated in 2.13.0 and then “removed and no longer called” in 2.16.0. We expected a warning on 2.19 and got none. Here are the same spider with the classic method, then with the new async def start(self):

Scrapy 2.19 spider using start_requests crawling 0 pages while the async start version scrapes 100 items

The first run closed in under a second with no item_scraped_count at all, while the second crawled all 10 pages and scraped 100 items. Two books ship spiders affected by the removal. Mitchell’s article.py has no start_urls, so it scrapes nothing, and the Hajba Splash and scrapy-selenium spiders build their rendered requests in a method Scrapy no longer calls, so they fetch their start_urls as plain, unrendered requests. Create the fixed spider:

vim quotes_start.py

Only the start method differs from the classic version, and pagination still uses response.follow:

import scrapy


class QuotesSpider(scrapy.Spider):
    name = "quotes"

    async def start(self):
        yield scrapy.Request("https://quotes.toscrape.com/", callback=self.parse)

    def parse(self, response):
        for q in response.css("div.quote"):
            yield {
                "author": q.css("small.author::text").get(),
                "text": q.css("span.text::text").get(),
            }
        next_page = response.css("li.next a::attr(href)").get()
        if next_page:
            yield response.follow(next_page, callback=self.parse)

Run it and write the items to a JSON file:

scrapy runspider quotes_start.py -o quotes.json

Spiders that only set start_urls, like the ones in Hands-On Web Scraping and most of chapter 8 in Mitchell, are unaffected by the removal, because the default start() reads that list. Our start_urls test spider scraped its 10 items on 2.19. If a spider must also run on Scrapy older than 2.13, keep both methods.

Error: “lxml.html.clean module is now a separate project lxml_html_clean”

You’ll hit this the moment you import requests-html, which some older tutorials and books recommend for JavaScript pages. The package still installs, but its last release is 0.10.0, and it imports a cleaner that lxml has since moved into a separate package. Installing that package makes the import work again:

requests-html import failing with ImportError lxml.html.clean module is now a separate project, fixed by installing lxml_html_clean

The install also pulled pyppeteer 2.0.0 and pinned urllib3 to 1.26.20, so keep requests-html in its own virtualenv if you use it at all. For new code, Playwright does the same job and is maintained.

Do the old BeautifulSoup method names still work?

Yes, but they warn. Code written for Beautiful Soup 3 uses camelCase names such as findAll, and Beautiful Soup 4.15 still runs them while printing a DeprecationWarning when warnings are on:

BeautifulSoup 4.15 DeprecationWarning for findAll while find_all runs silently

Swap findAll for find_all as you type the examples. It costs nothing, and your code won’t break when the old names finally go.

What should you learn that the books skip?

Async HTTP. None of the companion repos we searched uses httpx for scraping, yet a few lines of httpx.AsyncClient with asyncio.gather fetched five pages of the sandbox concurrently on our VM. Create the script:

vim httpx_async.py

Add the code we ran, which gathers pages 1 to 5 in one event loop:

import asyncio
import time

import httpx
from bs4 import BeautifulSoup


async def fetch(client, n):
    r = await client.get(f"https://quotes.toscrape.com/page/{n}/")
    return n, len(BeautifulSoup(r.text, "lxml").select("div.quote"))


async def main():
    start = time.perf_counter()
    async with httpx.AsyncClient(timeout=30) as client:
        for n, count in await asyncio.gather(*(fetch(client, n) for n in range(1, 6))):
            print(f"page {n}: {count} quotes")
    print(f"elapsed: {time.perf_counter() - start:.2f}s")


asyncio.run(main())

Run it with python httpx_async.py. All five pages came back with 10 quotes each in 0.88 seconds:

httpx AsyncClient with asyncio fetching five pages concurrently in 0.88 seconds

Concurrency is also how scrapers get blocked, so add a delay and respect robots.txt before you raise it. When a site does block you, our guide to residential, datacenter and mobile proxies explains which kind fits which job, and our explainer on how browser fingerprinting works covers the signals sites use to tell a headless browser from a person.

How do you get set up before chapter one?

Set up the environment once and the examples from every book will run in it, with the small fixes above. Work through this checklist in order:

  1. Your project has its own virtualenv, and which python points inside .venv.
  2. pip list shows Scrapy, Selenium 4 and Playwright, and you have noted them next to the versions the book used.
  3. python pw.py prints 10 quotes, which proves the Playwright browser and its system libraries are in place.
  4. Your Selenium code uses find_element(By....) and no executable_path or driver path argument.
  5. No Scrapy spider relies on start_requests() alone (keep it only next to start() for Scrapy older than 2.13), each one uses start_urls or async def start(self), and its run log shows item_scraped_count above zero.
  6. You have read the robots.txt file and the terms of use for the target site before pointing a crawler at it.

If you are still choosing your first Python book, our Python programming books list covers the language itself, and the data science books for Python pick up where scraping stops, once the data is on disk. If you also test the sites you scrape for weaknesses, our web application security books cover that side of HTTP. For PDFs of the pages you collect, print to PDF with headless Chrome works from the same Chromium that Playwright drives.

Keep reading

Best Machine Learning Books (2026): Textbooks, Python, Free PDFs AI Best Machine Learning Books (2026): Textbooks, Python, Free PDFs Best C++ and C Programming Books for 2026: Beginner to Advanced Books Best C++ and C Programming Books for 2026: Beginner to Advanced Install GitHub CLI (gh) on Linux, macOS, and Windows Programming Install GitHub CLI (gh) on Linux, macOS, and Windows Best Networking Books: TCP/IP, Routing and Troubleshooting Books Best Networking Books: TCP/IP, Routing and Troubleshooting Best AI Agent Books for 2026: Agentic AI, MCP and Multi-Agent Systems AI Best AI Agent Books for 2026: Agentic AI, MCP and Multi-Agent Systems Best Web Application Security and Bug Bounty Books Books Best Web Application Security and Bug Bounty Books

Leave a Comment

Press ESC to close