Repository navigation
Releases
Release v0.5.1
Small version bump, big browser upgrade! This version lets AI agents follow motion and control drag and drop, adds dedicated remote browser sessions, and brings fixes across parsing, spiders, proxies, and the shell π
With this update, we extend the limits of current AI models π₯
Warning
This release introduces breaking changes to remote browser connections. Connecting to remote browsers now requires a new browser session class. Check the remote-session migration guide before updating.
π New Stuff and quality of life changes
-
AI agents can now follow motion and take finer control of the mouse (Check the docs):
- Watch the page in motion: record a short video and share it as a sequence of frames with timestamps to help the agent understand animations, moving controls, and changes that a single screenshot can miss. Pair this with the existing ability to chain actions!
- Drag, inspect, adjust, and drop: hold a mouse button, move an item or slider, inspect the page while still holding it, then adjust and release. Supports left, right, and middle buttons.
- Reuse a logged-in remote browser or start with a separate browser context, so the agent can choose the right session for the job.
- Spend fewer tokens on tool descriptions with shorter text and less repeated guidance.
-
Dedicated remote browser sessions with control over login reuse.
RemoteSessionandAsyncRemoteSessionconnect to an existing Chromium browser through an HTTP or WebSocket CDP endpoint. They support the usual stealth session features, including Cloudflare solving. Both classes are also available in the interactive shell. (Check the docs)from scrapling.fetchers import RemoteSession with RemoteSession(cdp_url="http://localhost:9222", new_context=False) as session: page = session.fetch("https://example.com")
new_context=Falsereuses the browser's existing default context and login state. Its context settings stay in use, and its existing tabs remain open when Scrapling disconnects.- The default,
new_context=True, creates a separate context and closes it when the session ends. Scrapling uses its own tabs in either mode.
-
Choose your spider's log file encoding with
log_file_encoding="utf-8"or another encoding. The default,None, keeps Python's default encoding. (Check the docs)
π Bug Fixes
-
XPath expressions now return their values correctly, including
string(),count(), andboolean(). Empty strings, zero, and false remain available through the usual selector API, by @jayhemnani9910 in #552. -
Fixed parsing of HTML with no root element, including whitespace-only, comment-only, and doctype-only input, by @rushilnimbalker250 in #536.
-
Fixed
find_similar(similarity_threshold=0)when the reference has no attributes to compare, by @Master-Norna in #539. -
Fixed duplicate robots.txt fetches when concurrent requests reach the same domain, by @RayenMlayeh in #538.
-
Fixed temporary-file collisions during concurrent development-cache writes, by @xixihaha-ha in #546.
-
Malformed links no longer stop extraction of valid links on the same page, by @dajiaohuang in #548.
-
Fixed CSV feeds that use carriage returns as row separators, while preserving line breaks inside quoted fields, by @xunguangzlj-cloud in #551.
-
Truncated or damaged gzip feeds now follow the normal warning-and-skip path in sitemap, XML-feed, and CSV-feed spiders, by @cangen185-sudo in #522.
-
Fixed browser proxy authentication with percent-encoded usernames and passwords, by @kevin9327 in #545.
-
Improved browser cleanup when session setup or context closing fails, releasing owned resources while keeping borrowed remote contexts open.
-
Stopped shell and HTTP CLI cookie parsing from dropping browser cookie values with spaces or quotes, including the cookies that follow them, by @waw666waw666 in #541. Fixes #540.
-
Curl conversion now accepts
-Land--location, keeping the existing redirect behavior, by @jayhemnani9910 in #554. -
Fixed
scrapling extract fetch --ai-targetednot enabling ad blocking as documented, by @postoso in #527.
π Special thanks to the community for all the continuous testing and feedback
Big shoutout to our Platinum Sponsors
Release v0.5
Scrapling v0.5 upgrades almost everything: fetching, parsing, spiders, browser automation, AI tools, and docs in ten languages π
Warning
This release introduces a lot of breaking changes that don't fit here. Check the Migrating to v0.5 guide before updating.
π New Stuff and quality of life changes
-
A much more capable MCP server for AI agents that can solve almost any CAPTCHA (Check the docs):
- Complete browser workflows: fill forms, click, hover, scroll, use the keyboard, handle dialogs, and wait for results in one ordered sequence. Supports different field types and slower interactions, with actions stopping at the first error.
- Read and search the current page without reloading it: extract HTML, Markdown, or text from a whole page or one element. Searchable page snapshots show element roles, names, and positions, with focused views that retain the surrounding context.
- Run custom JavaScript and take focused screenshots: execute scripts, including async work, with access to the page's own JavaScript. Capture individual elements, the viewport, or the full page as PNG or JPEG.
- Inspect network activity and reuse logins: examine recorded requests and responses, save and restore browser login state, and interact with elements inside open or closed Shadow DOM. Read recorded traffic before closing the session.
- Use fewer tokens and get clearer errors: leaner tool descriptions and shared guidance reduce repeated text, while browser and request failures reach agents with useful error messages.
-
See the requests behind the page. Enable
record_requests=Trueto record completed traffic across loads, actions, and tabs, then inspect saved responses without sending requests again. (Check the docs)from scrapling.fetchers import StealthySession with StealthySession(record_requests=True) as session: session.fetch("https://example.com", network_idle=True) for response in session.network.search(url_pattern=r"/api/", limit=None): print(response.url, response.status, response.body)
- Filter by URL, method, resource type, or status; retrieve one response by ID. Inspect headers, sent data, API responses, HTTP errors, and redirects through the usual
ResponseAPI. - Retains supported text bodies with size limits, plus metadata and a reason when a body cannot be saved. The default limit is 1,000 requests. Works with sync and async sessions; saved Python history stays readable after close.
- Filter by URL, method, resource type, or status; retrieve one response by ID. Inspect headers, sent data, API responses, HTTP errors, and redirects through the usual
-
Save a login once and reuse it across runs.
save_state(path)andload_state(path)save cookies, local storage, and IndexedDB in Playwright-compatible JSON. Both sync and async browser sessions support them. (Check the docs)- Load state before fetching or reloading a page. Requires a started, shared browser context without
proxy_rotator; tabs, browser settings, andsessionStorageare not saved.
- Load state before fetching or reloading a page. Requires a started, shared browser context without
-
Extract open Shadow DOM with
pierce_shadow=True, then use CSS or XPath as usual. Available in browser fetchers, sessions, and the CLI's--pierce-shadowoption. Preserves slot content without changing the live page. HTML capture excludes closed roots and iframe documents. (Check the docs) -
Choose how your spider explores a site.
CrawlSpiderand its subclasses, includingContentSpider, supportcrawl_strategy="bfs"for oldest-first or"dfs"for newest-first scheduling. Priority still wins, and concurrent pages can finish in a different order. (Check the docs)
π Bug Fixes
-
Improved Cloudflare solving with delayed-widget detection, fewer unnecessary waits, and bounded passive-challenge and hidden-frame retry loops. Fixes #468 and #509.
-
Fixed browser startup from an elevated Windows shell. Fixes #463.
-
Kept the built-in stealth flags when adding custom browser flags by @alkaz-nodemaven in #430.
-
Improved charset parsing for browser responses, including declarations with different letter case, spaces, or quoted values.
-
Stopped logging the absence of a Cloudflare challenge as an error when no challenge needs to be solved. Fixes #459.
-
Corrected request fingerprints for query parameters by @yetval in #448, empty or falsy JSON bodies by @ac1982 in #480, and header insertion order by @SarwadnyaMahajan in #466.
-
Fixed request data leaking into followed links, including stale body-related headers, by @yetval in #462.
-
Prevented blocked responses from being stored as successful development-cache entries by @Kuang-xianxin in #446.
-
Preserved checkpoint files when a crawl fails with an exception by @SulimanAbdulrazzaq in #486.
-
Corrected robots.txt failure handling so failed lookups are not cached as permission to crawl, by @prateekraawat1 in #443.
-
Fixed link extraction with restricted selectors, including empty matches and selectors that target the link element itself, by @SulimanAbdulrazzaq in #452 and #483.
-
Fixed sitemap callback routing by @SulimanAbdulrazzaq in #458, and XML parsing with comments or processing instructions by @sup3dev in #450.
-
Handled leading byte-order marks in robots.txt and CSV feeds by @SulimanAbdulrazzaq in #453.
-
Fixed
stealthy_headershandling in spider HTTP sessions, passing the option to the session instead of directly tocurl_cffi, by @Yigtwxx in #475. -
Fixed dictionary form encoding for list and tuple values, while preserving request-body precedence and existing
Nonehandling, with contributions by @SulimanAbdulrazzaq in #485. -
Fixed case-insensitive merging of session and per-request HTTP headers by @SulimanAbdulrazzaq in #487.
-
Fixed CLI requests silently dropping empty or falsy JSON bodies by @SulimanAbdulrazzaq in #484.
-
Made Markdown and text file exports consistently use UTF-8 and prevented conversion errors from truncating an existing output file, by @SulimanAbdulrazzaq in #488.
-
Fixed
curl2fetcher()handling of curl's user-agent and proxy options by @SulimanAbdulrazzaq in #490. -
Fixed Markdown blocks running together when several selected elements are converted through
Response.markdown()or CLI extraction, with contributions by @Yigtwxx in #432. -
Fixed hidden-content cleaning for AI output to keep visible content such as
opacity:0.95, recognize more hidden styles, and remove hidden slots, with contributions by @Yigtwxx in #460. -
Fixed noise-tag removal stopping early around nested tags by @Yigtwxx in #474.
-
Fixed generated CSS and XPath selectors for the first of several same-tag siblings by @yellow-dog-dog in #495.
-
Resolved an lxml HTML parser deprecation warning by @afsuyadi in #493.
Docs
- The full website now speaks ten languages: English, Arabic, Spanish, Brazilian Portuguese, French, German, Simplified Chinese, Japanese, Russian, and Korean. Guides, API references, tutorials, and the changelog are translated, with Arabic RTL support and language switches that keep your page and section.
- Clearer guides, README examples, and setup steps, covering checkpoint data, browser encodings, saved state, network history, Shadow DOM, and crawl order.
- Updated the Agent Skill, its examples, references, and downloadable archive for the current browser and MCP tools.
- Moved the changelog to the website and added access to each page's Markdown source.
Project
- Refreshed Scrapling's logo, cover images, and website presentation.
- Expanded tests for browser actions, dialogs, extraction, screenshots, JavaScript, network recording, state files, Shadow DOM, and crawler fixes.
- Added a browser-test marker and separate browser, async, and parallel test runs. Improved Windows checkpoint and CSV test portability by @diegobravtraps-oss in #457.
- Updated the shared multilingual documentation build setup and contribution guidance.
_π Special thanks to the ...
Release v0.4.15
One of the biggest releases this year: a reworked MCP server, RAG-ready Markdown in one line, an improved Cloudflare solver, and browser tabs that stay open for automation π
Warning
This release introduces breaking changes to the MCP server. Check the breaking changes section before updating.
π New Stuff and quality of life changes
- Browser tabs now stay open and get reused across requests (Check the docs):
- All browser sessions keep their tabs after a request, and the next request reuses a free tab instead of opening a new one.
- Every request re-applies its own settings (timeouts, headers, resource blocking) to the tab it gets, so nothing leaks between requests.
- Tabs that hit an error are closed and replaced, and the new
close_pages()method closes every open tab. - The page you fetched stays loaded, so a
page_setupfunction on the next request runs on it before navigating away. That's the building block for chaining automation across requests.
- Turn any page into clean, LLM-ready Markdown with
Response.markdown()(Check the docs):from scrapling.fetchers import Fetcher markdown = Fetcher.get("https://example.com").markdown(main_content_only=True)
- Scripts, styles, and hidden prompt-injection content are always stripped first, the same as the cleaning the MCP server does.
- Pass
css_selectorto convert only the elements you need. - Available through the new
ragextra (pip install "scrapling[rag]"), which theai/shell/allextras include too.
- New
SiteToMarkdownSpidertemplate to crawl a whole website into a Markdown corpus for RAG pipelines (Check the docs):from scrapling.spiders import SiteToMarkdownSpider class DocsSpider(SiteToMarkdownSpider): name = "docs" start_urls = ["https://example.com/docs/"] allowed_domains = {"example.com"} output_dir = "docs_markdown" result = DocsSpider().start() result.items.to_jsonl("docs.jsonl")
- Yields one item per page with
url/title/markdown, and the optionaloutput_dirwrites one Markdown file per page. max_pagescaps the crawl, and since it builds onCrawlSpider, overridingrules()gives you full control over which links get followed.
- Yields one item per page with
- The MCP server is reworked (breaking) (Check the breaking changes and the docs):
- The 13 tools are now split into two modes: one-shot tools (
fetch,bulk_fetch,stealthy_fetch,bulk_stealthy_fetch) that always launch their own browser and show their real defaults, and session tools that work through a session opened once. - The new
session_fetchtool fetches through a browser session, whileopen_sessionnow holds the browser-level settings only and returns the session's effective settings for the AI agent. - The
gettool is renamed tomake_request, and it now supports any HTTP method. - The new
open_request_sessionandsession_make_requesttools give the AI persistent HTTP sessions that keep cookies and the browser fingerprint between requests. - This also ends fetches resetting the session's settings, first fixed by @Yigtwxx in #418.
- The 13 tools are now split into two modes: one-shot tools (
- The MCP server's HTTP transport now requires authentication and binds to localhost by default (breaking) by @yamantaka-singh in #414 (Fixes #413, check the docs):
- Pass
--auth-token(or theSCRAPLING_MCP_AUTH_TOKENenvironment variable) to require a bearer token, or--no-authto serve it unauthenticated on purpose. - Pass
--host 0.0.0.0to accept connections from the network.
- Pass
π Bug Fixes
- Cloudflare Turnstile/Interstitial solving now works regardless of the browser locale, and no longer loops forever on interactive challenges in headless mode. Stealth pages also stop crashing mid-solve, with contributions by @subediparas5 in #412. (Fixes #411 and #422)
- Fixed
find/find_allwithclass_silently missing multi-class elements by @yetval in #410, and blankclass_values and unescaped CSS string values by @yamantaka-singh in #417. - Fixed cached responses in
development_modelosing the request meta on replay by @Yigtwxx in #419. - Fixed HTTP requests with
retriesbelow 1 failing without sending the request by @Yigtwxx in #420.
Docs
- The website sections are restructured: a new "Using with AI" section holds the MCP server, the new Agent skill page, and the Building RAG systems guide, and the BeautifulSoup migration guide moved next to the Scrapy integration under "Integrations and migrations".
- Added a CHANGELOG.md to the repository with the notes of every release so far.
π Special thanks to the community for all the continuous testing and feedback
Big shoutout to our Platinum Sponsors
Release v0.4.14
A quick maintenance release to fix installation with uv π§
π Bug Fixes
- Fixed
uvrefusing to install v0.4.13 by default and silently falling back to an older version. The previous release required a prerelease version ofcurl_cffi, whichuvdoesn't allow unless explicitly enabled.
All dependencies now resolve to stable releases. (Fixes #407)
π Special thanks to the community for all the continuous testing and feedback
Big shoutout to our Platinum Sponsors
Release v0.4.13
A new update bringing feed spiders and a smarter MCP server π
Note
- Follow us on X for daily tips and tricks
- This will most likely be the last update before the major updates in v0.5
π New Stuff and quality of life changes
- New feed spider templates.
XMLFeedSpideriterates over the nodes of any XML feed (RSS, Atom, product feeds, etc.), andCSVFeedSpideriterates over CSV rows as dictionaries. Both decompress gzipped feeds automatically. (Check the docs)from scrapling.spiders import XMLFeedSpider class RSSSpider(XMLFeedSpider): name = "rss" start_urls = ["https://example.com/feed.xml"] async def parse_node(self, response, node): yield {"title": node.findtext("title"), "link": node.findtext("link")} result = RSSSpider().start()
- Upgraded the MCP server to MCP SDK v2 and made it smarter. The server now ships instructions that teach your AI agent how to use the tools efficiently; every tool declares annotations so clients like Claude Code can auto-approve the read-only ones; tool descriptions are leaner to save tokens; and the server advertises its version and logo to MCP clients. (Check the docs)
- Added a
scrapling-mcpcommand that maps directly toscrapling mcp, so registering Scrapling with MCP clients and registries that expect a single command is now a one-liner. - Unpinned Playwright/Patchright and browser versions. The generated browser User-Agent now always matches the exact Chromium version your installed Playwright/Patchright drives, so Scrapling no longer pins their versions and you can upgrade them freely. Run
scrapling install --forceafter updating to refresh the browsers.
π Bug Fixes
- Fixed importing Scrapling crashing with a
browserforgeValueError when the fingerprints data package lags behind the browser versions. (Fixes #394, #396, and #400) - Fixed the MCP bulk browser tools mis-sizing their page pools, which made
bulk_fetchfail on batches of more than 50 URLs andbulk_stealthy_fetchfetch all URLs through a single tab, by @Yigtwxx in #393.
Project
- New AI Contribution Policy: AI-assisted contributions are welcome but must be disclosed in the PR or issue; submissions that look like undisclosed AI output get labeled and closed.
π Special thanks to the community for all the continuous testing and feedback
Big shoutout to our Platinum Sponsors
Release v0.4.12
A release focused on making your spiders smarter about the websites they crawl
π New Stuff and quality of life changes
-
Spiders can now tune their own speed with AutoThrottle. Instead of guessing a
download_delaythat's either too slow or gets you banned, the spider measures how fast each website answers and adjusts the delay of every domain on its own. When a website starts blocking or rate-limiting you, it doubles the delay (or waits exactly what theRetry-Afterheader asks for) until that stops, then speeds back up. Yourdownload_delayand any robots.txtCrawl-delayare still respected as the minimum. (Check the docs)class MySpider(Spider): name = "adaptive" start_urls = ["https://example.com"] autothrottle_enabled = True autothrottle_start_delay = 2.0 autothrottle_max_delay = 30.0 autothrottle_block_backoff = True
-
Export your results to CSV and XML, next to the JSON/JSONL exporters you already had. Items that don't all share the same keys are still exported without losing anything, and nested values are written as JSON. (Check the docs)
result = MySpider().start() result.items.to_csv("products.csv") result.items.to_xml("products.xml")
-
The MCP server can now require authentication, so you can safely expose it instead of keeping it on your own machine. Any request without the token is rejected, and you can also restrict which hostnames the server answers to. (Check the docs)
scrapling mcp --http --auth-token "$(openssl rand -hex 32)" -
Browsers now accept CDP URLs over HTTP, not just WebSocket ones. So next to the
wss://endpoints managed browser providers hand out, you can now point any browser fetcher or MCP session at a Chrome you started yourself with--remote-debugging-port=9222. -
Published Docker images are now tagged with their release version instead of only
latest, so you can pin the exact version you want, by @JanRK in #384.
π Bug Fixes
-
Fixed cached responses losing all their cookies when the response came from a browser engine, which silently broke any session or auth logic relying on them while using the spiders' development mode, by @amitvijapur in #379. (Fixes #376)
-
Fixed
StealthyFetcherforcing theen-USlocale on every browser instead of following your system's, which made websites see a mismatch between your locale and your IP address and treat you as suspicious, like Google answering with 429s. (Fixes #381) -
Fixed a misleading error message in the storage system and removed a dead call left after inserts, by @fix2015 in #377.
Performance
get_all_text()is now O(nodes) instead of walking up the ancestors of every single text node, which makes it around 5-6x faster on deeply nested pages, by @yetval in #378.
π Special thanks to the community for all the continuous testing and feedback
Big shoutout to our Platinum Sponsors
Release v0.4.11
A solid update bringing the first platform spider template, a faster parser, and important fixes π
π New Stuff and quality of life changes
- Added
ShopifySpider, the first platform spider template! Extract every product from any Shopify-powered store through its JSON API without touching the website's HTML. Subclass it, set the store's domain, and you are done (Check the docs)from scrapling.spiders import ShopifySpider class MyStore(ShopifySpider): target_website = "example.com" result = MyStore().start()
- Added
--executable-pathto the CLI browser commands. Bothscrapling extract fetchandscrapling extract stealthy-fetchnow accept a custom Chromium-compatible browser executable, and fall back to theSCRAPLING_EXECUTABLE_PATHenvironment variable when the option isn't passed, bringing full parity with the MCP server (Solves #371)scrapling extract fetch "https://example.com" page.html --executable-path "/path/to/chromium"
π€ Quality of life changes
- Made
find_by_textandfind_by_regexup to ~2x faster whenfirst_matchis enabled (the default) by wrapping elements lazily so the search stops at the first match, by @yetval in #370 - Updated the benchmarks with the new numbers against the latest versions of all libraries.
- Updated contribution rules
π Bug Fixes
- Fixed the MCP server's fetch tools crashing on pages containing control characters with the error
All strings must be XML compatible, by @yetval in #368 (Fixes #366)
π Special thanks to the community for all the continuous testing and feedback
Big shoutout to our Platinum Sponsors:
Release v0.4.10
A new update with a brand-new Scrapy integration and a batch of community fixes π
π New Stuff and quality of life changes
-
Added a Scrapy integration so you can use Scrapling's parsing API inside your existing Scrapy projects without rewriting them. Put the
scrapling_responsedecorator on any spider callback, and the response it receives becomes a ScraplingResponsewhile Scrapy keeps handling the crawling (Check the docs):import scrapy from scrapling.integrations.scrapy import scrapling_response class QuotesSpider(scrapy.Spider): name = "quotes" start_urls = ["https://quotes.toscrape.com"] @scrapling_response def parse(self, response): # `response` is now a Scrapling Response first_quote = response.find_by_text("The world as we have created it", partial=True) for quote in [first_quote, *first_quote.find_similar()]: yield {"text": quote.get_all_text(strip=True)}
-
The MCP server can now use a custom Chromium-compatible browser for all browser-based tools. Set it once with
scrapling mcp --executable-path "/path/to/chromium"or theSCRAPLING_EXECUTABLE_PATHenvironment variable, or per request with theexecutable_pathargument, by @samrusani in #360 (Solves #347) -
Updated all browsers and fingerprints. Run
scrapling install --forceafter updating to refresh them.
π Bug Fixes
- Fixed garbled text (mojibake) from browser fetchers on non-UTF-8 websites by @yehudalevy-collab in #365 (Fixes #364).
- Fixed
LinkExtractornot filtering compound file extensions like.tar.gzby @renbkna in #359 (Fixes #349). - Fixed paused crawls losing their in-flight requests from checkpoints, so resuming no longer skips them by @yetval in #358.
- Fixed spiders calculating wrong crawl delays from robots.txt
Request-ratedirectives through the Protego upgrade, with tests aligned by @Disaster-Terminator in #355.
Docs
- Clarified how
init_scriptinteracts with Patchright's isolated execution context in stealth mode by @mturac in #353 (Solves #350). - Added the skills.sh install method for the agent skill by @ob-aion in #363.
π Special thanks to the community for all the continuous testing and feedback
Big shoutout to our Platinum Sponsors
Release v0.4.9
A maintenance update packed with community-reported fixes π οΈ
π New Stuff and quality of life changes
- Updated all browsers and fingerprints. Run
scrapling install --forceafter updating to refresh them. - Added a
--versionflag to the CLI by @ETM-Code in #303 (Solves #299)
π Bug Fixes
- Fixed the session-level
proxyargument being silently ignored in HTTP sessions, which could leak your real IP (Solves #295). Note that mixing a session-levelproxywith a per-requestproxiesargument (or vice versa) now raises an error instead of one being silently dropped. - Fixed browser navigations failing when combining
init_scriptwithuser_data_dir(Solves #294). - Fixed encoding detection when websites quote the charset value in the
Content-Typeheader by @Bortlesboat in #323. - Fixed an
IndexErrorin adaptive element relocation whenauto_saveis enabled by @Mubashirrrr in #340. - Fixed spiders' checkpoint and cache saving crashing on Windows by @MrStarkEG in #344.
- Fixed incorrect similarity scoring in
find_similarfor elements with mismatched attribute counts (Solves #322).
Docs
- Clarified that the default installation includes the parser engine only, and the fetchers/spiders need the extras (Solves #343).
- Fixed the Docker image name in the remaining examples by @evanclan in #315.
- Fixed a broken link in the contribution guide by @Bortlesboat in #320.
π Special thanks to the community for all the continuous testing and feedback
Big shoutout to our Platinum Sponsors
Release v0.4.8
A big spider update that takes the crawling framework to the next level π·οΈ
π New Stuff and quality of life changes
-
Added a
LinkExtractorprimitive inscrapling.spiders.LinkExtractorto pull URLs out of aResponse. There are a lot of controls (Check the docs)from scrapling.spiders import LinkExtractor extractor = LinkExtractor(allow=r"/posts/", deny_domains=["ads.example.com"])
-
Added
CrawlSpiderandCrawlRulegeneric spider templates so you no longer have to hand-write the same "follow links matching this pattern" boilerplate. Overriderules()to return a list ofCrawlRuleobjects, each pairing aLinkExtractor. (Check the docs)from scrapling.spiders import CrawlSpider, CrawlRule, LinkExtractor class QuotesSpider(CrawlSpider): name = "blog" start_urls = ["https://quotes.toscrape.com/"] def rules(self): return [ CrawlRule(LinkExtractor(allow=r"/author/"), callback=self.parse_author), CrawlRule(LinkExtractor(allow=r"/page/\d+/")), # pagination, no callback ] async def parse_author(self, response): yield { "name": response.css(".author-title::text").get(), "birthday": response.css(".author-born-date::text").get(), "url": response.url, }
-
Added a
SitemapSpidertemplate that seeds a crawl directly from a sitemap, orrobots.txtURLs. Handles gzip-compressed sitemaps, and a lot of controls and options. URLs are dispatched via the crawl rules as shown above for CrawlSpider. (Check the docs)from scrapling.spiders import SitemapSpider, CrawlRule, LinkExtractor class NewsSitemap(SitemapSpider): name = "news" sitemap_urls = ["https://example.com/robots.txt"] def rules(self): return [ CrawlRule(LinkExtractor(allow=r"/articles/"), callback=self.parse_article), ] async def parse_article(self, response): yield {"url": response.url, "title": response.css("h1::text").get()}
-
Adaptive relocation now defaults to a 40% similarity threshold instead of
0across all methods. This will make the adaptive feature work better. When nothing crosses the threshold, a warning now tells you the top score it did see, so you can lowerpercentagedeliberately if needed. -
Updated all browsers and fingerprints. Run a new
scrapling install --forceafter updating to refresh the browsers and fingerprints.
π Bug Fixes
- Fixed
Fetcher.configure(...)not applying to per-request calls. Same fix applied toAsyncFetcher. - Fixed incorrect request fingerprinting that caused duplicate requests in spiders by @yetval in #255.
- Fixed the Adaptive scraping engine staying silent on weak matches. Combined with the threshold change above, you now get a warning instead of a misleading "best guess" element when relocation fails.
Docs
- Refreshed older code examples across the documentation to match the current version.
- Improved the code copy-paste experience on the docs site and trimmed the agent skill so it uses fewer tokens per invocation.
π Special thanks to the community for all the continuous testing and feedback
Big shoutout to our Platinum Sponsors