Open SourceFrameworkFree

Scrapy

The veteran industrial-strength Python scraping framework: crawl millions of pages, extract with selectors, and export clean data. BSD-3-Clause, battle-tested for over a decade.

Homepage ↗Repo ↗

Alternatives

CrawleeAutoScraper

1 workflow use Scrapy

AutomationFreeMachine-verified

Scrape politely: honor robots.txt and a crawl delay (the part most skip)

Gate any scraper behind a robots.txt check and a crawl delay so you only fetch what a site allows, at a rate it allows, using nothing but the Python standard library.

Scrapy· beginner