Scrape politely: honor robots.txt and a crawl delay (the part most skip)

The Stack

Tested Against

Prerequisites

• Python 3
• A scraper to wrap (Scrapy, Crawl4AI, requests) for real use

Steps

1. 1

   Build the permission-and-pace gate and test it on a fixture

   Parse a robots.txt with urllib.robotparser, then check an allowed path passes, a disallowed path is blocked, and the crawl-delay is read so your scraper can wait between requests. CI runs this against a fixture. In real use you point set_url at the live robots.txt, which is the fenced step.

   cat > robots.txt <<'TXT'
   User-agent: *
   Disallow: /private/
   Allow: /public/
   Crawl-delay: 2
   TXT
   python3 - <<'PY'
   import sys
   from urllib.robotparser import RobotFileParser
   
   rp = RobotFileParser()
   with open("robots.txt") as f:
       rp.parse(f.read().splitlines())
   rp.modified()  # mark as read so can_fetch / crawl_delay are live
   
   ua = "FlowStacksBot"
   allowed = rp.can_fetch(ua, "https://site.example/public/page")
   blocked = rp.can_fetch(ua, "https://site.example/private/secret")
   delay = rp.crawl_delay(ua) or 0
   
   if not allowed:
       print("BAD: an allowed path was reported as blocked"); sys.exit(1)
   if blocked:
       print("BAD: a disallowed path was not blocked"); sys.exit(1)
   if delay < 1:
       print("BAD: crawl-delay was not read / honored"); sys.exit(1)
   
   print("scraping OK: robots.txt respected (allowed /public, blocked /private) and crawl-delay of " + str(int(delay)) + "s honored")
   PY

2. 2

   Wrap your real scraper (the network step, not checked by CI)

   Point the parser at the live site's robots.txt (rp.set_url + rp.read()), call can_fetch before every URL, and sleep the crawl-delay between requests. Scrapy has ROBOTSTXT_OBEY and DOWNLOAD_DELAY for the same thing. The live fetches are fenced.