Crawl4AI: a page to clean, LLM-ready markdown (no API key)

The Stack

Tested Against

Prerequisites

• pip install crawl4ai + its headless browser (only to actually crawl)

Steps

1. 1

   Write the run script and structure-check it

   Write crawl.py using AsyncWebCrawler with a cache mode (politeness and repeatability) that prints result.markdown. CI compiles it and checks it uses the right calls. Running it needs crawl4ai and a browser installed, so that step is fenced.

   cat > crawl.py <<'PY'
   import asyncio
   from crawl4ai import AsyncWebCrawler, CacheMode
   
   
   async def main():
       async with AsyncWebCrawler() as crawler:
           result = await crawler.arun(
               url="https://example.com",
               cache_mode=CacheMode.ENABLED,
           )
           print(result.markdown)
   
   
   asyncio.run(main())
   PY
   python3 - <<'CHECK'
   import py_compile, sys
   src = open("crawl.py").read()
   try:
       py_compile.compile("crawl.py", doraise=True)
   except py_compile.PyCompileError:
       print("BAD: crawl.py does not compile"); sys.exit(1)
   def need(tok, msg):
       if tok not in src:
           print("BAD: " + msg); sys.exit(1)
   need("AsyncWebCrawler", "script does not use crawl4ai AsyncWebCrawler")
   need(".arun(", "script never runs a crawl (arun)")
   need("markdown", "script does not request markdown output")
   need("CacheMode", "script sets no cache mode (politeness / repeatability)")
   print("config OK: crawl4ai script compiles and uses AsyncWebCrawler.arun to produce markdown with a cache mode set")
   CHECK

2. 2

   Run the crawl (the browser/network step, not checked by CI)

   pip install crawl4ai, install its browser, and run the script against a site you are allowed to crawl. It renders the page and prints clean markdown ready to chunk into a knowledge base. The fetch is fenced.