Design a web crawler
A crawler is a loop (fetch a page, find its links, fetch those) that becomes a design problem only at scale, where three constraints collide. A throughput target measured in thousands of pages a second, a politeness rule that limits how hard you may hit any one site, and a memory of billions of URLs you have already seen.
What you'll be able to do
- Derive pages per second, bandwidth and fetches in flight from a crawl size and a time window
- Design a frontier that enforces per-host politeness without a global lock, by partitioning on host
- Size a Bloom filter for the seen-URL set and explain what its false positives cost a crawler
- Make the crawl survive crashes and avoid traps, with checkpoints, retries and explicit limits
Before this: queues-and-backpressure, blob-storage-and-cdns
Log in to read the rest of this lesson
A crawler is a loop (fetch a page, find its links, fetch those) that becomes a design problem only at scale, where three constraints collide. A throughput target measured in thousands of pages a second, a politeness rule that limits how hard you may hit any one site, and a memory of billions of URLs you have already seen. This walkthrough runs about 26 minutes, with runnable code you can edit and re-run as you read. A new account gets 7 days of Pro free, no card required.
Log in to continueThe first module of every track is free to read on the web: see what's open in System design.