System design · Cases: ten designs, worked end to end

Design a web crawler

A crawler is a loop (fetch a page, find its links, fetch those) that becomes a design problem only at scale, where three constraints collide. A throughput target measured in thousands of pages a second, a politeness rule that limits how hard you may hit any one site, and a memory of billions of URLs you have already seen.

26 min read Full lesson in the app Patterns: politeness-queues, bloom-seen-set, partition-by-host

What you'll be able to do

Before this: queues-and-backpressure, blob-storage-and-cdns

Log in to read the rest of this lesson

A crawler is a loop (fetch a page, find its links, fetch those) that becomes a design problem only at scale, where three constraints collide. A throughput target measured in thousands of pages a second, a politeness rule that limits how hard you may hit any one site, and a memory of billions of URLs you have already seen. This walkthrough runs about 26 minutes, with runnable code you can edit and re-run as you read. A new account gets 7 days of Pro free, no card required.

Log in to continue

The first module of every track is free to read on the web: see what's open in System design.