Kafka and event streams: a log, not a queue
Kafka is a replicated, partitioned, append-only log that consumers read at their own pace, and nearly every design question about it (ordering, scaling, durability, replay) is really a question about partitions and offsets.
What you'll be able to do
- Describe topics, partitions, offsets and consumer groups, and say what each one guarantees
- Choose a partition key that preserves the ordering the product needs without creating a hot partition
- Size a topic's partition count and storage from the event rate and retention
- Configure acknowledgements and in-sync replicas for a stated durability requirement
- Decide between Kafka and a per-message queue, and explain the outbox pattern for publishing from a database
Before this: queues-and-backpressure, replication-and-consistency
Log in to read the rest of this lesson
Kafka is a replicated, partitioned, append-only log that consumers read at their own pace, and nearly every design question about it (ordering, scaling, durability, replay) is really a question about partitions and offsets. This walkthrough runs about 24 minutes, with runnable code you can edit and re-run as you read. A new account gets 7 days of Pro free, no card required.
Log in to continueThe first module of every track is free to read on the web: see what's open in System design.