Apache Kafka Explained: Topics, Partitions, and Consumer Groups

Apache Kafka is a distributed log for events — a system that lets one part of your architecture publish a stream of facts ("order placed," "payment captured," "email sent") and lets many other parts read that stream independently, at their own pace, without ever talking to each other directly. It is the backbone of most event-driven systems at scale, and it is also the piece newcomers most often misunderstand, because it looks like a message queue but behaves like an append-only database. This guide is Apache Kafka explained from the primitives up — topics, partitions, offsets, and consumer groups — so the whole model clicks. It sits in the Devgains architecture cluster next to event-driven architecture, because Kafka is the machinery that most event-driven designs actually run on.

Quick answer: what is Apache Kafka?

Apache Kafka is a distributed, append-only commit log for streaming events. Producers append records to the end of a log; consumers read forward through it and remember their position. The four primitives you must understand:

• Topic — a named stream of records (e.g. orders). This is the unit you publish to and subscribe from.
• Partition — a topic is split into ordered, append-only partitions. Partitions are the unit of parallelism and ordering.
• Offset — each record's sequential position within a partition. Consumers track offsets to know what they have already read.
• Consumer group — a set of consumers that cooperate to read a topic, with each partition assigned to exactly one consumer in the group.

The one-line mental model: Kafka is not a queue that deletes messages when you read them; it is a durable log you replay by moving a pointer. Reading does not remove data — it only advances your offset.

Why Kafka matters

Point-to-point integrations rot. When ten services all call each other directly, every new consumer of "an order was placed" means changing the order service. Kafka inverts that: the producer writes the event once and is done, and any number of consumers subscribe without the producer knowing they exist. That decoupling is why Kafka shows up everywhere from payments to analytics. Three properties make it worth the operational weight:

• Durability and replay. Records persist on disk for a configurable retention window (hours, days, or forever). A new consumer — or one recovering from a bug — can rewind and reprocess history. A traditional queue that deletes on delivery cannot do this.
• Horizontal scale through partitions. A single topic can be spread across dozens of partitions on many brokers, so throughput scales by adding partitions and consumers rather than by buying a bigger box.
• Ordering where it counts. Kafka guarantees order within a partition, which — with the right key — is exactly the ordering guarantee most systems actually need.

Kafka is the transport that makes patterns like the outbox pattern and the saga pattern practical across services.

Architecture: how the pieces fit

A Kafka deployment is a cluster of brokers (servers). Topics are split into partitions, and each partition is stored on one broker as the leader, with copies on other brokers as followers for fault tolerance (the replication.factor). Producers write to the leader; followers replicate.

Here is the flow of a single event:

1. A producer sends a record to a topic. If the record has a key, Kafka hashes it to pick a partition; without a key, it spreads records across partitions (round-robin-ish). Same key → same partition → guaranteed order for that key.
2. The partition leader appends the record to its log and assigns it the next offset. Followers replicate it; once enough replicas have it (governed by acks), the write is acknowledged.
3. Consumers in a consumer group each own a subset of the topic's partitions and read forward, periodically committing their offset so they can resume after a restart.

The critical constraint follows directly: a partition is consumed by at most one consumer within a group. So your maximum consumer parallelism for a topic equals its partition count. Ten partitions means at most ten active consumers in a group — an eleventh sits idle. This is the single most important number to get right up front, because you cannot easily reduce partitions later.

Multiple groups read the same topic independently. The analytics group and the fraud group both see every record, each tracking its own offsets — this is how one stream fans out to many consumers without duplicating the data.

Step-by-step: produce and consume a topic

Let's make it concrete. First, create a topic with several partitions so it can scale:

# Create an `orders` topic with 6 partitions and 3 replicas
kafka-topics.sh --create \
  --topic orders \
  --partitions 6 \
  --replication-factor 3 \
  --bootstrap-server broker1:9092

Six partitions means up to six consumers can process orders in parallel within one group. Replication-factor 3 means each partition survives losing two brokers.

Now a producer. The key is the important part — using orderId as the key guarantees that all events for a given order land in the same partition and are processed in order:

var props = new Properties();
props.put("bootstrap.servers", "broker1:9092");
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("acks", "all"); // wait for all in-sync replicas before acknowledging
 
try (var producer = new KafkaProducer<String, String>(props)) {
    // key = orderId → all events for this order share a partition and stay ordered
    var record = new ProducerRecord<>("orders", "order-42", "{\"status\":\"placed\"}");
    producer.send(record); // async; returns a Future you can await for the offset
}

acks=all is the durability knob: the write is only acknowledged once every in-sync replica has it, so an acknowledged record survives a broker failure. acks=1 (leader only) is faster but can lose data if the leader dies before followers catch up.

Now a consumer, joined to the billing group. Kafka automatically assigns partitions across every consumer in the group and rebalances when members join or leave:

var props = new Properties();
props.put("bootstrap.servers", "broker1:9092");
props.put("group.id", "billing");          // the consumer group
props.put("enable.auto.commit", "false");  // commit manually, after work succeeds
props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
 
try (var consumer = new KafkaConsumer<String, String>(props)) {
    consumer.subscribe(List.of("orders"));
    while (true) {
        var records = consumer.poll(Duration.ofMillis(500));
        for (var record : records) {
            process(record.value());        // do the real work first
        }
        consumer.commitSync();              // then commit the offset — at-least-once
    }
}

The ordering here matters: process, then commit. If the consumer crashes after processing but before committing, it reprocesses those records on restart — at-least-once delivery. That means your handlers must be idempotent, which is exactly what idempotency keys are for. Commit before processing and you get at-most-once (possible data loss) instead.

Kafka vs a traditional message queue

The takeaway: reach for Kafka when the event stream itself is valuable and many consumers need it; reach for a task queue when you're handing out units of work to be done once and discarded.

Best practices

• Choose a partition key deliberately. Order is only guaranteed per partition, so key by the entity whose events must stay ordered (userId, orderId). No key means no ordering guarantee.
• Over-provision partitions modestly. You can add partitions but not remove them, and adding them changes key→partition mapping. Start with headroom (e.g. 2–3× current consumers), not 500 partitions "just in case."
• Set acks=all for anything you can't lose, paired with min.insync.replicas=2, so a write is only acknowledged when it is genuinely replicated.
• Make consumers idempotent and commit offsets after successful processing to get safe at-least-once semantics.
• Right-size retention. Retention is your replay window and your disk bill. Match it to how far back a consumer might realistically need to rewind.
• Monitor consumer lag. Growing lag (offsets behind the log's end) is the earliest signal that consumers can't keep up — surface it in your metrics.

Common mistakes

• Treating Kafka as a queue that deletes on read. It doesn't. Reading advances an offset; retention deletes data, not consumption.
• One partition for an ordered topic, then wondering why it won't scale. One partition caps you at one consumer. Partition count is your parallelism ceiling.
• No message key, then expecting global ordering. Kafka orders per partition only; without a key your records scatter and per-entity order is lost.
• Committing offsets before processing. A crash then silently drops those records (at-most-once).
• Ignoring rebalances. Every time a consumer joins or leaves, partitions are reassigned and processing briefly pauses. Frequent restarts cause constant rebalancing and stalled throughput.

Key takeaways

• Kafka is a durable, replayable log, not a delete-on-read queue. Consumers move a pointer; the data stays.
• Partitions are the unit of ordering and parallelism. Order is guaranteed within a partition, and partition count caps consumer concurrency in a group.
• Keys route records to partitions. Same key → same partition → ordered.
• Consumer groups fan work out; multiple groups fan the same stream out to many readers.
• Durability and delivery are knobs (acks, min.insync.replicas, commit timing) — set them to match how much you can afford to lose.

FAQ

What is Apache Kafka used for? Kafka is used to move streams of events between systems: publishing facts like "order placed" or "metric recorded" once and letting many independent consumers read them for billing, analytics, search indexing, fraud detection, and more. It excels at high-throughput, durable, replayable event streaming and as the transport for event-driven architectures.

What is the difference between a topic and a partition? A topic is the named stream you publish to and subscribe from. A partition is one ordered, append-only slice of that topic. A topic is split into partitions so it can be stored across brokers and consumed in parallel — ordering is guaranteed within a partition, not across the whole topic.

What is a consumer group in Kafka? A consumer group is a set of consumers that cooperate to read a topic. Each partition is assigned to exactly one consumer in the group, so the group's members share the work. Different consumer groups read the same topic independently, each tracking its own offsets — that's how one stream fans out to many applications.

Does Kafka delete messages after they're read? No. Reading only advances a consumer's offset; the record stays on disk until it falls outside the topic's retention window (a time or size limit). This is why a new or recovering consumer can rewind and reprocess history — something a delete-on-consume queue cannot do.

How does Kafka guarantee message ordering? Kafka guarantees order within a single partition. Records are appended in arrival order and assigned increasing offsets. To keep all events for one entity ordered, give them the same message key so they hash to the same partition — Kafka does not guarantee ordering across partitions.

Conclusion

Apache Kafka stops being mysterious once you see it for what it is: an append-only log you publish to and replay from, split into partitions that provide both ordering and parallelism, read by consumer groups that each remember their offset. Get the partition key right, size partitions for your parallelism, tune acks and commit-after-process for the durability you need, and Kafka becomes the reliable spine of an event-driven system. From here, see how the events themselves are designed in event-driven architecture, how to publish them reliably with the outbox pattern, how to coordinate multi-service workflows with the saga pattern, and browse the full architecture category.

References

• Apache Kafka documentation: Introduction — the authoritative overview of topics, partitions, brokers, and the log.
• Kafka design: the log and partitions — how the commit log, replication, and delivery guarantees work.
• Consumer groups and offsets — how groups share partitions and track position.
• Producer acks and durability — the acknowledgement settings that control data safety.