How Kafka Works: Partitions, Offsets and Consumer Groups Explained

Most confusion about Kafka comes from one wrong assumption: that it is a message queue. It isn't. Kafka is a distributed, append-only log, and nearly every behaviour that surprises newcomers — why messages aren't deleted when you read them, why ordering breaks, why an extra consumer sits idle doing nothing — follows directly from that one design choice. This guide builds the model from the ground up.

Kafka is a log, not a queue

In a traditional queue (RabbitMQ, SQS, ActiveMQ), a message is a unit of work. A consumer takes it, the broker hands it out, and once acknowledged it is removed. The queue is a to-do list that shrinks as work gets done.

Kafka works nothing like that. A Kafka topic is an append-only file. Producers append records to the end. Consumers read from wherever they choose and simply remember how far they got. Reading changes nothing about the log itself.

That single difference produces the properties Kafka is known for:

• Many independent consumers. Ten different services can all read the same record, because reading is not consumption. Your analytics pipeline, your audit logger, and your email service each keep their own position.
• Replay. Since records aren't destroyed, you can rewind and reprocess. Deploy a bug fix, reset your position to yesterday, and run the corrected code over the same data.
• Enormous throughput. Appending to the end of a file and reading sequentially are the two operations disks and page caches are best at. There is no per-message bookkeeping to update.

Hold on to the mental image of a file you append to and read from at your own pace. Everything below is a refinement of it.

Topics and partitions

A topic is the name you publish to — orders, page-views, user-signups. It is a logical label, not a physical thing.

The physical thing is the partition. Each topic is split into one or more partitions, and each partition is an independent append-only log stored on a broker. When you create a topic with three partitions, you are creating three separate logs that happen to share a name.

Every record in a partition gets a sequential ID called its offset, starting at 0 and incrementing forever. Offsets are per-partition: partition 0 and partition 1 both have a record at offset 0, and they are entirely unrelated records.

Partitions exist for one reason: parallelism. A single log can only be written and read so fast, and it must fit on one machine. Splitting a topic into partitions spreads it across brokers and lets many consumers work simultaneously.

The cost of that parallelism is the single most important rule in Kafka:

Kafka guarantees ordering within a partition. It guarantees nothing across partitions.

Records in partition 0 come back in exactly the order they were written. But a record written to partition 0 and a record written to partition 1 have no defined order relative to each other — they are separate files being read by separate threads. If you publish OrderCreated and OrderCancelled for the same order and they land in different partitions, a consumer can genuinely see the cancellation first.

This trips up almost everyone once. "Kafka preserves order" is true, but only at partition scope. Which brings us to the mechanism that decides partition placement.

Producers and message keys

When a producer sends a record, something has to choose its partition. That choice is driven by the record's key.

A Kafka record is essentially a key, a value, a timestamp, and some optional headers. The value is your payload — typically JSON, Avro, or Protobuf. The key is optional, and its primary job is routing:

// key present: hash the key, mod the partition count
partition = hash(key) % numberOfPartitions

// key null: records are batched onto partitions
// (the "sticky partitioner" fills one batch, then switches)
partition = stickyBatchTarget()

Two consequences follow, and they are the whole reason keys matter:

1. The same key always routes to the same partition, so all records for that key stay strictly ordered relative to each other. Key your order events by order_id and every event for order 12345 lands in one partition, in order, forever.
2. Key choice determines load distribution. Because placement is a hash of the key, a poorly chosen key concentrates traffic. Keying by country when 80% of your users are in one country sends 80% of your traffic to one partition — a "hot partition" that becomes the bottleneck for the entire topic while its siblings idle.

The practical rule: choose the narrowest key that still gives you the ordering you actually need. order_id over customer_id, customer_id over region. And if you genuinely don't need ordering, leave the key null and let Kafka spread the load evenly.

One caveat worth knowing before you go to production: adding partitions to an existing keyed topic changes future routing for existing keys. Records already written don't move — they stay in whatever partition they were originally placed in, forever. But since placement is hash(key) % partitionCount, a larger partition count means new records for the same key can land in a different partition than the old ones did. The ordering guarantee holds within each partition as always; what breaks is the assumption that "same key" still means "same partition" across that resize boundary. Size your partition count with room to grow, so you resize rarely.

When you pull a record off a topic to inspect it, the value is usually a deeply nested JSON blob printed as one unreadable line. Paste it into the JSON Formatter to pretty-print it, or the JSON Viewer to expand and collapse the structure while you hunt for the field you care about — both run entirely in your browser, so payloads never leave your machine.

Offsets: the part people get wrong

An offset is just a position in a partition. But there are three distinct numbers that people routinely conflate, and telling them apart is what makes lag and replay make sense:

Consumer lag is the gap between the log-end offset and the committed offset — how far behind you are. It is the single most important number to alert on for any Kafka consumer.

The committed offset is the interesting one. Kafka stores it in an internal topic called __consumer_offsets, keyed by group, topic, and partition. Note what that means: the position belongs to the consumer group, not to the consumer process. Restart your service, deploy a new version, scale from two pods to ten — the group's position survives, because it was never held in your application's memory.

This is also why "reprocess from yesterday" is a routine operation rather than a data-recovery exercise. You are not restoring anything; you are just writing a smaller number into __consumer_offsets:

# how far behind is each partition? (LAG is the column that matters)
kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
  --describe --group order-processor

# TOPIC   PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG
# orders  0          15234           15240           6
# orders  1          15100           15980           880
# orders  2          15877           15880           3

# rewind the whole group to a point in time (consumers must be stopped)
kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
  --group order-processor --topic orders \
  --reset-offsets --to-datetime 2026-07-19T00:00:00.000 --execute

Notice partition 1 in that output: lag of 880 while its siblings sit at single digits. That asymmetry is the signature of a hot partition — a key-choice problem, not a consumer problem.

Every Kafka record also carries a timestamp, exposed as milliseconds since the Unix epoch. When you are correlating a lagging offset against your application logs, or working out what --to-datetime value to pass, a raw value like 1784505600000 is hard to reason about — drop it into the Timestamp Converter to turn it into a readable date (and mind the milliseconds-vs-seconds distinction, which is a classic source of "my reset went to 1970" confusion).

Consumer groups and rebalancing

A consumer group is a set of consumers that cooperate to read a topic. The rule that governs them is short:

Within a group, each partition is assigned to exactly one consumer. A consumer may hold several partitions.

So the partition count is a hard ceiling on parallelism inside a group. Four partitions and two consumers means two partitions each. Four partitions and four consumers means one each. Four partitions and five consumers means one consumer sits idle, receiving nothing at all — and this is the answer to one of the most common Kafka support questions.

Crucially, this limit applies within a group. Different groups are completely independent — the analytics group and the billing group both receive every record and track their own offsets. That is how Kafka does publish/subscribe: one group per subscriber.

When group membership changes — a consumer joins, crashes, or is deemed dead — Kafka triggers a rebalance to reassign partitions. Historically this was "eager": every consumer stopped, gave up all partitions, and waited for a fresh assignment, producing a brief total pause often called a stop-the-world rebalance.

Cooperative (incremental) rebalancing improves on this by moving only the partitions that actually need to change hands, letting everything else keep processing. It is worth knowing exactly what is on by default, because this is widely misreported: the default partition.assignment.strategy is [RangeAssignor, CooperativeStickyAssignor]. RangeAssignor — eager — is what actually runs by default; CooperativeStickyAssignor is present in the list only so you can migrate to it via a rolling restart without downtime. If you want cooperative rebalancing you must configure it explicitly.

Apache Kafka 4.0 (released March 2025) also introduced a redesigned, broker-coordinated consumer group protocol that removes much of the client-side rebalance dance entirely. If you are on 4.x it is worth reading up on; on 3.x the assignor configuration above is what governs your behaviour.

Retention and compaction

Since consuming doesn't delete anything, something else has to. Kafka offers two policies, set per topic via cleanup.policy.

Delete (the default) discards records once they age out or the partition grows too large:

# keep 7 days of data (the default), regardless of content
cleanup.policy=delete
retention.ms=604800000
retention.bytes=-1        # -1 = no size limit, time is the only trigger

Retention is the reason "replay from the beginning" has a floor: you can only rewind as far back as your retention window still holds data. It is also why a consumer that has been down longer than the retention period comes back to find its committed offset no longer exists in the log — a situation that surfaces as an OffsetOutOfRange condition.

Those retention values are configured in milliseconds, which makes them easy to misread by an order of magnitude — 604800000 is seven days but 60480000 is under a day, and the two differ by one digit. The Timestamp Converter is a quick sanity check when you are reading a duration or a record timestamp out of a config or a log line.

Compact takes a completely different approach: instead of deleting by age, it keeps the most recent record for every distinct key, indefinitely:

cleanup.policy=compact

# before compaction:
#   user-1 -> {"plan":"free"}
#   user-2 -> {"plan":"pro"}
#   user-1 -> {"plan":"pro"}      <-- newer value for user-1
#
# after compaction:
#   user-2 -> {"plan":"pro"}
#   user-1 -> {"plan":"pro"}      <-- only the latest survives

A compacted topic is a changelog: the current state of every key, durable forever. It is how Kafka Streams backs its state stores, and how __consumer_offsets itself works. Sending a record with a key and a null value writes a tombstone, which marks that key for deletion.

Compaction is the clearest case where the message key is doing real work beyond routing — on a compacted topic, the key defines record identity.

Brokers, replicas and leaders

A broker is one Kafka server; a cluster is several. Partitions are distributed across brokers, and each partition is replicated according to the topic's replication factor (3 is the common production choice).

For each partition, one replica is the leader and the rest are followers. All reads and writes for that partition go through its leader; followers continuously copy from it. If a leader's broker dies, one of the followers is promoted and clients transparently reconnect to the new leader.

The set of replicas currently caught up with the leader is the in-sync replica set (ISR). The ISR is what determines whether a write is considered durable, and it is central to Kafka's delivery guarantees — along with producer acknowledgements, idempotence, and the honest limits of "exactly-once". That is a big enough topic to deserve its own treatment, which is why it has one: Kafka Delivery Guarantees picks up exactly here.

One piece of version context worth carrying: Kafka historically depended on ZooKeeper for cluster metadata. KRaft mode, which moves that responsibility into Kafka itself, became production-ready in 3.3 and ZooKeeper was removed entirely in Kafka 4.0. If you are reading older tutorials that walk you through starting a ZooKeeper process, they predate 4.0.

Where schemas fit in

Everything so far treats a record's value as opaque bytes — Kafka genuinely does not care what is inside. That flexibility becomes a liability the moment more than one team is involved, because nothing stops a producer from changing the shape of its payload and silently breaking every downstream consumer.

That is the problem a schema registry solves: producers and consumers agree on a contract, and the registry enforces that new versions of it stay compatible. If you are running JSON on your topics today, the same compatibility questions apply even without Avro — adding a required field is a breaking change whatever your serialization format.

Our companion guide covers this in depth: Kafka Avro Schema: Backward Compatibility Explained walks through schema evolution, the backward/forward/full compatibility modes, and which changes are safe versus which will break your consumers at deserialization time.

Putting the model together

If you remember five things from this guide, make them these:

1. Kafka is an append-only log. Reading doesn't delete; consumers just track a position.
2. A topic is a logical name; partitions are the real logs. Ordering is guaranteed per partition only.
3. The message key picks the partition — so it controls both ordering and load balance.
4. Within a group, one partition goes to exactly one consumer. Partition count caps your parallelism.
5. Retention deletes by age or size; compaction keeps the latest record per key forever.

Almost every Kafka question that starts with "why is it doing that?" resolves to one of those five.