Real-World Kafka Problems and How to Fix Them

Kafka is reliable in production and confusing in production, and those aren't contradictory. Most incidents aren't Kafka breaking — they're Kafka doing exactly what it was configured to do, in a way nobody expected. This guide walks through eight failures you will actually meet, what each symptom really means, and how to fix it. If the concepts feel shaky, How Kafka Works builds the model these problems all sit on.

1. The consumer group rebalances forever

Symptom: throughput collapses. Logs fill with revocation and reassignment messages, and periodically:

org.apache.kafka.clients.consumer.CommitFailedException: Offset commit cannot be
completed since the consumer is not part of an active group for auto partition
assignment; it is likely that the consumer was kicked out of the group.

What's really happening: your consumer is taking too long between calls to poll(). Heartbeats run on a separate background thread and keep reassuring the coordinator the process is alive, so only the missed poll() itself exposes the stall — and max.poll.interval.ms (default 5 minutes) is what catches it. The full mechanism, including why session.timeout.ms doesn't catch this case, is covered in CommitFailedException: Offset commit cannot be completed.

Then the loop closes on itself: the coordinator evicts the consumer and rebalances, the replacement picks up the same oversized batch, takes just as long, and gets evicted too. The group never stabilises.

Notice what this costs beyond the rebalance itself: the evicted consumer typically did finish processing that batch — it just never got to commit before eviction. The partition's new owner has no way to know that, so it re-fetches from the last committed offset and reprocesses those same records. That's ordinary at-least-once behaviour, not corruption, but it's only harmless if your processing is idempotent (see Kafka Delivery Guarantees for what that requires in practice).

The fix is almost always to make each batch smaller rather than to extend the deadline:

# preferred: fewer records per poll, so each batch finishes well inside the window
max.poll.records=100              # default is 500

# only if the work genuinely takes this long (e.g. large batch DB writes)
max.poll.interval.ms=600000       # 10 minutes

Do the arithmetic before you pick a number: if one record takes 200 ms and you fetch 500 of them, that batch needs 100 seconds. Comfortably inside 5 minutes — until a slow dependency doubles your per-record time and it isn't. Size max.poll.records so the worst realistic batch still finishes with room to spare.

When a consumer misbehaves in one environment but not another, the cause is often a config that drifted rather than the code. Exporting both configs and running them through JSON Diff shows you the changed keys immediately, which is faster than reading two long property files side by side.

2. Consumer lag grows and never drains

Symptom: lag climbs steadily. The consumer is running and healthy, just permanently behind.

kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
  --describe --group order-processor

# TOPIC   PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG
# orders  0          15234           15240           6
# orders  1          15100           98700           83600      <-- one partition
# orders  2          15877           15880           3

Partitions 0 and 2 are the same healthy numbers from the describe output in How Kafka Works — single-digit lag, nothing to see. Partition 1 is the same partition, later, once the backlog behind it has actually piled up. This is what "hot partition" looks like once it's had time to compound.

Read the shape of the lag before you touch anything — it tells you which of two completely different problems you have:

• Lag spread evenly across all partitions → you are genuinely under-provisioned. Consumers cannot keep up with the incoming rate. Add consumers (up to the partition count), or make processing faster — the usual levers are moving slow work (a DB write, an external call) onto a separate async worker so poll() keeps cycling, batching those writes instead of doing one round-trip per record, and raising fetch.min.bytes so the consumer pulls fewer, larger batches instead of paying per-request overhead on every poll.
• Lag concentrated on one partition → a hot partition. Adding consumers will do nothing, because that partition is already assigned to exactly one consumer and no other consumer in the group is permitted to help. See problem 7.

If lag is even and you want more parallelism, remember the ceiling: consumers beyond the partition count sit idle. Scaling from 3 consumers to 10 against a 3-partition topic adds seven idle processes and zero throughput.

When you are reconciling lag against your application logs to find when the backlog started, Kafka's record timestamps come through as epoch milliseconds. The Timestamp Converter turns a value like 1784505600000 into a readable date — and catches the classic seconds-versus-milliseconds mix-up that otherwise puts your incident in 1970.

Everything in this list is reactive — found after someone notices something is slow. An alert on records-lag-max (or the equivalent from your monitoring stack) crossing a threshold would have caught most of the problems in this article before a person did, this one included.

3. Deserialization breaks after a schema change

Symptom: consumers were fine, someone deployed a producer change, and now every record fails:

org.apache.kafka.common.errors.SerializationException: Unknown magic byte!

What's really happening: this one is not about your schema being wrong — it is about the byte layout. A Schema Registry serializer doesn't just write your payload. It prefixes a magic byte of 0x0 followed by a four-byte schema ID:

[0x00][4-byte schema ID][serialized payload]
  ^
  the "magic byte" the deserializer checks first

The consumer's KafkaAvroDeserializer reads byte zero, expects 0x0, finds something else, and throws immediately — before it ever looks at your data. The usual cause is a producer writing with StringSerializer or a plain JSON serializer while the consumer reads with an Avro deserializer. Sometimes it's a single legacy record written before the registry was introduced, poisoning an otherwise healthy topic.

Worth knowing: a plain JSON payload starts with {, which is 0x7B — so the very common "we send JSON, we read Avro" mismatch produces this error every single time, not intermittently. Confirming which side is wrong costs one command: read the offending record with the plain kafka-console-consumer (no schema deserializer configured), the same technique used in problem 8 below. Readable JSON means the producer never used the registry; unreadable binary means the problem is on the consumer's configuration instead.

The full diagnosis, including how to confirm what is actually on the topic: SerializationException: Unknown magic byte!. And if your problem turns out to be schema compatibility rather than wire format, Kafka Avro Schema: Backward Compatibility Explained covers which changes are safe.

4. The producer times out and drops records

Symptom: the application looks healthy, but records never arrive:

org.apache.kafka.common.errors.TimeoutException: Expiring 3 record(s) for
orders-0:120000 ms has passed since batch creation

What's really happening: those records never left your process. The producer buffers records in memory and sends them in batches; if a batch cannot be delivered within delivery.timeout.ms (default 120000 ms — note that the number in the message is that timeout), it is expired and discarded.

Three causes, in the order worth checking:

1. The broker is unreachable — wrong bootstrap.servers, a firewall, or a broker that is genuinely down.
2. Metadata can't be fetched — the topic doesn't exist and auto-creation is off, or advertised listeners are misconfigured so the client resolves an address it cannot reach (see problem 6).
3. The buffer is saturated — you are producing faster than the link drains, and batches queue until they age out.

A detail worth internalising: this is a silent data loss path unless you are checking the result of your sends. Fire-and-forget producer.send(record) without inspecting the returned future or supplying a callback means expired batches vanish with only a log line. Always handle the failure case.

Full walkthrough: TimeoutException: Expiring N record(s).

5. A consumer restarts and skips (or replays) everything

Symptom: after downtime, a consumer either jumps to the newest data and silently skips the backlog, or reprocesses far more than expected. Often accompanied by:

Fetch offset 15234 is out of range for partition orders-0, resetting offset
org.apache.kafka.clients.consumer.OffsetOutOfRangeException

What's really happening: the committed offset no longer exists in the log. Your consumer was down longer than retention.ms, so the segments containing offset 15234 were deleted while it was away. The consumer asks for a position the broker no longer has.

What happens next is decided entirely by auto.offset.reset, and both options lose something:

The same setting explains a different puzzle: a brand-new consumer group started against an existing topic has no committed offset either, so with the default latest it reads nothing already in the log. Teams routinely conclude the topic is empty when in fact their consumer chose to start at the end.

A decision rule that saves relitigating this every time: if the data is financial, an audit trail, or anything where silently skipping or silently duplicating a record would be worse than a loud failure, set auto.offset.reset=none and handle the exception explicitly. Reserve latest and earliest for consumers where a small, understood gap or a burst of reprocessing is genuinely tolerable — metrics and analytics pipelines are the common case.

Details and recovery options: OffsetOutOfRangeException: fetch position is out of range.

6. The client cannot reach the cluster

Symptom: connection succeeds, then metadata warnings repeat forever and nothing is produced or consumed:

WARN [Consumer clientId=consumer-1, groupId=orders] Error while fetching metadata
with correlation id 7 : {orders=LEADER_NOT_AVAILABLE}

What's really happening: in Docker and Kubernetes this is nearly always advertised.listeners. The client connects to the bootstrap address successfully, and the broker replies with the address clients should actually use for the partition leader. If that advertised address is an internal container hostname, an external client cannot resolve it — so the initial connection works and everything afterwards fails, which is what makes the symptom so confusing.

Kubernetes has its own well-known variant of the same root cause. Kafka on Kubernetes is commonly deployed as a StatefulSet behind a headless service, which gives each broker pod a stable per-pod DNS name like kafka-0.kafka-headless.kafka.svc.cluster.local — and that's frequently what gets baked into advertised.listeners, because it survives pod rescheduling in a way a raw pod IP doesn't. The catch: that DNS name only resolves inside the cluster's DNS. A client running outside the Kubernetes cluster entirely can reach the bootstrap service, then fails identically to the Docker case the moment it's handed that internal name. The fix is the same shape as the Docker case: a separate listener advertising an address actually reachable from outside, such as a LoadBalancer or NodePort address, rather than the internal headless-service DNS name — the full working listener configuration is in LEADER_NOT_AVAILABLE / Error while fetching metadata.

The other benign case: the topic was just created (or auto-created on first use) and leader election hasn't finished. That version resolves itself within a second or two. If the warning repeats indefinitely, it is a listener configuration problem.

Full diagnosis including working Docker listener configuration: LEADER_NOT_AVAILABLE / Error while fetching metadata.

7. One partition is hot and everything queues behind it

Symptom: the uneven lag from problem 2. One partition is buried; the rest are idle.

What's really happening: your message key has poor distribution. Partition placement is hash(key) % partitionCount, so key cardinality directly determines balance:

# BAD — low cardinality, and real traffic is heavily skewed
key = "US"                    # 80% of records land on one partition

# BAD — one tenant dwarfs the others
key = tenant_id               # your largest customer saturates its partition

# GOOD — high cardinality, spreads evenly
key = order_id                # still keeps all events for one order in order

The tempting fix — add consumers — does nothing at all. A partition is assigned to exactly one consumer in a group, so the busy one cannot be helped by idle peers. Adding partitions doesn't retroactively fix it either: records already written stay exactly where they are, and a larger partition count only changes where new records for a given key route to going forward — it doesn't rebalance the backlog that's already piled up on the hot one.

The real fix is choosing a higher-cardinality key, which means a producer change. Pick the narrowest key that still preserves the ordering you actually require: order_id rather than customer_id, customer_id rather than region. If a record needs no ordering guarantee at all, send it with a null key and let Kafka distribute it evenly.

That's the durable fix, and it takes a deploy. If you're mid-incident and need a stopgap: there isn't a great one. Increasing that one consumer's max.poll.records and processing the batch concurrently with an internal worker pool can squeeze out some throughput, but you're still bottlenecked on a single partition being read by a single consumer — and processing out of order within that pool means accepting the ordering loss you were trying to avoid. Treat it as buying time, not a fix.

8. Messages "disappeared"

Symptom: records that were definitely produced cannot be found.

Work through four possibilities in this order:

1. Retention expired them. Past retention.ms (default 7 days), segments are deleted regardless of whether anyone read them. Perfectly normal, frequently forgotten.
2. Compaction superseded them. On a topic with cleanup.policy=compact, only the newest record per key survives. If you are looking for an older value for a key, it is gone by design. A record with a null value is a tombstone, marking that key for removal.
3. Your consumer started at the end. The auto.offset.reset=latest case from problem 5 — the records exist, you just never asked for them.
4. They were never written. The expired-batch case from problem 4. Records dropped in the producer buffer never existed on any broker.

Confirm what is genuinely on the topic before theorising, by reading it directly from the beginning:

kafka-console-consumer.sh --bootstrap-server localhost:9092 \
  --topic orders --from-beginning --max-messages 5 \
  --property print.key=true --property print.timestamp=true

# CreateTime:1784505600000  order-1001  {"id":1001,"status":"created"}
# CreateTime:1784505601000  order-1002  {"id":1002,"status":"created"}

Those payloads arrive as a single dense line each. Paste one into the JSON Formatter to pretty-print it, or the JSON Viewer to expand a deeply nested record — useful when you need to confirm whether a field you expected is actually present, or whether the producer quietly stopped sending it.

Before you share that log: redact it

Every problem above ends with someone pasting a stack trace into a ticket, a Slack channel, or a Stack Overflow question. Kafka client logs are unusually leaky, and it is worth knowing exactly what you are handing over:

• Internal broker hostnames and ports — every connection log line carries your cluster topology.
• Topic and consumer group names — these describe your domain model and often your customers.
• Credentials — SASL and SSL failures are the worst offenders, echoing usernames and fragments of JAAS configuration directly into the error message.

Run the log through the Log Redactor before it leaves your machine. It strips hostnames, tokens, credentials, and other identifiers entirely in your browser — the unredacted text is never uploaded anywhere, which matters when the thing you are trying not to leak is in the text you would otherwise be uploading to some redaction service.

A triage order that works

When something is wrong and you don't yet know what, this sequence gets you to the answer fastest:

1. Check lag, and check its shape. kafka-consumer-groups.sh --describe. Even lag across partitions is a capacity problem; lag on one partition is a key problem.
2. Read the consumer log for rebalances. Repeated revoke/assign cycles mean problem 1, not a slow database.
3. Confirm records exist on the topic. --from-beginning --max-messages 5. This splits "not produced" from "not consumed" in one command.
4. Check producer callbacks. If nobody inspects send results, expired batches are invisible.
5. Compare configs across environments. Most "it works in staging" incidents are one drifted setting.

Kafka rewards understanding the model over memorising fixes. Nearly every problem here is a consequence of three facts: partitions are independent logs, a partition belongs to exactly one consumer per group, and nothing is deleted on read. Hold those and the failures stop being mysterious.