Kafka: OffsetOutOfRangeException — Fetch offset is out of range for partition

Common symptoms

You may not see the exact exception name first — it often shows up as one of these instead:

• An exception, or the "resetting offset" WARN, fires right after a consumer restarts following downtime, with no code change to explain it
• A large chunk of messages appears to vanish with no error at all — a silent skip under the default auto.offset.reset=latest
• A brand-new consumer group reports zero lag immediately, as if the topic had no data
• A consumer suddenly reprocesses a huge volume of records it already handled — a symptom of earliest, not latest
• kafka-consumer-groups.sh --describe shows CURRENT-OFFSET jump to equal LOG-END-OFFSET right after a restart, with no obvious cause

The exact error string

Fetch offset 15234 is out of range for partition orders-0, resetting offset
org.apache.kafka.clients.consumer.OffsetOutOfRangeException

This means offset 15234 is no longer a valid position to read from in this partition. Either it's below the earliest offset the broker still retains, or above the current log-end offset. In practice the first case — retention or compaction having removed it — is by far the more common one.

Why the offset stopped existing

A committed offset is just a number stored in __consumer_offsets; nothing about committing it reserves or protects the record it points at. The record ages out (or gets compacted away) on exactly the same schedule as everything else in the partition, regardless of whether any consumer's position still refers to it:

partition orders-0, offsets increasing left → right

  ...          14000    15000   15234   16000   17000   ← log-end-offset
   |------------|--------|--------|--------|--------|
                                     ^
                             consumer committed here (15234)

  retention.ms elapses while the consumer is down; segments
  holding offsets up through ~15900 get deleted:

   (deleted, through ~15900)                16000   17000
   X--------------------------------------- |--------|
                    ^
            offset 15234 no longer exists anywhere

  consumer restarts, asks for offset 15234:
  → Fetch offset 15234 is out of range for partition orders-0

cleanup.policy=delete
retention.ms=604800000     # 7 days — the default

# consumer group "order-processor" commits offset 15234, then goes down
# for 8 days. retention.ms elapses while it's away and deletes the
# segment containing offset 15234 — regardless of any committed position.
#
# consumer restarts, asks for offset 15234:
# -> Fetch offset 15234 is out of range for partition orders-0, resetting offset

The rule of thumb: if a consumer can plausibly be down longer than your retention window, this error is not a bug, it's the retention policy working as designed. The fix is either a longer retention window, faster incident response, or accepting that a long-enough outage costs you the gap.

What happens next: auto.offset.reset

Once the committed offset is invalid, one setting decides the consumer's fate — and it's worth reading all three options before assuming the default is what you want:

latest is the default, and it's the option most likely to surprise you: unless you're actively watching for this exception in your logs, a silent skip produces no error, no crash, no visible symptom — consumption just resumes at the current end, and the gap is gone without a trace.

# if silently skipping data is unacceptable for this consumer, prefer:
auto.offset.reset=none

// then handle it explicitly in code, so the gap is visible and logged
try {
    ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(500));
} catch (org.apache.kafka.clients.consumer.OffsetOutOfRangeException e) {
    log.error("Offset out of range for {} — data gap detected, alerting", e.partitions());
    // decide deliberately: seekToBeginning(), seekToEnd(), or page a human
}

Why a brand-new consumer group reads nothing

This is the same setting causing a different, more common source of confusion. A consumer group that has never committed an offset for a partition is in an equivalent position to one whose offset is out of range — there's no valid position to resume from, so auto.offset.reset applies identically:

# topic "orders" has 50,000 existing records
# a brand-new consumer group "new-analytics-job" starts consuming it
# with the default auto.offset.reset=latest:

kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
  --describe --group new-analytics-job

# TOPIC   PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG
# orders  0          50000           50000           0     <-- starts at the END

# the 50,000 existing records are never read by this group —
# not because the topic is empty, but because latest was chosen

If a new consumer group is meant to process everything already on the topic — a common requirement for a newly added analytics or audit consumer — set auto.offset.reset=earliest for that group, or explicitly seek to the beginning on startup rather than relying on the default.

The other common trigger: a topic deleted and recreated with the same name

This is the second most frequent real-world cause, and it's easy to miss because nothing about the setup looks wrong. Deleting a topic and creating a new one with the same name gives you a completely fresh partition, starting again from offset 0. Kafka does not reset a consumer group's committed offsets when this happens — __consumer_offsets still holds whatever was committed against the old topic, because as far as that internal topic is concerned nothing told it to forget:

# old topic "orders" reaches offset 90,000 before being deleted:
kafka-topics.sh --bootstrap-server localhost:9092 --delete --topic orders

# new topic "orders" is created fresh — starts again at offset 0:
kafka-topics.sh --bootstrap-server localhost:9092 --create --topic orders --partitions 3

# consumer group "order-processor" still has a committed offset from
# the OLD topic sitting in __consumer_offsets, e.g. 89500 — but the
# new topic's log-end-offset might only be 200 so far:
kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
  --describe --group order-processor

# TOPIC   PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG
# orders  0          89500           200             -89300   ← negative lag, nonsensical

# → Fetch offset 89500 is out of range for partition orders-0, resetting offset

The tell here is different from the retention case: instead of a plausible small gap, the committed offset is now vastly higher than the log-end-offset, and lag briefly reports a nonsensical negative number. That combination is the signature of a delete-and-recreate, not an expired retention window. The fix is to reset the group's offsets explicitly once, immediately after recreating the topic, rather than letting the mismatch surface as a production incident:

kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
  --group order-processor --topic orders \
  --reset-offsets --to-earliest --execute

If topics get recreated as part of a routine process (a test harness, a CI pipeline, an environment reset script), build this reset into that same process rather than relying on someone remembering it during an incident later.

Log compaction: related, but it doesn't cause this exception by itself

Compaction gets blamed for this error more often than it deserves. Two things are worth correcting. First, compaction isn't instant: the log cleaner reclaims space in the background on a schedule governed by min.cleanable.dirty.ratio and min.compaction.lag.ms, and it never touches the active segment currently being written to. A newer record superseding an older one for the same key doesn't delete the old one on the spot — it only marks it eligible for removal whenever the cleaner next processes that segment.

Second, and more importantly: fetching a compacted-away offset does not throw OffsetOutOfRangeException. Kafka offsets are never reassigned, so compaction leaves gaps in the sequence rather than invalid positions. Asking for a removed offset just returns the next surviving record at a higher offset, silently:

cleanup.policy=compact
# consumer's position is offset 200, an old value for key "user-42"
# a newer record for "user-42" was written at offset 800; the cleaner
# later removes offset 200's record from disk — no exception is thrown

# the consumer's next fetch just returns whatever the next surviving
# offset actually is (maybe 201, maybe 640) — reading continues,
# it just silently skips the record that got compacted away

What actually triggers OffsetOutOfRangeException is simpler than "compaction removed my record": the broker compares the requested offset against log.start.offset (the earliest offset still on disk) and the log-end offset, and rejects anything outside that range. Pure compaction rewrites segments in place without deleting them outright, so it doesn't advance log.start.offset by itself. That only happens when whole segments are deleted — a topic combining compaction with deletion (cleanup.policy=compact,delete), or an old segment aging out via retention.ms/retention.bytes in the ordinary way. So compaction alone produces silent key-level skips, not this exception; the exception itself is always a log.start.offset question, exactly like the time-based case above.

Recovering (or accepting) the gap

Once records are actually deleted — by either mechanism — there is no configuration that retrieves them; they no longer exist anywhere in the cluster. Your realistic options are decided in advance, not after the fact:

• Extend retention.ms well beyond your worst realistic downtime, if the data's value justifies the storage cost.
• Use compaction for state you must never lose — a compacted topic keeps the latest value per key forever, sidestepping time-based expiry entirely. It still discards older values for a key once the cleaner runs, so it's a fit for "latest state per key," not for preserving full history.
• Accept the gap deliberately, with auto.offset.reset=none and explicit handling, so a real outage produces a loud, visible alert instead of a silent skip.

Whichever you choose, decide it before the incident. The worst outcome is discovering your production auto.offset.reset default only when it's already skipped a week of data.

Diagnosing the gap

kafka-consumer-groups.sh: lag, or a skip?

Comparing a group's --describe output before and after a restart tells you whether the consumer caught up normally or whether latest just erased a backlog:

kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
  --describe --group order-processor

# before the restart — real lag, offset still valid:
# TOPIC   PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG
# orders  0          15234           17000           1766

# right after a reset triggered by this error:
# TOPIC   PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG
# orders  0          17000           17000           0     ← jumped straight to the end

CURRENT-OFFSET below LOG-END-OFFSET is normal lag — the consumer is behind but its position is still valid. If CURRENT-OFFSET suddenly equals LOG-END-OFFSET immediately after a restart, with no corresponding burst of processing in your logs, that's the signature of an auto.offset.reset=latest skip, not the consumer catching up.

kcat: is it the consumer, or the data?

One command settles whether the records are actually gone or whether the problem is purely in the consumer's own offset handling:

kcat -b localhost:9092 -C -t orders -o beginning

# if kcat successfully reads records starting from the earliest
# available offset, the data isn't gone — the problem is on the
# consumer side (auto.offset.reset, a bad seek, or a commit bug)
#
# if kcat's own earliest available offset is already past 15234,
# the data has genuinely been deleted by retention or compaction —
# no consumer-side configuration change recovers it

This is the fastest way to separate "my consumer is misconfigured" from "the data is actually gone" — and that distinction changes what's actually worth fixing.

Debugging checklist

• ✓ Check auto.offset.reset for the affected consumer group — this decided what already happened
• ✓ Compare how long the consumer was down against retention.ms for the topic
• ✓ If it's a compacted topic, remember pure compaction skips keys silently — it doesn't throw this exception unless paired with delete
• ✓ New consumer group reading nothing? Confirm whether it needs earliest instead of the default latest
• ✓ Catching auto.offset.reset=none exceptions? Handle both OffsetOutOfRangeException and NoOffsetForPartitionException, or catch their shared parent InvalidOffsetException
• ✓ Use kcat -o beginning to confirm whether the data is really gone, before assuming it's a consumer misconfiguration
• ✓ If silent skipping is unacceptable, switch to auto.offset.reset=none and handle the exception explicitly
• ✓ Deleted records cannot be recovered — plan retention and compaction policy for this in advance, not after