Kafka: CommitFailedException: Offset commit cannot be completed since the group has already rebalanced

Common symptoms

The exception name itself is only one way this surfaces. Watch for any of these too:

• Consumer throughput periodically collapses to near-zero, then partially recovers, in a repeating cycle
• Logs fill with partition revoked / partition assigned messages with no corresponding deploy or crash
• A downstream system sees the same batch of records processed more than once
• kafka-consumer-groups.sh --describe shows the group's member count changing between runs with no scaling event

The exact error string

org.apache.kafka.clients.consumer.CommitFailedException: Offset commit cannot be
completed since the consumer is not part of an active group for auto partition
assignment; it is likely that the consumer was kicked out of the group.

# a differently-worded but equally common variant, seen across other client
# versions and libraries, for the identical root cause:
# Commit cannot be completed since the group has already rebalanced and
# assigned the partitions to another member.

Both wordings describe the same event: the consumer was removed from its group before this commit arrived, so the commit is rejected outright. This is nearly always a symptom of one specific cause: too much time passed between two consecutive calls to poll().

Why "the consumer is clearly alive" doesn't help

The confusing part of this error is that the process looks completely healthy — no crash, no exception, no network partition. That's because two entirely separate checks decide whether a consumer is "alive" from the group coordinator's point of view, and only one of them is about crashing:

Heartbeats are sent by a background thread that runs independently of your application code. A consumer stuck in a slow database write keeps heartbeating the entire time — the process is genuinely alive, and session.timeout.ms never fires. The only thing that catches a stuck consumer is the second check: has poll() been called again within max.poll.interval.ms? If not, the coordinator concludes processing has stalled, evicts the consumer, and triggers a rebalance — and that eviction is what makes the pending commit fail.

How the rebalance loop forms

A single slow batch would just cause one eviction. What turns it into a recurring incident is that the replacement consumer typically inherits the exact same problem:

Fix 1: shrink the batch (preferred)

The default max.poll.records is 500. If your per-record processing time is even modest, 500 records can easily exceed 5 minutes:

// ❌ default batch size, with per-record work that adds up
props.put("max.poll.records", 500);   // 500 × 700ms/record = 350s > 300s limit

// ✅ smaller batches finish comfortably inside the window
props.put("max.poll.records", 100);   // 100 × 700ms/record = 70s — plenty of margin

Do this arithmetic explicitly rather than guessing: measure your realistic worst-case per-record time (including occasional slow dependency calls, not just the happy path) and size max.poll.records so the total stays well under max.poll.interval.ms — leave real margin, not just enough to pass on a good day.

Fix 2: raise the interval, if the work genuinely needs it

Sometimes each record really does require several minutes — a large batch write, a slow external call you can't avoid. In that case, extend the deadline rather than fighting it:

props.put("max.poll.interval.ms", 600000);   // 10 minutes

This is a legitimate fix, not a workaround — but it comes with a real trade-off: it also lengthens how long a genuinely stuck or crashed consumer's partitions sit unprocessed before anyone notices, since eviction is the only mechanism that would catch it. Use it deliberately, not as a first reflex.

Fix 3: move slow work off the poll loop

The most robust fix restructures the consumer so poll() keeps getting called frequently regardless of processing time, by handing records to a separate worker. Done naively, though, this trades one bug for two worse ones, so the sketch below is deliberately more than a bare hand-off:

// sketch — bounds memory and only commits work that actually finished.
// wire the details (thread pool, tracking structure) to your framework.
int inFlight = 0;
Map<TopicPartition, OffsetAndMetadata> readyToCommit = new HashMap<>();

while (running) {
    if (inFlight >= MAX_IN_FLIGHT) {
        consumer.pause(consumer.assignment());   // stop pulling more until the queue drains
    } else {
        consumer.resume(consumer.assignment());
    }

    ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
    for (ConsumerRecord<String, String> record : records) {
        inFlight++;
        workQueue.submit(() -> {
            process(record);
            inFlight--;
            // only mark this offset committable once the work is DONE —
            // never commit ahead of completed processing
            markReadyToCommit(readyToCommit, record);
        });
    }
    consumer.commitAsync(drain(readyToCommit), null);
}

Two failure modes make the bare "just submit to a queue" version dangerous enough to avoid in production: an unbounded queue grows without limit if the workers can't keep up, eventually exhausting memory; and committing on the poll loop's own schedule — rather than after work actually completes — can commit an offset for a record that's still sitting in the queue, so a crash loses it silently instead of it being redelivered. pause()/resume() bound the first problem by refusing to pull more work while the backlog is full; tracking completion and only committing finished offsets fixes the second. This decouples "how often does the coordinator hear from me" from "how long does my business logic take" — but it's meaningfully more code than the config-only fixes above, and worth it only if per-record time is genuinely variable and hard to bound.

Fix 4: cooperative rebalancing reduces the blast radius (but isn't the fix)

It's worth knowing what cooperative rebalancing does and doesn't help with here. Eager rebalancing is what actually runs by default — see How Kafka Works for why that's widely misreported and what the assignor list actually contains. To switch to cooperative:

props.put("partition.assignment.strategy",
    "org.apache.kafka.clients.consumer.CooperativeStickyAssignor");

With eager rebalancing, every consumer in the group gives up all its partitions and stops during a rebalance, even the ones that were behaving fine. Cooperative rebalancing only reassigns the partitions that actually need to move, so the rest of the group keeps processing uninterrupted. That materially reduces the damage a rebalance does — but the specific consumer that exceeded max.poll.interval.ms is still evicted, and its commit still fails with this exact exception. Cooperative rebalancing limits the blast radius; it doesn't prevent the underlying cause.

group.instance.id (static membership, KIP-345) is worth knowing about here, but it solves a neighbouring problem. Giving each consumer a stable ID means a restart within session.timeout.ms doesn't trigger a rebalance at all — the returning member reclaims its old partitions. That removes a lot of needless churn from rolling deploys, but it does nothing for slow polling: a consumer that blows past max.poll.interval.ms is still evicted and its commit still fails. Use it to reduce rebalance frequency, not to fix this error.

props.put("group.instance.id", "order-processor-" + podOrdinal);  // stable per pod/instance
props.put("session.timeout.ms", "45000");                          // window to reclaim on restart

Is this actually dangerous?

By itself, no. The records the evicted consumer processed but never got to commit will be redelivered to whichever consumer inherits the partition — standard at-least-once behavior, assuming your processing is idempotent (see Kafka Delivery Guarantees for what that actually requires). No data is lost.

The real damage is throughput: while a rebalance is in progress the group pauses, and if the slow-batch cause repeats on every cycle — as in the loop diagram above — the group can spend more time rebalancing than processing.

Debugging checklist

• ✓ Confirm it's a poll-interval issue, not a crash: check logs for repeated join/rebalance messages, not exceptions from your processing code
• ✓ Measure your realistic per-record processing time, including slow-path cases
• ✓ Reduce max.poll.records first — the lowest-risk fix
• ✓ Raise max.poll.interval.ms only if the work genuinely needs the extra time
• ✓ For variable/unbounded per-record time, move processing off the poll thread
• ✓ Consider CooperativeStickyAssignor to limit collateral impact on the rest of the group
• ✓ Verify: after the fix, kafka-consumer-groups.sh --describe shows a stable member count with no repeated rebalances