Kafka Delivery Guarantees: acks, ISR and Exactly-Once Explained

"Exactly-once" is the most oversold phrase in streaming. Kafka does support it — genuinely, not as marketing — but only inside a boundary that most teams discover the hard way. This guide works through what each acknowledgement level actually promises, the replication setting that quietly cancels your durability guarantee, what the idempotent producer does and doesn't dedupe, and where exactly-once stops applying. It assumes you know what partitions and offsets are; if not, start with How Kafka Works.

Three guarantees, and which one you actually have

Every messaging system lands somewhere among three delivery semantics:

Most production Kafka systems run at-least-once and make duplicates harmless. That is not a compromise or a failure to configure things properly — it is usually the correct engineering decision, and the last section covers why.

Importantly, the guarantee isn't a single setting. It emerges from two independent halves: what the producer does to get a record durably into the log, and what the consumer does about committing its position. You can get either half wrong on its own.

The producer half: acks

The acks setting controls how much confirmation a producer waits for before considering a send successful.

acks=0 has narrow legitimate uses — high-volume metrics or clickstream data where losing a fraction of a percent genuinely doesn't matter and throughput is everything. For anything you'd be unhappy to lose, it's the wrong choice.

acks=1 is the deceptive one. The leader confirms the write and your producer moves on, but the followers may not have copied it yet. If that broker dies in the window before replication, the record is gone — and the producer already reported success, so nothing in your application ever learns about it.

acks=all is what you want for important data. But it comes with a large asterisk.

The ISR trap: why acks=all can mean nothing

Each partition has a leader and some followers. The set of replicas currently caught up with the leader is the in-sync replica set (ISR). A follower that falls too far behind — slow disk, network trouble, a restart — is dropped from the ISR until it catches up.

Now read acks=all carefully. It means every replica currently in the ISR, not every replica that exists. And the ISR shrinks under exactly the conditions where you most need durability.

Picture a topic with replication factor 3. Two followers get slow and drop out of the ISR, which now contains only the leader. A producer with acks=all sends a record. The leader writes it, sees that every member of the ISR (itself) has the record, and acknowledges. Your producer is told the write is durable. It exists on one broker. That broker's disk fails, and the record is gone — despite acks=all.

The setting that prevents this is min.insync.replicas, configured on the broker or topic. It sets the floor: if the ISR is smaller than this, writes with acks=all are rejected rather than silently accepted with weak durability.

# DANGEROUS — acks=all degrades to acks=1 whenever followers fall behind
#   replication.factor=3
#   min.insync.replicas=1        <-- ISR may shrink to the leader alone

# SAFE — the standard production pairing
replication.factor=3
min.insync.replicas=2            # at least 2 copies, or the write is refused
# producer:
acks=all

With min.insync.replicas=2 and replication factor 3, you can lose one broker and keep accepting writes. Lose two and producers start receiving NotEnoughReplicasException — which is the system correctly refusing to accept data it cannot store safely. An error you can see and alert on beats silent data loss.

A related setting decides what happens if every in-sync replica is lost outright — not just below the min.insync.replicas floor, but gone entirely. unclean.leader.election.enable (default false) controls whether Kafka is allowed to elect a leader from an out-of-sync replica in that scenario. Leaving it off means the partition simply stays unavailable until an in-sync replica returns — the safe default. Turning it on trades availability for consistency: the partition comes back online sooner, but the newly elected leader may be missing records the old leader had already acknowledged. Treat it the same way as min.insync.replicas=1 — a deliberate, rare exception for availability-over-durability workloads, not a default to flip during an incident.

Configuration drift between environments is a common way this goes wrong: staging is set up correctly, production was provisioned earlier with different defaults, and nobody notices until a broker dies. If you keep broker or topic configs as JSON, the JSON Diff tool is a quick way to compare a known-good config against a suspect one and see exactly which keys differ.

Where duplicates come from

Set acks=all and min.insync.replicas=2 and you have solved data loss. You have simultaneously created data duplication, and it's worth seeing why, because the mechanism is unavoidable rather than a bug.

A producer sends a record. The broker writes it successfully and sends an acknowledgement. The acknowledgement is lost on the network. The producer's request times out.

From the producer's point of view, the send failed. It has no way to distinguish "the broker never got it" from "the broker got it and the reply vanished." So it retries — and the record is written twice.

This is intrinsic to any at-least-once system: an acknowledgement that can be lost forces a choice between retrying (risking duplicates) and not retrying (risking loss). Kafka's answer is to retry, then eliminate the duplicate at the broker.

The idempotent producer

Idempotence solves precisely the retry-duplicate problem above. The producer is assigned a producer ID (PID), and attaches a monotonically increasing sequence number per partition. The broker remembers the last sequence number it accepted for each PID and partition, so a retried record arrives with a sequence number it has already seen and is discarded rather than appended.

Since Kafka 3.0, enable.idempotence defaults to true, which in turn implies acks=all and effectively unlimited retries. On 3.0 and later you likely have it on without having asked for it.

There is an important subtlety in how that default interacts with your other settings:

# Kafka 3.0+ — idempotence is ON by default, nothing to do
# (implies acks=all, retries=Integer.MAX_VALUE, max.in.flight <= 5)

# TRAP: setting a conflicting value WITHOUT naming enable.idempotence
#   acks=1                       <-- idempotence is now silently DISABLED
#
# Setting both explicitly throws ConfigException at startup instead:
#   enable.idempotence=true
#   acks=1                       <-- ConfigException — this one you'll notice

So the dangerous case is the quiet one: someone tunes acks=1 for throughput, and idempotence silently switches off along with it. If you rely on idempotence, set enable.idempotence=true explicitly so that a conflicting change fails loudly instead of degrading in silence.

Now the limits, which matter more than the feature:

• It only covers one producer session. A restarted producer gets a new PID, and the broker has no way to connect it to the old one. Records re-sent after a restart are new records.
• It only covers retries of the same send. If your application logic calls send() twice with the same payload, those are two legitimate records and both are written.
• It is per-partition. Sequence tracking is scoped to a PID and partition pair.

Idempotence makes the producer safe against network flakiness. It does not make your pipeline exactly-once.

Transactions and the real boundary of exactly-once

Kafka transactions let a producer write to multiple partitions and commit consumer offsets as one atomic unit. That is what makes a consume-transform-produce loop exactly-once: reading an input record, writing an output record, and marking the input consumed either all happen or none do.

props.put("transactional.id", "order-processor-1");   // required — initTransactions()
                                                        // throws IllegalStateException without it
producer.initTransactions();

while (true) {
    ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
    producer.beginTransaction();
    try {
        for (ConsumerRecord<String, String> record : records) {
            producer.send(new ProducerRecord<>("output-topic", transform(record)));
        }
        // committing offsets INSIDE the transaction is what makes this atomic
        producer.sendOffsetsToTransaction(offsetsOf(records), consumer.groupMetadata());
        producer.commitTransaction();
    } catch (Exception e) {
        producer.abortTransaction();
    }
}

transactional.id must be set for any of this to work at all — it's what lets the broker recognize the same producer instance across restarts and fence off a zombie if two instances with the same ID ever run at once. Give each producer instance a stable, unique ID (not a random one generated per run), or initTransactions() fails immediately.

Consumers must also set isolation.level=read_committed, otherwise they will read records from transactions that were later aborted. The default is read_uncommitted, so this is an easy step to miss.

And here is the boundary that catches teams out:

Kafka transactions are atomic across Kafka only. Topics and consumer offsets are inside the transaction. Your database, your HTTP calls, and your emails are not.

If your consumer reads a record, inserts a row into Postgres, and commits the offset, no Kafka transaction can undo that insert when the commit fails. The row is already there. Kafka cannot roll back a system it doesn't control.

This is why "we'll just turn on exactly-once" rarely solves the problem people actually have. Exactly-once covers Kafka-to-Kafka stream processing — the domain Kafka Streams operates in, where enabling it is genuinely a config change. The moment your side effect leaves Kafka, the guarantee stops at the boundary and the responsibility becomes yours.

The consumer half: when you commit

Producer settings determine whether a record reaches the log. Consumer commit timing determines what happens when your application crashes mid-processing. There is no single setting for this — it is about ordering.

// AT-MOST-ONCE — commit first. A crash before processing loses the record.
records = consumer.poll(timeout);
consumer.commitSync();
process(records);                 // crash here => records never processed

// AT-LEAST-ONCE — process first. A crash before commit redelivers them.
records = consumer.poll(timeout);
process(records);
consumer.commitSync();            // crash here => records processed again

The default, enable.auto.commit=true, commits periodically (every auto.commit.interval.ms, default 5000) during calls to poll(). That yields at-least-once behaviour in practice, with a wrinkle: because commits happen on a timer rather than when your processing finishes, the redelivery window after a crash can span several seconds of already-processed records.

For anything where the redelivery set matters, turn auto-commit off and commit deliberately after processing.

Two timeouts govern whether the group thinks you are alive, and confusing them is a common source of mysterious rebalances:

Heartbeats come from a background thread independent of your processing code, so a consumer stuck in slow work keeps heartbeating while failing to poll — which is exactly what max.poll.interval.ms exists to catch. The full mechanism, and how it produces CommitFailedException, is covered in CommitFailedException: Offset commit cannot be completed.

Note the version detail on the first row: session.timeout.ms defaulted to 10 seconds before Kafka 3.0 and was raised to 45 seconds by KIP-735, specifically to stop transient network blips from triggering needless rebalances. Older tutorials showing 10000 predate that change.

These values are all milliseconds, and a misplaced digit turns five minutes into thirty seconds. When you're reconciling a timeout against timestamps in a consumer log to work out what actually expired, the Timestamp Converter is handy for turning epoch-millisecond log values into readable times.

What to do in practice: idempotent consumers

Given all that, here is the pattern most production systems settle on — and it deliberately does not use Kafka transactions:

1. Configure the producer for durability: acks=all, min.insync.replicas=2, replication factor 3, idempotence explicitly on.
2. Accept at-least-once on the consumer: process first, commit after.
3. Make processing idempotent so a duplicate is harmless.

Step 3 is the one that does the work. If handling the same record twice produces the same result, duplicates stop being a correctness problem and become a minor efficiency one. Usually that means giving each event a stable ID and having your write ignore an ID it has already seen:

-- upsert keyed by the event's own ID: replaying the record changes nothing
INSERT INTO processed_orders (event_id, order_id, amount)
VALUES ($1, $2, $3)
ON CONFLICT (event_id) DO NOTHING;

This is why so many mature Kafka systems never enable exactly-once. Idempotent writes are simpler to reason about, carry no transactional throughput penalty, and — unlike Kafka transactions — keep working when the write target is a database, a search index, or a third-party API. Effectively-once through idempotence covers the cases exactly-once cannot reach.

Reach for transactions when you are doing Kafka-to-Kafka stream processing and atomic multi-partition writes genuinely matter. For everything else, make the operation idempotent and sleep better.

Summary

• acks=all alone is not durability — pair it with min.insync.replicas=2 or the ISR can shrink to one.
• Idempotence is on by default since Kafka 3.0, but a conflicting setting disables it silently. Set it explicitly.
• The idempotent producer dedupes retries within one session — not restarts, not duplicate application logic.
• Exactly-once is real, and bounded to Kafka topics plus offsets. It does not extend to your database.
• At-least-once plus idempotent processing is the pragmatic default for most systems.