Kafka Offset

In Apache Kafka, an offset is a unique number that identifies the position of a message within a partition. Think of it like a page number in a book: each message in a partition gets a sequential number, starting from 0.

This guide explains what offsets are, how consumers commit them, where Kafka stores them, how automatic and manual commits affect message loss and duplicates, and how to rewind, reset and monitor offsets, with formatted Java, command-line and Spring Boot examples. It builds on Kafka Partition and Kafka Consumer.

1. What Is a Kafka Offset?

Every time a producer writes a message to a partition, Kafka appends it to the end of the partition’s log and gives it the next offset number. Offsets help Kafka consumers know:

• Which messages have already been read.
• Which message should be read next.
• How to resume reading after a restart or failure.

Topic: orders, Partition 0

 offset:   0       1       2       3       4       5
        +-------+-------+-------+-------+-------+-------+
        | msg A | msg B | msg C | msg D | msg E | msg F |  <- new messages
        +-------+-------+-------+-------+-------+-------+     are appended here
                                    ^
                         consumer's next read position

In simple words, a Kafka offset tracks message consumption. An offset is only meaningful together with its topic and partition: “offset 3” in partition 0 and “offset 3” in partition 1 are completely different messages.

2. Key Features of Kafka Offsets

3. Simple Example of Kafka Offsets

Let’s understand Kafka offsets with a simple scenario: the topic is order-topic and we look at Partition 0.

If a consumer has successfully processed messages up to offset 1:

• Messages with offsets 0 and 1 are already consumed.
• The next message to read is offset 2, so the consumer commits 2.
• If the consumer crashes and restarts, Kafka lets it continue from offset 2 instead of starting from the beginning.

Remember: the committed offset is the offset of the next message to read, which is the last processed offset plus one.

4. Important Offset Positions in a Partition

Several named positions describe the state of a partition and of a consumer reading it:

 log start   committed     current       high        log end
  offset      offset       position    watermark      offset
    |            |             |            |            |
    v            v             v            v            v
    [ 100 ... 140 | 141 ... 145 | 146 ... 159 | 160 ... 161 ]
      processed     fetched, not   available    not yet fully
      and saved     yet committed  to read      replicated

Offsets in a partition are not always contiguous. Log compaction removes older records with the same key, and transactions add internal marker records, so a consumer may see offset 7 followed by offset 9. Code should never assume “next offset = previous + 1” when reading.

5. Where Kafka Stores Consumer Offsets

Consumer offsets are stored inside Kafka itself, in an internal compacted topic called __consumer_offsets (50 partitions by default). Each record says, in effect, “group G, topic T, partition P has committed offset N.”

• Offsets are stored per consumer group (group.id), not per consumer instance, so any member of the group can continue where another left off.
• A broker called the group coordinator handles commits and offset lookups for each group.
• Committed offsets for an empty group are deleted after offsets.retention.minutes, which defaults to 7 days.

Very old Kafka versions kept offsets in ZooKeeper; modern clients and brokers always use __consumer_offsets.

6. Kafka Offset Commit

When a consumer finishes processing messages, it commits the offset. This tells Kafka, “I have safely processed messages up to this point.” Offsets can be committed in two ways:

Manual commit gives better control and is commonly used in production systems.

7. Automatic Offset Commit

# Automatic commit is on by default in the plain Java client
enable.auto.commit=true

# How often offsets are committed (default 5000 ms)
auto.commit.interval.ms=5000

Auto commit runs inside poll(): when the interval has passed, the client commits the positions returned by the previous poll. This is safe as long as your code finishes processing each batch before calling poll() again. Problems appear when you hand messages to other threads: the offsets can be committed before those threads finish, and a crash then loses the unfinished messages. If the consumer crashes between two commits, the messages since the last commit are delivered again.

8. Manual Offset Commit in Java

With enable.auto.commit=false, the application decides when to commit. The most common pattern is to process the whole batch and then commit:

package com.example.kafka;

import java.time.Duration;
import java.util.List;
import java.util.Properties;

import org.apache.kafka.clients.consumer.ConsumerConfig;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.apache.kafka.clients.consumer.KafkaConsumer;
import org.apache.kafka.common.serialization.StringDeserializer;

public class ManualCommitConsumer {

    public static void main(String[] args) {
        Properties props = new Properties();
        props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
        props.put(ConsumerConfig.GROUP_ID_CONFIG, "order-service");
        props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
        props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
        props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, false);
        props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");

        try (KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props)) {
            consumer.subscribe(List.of("order-topic"));

            while (true) {
                ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(500));

                for (ConsumerRecord<String, String> record : records) {
                    System.out.printf("partition=%d offset=%d value=%s%n",
                            record.partition(), record.offset(), record.value());
                    processOrder(record.value());
                }

                if (!records.isEmpty()) {
                    consumer.commitSync();   // saves the positions after this batch
                }
            }
        }
    }

    private static void processOrder(String order) {
        // business logic, for example saving the order to a database
    }
}

A common combination is commitAsync() inside the loop for speed, plus one final commitSync() when the consumer shuts down.

Committing a specific offset

To save progress more often than once per batch, commit an exact offset per partition. Always commit offset + 1:

Map<TopicPartition, OffsetAndMetadata> toCommit = Map.of(
        new TopicPartition(record.topic(), record.partition()),
        new OffsetAndMetadata(record.offset() + 1));   // next offset to read

consumer.commitSync(toCommit);

9. Offsets and Delivery Guarantees

The moment you commit, before or after processing, decides what happens when a consumer crashes:

At-least-once is the usual choice. Make processing idempotent, for example by ignoring an order ID that was already saved, so that a repeated message causes no harm.

For exactly-once in a read-process-write flow, a transactional producer includes the consumer’s offsets in its transaction:

producer.beginTransaction();

for (ConsumerRecord<String, String> record : records) {
    producer.send(new ProducerRecord<>("order-events", record.key(), transform(record.value())));
}

producer.sendOffsetsToTransaction(nextOffsets(records), consumer.groupMetadata());
producer.commitTransaction();   // output and offsets are saved together, or not at all

Here nextOffsets(records) is a small helper that builds the map of each partition’s last offset plus one. Downstream consumers should use isolation.level=read_committed.

10. Where a New Consumer Starts: auto.offset.reset

When a consumer group has no committed offset for a partition (a brand-new group, or offsets that expired), the auto.offset.reset setting decides where to start:

This setting is used only when no valid committed offset exists. Once the group has committed, it always resumes from the committed offset. It is also used when the committed offset is no longer available because retention already deleted those messages.

11. Reading from a Specific Offset

The consumer can move its position at any time with the seek methods. This is useful for replaying data, skipping a bad message, or starting from a point in time:

TopicPartition partition0 = new TopicPartition("order-topic", 0);
consumer.assign(List.of(partition0));

// 1. Jump to an exact offset
consumer.seek(partition0, 120L);

// 2. Go to the oldest or newest message
consumer.seekToBeginning(List.of(partition0));
consumer.seekToEnd(List.of(partition0));

// 3. Start from a point in time, for example one hour ago
long oneHourAgo = Instant.now().minus(Duration.ofHours(1)).toEpochMilli();
Map<TopicPartition, OffsetAndTimestamp> found =
        consumer.offsetsForTimes(Map.of(partition0, oneHourAgo));

OffsetAndTimestamp result = found.get(partition0);
if (result != null) {
    consumer.seek(partition0, result.offset());
}

A seek changes only the current position. The new position is saved for the group when the consumer next commits. When using subscribe() instead of assign(), seek inside a rebalance listener, after partitions have been assigned.

12. Resetting Offsets with the Command Line

The kafka-consumer-groups.sh tool shows and changes a group’s committed offsets. Stop all consumers in the group first; offsets of an active group cannot be reset.

# Show committed offsets and lag for a group
bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
  --describe --group order-service

# Preview a reset to the earliest offsets (nothing changes yet)
bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
  --group order-service --topic order-topic \
  --reset-offsets --to-earliest --dry-run

# Apply the reset
bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
  --group order-service --topic order-topic \
  --reset-offsets --to-earliest --execute

Without --execute, the tool only prints the planned result, so preview every reset first.

13. Consumer Lag: Measuring How Far Behind You Are

Consumer lag is how many messages a group still has to read in a partition:

lag = latest available offset (high watermark) - committed offset

Example: high watermark = 1,000, committed offset = 950  ->  lag = 50

GROUP          TOPIC        PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG
order-service  order-topic  0          950             1000            50
order-service  order-topic  1          1203            1203            0
order-service  order-topic  2          880             1410            530

• Low, stable lag is normal: consumers are keeping up.
• Steadily growing lag means consumers are too slow; add consumers (up to the number of partitions) or speed up processing.
• Lag on one partition only often means a stuck consumer or a “hot” key sending most traffic to that partition.

Track lag continuously with consumer metrics, a monitoring stack such as Prometheus with a Kafka exporter, or tools like Kafka UI, and alert when it keeps rising.

14. Offsets During a Rebalance

When consumers join or leave a group, Kafka rebalances the partitions among them. The new owner of a partition starts from its last committed offset, so anything processed but not yet committed is processed again. Commit before losing a partition with a ConsumerRebalanceListener:

Map<TopicPartition, OffsetAndMetadata> processed = new HashMap<>();

consumer.subscribe(List.of("order-topic"), new ConsumerRebalanceListener() {

    @Override
    public void onPartitionsRevoked(Collection<TopicPartition> partitions) {
        consumer.commitSync(processed);     // save progress before giving partitions away
        processed.clear();
    }

    @Override
    public void onPartitionsAssigned(Collection<TopicPartition> partitions) {
        // optional: seek to offsets stored elsewhere, for example in a database
    }
});

// in the poll loop, after processing each record:
processed.put(new TopicPartition(record.topic(), record.partition()),
              new OffsetAndMetadata(record.offset() + 1));

15. Managing Offsets in Spring Boot

Spring for Apache Kafka disables the client’s auto commit and lets the listener container commit for you. By default (AckMode.BATCH) it commits after all records from one poll are processed. For full control, switch to manual acknowledgement:

spring:
  kafka:
    bootstrap-servers: localhost:9092
    consumer:
      group-id: order-service
      auto-offset-reset: earliest
    listener:
      ack-mode: manual

package com.example.kafka;

import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.kafka.support.Acknowledgment;
import org.springframework.stereotype.Component;

@Component
public class OrderListener {

    private final OrderService orderService;

    public OrderListener(OrderService orderService) {
        this.orderService = orderService;
    }

    @KafkaListener(topics = "order-topic")
    public void onOrder(ConsumerRecord<String, String> record, Acknowledgment ack) {
        orderService.process(record.value());
        ack.acknowledge();   // offset is committed only after successful processing
    }
}

If the listener throws an exception, the offset is not committed. The default error handler retries the record a few times and then logs and skips it; configure a dead-letter topic so failed messages are kept for investigation.

16. Why Kafka Offsets Are Important

• Prevents message loss: consumers resume exactly where they stopped.
• Limits duplicate processing: committing after processing means only a small window of messages can be repeated.
• Enables reliable, scalable processing: any member of a group can take over a partition and continue from its offset.
• Supports recovery and reprocessing: moving offsets back replays history, for example after fixing a bug.
• Allows independent consumers: many applications read the same topic, each with its own offsets.

17. Best Practices and Common Mistakes

18. Summary

A Kafka offset is a number that represents the position of a message in a partition. It helps consumers track what they have already read and what comes next. Consumer groups save their progress as committed offsets in the __consumer_offsets topic, either automatically or manually; committing after processing gives at-least-once delivery, and transactions give exactly-once for Kafka-to-Kafka flows. With auto.offset.reset, seek and the kafka-consumer-groups.sh tool you control where consumers start and can replay history, and consumer lag shows whether they are keeping up.

In short, Kafka offsets are the backbone of message tracking in Kafka: they make Kafka reliable, fault-tolerant and flexible by allowing safe recovery and message replay.

19. Frequently Asked Questions (FAQ)

new OffsetAndMetadata(record.offset() + 1)

consumer.seek(new TopicPartition("order-topic", 0), 120L);

bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
  --group order-service --topic order-topic \
  --reset-offsets --to-earliest --execute

@KafkaListener(topics = "order-topic")
public void onOrder(String order, Acknowledgment ack) {
    orderService.process(order);
    ack.acknowledge();
}