Kafka Fault Tolerance: Replication and Failure Recovery Explained

Kafka fault tolerance is Kafka’s ability to keep working when something goes wrong, such as a server crash, a network problem or a disk failure. Instead of stopping the flow of data, Kafka is designed to protect your messages and keep applications running.

This guide explains each building block, replication, leaders and followers, in-sync replicas, leader election and durable storage, and shows how to configure topics, producers, consumers and Spring Boot for safety, with Java examples, monitoring tips and best practices.

1. What Is Kafka Fault Tolerance?

In simple words, Kafka fault tolerance ensures that messages are not lost and that applications can still read and write data, even when some parts of the system fail.

Kafka achieves this by keeping several copies of every piece of data on different servers, and by automatically switching to a healthy copy when a server fails. Think of it like keeping important documents in three different safes in three different buildings: if one building has a problem, you still have the documents.

2. Why Fault Tolerance Is Important in Kafka

Kafka is used in real-time systems such as payment processing, order tracking, logging and analytics. In these systems, data loss or downtime can cause serious problems: lost orders, wrong account balances or missing audit records. Fault tolerance helps Kafka:

• Prevent data loss, even when a server fails.
• Stay highly available, so producers and consumers keep working.
• Recover automatically, without manual intervention.
• Scale safely to large, distributed clusters where failures are normal.

3. What Can Fail in a Kafka System?

4. Data Replication

Kafka stores data in topics, which are divided into partitions. Each partition is copied to several brokers. This is called replication, and the number of copies is the replication factor.

            Broker 1          Broker 2          Broker 3
Partition 0 [Leader]          [Follower]        [Follower]
Partition 1 [Follower]        [Leader]          [Follower]
Partition 2 [Follower]        [Follower]        [Leader]

Every partition has 3 copies on 3 different brokers.
Leaders are spread out, so each broker shares the work.

With a replication factor of 3, a partition survives the loss of two brokers without losing data. A replication factor of 3 is the common choice for production; a factor of 1 means no fault tolerance at all.

5. Leaders and Followers

For each partition, one replica is the leader and the others are followers:

• The leader handles all writes from producers and, by default, all reads from consumers.
• Followers continuously copy (replicate) new data from the leader.
• If the leader fails, one of the up-to-date followers becomes the new leader.

Producers and consumers always talk to the current leader. When the leader changes, the client libraries discover the new leader automatically and continue.

6. In-Sync Replicas (ISR) and min.insync.replicas

Kafka keeps a list of the replicas that are fully caught up with the leader. This list is called the In-Sync Replicas (ISR). A follower that falls too far behind, by default for more than 30 seconds (replica.lag.time.max.ms), is removed from the ISR until it catches up again.

Two rules make the ISR the core of Kafka’s safety:

1. Normally, only replicas in the ISR can become leader, so a new leader never misses acknowledged data.
2. With acks=all, a write is confirmed only when all current ISR members have it.

6.1 min.insync.replicas

What if the ISR shrinks to just the leader? Then acks=all would accept writes stored on only one broker. The topic setting min.insync.replicas prevents this: it sets the minimum number of in-sync replicas that must be available for a write to succeed.

The classic, recommended combination is replication factor 3, min.insync.replicas=2 and producers using acks=all. If too few replicas are in sync, producers receive a NotEnoughReplicasException instead of silently risking data loss.

7. Automatic Leader Election and the Controller

One broker in the cluster acts as the controller. It watches the brokers, and when a broker fails it chooses new leaders for all partitions that were led by that broker, picking from their ISR. This happens automatically, usually within seconds.

In KRaft mode, the controllers themselves are replicated, usually three of them, so losing one controller does not stop the cluster. See Kafka KRaft and Kafka ZooKeeper for details.

8. Unclean Leader Election

What if every in-sync replica of a partition is down, and only an out-of-date replica is left? Kafka must choose between:

• Waiting for an in-sync replica to come back: no data loss, but the partition is unavailable meanwhile.
• Electing the out-of-date replica (an unclean leader election): the partition becomes available at once, but messages it never received are lost.

# false (default): prefer consistency, never lose acknowledged data
unclean.leader.election.enable=false

Keep the default (false) for important data such as payments and orders. Some teams enable it only for data where availability matters more than completeness, such as non-critical metrics.

9. Durable Storage

Kafka writes every message to a log on disk, not only to memory. Each partition is a sequence of log segment files, and data stays there according to the topic’s retention settings, for example seven days, whether or not it has been read.

• If a broker restarts, its data is still on disk, and it catches up on anything it missed from the current leader.
• Consumers can re-read old messages, which helps recover from bugs in consumer code.

10. Producer Acknowledgements and Safe Producers

Producers choose how safely each write must be stored with the acks setting:

Since Kafka 3.0, producers use acks=all and idempotence by default. The idempotent producer gives each message a sequence number, so if a network glitch causes a retry, the broker stores the message only once. Here is a safe producer with the settings written out explicitly:

package com.example.kafka;

import java.util.Properties;

import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerConfig;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.common.serialization.StringSerializer;

public class SafeOrderProducer {

    public static void main(String[] args) {
        Properties props = new Properties();
        props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "broker1:9092,broker2:9092,broker3:9092");
        props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
        props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());

        // Fault-tolerance settings
        props.put(ProducerConfig.ACKS_CONFIG, "all");                  // wait for all in-sync replicas
        props.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, true);     // no duplicates on retry
        props.put(ProducerConfig.DELIVERY_TIMEOUT_MS_CONFIG, 120_000); // keep retrying for up to 2 minutes

        try (KafkaProducer<String, String> producer = new KafkaProducer<>(props)) {
            ProducerRecord<String, String> record =
                    new ProducerRecord<>("orders", "order-1001", "{\"orderId\":\"1001\",\"total\":59.90}");

            producer.send(record, (metadata, exception) -> {
                if (exception != null) {
                    // All retries failed: log it, alert, or store the order for a later attempt
                    System.err.println("Could not send order: " + exception.getMessage());
                } else {
                    System.out.printf("Stored in partition %d at offset %d%n",
                            metadata.partition(), metadata.offset());
                }
            });
        }
    }
}

List several brokers in bootstrap.servers, so the producer can still connect if one of them is down when it starts. Always check the send result: a failed send that is ignored is lost data, whatever the broker settings.

11. Creating a Fault-Tolerant Topic

Fault tolerance starts when the topic is created. With the command-line tool:

kafka-topics.sh --bootstrap-server broker1:9092 \
  --create --topic orders \
  --partitions 6 \
  --replication-factor 3 \
  --config min.insync.replicas=2

# Check where the replicas are and which are in sync
kafka-topics.sh --bootstrap-server broker1:9092 --describe --topic orders

Topic: orders  PartitionCount: 6  ReplicationFactor: 3  Configs: min.insync.replicas=2
  Partition: 0  Leader: 1  Replicas: 1,2,3  Isr: 1,2,3
  Partition: 1  Leader: 2  Replicas: 2,3,1  Isr: 2,3,1
  Partition: 2  Leader: 3  Replicas: 3,1,2  Isr: 3,1,2
  ...

Or from Java, using the Admin client:

package com.example.kafka;

import java.util.List;
import java.util.Map;
import java.util.Properties;

import org.apache.kafka.clients.admin.Admin;
import org.apache.kafka.clients.admin.AdminClientConfig;
import org.apache.kafka.clients.admin.NewTopic;

public class CreateOrdersTopic {

    public static void main(String[] args) throws Exception {
        Properties props = new Properties();
        props.put(AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG, "broker1:9092,broker2:9092,broker3:9092");

        try (Admin admin = Admin.create(props)) {
            NewTopic orders = new NewTopic("orders", 6, (short) 3)      // 6 partitions, 3 copies each
                    .configs(Map.of("min.insync.replicas", "2"));

            admin.createTopics(List.of(orders)).all().get();
            System.out.println("Topic 'orders' created");
        }
    }
}

Set safe defaults on the brokers too, so topics created automatically are not left with a single copy: default.replication.factor=3 and min.insync.replicas=2.

12. Consumer Fault Tolerance: Offsets and Rebalancing

Kafka tracks what each consumer group has read using offsets. Committed offsets are stored in Kafka itself, in the internal, replicated __consumer_offsets topic, so they survive broker failures too.

• If a consumer crashes, the group rebalances: its partitions are given to the remaining consumers.
• They continue from the last committed offset, so no message is skipped.
• Messages processed but not yet committed may be processed again, so processing should be idempotent.

To avoid losing messages, commit offsets only after processing them:

package com.example.kafka;

import java.time.Duration;
import java.util.List;
import java.util.Properties;

import org.apache.kafka.clients.consumer.ConsumerConfig;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.clients.consumer.KafkaConsumer;
import org.apache.kafka.common.serialization.StringDeserializer;

public class OrderConsumer {

    public static void main(String[] args) {
        Properties props = new Properties();
        props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "broker1:9092,broker2:9092,broker3:9092");
        props.put(ConsumerConfig.GROUP_ID_CONFIG, "shipping-service");
        props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
        props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
        props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, false);   // we commit ourselves
        props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");

        try (KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props)) {
            consumer.subscribe(List.of("orders"));

            while (true) {
                var records = consumer.poll(Duration.ofSeconds(1));

                for (ConsumerRecord<String, String> record : records) {
                    createShipment(record.key(), record.value());   // must be idempotent
                }

                // Commit only after the whole batch was processed
                if (!records.isEmpty()) {
                    consumer.commitSync();
                }
            }
        }
    }

    private static void createShipment(String orderId, String orderJson) {
        System.out.println("Creating shipment for order " + orderId);
    }
}

See Kafka Consumer for more on consumer groups and rebalancing.

13. Fault-Tolerant Settings in Spring Boot

With Spring for Apache Kafka, the same safety settings go into application.yml:

spring:
  kafka:
    bootstrap-servers: broker1:9092,broker2:9092,broker3:9092
    producer:
      acks: all
      properties:
        enable.idempotence: true
        delivery.timeout.ms: 120000
    consumer:
      group-id: shipping-service
      enable-auto-commit: false
      auto-offset-reset: earliest
    listener:
      ack-mode: record     # commit after each record is processed successfully

A topic can be declared as a bean, and Spring Boot creates it at startup if it does not exist:

package com.example.order.config;

import org.apache.kafka.clients.admin.NewTopic;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.kafka.config.TopicBuilder;

@Configuration
public class KafkaTopicConfig {

    @Bean
    public NewTopic ordersTopic() {
        return TopicBuilder.name("orders")
                .partitions(6)
                .replicas(3)
                .config("min.insync.replicas", "2")
                .build();
    }
}

For messages that keep failing in a listener, add a DefaultErrorHandler with a dead-letter topic, so one bad message does not block the partition.

14. Rack Awareness and Multiple Data Centers

Replicas on three brokers in the same rack or availability zone can all be lost together, for example in a power or network outage. Rack awareness spreads the copies:

# The rack or availability zone this broker runs in
broker.rack=eu-west-1a

When creating topics, Kafka then places the replicas of each partition in different racks, so the loss of a whole zone still leaves copies elsewhere. Consumers can even read from a nearby follower in their own zone, reducing cross-zone traffic, when follower fetching is enabled.

For protection against losing an entire data center or region, copy data to a second cluster with MirrorMaker 2.

15. Example: A Broker Fails in an Online Shop

Imagine an online shop that sends every order to the orders topic, with replication factor 3, min.insync.replicas=2 and acks=all. Each order message is stored on three brokers.

Before:   Partition 0  Leader: 1  Replicas: 1,2,3  Isr: 1,2,3

Broker 1 crashes
  -> the controller notices the failure
  -> it picks Broker 2 (in the ISR) as the new leader

After:    Partition 0  Leader: 2  Replicas: 1,2,3  Isr: 2,3

Broker 1 restarts and catches up
          Partition 0  Leader: 2  Replicas: 1,2,3  Isr: 2,3,1

1. Broker 1 crashes. The controller detects it and makes broker 2 the new leader of partition 0.
2. Producers discover the new leader and keep sending orders. Two in-sync replicas remain, so acks=all writes still succeed.
3. Consumers continue reading from the new leader, from their last committed offset.
4. When broker 1 comes back, it copies the messages it missed and rejoins the ISR.

The shop keeps running without losing a single acknowledged order. That is Kafka fault tolerance in action. If a second broker failed while broker 1 was down, writes would pause with NotEnoughReplicasException rather than risk data loss, and reads would continue from the remaining replica.

16. Durability vs Availability

17. Monitoring Fault Tolerance

# Show only partitions that are missing replicas
kafka-topics.sh --bootstrap-server broker1:9092 --describe --under-replicated-partitions

18. Best Practices

1. Use replication factor 3 for production topics, including internal topics.
2. Set min.insync.replicas=2 and use acks=all for important data.
3. Keep idempotence enabled on producers.
4. Keep unclean leader election disabled for critical topics.
5. List several bootstrap servers in every client.
6. Commit consumer offsets after processing, and make processing idempotent.
7. Spread brokers across racks or zones and set broker.rack.
8. Run at least three brokers, and three KRaft controllers.
9. Monitor under-replicated and offline partitions, and alert on them.
10. Test failures: stop a broker in a test environment and confirm that producers and consumers continue.

19. Common Mistakes to Avoid

20. Summary

Kafka fault tolerance is the backbone of its reliability. By replicating every partition across brokers, tracking in-sync replicas, electing new leaders automatically through the controller, storing data durably on disk and keeping consumer offsets in replicated storage, Kafka keeps data safe and available when servers fail. Configure it well: replication factor 3, min.insync.replicas=2, acks=all, idempotent producers, offsets committed after processing, and replicas spread across zones. In simple terms, Kafka is designed to expect failures and handle them gracefully, which makes it a trusted choice for real-time, large-scale data systems. Continue with Kafka Cluster to see how brokers work together.

21. Frequently Asked Questions (FAQ)