Kafka Cluster

A Kafka cluster is a group of Kafka servers that work together to store, process and distribute data in real time. This guide explains how a cluster works, its components and features, how many brokers you need, and how a cluster handles failures and growth, with a Docker Compose setup, command-line steps and Java and Spring Boot examples.

It builds on Kafka Broker and Kafka KRaft.

1. What Is a Kafka Cluster?

A Kafka cluster is a group of multiple Kafka servers (called brokers) that work together to store, process and distribute data in real time. Instead of relying on a single server, Kafka spreads data across many servers, which makes the system fast, reliable and scalable.

In simple terms, think of a Kafka cluster as a team of servers that share the work of handling large amounts of data, so applications can send and receive messages smoothly without delays. If one team member is absent, the others cover for them.

2. How Does a Kafka Cluster Work?

A Kafka cluster receives data from producers (applications that send data), stores it safely, and delivers it to consumers (applications that read data):

• Data is stored in topics, which are like categories or channels.
• Each topic is divided into partitions for better performance.
• Partitions, and copies of them, are distributed across the brokers in the cluster.
• Controllers keep track of the cluster’s metadata and decide which broker leads each partition.

              Producers (order app, web app)
                          |
                          v
 +-------------------- Kafka cluster --------------------+
 |   Broker 1          Broker 2          Broker 3        |
 |   orders-P0 (L)     orders-P1 (L)     orders-P2 (L)   |
 |   orders-P1 (F)     orders-P2 (F)     orders-P0 (F)   |
 |   orders-P2 (F)     orders-P0 (F)     orders-P1 (F)   |
 |                                                       |
 |   KRaft controllers: manage metadata and leaders      |
 +-------------------------------------------------------+
                          |
                          v
              Consumers (billing, inventory)

  L = leader replica    F = follower replica

Each broker leads some partitions and keeps copies of others. This spreads the work evenly and lets Kafka handle millions of messages per second with high reliability.

3. Key Components of a Kafka Cluster

Older Kafka versions used ZooKeeper to store metadata. Kafka 4.0 and later use only KRaft controllers.

4. Features of a Kafka Cluster

5. Simple Example of a Kafka Cluster

Imagine an e-commerce website with one Kafka cluster:

All these applications connect to the same Kafka cluster. The cluster spreads the orders and payments partitions across its brokers and keeps copies on other brokers, so data is delivered reliably and quickly even if one server goes down.

6. How Many Brokers Does a Cluster Need?

Size a cluster by looking at peak write throughput, total retained data (including copies), the number of partitions, and how much spare capacity you need when one broker is down. In production, run 3 or 5 dedicated KRaft controllers in addition to the brokers.

7. Running a 3-Node Kafka Cluster with Docker Compose

The Docker Compose file below starts a 3-node cluster on your computer. For simplicity, each node is both a broker and a controller (combined mode). Each node has an internal listener for cluster traffic and an external port for applications on your machine.

x-kafka-common: &kafka-common
  CLUSTER_ID: "5L6g3nShT-eMCtK--X86sw"
  KAFKA_PROCESS_ROLES: broker,controller
  KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka-1:9093,2@kafka-2:9093,3@kafka-3:9093
  KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
  KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
  KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,CONTROLLER:PLAINTEXT,EXTERNAL:PLAINTEXT
  KAFKA_DEFAULT_REPLICATION_FACTOR: 3
  KAFKA_MIN_INSYNC_REPLICAS: 2
  KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 3
  KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 3
  KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 2

services:
  kafka-1:
    image: apache/kafka:4.0.0
    container_name: kafka-1
    ports: ["29092:29092"]
    environment:
      <<: *kafka-common
      KAFKA_NODE_ID: 1
      KAFKA_LISTENERS: INTERNAL://:9092,CONTROLLER://:9093,EXTERNAL://:29092
      KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka-1:9092,EXTERNAL://localhost:29092

  kafka-2:
    image: apache/kafka:4.0.0
    container_name: kafka-2
    ports: ["39092:39092"]
    environment:
      <<: *kafka-common
      KAFKA_NODE_ID: 2
      KAFKA_LISTENERS: INTERNAL://:9092,CONTROLLER://:9093,EXTERNAL://:39092
      KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka-2:9092,EXTERNAL://localhost:39092

  kafka-3:
    image: apache/kafka:4.0.0
    container_name: kafka-3
    ports: ["49092:49092"]
    environment:
      <<: *kafka-common
      KAFKA_NODE_ID: 3
      KAFKA_LISTENERS: INTERNAL://:9092,CONTROLLER://:9093,EXTERNAL://:49092
      KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka-3:9092,EXTERNAL://localhost:49092

docker compose up -d
docker compose ps

All three nodes share the same CLUSTER_ID and controller list, so they form one cluster. Applications on your machine connect to localhost:29092,localhost:39092,localhost:49092. In production, run controllers and brokers on separate machines and secure all listeners.

8. Creating a Replicated Topic

docker exec -it kafka-1 /opt/kafka/bin/kafka-topics.sh \
  --bootstrap-server kafka-1:9092 \
  --create --topic orders --partitions 3 --replication-factor 3

docker exec -it kafka-1 /opt/kafka/bin/kafka-topics.sh \
  --bootstrap-server kafka-1:9092 --describe --topic orders

Topic: orders  PartitionCount: 3  ReplicationFactor: 3  Configs: min.insync.replicas=2
  Partition: 0  Leader: 1  Replicas: 1,2,3  Isr: 1,2,3
  Partition: 1  Leader: 2  Replicas: 2,3,1  Isr: 2,3,1
  Partition: 2  Leader: 3  Replicas: 3,1,2  Isr: 3,1,2

Each partition has a different leader, so all three brokers share the work, and every partition has copies on all three brokers.

9. Important Cluster Configuration

A handful of broker settings decide how safe and how available your cluster is. They live in each node's server.properties file (or in environment variables in Docker).

The combination replication factor 3 + min.insync.replicas=2 + producer acks=all is the standard way to survive the loss of one broker without losing or refusing acknowledged writes.

10. Connecting to a Kafka Cluster from Java

Clients connect to a cluster through bootstrap.servers. They only need a few brokers in this list: after connecting, they download the full list of brokers and partition leaders and then talk directly to the right broker for each partition. List at least two or three brokers so the client can still start if one is down.

package com.example.kafka;

import java.util.Collection;
import java.util.Properties;

import org.apache.kafka.clients.admin.Admin;
import org.apache.kafka.clients.admin.AdminClientConfig;
import org.apache.kafka.clients.admin.DescribeClusterResult;
import org.apache.kafka.common.Node;

public class ClusterInfo {

    public static void main(String[] args) throws Exception {
        Properties props = new Properties();
        props.put(AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG,
                "localhost:29092,localhost:39092,localhost:49092");

        try (Admin admin = Admin.create(props)) {
            DescribeClusterResult cluster = admin.describeCluster();

            System.out.println("Cluster ID: " + cluster.clusterId().get());

            Collection<Node> brokers = cluster.nodes().get();
            System.out.println("Brokers online: " + brokers.size());
            for (Node broker : brokers) {
                System.out.printf("  id=%d host=%s port=%d rack=%s%n",
                        broker.id(), broker.host(), broker.port(), broker.rack());
            }
        }
    }
}

Cluster ID: 5L6g3nShT-eMCtK--X86sw
Brokers online: 3
  id=1 host=localhost port=29092 rack=null
  id=2 host=localhost port=39092 rack=null
  id=3 host=localhost port=49092 rack=null

A producer or consumer uses the same bootstrap.servers value. Nothing else in the client code changes between a one-broker and a hundred-broker cluster.

Creating a Replicated Topic from Java

The Admin client can also create topics. Here the topic gets 6 partitions, 3 replicas and min.insync.replicas=2, so it is spread across the whole cluster and survives one broker failure:

package com.example.kafka;

import java.util.List;
import java.util.Map;
import java.util.Properties;

import org.apache.kafka.clients.admin.Admin;
import org.apache.kafka.clients.admin.AdminClientConfig;
import org.apache.kafka.clients.admin.NewTopic;

public class CreateOrdersTopic {

    public static void main(String[] args) throws Exception {
        Properties props = new Properties();
        props.put(AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG,
                "localhost:29092,localhost:39092,localhost:49092");

        try (Admin admin = Admin.create(props)) {
            // 6 partitions, 3 copies of each, at least 2 copies must confirm a write
            NewTopic orders = new NewTopic("orders", 6, (short) 3)
                    .configs(Map.of("min.insync.replicas", "2"));

            admin.createTopics(List.of(orders)).all().get();   // wait for the result
            System.out.println("Topic 'orders' created");
        }
    }
}

Producing Safely to a Cluster

These producer settings make sure a message is only reported as sent after it is stored on at least two brokers, and that retries after a broker failure never create duplicates:

package com.example.kafka;

import java.util.Properties;

import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerConfig;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.common.serialization.StringSerializer;

public class ReliableOrderProducer {

    public static void main(String[] args) {
        Properties props = new Properties();
        props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG,
                "localhost:29092,localhost:39092,localhost:49092");
        props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
        props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class);

        // Wait until all in-sync replicas (at least min.insync.replicas) have the record
        props.put(ProducerConfig.ACKS_CONFIG, "all");
        // Retries after a leader change never write the same record twice
        props.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, "true");
        // Keep retrying for up to 2 minutes while the cluster elects new leaders
        props.put(ProducerConfig.DELIVERY_TIMEOUT_MS_CONFIG, 120_000);

        try (KafkaProducer<String, String> producer = new KafkaProducer<>(props)) {
            ProducerRecord<String, String> record =
                    new ProducerRecord<>("orders", "order-1001", "Order placed");

            producer.send(record, (metadata, error) -> {
                if (error != null) {
                    System.err.println("Not stored: " + error.getMessage());
                } else {
                    System.out.printf("Stored in partition %d at offset %d%n",
                            metadata.partition(), metadata.offset());
                }
            });
        }
    }
}

If a broker fails while this program is sending, the producer waits for a new leader and retries. With fewer than two in-sync replicas left, the write fails with a NotEnoughReplicasException instead of being stored unsafely.

11. Connecting from Spring Boot

spring:
  kafka:
    bootstrap-servers: localhost:29092,localhost:39092,localhost:49092
    producer:
      acks: all
    consumer:
      group-id: inventory-service

KafkaTemplate and @KafkaListener then work with the whole cluster automatically. Using acks=all makes the producer wait until the in-sync replicas have each message.

12. What Happens When a Broker Fails?

1. The controllers notice that the broker has stopped sending heartbeats.
2. For each partition the broker was leading, the active controller picks a new leader from the in-sync replicas on other brokers.
3. Clients get updated metadata and switch to the new leaders automatically, usually within seconds.
4. When the broker comes back, it catches up from the current leaders and rejoins the in-sync replicas. Leadership is later moved back to balance the load.

Whether any data can be lost depends on the replication settings:

See Kafka Fault Tolerance for a detailed explanation.

13. Adding Brokers and Rebalancing Partitions

A new broker joins the cluster as soon as it starts with the same cluster ID and controller list. However, Kafka does not move existing partitions to it automatically; only newly created partitions use it. To spread existing data, reassign partitions:

# 1. List the topics to move
echo '{"version":1,"topics":[{"topic":"orders"}]}' > topics.json

# 2. Ask Kafka to propose a new assignment
bin/kafka-reassign-partitions.sh --bootstrap-server localhost:29092 \
  --topics-to-move-json-file topics.json --broker-list "1,2,3,4" --generate

# 3. Save the proposed assignment as plan.json, then apply it with a throttle (bytes/sec)
bin/kafka-reassign-partitions.sh --bootstrap-server localhost:29092 \
  --reassignment-json-file plan.json --execute --throttle 50000000

# 4. Check progress until all moves are complete
bin/kafka-reassign-partitions.sh --bootstrap-server localhost:29092 \
  --reassignment-json-file plan.json --verify

The throttle limits how fast data is copied, so the move does not slow down normal traffic. Running --verify after completion also removes the throttle. Removing a broker works the same way in reverse: move its partitions away first, then shut it down.

14. Rolling Restarts and Upgrades

A Kafka cluster can be restarted or upgraded one broker at a time while applications keep running. This is called a rolling restart:

1. Check that no partitions are under-replicated.
2. Stop one broker gracefully; its partition leaders move to other brokers.
3. Apply the change (configuration, operating system patch or new Kafka version) and start the broker.
4. Wait until it has caught up and all partitions are fully in sync again.
5. Repeat for the next broker.

With replication factor 3 and min.insync.replicas=2, producers using acks=all keep working while one broker is down for maintenance.

15. Racks, Zones and Multiple Data Centers

If all copies of a partition sit in the same server rack or cloud availability zone, one power or network failure could take them all down. Tell each broker where it runs, and Kafka places the copies of each partition in different racks:

# Set a different value per rack or availability zone
broker.rack=zone-a

Consumers can also read from a follower in their own zone, reducing cross-zone network costs, by setting client.rack on the consumer and a rack-aware replica selector on the brokers.

A single cluster should stay within one region, because brokers need fast, reliable connections to each other. To copy data between clusters in different regions or data centers, use Kafka MirrorMaker.

16. Monitoring Cluster Health

These metrics are available through JMX and can be collected with tools such as Prometheus and shown in Grafana or another dashboard. Alert on under-replicated and offline partitions immediately.

17. Checking Cluster Health from Java

Dashboards are best for day-to-day monitoring, but a small program is handy for scripts and health checks. This one lists every partition that has fewer in-sync replicas than replicas (under-replicated) or no leader at all (offline):

package com.example.kafka;

import java.util.Map;
import java.util.Properties;
import java.util.Set;

import org.apache.kafka.clients.admin.Admin;
import org.apache.kafka.clients.admin.AdminClientConfig;
import org.apache.kafka.clients.admin.TopicDescription;
import org.apache.kafka.common.TopicPartitionInfo;

public class ClusterHealthCheck {

    public static void main(String[] args) throws Exception {
        Properties props = new Properties();
        props.put(AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG,
                "localhost:29092,localhost:39092,localhost:49092");

        try (Admin admin = Admin.create(props)) {
            Set<String> topics = admin.listTopics().names().get();
            Map<String, TopicDescription> descriptions =
                    admin.describeTopics(topics).allTopicNames().get();

            int problems = 0;
            for (TopicDescription topic : descriptions.values()) {
                for (TopicPartitionInfo p : topic.partitions()) {
                    boolean offline = p.leader() == null || p.leader().isEmpty();
                    boolean underReplicated = p.isr().size() < p.replicas().size();

                    if (offline || underReplicated) {
                        problems++;
                        System.out.printf("%s-%d: replicas=%d in-sync=%d %s%n",
                                topic.name(), p.partition(), p.replicas().size(),
                                p.isr().size(), offline ? "OFFLINE" : "UNDER-REPLICATED");
                    }
                }
            }
            System.out.println(problems == 0
                    ? "Cluster is healthy"
                    : problems + " problem(s)");
        }
    }
}

orders-1: replicas=3 in-sync=2 UNDER-REPLICATED
orders-4: replicas=3 in-sync=2 UNDER-REPLICATED
2 problem(s)

When the stopped broker comes back and catches up, the same program prints Cluster is healthy again.

18. Securing a Kafka Cluster

By default, Kafka listeners accept plain, unauthenticated connections. A production cluster should protect three things:

• Encryption: use TLS (SSL or SASL_SSL listeners) so data cannot be read on the network.
• Authentication: make clients and brokers prove who they are with SASL (for example SCRAM or OAUTHBEARER) or TLS client certificates.
• Authorization: use ACLs so each application can only read or write the topics it needs.

On the client side, connecting to a secured cluster only needs a few extra properties:

bootstrap.servers=kafka-1.example.com:9093,kafka-2.example.com:9093,kafka-3.example.com:9093
security.protocol=SASL_SSL
sasl.mechanism=SCRAM-SHA-512
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required \
    username="orders-service" \
    password="${ORDERS_SERVICE_PASSWORD}";
ssl.truststore.location=/etc/kafka/truststore.jks
ssl.truststore.password=${TRUSTSTORE_PASSWORD}

Keep passwords in a secrets manager or environment variables, never in source code. See Kafka Security for the full setup.

19. Best Practices

20. Common Mistakes

• Replication factor 1 in production: one disk failure loses data permanently.
• acks=all without min.insync.replicas=2: with only one copy left in sync, "all" can mean a single broker, so the guarantee quietly weakens.
• Only one broker in bootstrap.servers: clients cannot start while that broker is down.
• Wrong advertised listeners: clients connect to the bootstrap broker but then fail, because brokers advertise addresses the clients cannot reach (common with Docker and cloud networks).
• Adding brokers and forgetting reassignment: new brokers stay empty until partitions are moved to them.
• Restarting several brokers at once: partitions can go offline; always restart one broker at a time and wait for zero under-replicated partitions.
• Letting disks fill up: a full disk stops the broker; set retention limits and alert on disk usage.
• Enabling unclean leader election to "fix" an outage: it brings partitions back quickly but can silently lose acknowledged data.

21. Summary

A Kafka cluster is a powerful, distributed system designed to handle real-time data streaming. Brokers store partitions and their copies, controllers manage the cluster’s metadata, and clients connect through a few bootstrap brokers to reach the right leader for each partition. With enough brokers, replication factor 3, min.insync.replicas=2 and acks=all, a cluster survives broker failures without losing acknowledged data, and it can grow by adding brokers and reassigning partitions.

Because of these features, Kafka clusters are widely used in modern systems for event streaming, log aggregation, data pipelines and microservices communication.

Frequently Asked Questions (FAQ)

admin.describeCluster().nodes().get()
     .forEach(node -> System.out.println(node.id() + " " + node.host()));

spring:
  kafka:
    bootstrap-servers: broker1:9092,broker2:9092,broker3:9092