Kafka KRaft

Kafka KRaft stands for Kafka Raft metadata mode. It is the way Apache Kafka manages its internal metadata by itself, without ZooKeeper. Since Kafka 4.0, KRaft is the only supported mode.

This guide explains why KRaft was introduced, how controllers and brokers work together, how the metadata quorum survives failures, and how to set up, inspect and monitor a KRaft cluster, with formatted configuration, command-line and Java examples. It builds on Kafka Architecture and Kafka Cluster.

1. What Is Kafka KRaft?

Every Kafka cluster needs metadata: information about brokers, topics, partitions, partition leaders, configurations and access rules. Earlier, Kafka stored this information in a separate system, Apache ZooKeeper. With KRaft, Kafka stores and manages all of it itself.

In simple words, KRaft removes ZooKeeper and makes Kafka a self-managed and simpler system: one product to install, configure, secure and monitor.

 Before (ZooKeeper mode)                  Now (KRaft mode)

 +------------------+                     +--------------------------+
 | ZooKeeper        |  <- metadata        | Kafka controllers (Raft) |  <- metadata
 | ensemble         |                     +--------------------------+
 +------------------+                                  |
          |                                            v
 +------------------+                     +--------------------------+
 | Kafka brokers    |  <- data            | Kafka brokers            |  <- data
 +------------------+                     +--------------------------+
   two systems                              one system

2. Why Was Kafka KRaft Introduced?

ZooKeeper worked well for many years, but it added extra complexity. Teams had to install, manage, monitor and secure two distributed systems instead of one. It also limited Kafka’s growth:

• Metadata had to be copied between ZooKeeper and the active controller, which was slow for clusters with many partitions.
• When the controller changed, the new one had to load all metadata from ZooKeeper before it could act, delaying recovery.
• ZooKeeper had its own configuration, security model and upgrade process.

Kafka KRaft was introduced to:

• Simplify Kafka’s architecture.
• Reduce operational overhead.
• Improve scalability to millions of partitions.
• Increase reliability and speed up recovery.

3. Key Features of Kafka KRaft

4. KRaft Version Timeline

5. Node Roles: Brokers and Controllers

In KRaft mode, every Kafka node has one or two roles, set with process.roles:

Among the controllers, one is the active controller (the Raft leader). It makes all metadata decisions, such as creating topics and choosing partition leaders. The other controllers are followers that keep an up-to-date copy of the metadata log, ready to take over.

6. How Kafka KRaft Works

In a KRaft-based cluster:

• Some Kafka nodes act as controllers.
• Controllers manage metadata using the Raft protocol.
• One controller becomes the leader (active controller); the others are followers.
• Brokers continuously fetch new metadata records from the active controller and keep a local copy.

All metadata changes, like creating a topic, go through the leader. The leader appends them to the metadata log, and a change is committed once a majority of controllers have stored it. Brokers then apply it.

   Admin: create topic "orders"
              |
              v
   +---------------------+   replicate    +---------------------+
   | Controller 1        | -------------> | Controller 2        |
   | ACTIVE (leader)     | -------------> | Controller 3        |
   | __cluster_metadata  |                | (followers)         |
   +---------------------+                +---------------------+
              |
              |  brokers fetch new metadata records
              v
   +----------+  +----------+  +----------+
   | Broker 1 |  | Broker 2 |  | Broker 3 |   <- create partitions, serve clients
   +----------+  +----------+  +----------+

To keep the log from growing forever, controllers regularly write snapshots of the full metadata state. A new or restarted node loads the latest snapshot and then reads only the records after it.

7. Simple Example of Kafka KRaft

Imagine a small Kafka cluster with 3 nodes running in combined mode (each node is both a broker and a controller): Node 1, Node 2 and Node 3. In KRaft mode:

• Node 1 is elected as the controller leader.
• Node 2 and Node 3 are controller followers.

Now, when you create a new topic:

1. The request goes to the controller leader (Node 1).
2. The leader writes the metadata change to the Raft log.
3. The change is replicated to Node 2 and Node 3.
4. Once a majority has stored it, the change is committed, and the brokers create the topic’s partitions.

If Node 1 fails, Node 2 and Node 3 hold an election and one of them becomes the new controller leader within moments. No ZooKeeper is involved at any step. In production, the same process runs on dedicated controller nodes instead of combined nodes.

8. Controller Quorum and Fault Tolerance

Raft needs a majority of controllers to agree on every change. That decides how many controller failures a cluster can survive:

If the majority is lost, the cluster cannot make metadata changes (such as creating topics or electing new partition leaders), although brokers can keep serving existing partitions for a while. Place controllers in different racks or availability zones so one failure cannot take out the majority.

9. Running a Single-Node KRaft Cluster

For learning, one node can play both roles. Kafka 4.x ships with a ready-made configuration in config/server.properties. The important settings are:

# This node is both a broker and a controller
process.roles=broker,controller
node.id=1

# The controller quorum (just this node)
controller.quorum.bootstrap.servers=localhost:9093

# Client listener on 9092, controller listener on 9093
listeners=PLAINTEXT://:9092,CONTROLLER://:9093
advertised.listeners=PLAINTEXT://localhost:9092
controller.listener.names=CONTROLLER
listener.security.protocol.map=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT

log.dirs=/tmp/kraft-combined-logs

# 1. Generate a cluster ID
KAFKA_CLUSTER_ID="$(bin/kafka-storage.sh random-uuid)"

# 2. Format the log directory (first start only)
bin/kafka-storage.sh format --standalone -t "$KAFKA_CLUSTER_ID" -c config/server.properties

# 3. Start the node
bin/kafka-server-start.sh config/server.properties

Formatting writes the cluster ID and initial metadata into the log directory. Every node in a cluster must be formatted with the same cluster ID.

10. A Production-Style KRaft Cluster

In production, controllers and brokers usually run on separate machines. Below are example settings for three dedicated controllers and any number of brokers.

process.roles=controller
node.id=1

# All controller voters: id@host:port
controller.quorum.voters=1@controller-1:9093,2@controller-2:9093,3@controller-3:9093

listeners=CONTROLLER://controller-1:9093
controller.listener.names=CONTROLLER
log.dirs=/var/lib/kafka/metadata

process.roles=broker
node.id=4

# Where to find the controllers
controller.quorum.voters=1@controller-1:9093,2@controller-2:9093,3@controller-3:9093
controller.listener.names=CONTROLLER

listeners=PLAINTEXT://broker-4:9092
advertised.listeners=PLAINTEXT://broker-4.example.com:9092
inter.broker.listener.name=PLAINTEXT
listener.security.protocol.map=PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT
log.dirs=/var/lib/kafka/data

Controllers 2 and 3 use the same file with their own node.id and host name. Every node.id in the cluster must be unique. Format each node with the shared cluster ID before its first start, and secure the listeners with TLS and authentication in real deployments (see Kafka Security).

Kafka 3.9 and later also support a dynamic quorum, configured with controller.quorum.bootstrap.servers, which lets you add or remove controllers without restarting the whole quorum.

11. Checking the Metadata Quorum

The kafka-metadata-quorum.sh tool shows the health of the controller quorum:

bin/kafka-metadata-quorum.sh --bootstrap-server localhost:9092 describe --status

ClusterId:              Mk3OEYBSD34fcwNTJENDM2Qk
LeaderId:               1
LeaderEpoch:            7
HighWatermark:          10432
MaxFollowerLag:         0
MaxFollowerLagTimeMs:   12
CurrentVoters:          [1,2,3]
CurrentObservers:       [4,5,6]

Use describe --replication to see each node’s position in the metadata log. Monitor the quorum with JMX metrics as well, for example the active controller count, which should be exactly 1 across the cluster.

12. Reading Quorum Information in Java

The Java Admin client can read the same information, which is useful for health checks and dashboards:

package com.example.kafka;

import java.util.Properties;

import org.apache.kafka.clients.admin.Admin;
import org.apache.kafka.clients.admin.AdminClientConfig;
import org.apache.kafka.clients.admin.QuorumInfo;

public class QuorumInfoReader {

    public static void main(String[] args) throws Exception {
        Properties props = new Properties();
        props.put(AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");

        try (Admin admin = Admin.create(props)) {
            // Details of the Raft metadata quorum
            QuorumInfo quorum = admin.describeMetadataQuorum().quorumInfo().get();
            System.out.println("Active controller (leader) ID: " + quorum.leaderId());
            System.out.println("Leader epoch:                  " + quorum.leaderEpoch());

            for (QuorumInfo.ReplicaState voter : quorum.voters()) {
                System.out.printf("Voter %d at metadata offset %d%n",
                        voter.replicaId(), voter.logEndOffset());
            }
            System.out.println("Observers (brokers): " + quorum.observers().size());
        }
    }
}

Active controller (leader) ID: 1
Leader epoch:                  7
Voter 1 at metadata offset 10432
Voter 2 at metadata offset 10432
Voter 3 at metadata offset 10431
Observers (brokers): 3

Client applications such as producers, consumers and Spring Boot services need no changes for KRaft. They connect to brokers exactly as before; only the cluster’s internal metadata handling is different.

13. Kafka with ZooKeeper vs Kafka KRaft

14. Moving from ZooKeeper to KRaft

Existing ZooKeeper-based clusters must move to KRaft before upgrading to Kafka 4.0. Kafka 3.x provides an online migration that keeps the cluster running. At a high level:

1. Upgrade the cluster to a recent 3.x version (3.9 is the recommended bridge release).
2. Deploy a new KRaft controller quorum with migration enabled.
3. Restart the brokers one at a time so they register with the new controllers; metadata is copied from ZooKeeper.
4. Restart the brokers again in pure KRaft mode.
5. Finalize the migration, remove ZooKeeper settings, and shut down ZooKeeper.
6. Upgrade to Kafka 4.x.

Test the migration in a non-production environment first, and follow the official migration guide for your exact version, because finalizing the migration cannot be undone. See Kafka ZooKeeper for background on the old mode.

15. When Should You Use Kafka KRaft?

For any new cluster, KRaft is the only choice: Kafka 4.0 and later do not support ZooKeeper. KRaft is especially helpful if:

• You are setting up a new Kafka cluster.
• You want simpler deployment and maintenance.
• You plan to scale Kafka to many topics and partitions.
• You want to avoid managing and securing ZooKeeper.
• You want to upgrade to Kafka 4.x and later.

16. Best Practices

17. Summary

Kafka KRaft is a major improvement in Kafka’s architecture. It removes ZooKeeper and lets Kafka manage its own metadata using the Raft consensus algorithm: a small quorum of controllers stores the metadata log, one active controller makes decisions, and brokers follow the log to stay up to date. KRaft became production-ready in Kafka 3.3 and is the only mode from Kafka 4.0 onward.

With KRaft, Kafka becomes:

• Simpler to manage.
• More scalable.
• Faster to recover.
• Ready for the future.

If you are learning or working with modern Kafka, understanding KRaft is essential, because it is how every current Kafka cluster runs.

18. Frequently Asked Questions (FAQ)

bin/kafka-metadata-quorum.sh --bootstrap-server localhost:9092 describe --status