Kafka Architecture

Apache Kafka is a distributed event streaming platform used to move data between applications in real time. It handles very large volumes of data quickly and reliably, and is widely used in microservices, data pipelines and real-time analytics.

This guide explains Kafka’s architecture piece by piece: producers, topics, partitions, brokers, replication, the controller, consumers, consumer groups and offsets. It then follows a message through the whole system, with diagrams and Java and Spring Boot examples.

1. What Is Kafka Architecture?

Kafka architecture describes how Kafka’s components work together to collect, store and distribute data. In simple words, Kafka is a central hub: some applications send messages to it, and other applications read those messages whenever they need them.

Kafka is distributed: it runs on several machines that work together as one cluster. That makes it fast, scalable and fault-tolerant.

A helpful picture is a newspaper archive. Reporters (producers) add new articles to sections (topics). Every article is kept in order and stored for a while. Readers (consumers) come at different times and read from wherever they stopped, without removing anything for other readers.

2. The Big Picture

 Producers                    Kafka Cluster                         Consumers
                  +--------------------------------------+
 Order Service -> |  Broker 1     Broker 2     Broker 3  | -> Payment Service
                  |  orders-P0    orders-P1    orders-P2 |    (group: payments)
 Web Clicks ----> |  clicks-P1    clicks-P0    clicks-P2 | -> Analytics Service
                  +--------------------------------------+    (group: analytics)
                                   |
                         [ KRaft controllers ]
                     (cluster metadata, leader election)

Producers write to topics, topics are split into partitions spread across brokers, and consumers in consumer groups read from those partitions. A small group of controllers keeps track of the cluster.

3. Main Components of Kafka Architecture

Multiple consumer groups can read the same data independently without affecting each other. For example, the payment service and the notification service each receive every order.

4. What a Kafka Message (Record) Contains

5. Topics, Partitions and the Commit Log

Each partition is an ordered, append-only log: new messages are always added at the end, and existing messages are never changed. Every message gets the next offset number in its partition.

Partition 0:  [0] [1] [2] [3] [4] [5]  <- new messages added here
Partition 1:  [0] [1] [2] [3]          <- new messages added here
Partition 2:  [0] [1] [2] [3] [4]      <- new messages added here

Order is guaranteed inside one partition, not across partitions.

• Partitions enable parallelism: different partitions can live on different brokers and be read by different consumers at the same time.
• Ordering is guaranteed only within a partition. Use a key, such as the order ID, to keep related messages in the same partition and therefore in order.
• On disk, each partition is stored as a series of segment files, which makes it easy to delete old data in whole files.

Learn more in Kafka Topic and Kafka Partition.

6. Brokers and the Cluster

A broker is a Kafka server. It stores partitions on its disks, accepts writes from producers and serves reads to consumers. A cluster is several brokers working together; production clusters usually have at least three.

• Partitions of a topic are spread across brokers, so the load is shared.
• Clients only need the addresses of one or a few brokers (the bootstrap servers). From any broker, they learn where every partition lives.
• Adding brokers increases capacity; partitions can then be moved onto the new brokers.

See Kafka Broker and Kafka Cluster.

7. Replication: Leaders and Followers

Each partition is copied to several brokers. The number of copies is the replication factor, usually 3. For each partition:

• One replica is the leader, which handles writes and, by default, reads.
• The others are followers, which copy new data from the leader.
• Followers that are fully caught up form the in-sync replicas (ISR).
• If the leader’s broker fails, an in-sync follower becomes the new leader automatically.

              Broker 1     Broker 2     Broker 3
orders-P0     LEADER       follower     follower
orders-P1     follower     LEADER       follower
orders-P2     follower     follower     LEADER

Leaders are spread across brokers so every broker shares the work. For a deep dive, see Kafka Fault Tolerance.

8. The Controller: ZooKeeper and KRaft

The controller manages the cluster: it tracks which brokers are alive, stores the metadata about topics and partitions, and elects new partition leaders when brokers fail.

KRaft makes Kafka simpler to operate and lets clusters handle far more partitions. Learn more in Kafka KRaft and Kafka ZooKeeper.

9. How Producers Write Data

1. The producer serializes the key and value into bytes, for example JSON.
2. A partitioner chooses the partition: by the key’s hash if there is a key, or spreading messages across partitions if there is not.
3. Messages are batched and optionally compressed for efficiency.
4. The batch is sent to the leader of the chosen partition.
5. The leader appends it to the log; followers copy it.
6. The producer receives an acknowledgement according to its acks setting (acks=all waits for all in-sync replicas).

See Kafka Producer for all producer settings.

10. How Consumers and Consumer Groups Read Data

Consumers pull data from Kafka at their own pace; Kafka does not push it. Consumers that share a group.id form a consumer group:

• Each partition is read by exactly one consumer in the group at a time, so each message is processed once per group.
• Adding consumers to a group spreads partitions across them, increasing processing speed, up to one consumer per partition.
• Different groups read the same topic independently: each gets every message.
• If a consumer joins, leaves or crashes, the group rebalances and partitions are reassigned.

                 orders-P0   orders-P1   orders-P2
Group "payments":
  consumer A        X           X
  consumer B                                X

Group "notifications":
  consumer C        X           X           X

Each group receives every message; inside a group, partitions are shared.

See Kafka Consumer for consumer settings and rebalancing.

11. Offsets and Retention

Each consumer group stores its progress as a committed offset per partition, in the internal __consumer_offsets topic. If a consumer restarts, it continues from its last committed offset.

Unlike a traditional message queue, Kafka does not delete messages when they are read. Data stays until the topic’s retention rule removes it:

Because data is kept, consumers can replay old messages, for example to rebuild a database or fix a bug. See Kafka Offset.

12. How Kafka Architecture Works: Step by Step

1. A producer sends a message with key order-1001 to the orders topic.
2. The key decides the partition, for example partition 1.
3. The message goes to the leader broker of partition 1, which appends it to the log at the next offset.
4. Followers on other brokers copy the message.
5. The producer receives an acknowledgement.
6. Consumers in each interested group fetch the message, process it at their own speed and commit their offset.
7. The message stays in the log until retention removes it.

13. A Complete Java Example

This example shows the core parts working together: a producer, a topic with keys, and a consumer in a consumer group. Add the client library first:

<dependency>
    <groupId>org.apache.kafka</groupId>
    <artifactId>kafka-clients</artifactId>
    <version>4.0.0</version>
</dependency>

package com.example.kafka;

import java.util.Properties;

import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerConfig;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.clients.producer.RecordMetadata;
import org.apache.kafka.common.serialization.StringSerializer;

public class OrderEventProducer {

    public static void main(String[] args) throws Exception {
        Properties props = new Properties();
        props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
        props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
        props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
        props.put(ProducerConfig.ACKS_CONFIG, "all");

        try (KafkaProducer<String, String> producer = new KafkaProducer<>(props)) {
            for (int i = 1001; i <= 1005; i++) {
                String key = "order-" + i;   // same key -> same partition -> kept in order
                String value = "{\"orderId\":\"" + i + "\",\"status\":\"CREATED\"}";

                RecordMetadata meta = producer.send(new ProducerRecord<>("orders", key, value)).get();
                System.out.printf("%s -> partition %d, offset %d%n", key, meta.partition(), meta.offset());
            }
        }
    }
}

order-1001 -> partition 2, offset 0
order-1002 -> partition 0, offset 0
order-1003 -> partition 2, offset 1
order-1004 -> partition 1, offset 0
order-1005 -> partition 0, offset 1

package com.example.kafka;

import java.time.Duration;
import java.util.List;
import java.util.Properties;

import org.apache.kafka.clients.consumer.ConsumerConfig;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.apache.kafka.clients.consumer.KafkaConsumer;
import org.apache.kafka.common.serialization.StringDeserializer;

public class OrderEventConsumer {

    public static void main(String[] args) {
        Properties props = new Properties();
        props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
        props.put(ConsumerConfig.GROUP_ID_CONFIG, "payment-service");   // consumer group
        props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
        props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
        props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest"); // new group: start at the beginning

        try (KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props)) {
            consumer.subscribe(List.of("orders"));

            while (true) {
                ConsumerRecords<String, String> records = consumer.poll(Duration.ofSeconds(1));

                for (ConsumerRecord<String, String> record : records) {
                    System.out.printf("partition=%d offset=%d key=%s value=%s%n",
                            record.partition(), record.offset(), record.key(), record.value());
                }
                // Offsets are committed automatically by default (enable.auto.commit=true)
            }
        }
    }
}

Start the consumer twice with the same group.id, and Kafka splits the partitions between the two instances. Start it with a different group.id, and that instance receives every message independently.

14. Kafka Architecture in a Spring Boot Application

In Spring Boot, the same architecture takes much less code. Add spring-kafka, then configure the connection:

spring:
  kafka:
    bootstrap-servers: localhost:9092
    producer:
      acks: all
      key-serializer: org.apache.kafka.common.serialization.StringSerializer
      value-serializer: org.apache.kafka.common.serialization.StringSerializer
    consumer:
      group-id: notification-service
      auto-offset-reset: earliest
      key-deserializer: org.apache.kafka.common.serialization.StringDeserializer
      value-deserializer: org.apache.kafka.common.serialization.StringDeserializer

package com.example.shop;

import org.apache.kafka.clients.admin.NewTopic;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.kafka.config.TopicBuilder;
import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.stereotype.Component;
import org.springframework.stereotype.Service;

@Configuration
class TopicConfig {

    // Created automatically at startup if it does not exist
    @Bean
    NewTopic ordersTopic() {
        return TopicBuilder.name("orders").partitions(3).replicas(1).build();   // use 3 replicas in production
    }
}

@Service
class OrderPublisher {

    private final KafkaTemplate<String, String> kafka;

    OrderPublisher(KafkaTemplate<String, String> kafka) {
        this.kafka = kafka;
    }

    void orderCreated(String orderId, String json) {
        kafka.send("orders", orderId, json);   // producer
    }
}

@Component
class NotificationListener {

    @KafkaListener(topics = "orders", groupId = "notification-service")   // consumer group
    void onOrder(String json) {
        System.out.println("Sending confirmation email for " + json);
    }
}

15. The Kafka Ecosystem: Connect, Streams and Schema Registry

16. Why Kafka Is So Fast

17. Key Features of Kafka

• High throughput: handles millions of messages per second on a well-sized cluster.
• Scalability: add brokers and partitions to grow capacity without downtime.
• Fault tolerance: data is replicated across brokers to prevent loss.
• Durability: messages are stored on disk and survive restarts.
• Real-time processing: data can be consumed as soon as it is produced.
• Loose coupling: producers and consumers do not know about each other.
• Replay capability: consumers can re-read old messages whenever required.
• Ordering: messages with the same key are kept in order.

18. Example: An Online Shopping Website

Imagine an online shopping website:

1. The order service sends each new order to the orders topic.
2. The payment service (group payments) reads the orders and charges customers.
3. The notification service (group notifications) reads the same orders to send emails and text messages.
4. An analytics service (group analytics) reads them to update sales dashboards.

                          +--> [payments]       Payment Service
Order Service --> orders -+--> [notifications]  Notification Service
                          +--> [analytics]      Analytics Service

All these services work independently, without calling each other directly. A new service, such as fraud detection, can start reading the orders topic tomorrow without any change to the order service. If the notification service is down for an hour, it simply catches up from its last offset when it returns. This makes the system flexible and scalable. See Kafka Use Cases for more examples.

19. Best Practices for Designing with Kafka

1. Name topics clearly, for example by domain and event: orders.created, payments.completed.
2. Choose partition counts with growth in mind: partitions limit how many consumers in a group can work in parallel.
3. Use keys for messages that must stay in order, such as all events of one order.
4. Use replication factor 3 and acks=all for important data.
5. Give each application its own consumer group.
6. Make consumers idempotent, because messages may be delivered more than once.
7. Agree on message formats, ideally with schemas, and keep changes backward compatible.
8. Set retention based on how long data may need to be replayed.
9. Use KRaft for new clusters.
10. Monitor consumer lag, under-replicated partitions and broker health.

20. Summary

Kafka architecture is designed for large-scale, real-time data streaming. Producers send data; topics and partitions organize it as ordered, append-only logs; brokers store it, with replication for safety; a controller, now using KRaft, manages the cluster; and consumers in consumer groups read it at their own pace, tracking progress with offsets. Data stays available for replay according to retention rules. With high throughput, scalability, fault tolerance and loose coupling, Kafka has become a core building block of modern distributed systems.

21. Frequently Asked Questions (FAQ)

docker run -d --name kafka -p 9092:9092 apache/kafka:latest