Apache Kafka Introduction

Apache Kafka is a distributed event streaming platform used to publish, store and process large amounts of data in real time. In simple words, Kafka helps different applications talk to each other by sending messages (events) quickly, reliably and at scale.

This beginner’s guide explains what Kafka is, why it is needed, its core concepts and how it works, and then gets you started: running Kafka locally, sending your first messages from the command line, and writing your first Java and Spring Boot programs.

1. What Is Apache Kafka?

Kafka’s distributed architecture makes it horizontally scalable: new servers can be added to a cluster easily, and consumers can read messages at their own pace.

Kafka was originally developed at LinkedIn to handle its huge flow of activity data and logs. It was released as open source in 2011 and became a top-level Apache Software Foundation project in 2012. Since then it has grown into a general-purpose platform for handling data streams in many scenarios. Kafka itself is written in Java and Scala, but clients exist for a wide range of programming languages.

Kafka is commonly used in modern applications such as banking systems, e-commerce platforms, real-time analytics, logging systems and microservices architectures.

2. Why Kafka Is Needed

In traditional systems, applications communicate directly with each other. As the system grows, this direct communication becomes slow, complex and difficult to manage: every new service needs new connections, and if one service is down, others fail too.

Without Kafka:                              With Kafka:

 Orders ----> Inventory                      Orders ---+
 Orders ----> Payments                                 v
 Orders ----> Email                            [ Kafka: orders topic ]
 Orders ----> Analytics                          |     |     |      |
                                          Inventory Payments Email Analytics
 Each new service = new connection         New service = just start reading

Kafka solves this problem by acting as a central messaging system that decouples producers (senders) and consumers (receivers). Producers write events once; any number of consumers read them independently, even if they are temporarily offline.

3. Key Features of Apache Kafka

See Kafka Features for a detailed look at each feature.

4. Core Kafka Concepts

5. How Kafka Works

Kafka runs on a group of servers (brokers) working together. If you need more power, you add more servers. Programs that read data from Kafka can do so whenever they are ready, without being forced to keep up with the sender.

Kafka saves every message in order, one after another, like entries in a diary. This is called a log. Each message has an optional key and a value. Many traditional message queues remove a message once it has been read. Kafka instead keeps messages for as long as you choose, so you can go back and read them again. This is useful when you need to replay past events or rebuild something from its history.

Topic "orders", partition 0

 offset:   0          1             2              3
        +--------+-----------+--------------+-------------+
        | Order  | Payment   | Order        | Order       |  <- new events are
        | placed | received  | packed       | shipped     |     added at the end
        +--------+-----------+--------------+-------------+
              ^                                 ^
     Analytics reads here              Email service reads here

6. Simple Kafka Flow

1. A producer sends a message to a Kafka topic.
2. Kafka stores the message safely on disk and copies it to other brokers.
3. A consumer reads the message from the topic.
4. The consumer processes the message and records how far it has read.

7. Simple Real-World Example

Imagine an online shopping application:

• A user places an order.
• The order service sends an event called “Order Placed” to Kafka.
• Several services consume this event independently:

All these services work independently, and Kafka makes sure each of them receives the event reliably and in real time. If the notification service is down for a while, it simply catches up when it comes back, without the order service noticing.

8. Kafka Architecture at a Glance

 Producers                    Kafka cluster                      Consumers
                 +-------------------------------------+
 Order app --->  |  Broker 1     Broker 2    Broker 3  |  ---> Billing (group A)
 Web app   --->  |  orders-P0    orders-P1   orders-P2 |  ---> Analytics (group B)
                 |  (+ copies of other partitions)     |
                 |  KRaft controllers manage metadata  |
                 +-------------------------------------+

• Brokers store partitions and serve producers and consumers.
• Each partition has a leader and followers on different brokers.
• Controllers manage cluster metadata, such as which broker leads each partition. Modern Kafka does this with its built-in KRaft protocol; older versions used a separate ZooKeeper cluster, which Kafka 4.0 no longer supports.

See Kafka Architecture for the full picture.

9. Running Kafka on Your Computer

The quickest way to try Kafka is with Docker:

docker run -d --name kafka -p 9092:9092 apache/kafka:4.0.0

Or download Kafka from the official Apache Kafka website, unpack it, and start a single-node cluster. Kafka 4.x brokers need Java 17 or later:

cd kafka_2.13-4.0.0

# 1. Create a cluster ID and format the storage directory (first time only)
KAFKA_CLUSTER_ID="$(bin/kafka-storage.sh random-uuid)"
bin/kafka-storage.sh format --standalone -t "$KAFKA_CLUSTER_ID" -c config/server.properties

# 2. Start the broker
bin/kafka-server-start.sh config/server.properties

Kafka now listens on localhost:9092. This single-node setup is for learning only; production clusters use several brokers.

10. Your First Topic, Producer and Consumer

Open new terminals in the Kafka folder and run these commands. (With Docker, run them inside the container with docker exec -it kafka /opt/kafka/bin/<script>.)

bin/kafka-topics.sh --create --topic orders \
  --bootstrap-server localhost:9092 --partitions 3

bin/kafka-console-producer.sh --topic orders --bootstrap-server localhost:9092
> Order placed: book
> Order placed: laptop

bin/kafka-console-consumer.sh --topic orders --from-beginning \
  --bootstrap-server localhost:9092

The consumer prints every message, including those sent before it started, because of --from-beginning. Start a second consumer and it receives the same messages too.

11. Your First Kafka Program in Java

Add the client library to your Maven project:

<dependency>
    <groupId>org.apache.kafka</groupId>
    <artifactId>kafka-clients</artifactId>
    <version>4.0.0</version>
</dependency>

package com.example.kafka;

import java.util.Properties;

import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerConfig;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.common.serialization.StringSerializer;

public class HelloKafkaProducer {

    public static void main(String[] args) {
        Properties props = new Properties();
        props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
        props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
        props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class);

        try (KafkaProducer<String, String> producer = new KafkaProducer<>(props)) {
            producer.send(new ProducerRecord<>("orders", "order-1", "Order placed: book"));
            System.out.println("Message sent");
        }
    }
}

package com.example.kafka;

import java.time.Duration;
import java.util.List;
import java.util.Properties;

import org.apache.kafka.clients.consumer.ConsumerConfig;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.clients.consumer.KafkaConsumer;
import org.apache.kafka.common.serialization.StringDeserializer;

public class HelloKafkaConsumer {

    public static void main(String[] args) {
        Properties props = new Properties();
        props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
        props.put(ConsumerConfig.GROUP_ID_CONFIG, "hello-group");
        props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
        props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
        props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);

        try (KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props)) {
            consumer.subscribe(List.of("orders"));

            while (true) {
                for (ConsumerRecord<String, String> record : consumer.poll(Duration.ofMillis(500))) {
                    System.out.printf("Received key=%s value=%s%n", record.key(), record.value());
                }
            }
        }
    }
}

Run the consumer, then the producer, and the consumer prints the new order. See Kafka Producer and Kafka Consumer for the details.

12. Kafka with Spring Boot

With Spring Boot, add the spring-kafka dependency and set the broker address:

spring:
  kafka:
    bootstrap-servers: localhost:9092
    consumer:
      group-id: order-notifications
      auto-offset-reset: earliest

package com.example.kafka;

import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.stereotype.Service;

@Service
public class OrderEvents {

    private final KafkaTemplate<String, String> kafkaTemplate;

    public OrderEvents(KafkaTemplate<String, String> kafkaTemplate) {
        this.kafkaTemplate = kafkaTemplate;
    }

    // Producer: publish an event
    public void orderPlaced(String orderId, String details) {
        kafkaTemplate.send("orders", orderId, details);
    }

    // Consumer: react to events
    @KafkaListener(topics = "orders")
    public void onOrder(String details) {
        System.out.println("Sending confirmation for: " + details);
    }
}

Spring Boot creates the producer and consumer, and uses String serializers by default.

13. Kafka vs Traditional Message Queues

See Kafka vs RabbitMQ for a detailed comparison.

14. Where Kafka Is Commonly Used

See Kafka Use Cases for real-world examples.

15. The Kafka Ecosystem

16. Advantages and Limitations

17. What to Learn Next

1. Kafka Architecture: how the pieces fit together.
2. Kafka Topic, Kafka Partition and Kafka Offset: how data is organized.
3. Kafka Producer and Kafka Consumer: writing and reading data.
4. Kafka Broker and Kafka Cluster: running Kafka.
5. Kafka Fault Tolerance and Kafka Security: production readiness.

18. Summary

Apache Kafka is a powerful and reliable platform for handling real-time data streams. It stores events in ordered, replicated logs called topics, lets producers write them once, and lets any number of consumers read them independently and replay them later. With features like high throughput, scalability, fault tolerance and durability, Kafka helps applications communicate efficiently by decoupling producers and consumers, and it has become a core technology in modern system design.

In short, if you need to process large volumes of data in real time and build scalable, event-driven systems, Kafka is an excellent choice.

19. Frequently Asked Questions (FAQ)

docker run -d --name kafka -p 9092:9092 apache/kafka:4.0.0

kafkaTemplate.send("orders", orderId, details);

@KafkaListener(topics = "orders")
public void onOrder(String details) { ... }