Apache Kafka Introduction

Apache Kafka is a distributed event streaming platform used to publish, store and process large amounts of data in real time. In simple words, Kafka helps different applications talk to each other by sending messages (events) quickly, reliably and at scale.

This beginner’s guide explains what Kafka is, why it is needed, its core concepts and how it works, and then gets you started: running Kafka locally, sending your first messages from the command line, and writing your first Java and Spring Boot programs.

1. What Is Apache Kafka?

Kafka’s distributed architecture makes it horizontally scalable: new servers can be added to a cluster easily, and consumers can read messages at their own pace.

Kafka was originally developed at LinkedIn to handle its huge flow of activity data and logs. It was released as open source in 2011 and became a top-level Apache Software Foundation project in 2012. Since then it has grown into a general-purpose platform for handling data streams in many scenarios. Kafka itself is written in Java and Scala, but clients exist for a wide range of programming languages.

Kafka is commonly used in modern applications such as banking systems, e-commerce platforms, real-time analytics, logging systems and microservices architectures.

2. Why Kafka Is Needed

In traditional systems, applications communicate directly with each other. As the system grows, this direct communication becomes slow, complex and difficult to manage: every new service needs new connections, and if one service is down, others fail too.

Kafka solves this problem by acting as a central messaging system that decouples producers (senders) and consumers (receivers). Producers write events once; any number of consumers read them independently, even if they are temporarily offline.

3. Key Features of Apache Kafka

See Kafka Features for a detailed look at each feature.

4. Kafka Architecture

Here is a simple diagram showing the main components of a Kafka system:

Read more about Kafka Architecture

5. Core Kafka Concepts

6. How Kafka Works

Kafka runs on a group of servers (brokers) working together. If you need more power, you add more servers. Programs that read data from Kafka can do so whenever they are ready, without being forced to keep up with the sender.

Kafka saves every message in order, one after another, like entries in a diary. This is called a log. Each message has an optional key and a value. Many traditional message queues remove a message once it has been read. Kafka instead keeps messages for as long as you choose, so you can go back and read them again. This is useful when you need to replay past events or rebuild something from its history.

7. Simple Kafka Flow

1. A producer sends a message to a Kafka topic.
2. Kafka stores the message safely on disk and copies it to other brokers.
3. A consumer reads the message from the topic.
4. The consumer processes the message and records how far it has read.

8. Running Kafka on Your Computer

The quickest way to try Kafka is with Docker:

docker run -d --name kafka -p 9092:9092 apache/kafka:4.0.0

Or download Kafka from the official Apache Kafka website, unpack it, and start a single-node cluster. Kafka 4.x brokers need Java 17 or later:

cd kafka_2.13-4.0.0

# 1. Create a cluster ID and format the storage directory (first time only)
KAFKA_CLUSTER_ID="$(bin/kafka-storage.sh random-uuid)"
bin/kafka-storage.sh format --standalone -t "$KAFKA_CLUSTER_ID" -c config/server.properties

# 2. Start the broker
bin/kafka-server-start.sh config/server.properties

Kafka now listens on localhost:9092. This single-node setup is for learning only; production clusters use several brokers.

9. Your First Topic, Producer and Consumer

Open new terminals in the Kafka folder and run these commands. (With Docker, run them inside the container with docker exec -it kafka /opt/kafka/bin/<script>.)

bin/kafka-topics.sh --create --topic orders \
  --bootstrap-server localhost:9092 --partitions 3

bin/kafka-console-producer.sh --topic orders --bootstrap-server localhost:9092
> Order placed: book
> Order placed: laptop

bin/kafka-console-consumer.sh --topic orders --from-beginning \
  --bootstrap-server localhost:9092

The consumer prints every message, including those sent before it started, because of --from-beginning. Start a second consumer and it receives the same messages too.

10. Your First Kafka Program in Java

Add the client library to your Maven project:

<dependency>
    <groupId>org.apache.kafka</groupId>
    <artifactId>kafka-clients</artifactId>
    <version>4.0.0</version>
</dependency>

package com.javabytechie.kafka;

import java.util.Properties;

import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerConfig;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.common.serialization.StringSerializer;

public class HelloKafkaProducer {

    public static void main(String[] args) {
        Properties props = new Properties();
        props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
        props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
        props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class);

        try (KafkaProducer<String, String> producer = new KafkaProducer<>(props)) {
            producer.send(new ProducerRecord<>("orders", "order-1", "Order placed: book"));
            System.out.println("Message sent");
        }
    }
}

package com.javabytechie.kafka;

import java.time.Duration;
import java.util.List;
import java.util.Properties;

import org.apache.kafka.clients.consumer.ConsumerConfig;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.clients.consumer.KafkaConsumer;
import org.apache.kafka.common.serialization.StringDeserializer;

public class HelloKafkaConsumer {

    public static void main(String[] args) {
        Properties props = new Properties();
        props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
        props.put(ConsumerConfig.GROUP_ID_CONFIG, "hello-group");
        props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
        props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
        props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);

        try (KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props)) {
            consumer.subscribe(List.of("orders"));

            while (true) {
                for (ConsumerRecord<String, String> record : consumer.poll(Duration.ofMillis(500))) {
                    System.out.printf("Received key=%s value=%s%n", record.key(), record.value());
                }
            }
        }
    }
}

Run the consumer, then the producer, and the consumer prints the new order. See Kafka Producer and Kafka Consumer for the details.

11. Kafka with Spring Boot

With Spring Boot, add the spring-kafka dependency and set the broker address:

spring:
  kafka:
    bootstrap-servers: localhost:9092
    consumer:
      group-id: order-notifications
      auto-offset-reset: earliest

package com.javabytechie.kafka;

import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.stereotype.Service;

@Service
public class OrderEvents {

    private final KafkaTemplate<String, String> kafkaTemplate;

    public OrderEvents(KafkaTemplate<String, String> kafkaTemplate) {
        this.kafkaTemplate = kafkaTemplate;
    }

    // Producer: publish an event
    public void orderPlaced(String orderId, String details) {
        kafkaTemplate.send("orders", orderId, details);
    }

    // Consumer: react to events
    @KafkaListener(topics = "orders")
    public void onOrder(String details) {
        System.out.println("Sending confirmation for: " + details);
    }
}

Spring Boot creates the producer and consumer, and uses String serializers by default.

12. Kafka vs Traditional Message Queues

See Kafka vs RabbitMQ for a detailed comparison.

13. Where Kafka Is Commonly Used

See Kafka Use Cases for real-world examples.

14. Advantages and Limitations

14.1 Advantages

• Very high throughput and low latency.
• Scales horizontally by adding brokers.
• Durable, replicated storage.
• Many consumers and replay of history.
• Large ecosystem and community.

14.2 Limitations

• More moving parts than a simple queue; needs operational know-how.
• Ordering only within a partition.
• Not designed for request-reply or per-message priorities.
• Not a query database; data is read in order, not searched.
• Default message size limit of about 1 MB.

15. Summary

Apache Kafka is a powerful and reliable platform for handling real-time data streams. It stores events in ordered, replicated logs called topics, lets producers write them once, and lets any number of consumers read them independently and replay them later. With features like high throughput, scalability, fault tolerance and durability, Kafka helps applications communicate efficiently by decoupling producers and consumers, and it has become a core technology in modern system design.

In short, if you need to process large volumes of data in real time and build scalable, event-driven systems, Kafka is an excellent choice.

Frequently Asked Questions (FAQ)

docker run -d --name kafka -p 9092:9092 apache/kafka:4.0.0

kafkaTemplate.send("orders", orderId, details);

@KafkaListener(topics = "orders")
public void onOrder(String details) { ... }