00:00

Help Us Improve

We're sorry this tutorial wasn't helpful. Please tell us what we can do better.

Free Apache Kafka Course

Apache Kafka Tutorial

Learn Apache Kafka step by step, from your first topic to production-ready event streaming. Clear explanations, diagrams and working Java and Spring Boot examples, written for developers.

Start Learning View Learning Path
  • 31 tutorials
  • 11 learning tracks
  • Java & Spring Boot code
  • Beginner to advanced

What Is Apache Kafka?

Apache Kafka is an open-source distributed event streaming platform. Applications called producers write events (messages) to Kafka, Kafka stores them safely and in order, and other applications called consumers read them, in real time or later. It is used for data pipelines, event-driven microservices, real-time analytics and system integration at companies of every size.

TypeDistributed event log
OriginLinkedIn, open-sourced 2011
LicenseApache License 2.0
MetadataKRaft (no ZooKeeper in 4.0+)

What You Will Learn

  • How Kafka works: topics, partitions, offsets, brokers and clusters
  • Writing producers and consumers in plain Java
  • Consumer groups, scaling and rebalancing
  • Delivery guarantees: acknowledgements, idempotence and transactions
  • Fault tolerance, replication and disaster recovery
  • KRaft mode and how it replaced ZooKeeper
  • Kafka Streams and Kafka Connect for processing and integration
  • Building Kafka applications with Spring Boot
  • Securing, tuning and running Kafka in production

Who this is for

Java and Spring Boot developers, backend engineers building microservices, and anyone preparing for Kafka or system design interviews.

Prerequisites

Basic Java and comfort with the command line. Knowing Spring Boot helps for the Spring Kafka track, but is not required to start.

Kafka Architecture at a Glance

Producers send events to topics. Each topic is split into partitions that are stored and replicated across the brokers of a cluster. Consumers in a consumer group share the partitions and track their progress with offsets. A small group of KRaft controllers manages the cluster's metadata.

ProducerPublishes events to topics ConsumerReads events from topics TopicA named stream of events PartitionAn ordered slice of a topic BrokerA Kafka server storing data OffsetA record's position in a partition

Read the full Kafka Architecture tutorial

Kafka Learning Path

Follow the tracks in order, or jump straight to the topic you need. Each tutorial explains the idea in simple words, then shows working code and answers common questions.

07

Spring Kafka

Build Kafka producers and consumers with Spring Boot.

8 more tutorials coming soon

08

Kafka Security

Encryption, authentication and authorization for Kafka clusters.

6 more tutorials coming soon

09

Kafka Monitoring & Operations

CLI tools, monitoring, consumer lag and production checklists.

Tutorials for this track are being written.

6 more tutorials coming soon

10

Kafka Architecture & Design

Event-driven design patterns built on Kafka.

Tutorials for this track are being written.

6 more tutorials coming soon

11

Kafka Interview Questions

Questions and answers to prepare for Kafka interviews.

Tutorials for this track are being written.

5 more tutorials coming soon

Quick Start: Your First Kafka Program in Java

With a Kafka broker running on localhost:9092 and the kafka-clients library on the classpath, these two small programs send an event to the orders topic and read it back.

HelloKafkaProducer.java 
import java.util.Properties;

import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerConfig;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.common.serialization.StringSerializer;

public class HelloKafkaProducer {

    public static void main(String[] args) {
        Properties props = new Properties();
        props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
        props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
        props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class);

        try (KafkaProducer<String, String> producer = new KafkaProducer<>(props)) {
            producer.send(new ProducerRecord<>("orders", "order-1", "Order placed"));
            System.out.println("Event sent to topic 'orders'");
        }
    }
}
HelloKafkaConsumer.java 
import java.time.Duration;
import java.util.List;
import java.util.Properties;

import org.apache.kafka.clients.consumer.ConsumerConfig;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.clients.consumer.KafkaConsumer;
import org.apache.kafka.common.serialization.StringDeserializer;

public class HelloKafkaConsumer {

    public static void main(String[] args) {
        Properties props = new Properties();
        props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
        props.put(ConsumerConfig.GROUP_ID_CONFIG, "hello-group");
        props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
        props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
        props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);

        try (KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props)) {
            consumer.subscribe(List.of("orders"));

            while (true) {
                var records = consumer.poll(Duration.ofSeconds(1));
                for (ConsumerRecord<String, String> record : records) {
                    System.out.printf("key=%s value=%s offset=%d%n",
                            record.key(), record.value(), record.offset());
                }
            }
        }
    }
}

Set up Kafka locally and run these step by step in Kafka Introduction

Why Developers Choose Kafka

High Throughput

Batching and sequential disk writes let a cluster handle very large volumes of events with low latency.

Horizontal Scaling

Add brokers and partitions to grow capacity, and add consumers to process in parallel.

Durable Storage

Events are written to disk and replicated, and can be kept for days, months or forever.

Replay

Consumers can re-read old events at any time by moving their offset, which makes recovery and new features easy.

Fault Tolerance

When a broker fails, another replica takes over automatically, with no acknowledged data lost.

Rich Ecosystem

Kafka Connect, Kafka Streams and clients for most languages connect Kafka to almost anything.

Explore all Kafka features

Common Kafka Use Cases

Event-Driven Microservices

Services publish events instead of calling each other directly, so they stay loosely coupled.

Data Pipelines

Move data reliably between databases, data warehouses, search engines and cloud storage.

Real-Time Analytics

Process clicks, orders and sensor readings as they happen for dashboards and alerts.

Log & Metrics Collection

Gather logs and metrics from many servers into one place for monitoring.

See real-world Kafka use cases

Kafka Compared

Kafka vs RabbitMQ Log-based streaming versus a classic message broker: storage, routing, ordering and when to use each. Kafka vs JMS An event streaming platform versus the Java messaging standard, with Java examples of both. ZooKeeper vs KRaft How Kafka manages its cluster metadata, and why KRaft replaced ZooKeeper.