Apache Kafka Topic

In Apache Kafka, a topic is a named category or channel where related messages are stored and organized. Producers write messages to topics, and consumers read from them.

This guide explains how topics work, how they relate to partitions, offsets and replication, and how to create, configure, change and delete them with the command line, Java and Spring Boot, plus best practices for naming, retention and partition counts.

1. What Is a Kafka Topic?

Think of a Kafka topic like a TV channel. Each channel focuses on one kind of content, and viewers choose which channels to watch. In the same way, each topic holds one kind of data, such as orders or page views, and applications choose which topics to read.

Kafka does not change the data in a topic. It only stores and delivers it efficiently, in order, to everyone who reads the topic.

2. Key Features of a Kafka Topic

3. Topics, Partitions and Offsets

A topic is a logical name. Physically, it is made of one or more partitions, and each partition is an ordered log of messages. Each message in a partition has a position number called its offset.

orders-0:  [0] [1] [2] [3] [4]
orders-1:  [0] [1] [2]
orders-2:  [0] [1] [2] [3]

Each partition has its own offsets, starting at 0.
Messages with the same key always go to the same partition.

• Partitions let a topic be spread across several brokers and read by several consumers in parallel.
• Order is guaranteed within a partition, not across the whole topic.
• Consumers remember their progress as offsets, so they can continue after a restart or replay older messages.

Learn more in Kafka Partition and Kafka Offset.

4. Topic Replication

Each partition of a topic can be copied to several brokers. The replication factor is set per topic: with a replication factor of 3, every partition has three copies on three different brokers. One copy is the leader; the others are followers that stay in sync. If the leader’s broker fails, a follower takes over.

Use a replication factor of 3 for important production topics, together with min.insync.replicas=2. See Kafka Fault Tolerance.

5. Naming Topics Well

Topic names can contain letters, digits, dots (.), underscores (_) and hyphens (-), up to 249 characters. Clear, consistent names make a cluster much easier to understand:

• Pick one style for the whole organization, for example domain.entity.event.
• Avoid mixing dots and underscores in the same names: Kafka treats them as equal in metric names, so my.topic and my_topic can collide.
• Do not put changing details, such as team names or environments, in topic names if they may change; use separate clusters for environments.

6. Creating and Managing Topics with the CLI

# Create a topic with 6 partitions and 3 copies of each
bin/kafka-topics.sh --bootstrap-server localhost:9092 \
  --create --topic shop.orders.created \
  --partitions 6 --replication-factor 3 \
  --config retention.ms=604800000 \
  --config min.insync.replicas=2

# List all topics
bin/kafka-topics.sh --bootstrap-server localhost:9092 --list

# Show partitions, leaders, replicas and in-sync replicas
bin/kafka-topics.sh --bootstrap-server localhost:9092 \
  --describe --topic shop.orders.created

# Write a few test messages (type one per line, Ctrl+C to stop)
bin/kafka-console-producer.sh --bootstrap-server localhost:9092 \
  --topic shop.orders.created

# Read all messages from the beginning
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 \
  --topic shop.orders.created --from-beginning

604800000 milliseconds is 7 days. On a single-broker development setup, use --replication-factor 1, because a topic cannot have more copies than there are brokers.

7. Creating Topics with the Java Admin Client

Applications and setup tools can create topics with the Admin client, which is part of kafka-clients:

package com.example.kafka;

import java.util.List;
import java.util.Map;
import java.util.Properties;
import java.util.Set;
import java.util.concurrent.ExecutionException;

import org.apache.kafka.clients.admin.Admin;
import org.apache.kafka.clients.admin.AdminClientConfig;
import org.apache.kafka.clients.admin.NewTopic;
import org.apache.kafka.common.errors.TopicExistsException;

public class TopicSetup {

    public static void main(String[] args) throws Exception {
        Properties props = new Properties();
        props.put(AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");

        try (Admin admin = Admin.create(props)) {
            NewTopic orders = new NewTopic("shop.orders.created", 6, (short) 3)
                    .configs(Map.of(
                            "retention.ms", "604800000",       // 7 days
                            "min.insync.replicas", "2"));

            try {
                admin.createTopics(List.of(orders)).all().get();
                System.out.println("Topic created");
            } catch (ExecutionException e) {
                if (e.getCause() instanceof TopicExistsException) {
                    System.out.println("Topic already exists");
                } else {
                    throw e;
                }
            }

            Set<String> names = admin.listTopics().names().get();
            System.out.println("Topics: " + names);
        }
    }
}

8. Topics in Spring Boot

With Spring for Apache Kafka, declare topics as beans. At startup, Spring Boot’s KafkaAdmin creates any that do not exist yet:

package com.example.shop.config;

import org.apache.kafka.clients.admin.NewTopic;
import org.apache.kafka.common.config.TopicConfig;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.kafka.config.TopicBuilder;

@Configuration
public class KafkaTopicsConfig {

    @Bean
    public NewTopic ordersCreated() {
        return TopicBuilder.name("shop.orders.created")
                .partitions(6)
                .replicas(3)
                .config(TopicConfig.RETENTION_MS_CONFIG, "604800000")
                .config(TopicConfig.MIN_IN_SYNC_REPLICAS_CONFIG, "2")
                .build();
    }

    @Bean
    public NewTopic customerProfiles() {
        // Keeps only the latest profile per customer ID
        return TopicBuilder.name("shop.customers.profile")
                .partitions(3)
                .replicas(3)
                .compact()
                .build();
    }
}

Spring does not delete or shrink existing topics. If a bean asks for more partitions than a topic has, Spring can add them; it never reduces them.

9. Writing to and Reading from a Topic

A producer writes to a topic by name, and a consumer subscribes to it by name:

@Service
class OrderEvents {

    private final KafkaTemplate<String, String> kafka;

    OrderEvents(KafkaTemplate<String, String> kafka) {
        this.kafka = kafka;
    }

    void orderCreated(String orderId, String json) {
        // The key (order ID) decides the partition, keeping one order's events in order
        kafka.send("shop.orders.created", orderId, json);
    }
}

@Component
class InventoryListener {

    @KafkaListener(topics = "shop.orders.created", groupId = "inventory-service")
    void onOrderCreated(ConsumerRecord<String, String> record) {
        System.out.printf("partition=%d offset=%d key=%s%n",
                record.partition(), record.offset(), record.key());
    }
}

See Kafka Producer and Kafka Consumer for details.

10. Important Topic Settings

# Show a topic's settings
bin/kafka-configs.sh --bootstrap-server localhost:9092 \
  --entity-type topics --entity-name shop.orders.created --describe

# Change retention to 3 days while the topic is in use
bin/kafka-configs.sh --bootstrap-server localhost:9092 \
  --entity-type topics --entity-name shop.orders.created \
  --alter --add-config retention.ms=259200000

11. Retention vs Log Compaction

Before compaction:
  [cust-1: Pune] [cust-2: Delhi] [cust-1: Mumbai] [cust-3: Goa] [cust-2: Jaipur]

After compaction:
  [cust-1: Mumbai] [cust-3: Goa] [cust-2: Jaipur]

12. Choosing the Number of Partitions

The number of partitions limits how many consumers in one group can work in parallel, so plan for future growth:

partitions >= peak messages per second / messages per second one consumer can handle

Example: peak 6,000 msg/s, one consumer handles 1,000 msg/s -> at least 6 partitions
Add room for growth -> 12 partitions

• Many teams start with 3–12 partitions for normal topics.
• You can add partitions later, but not remove them.
• Adding partitions changes which partition each key maps to, so ordering by key is only guaranteed for new messages after the change.
• Avoid thousands of partitions per topic without a reason; each one uses broker resources.

13. Changing and Deleting Topics

# Increase partitions from 6 to 12 (cannot decrease)
bin/kafka-topics.sh --bootstrap-server localhost:9092 \
  --alter --topic shop.orders.created --partitions 12

# Delete a topic (allowed by default: delete.topic.enable=true)
bin/kafka-topics.sh --bootstrap-server localhost:9092 \
  --delete --topic shop.orders.test

• A topic cannot be renamed. Create a new topic and move producers and consumers to it.
• Changing the replication factor needs a partition reassignment with kafka-reassign-partitions.sh.
• Deleting a topic removes all its data permanently; make sure no application still needs it.

14. Internal Topics

Do not write to or delete these topics yourself. Make sure internal topics also have a replication factor of 3 in production.

15. Example: An Online Shopping Application

Imagine an online shop. When a customer places an order, the application sends an event to the topic shop.orders.created. Several services read it:

• Inventory service (group inventory-service) updates stock.
• Payment service (group payment-service) processes the payment.
• Notification service (group notification-service) sends an email or SMS.

                                    +--> inventory-service
Order Service --> shop.orders.created --+--> payment-service
                                    +--> notification-service

All three read the same messages, each in its own consumer group, at its own speed. Producers do not need to know who reads the data, and consumers can join or leave at any time. That is why topics are the foundation of real-time, event-driven systems.

16. Best Practices

1. Use a clear naming convention and document it.
2. Create topics deliberately; disable automatic topic creation in production (auto.create.topics.enable=false).
3. Use replication factor 3 and min.insync.replicas=2 for important topics.
4. Plan partition counts for future load.
5. Set retention based on how long data might need to be replayed.
6. Use compaction for “latest value per key” data.
7. One kind of event per topic, or a clear, documented set of related events.
8. Agree on message formats, ideally with schemas, and version breaking changes.
9. Manage topics as code, for example with Spring NewTopic beans, Terraform or GitOps tools.

17. Common Mistakes to Avoid

18. Summary

A Kafka topic is a named, append-only stream of related messages. Physically it is split into partitions, each an ordered log with offsets, replicated across brokers for safety. Topics decouple producers from consumers: producers write once, and any number of consumer groups read independently. Create topics deliberately with the CLI, the Java Admin client or Spring Boot beans; choose clear names, enough partitions and the right retention or compaction policy; and use replication factor 3 for important data. In simple terms, Kafka topics are organized data streams that help applications talk to each other efficiently, and they are the backbone of modern event-driven architectures.

19. Frequently Asked Questions (FAQ)

bin/kafka-topics.sh --bootstrap-server localhost:9092 \
  --create --topic orders --partitions 6 --replication-factor 3

bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 \
  --topic orders --from-beginning --property print.key=true