Kafka Partition

In Apache Kafka, a partition is a smaller, ordered part of a topic. A topic is divided into one or more partitions so that data can be stored, processed and read in parallel.

This guide explains why partitions exist, what is inside them, how producers choose a partition, how consumer groups share partitions, how ordering and replication work, and how to choose and change the number of partitions, with formatted Java, Spring Boot and command-line examples. It builds on Kafka Topic and leads into Kafka Offset.

1. What Is a Kafka Partition?

You can think of a Kafka topic as a large book and partitions as the chapters of that book. Each chapter holds part of the content, so many people can read the book at the same time.

Topic: order-events

Partition 0:  [ 0 ][ 1 ][ 2 ][ 3 ][ 4 ]  ->  new messages added at the end
Partition 1:  [ 0 ][ 1 ][ 2 ]            ->
Partition 2:  [ 0 ][ 1 ][ 2 ][ 3 ]       ->

Each partition is its own ordered log with its own offsets.

Every message in a topic belongs to exactly one partition. Inside that partition it gets a position number called an offset.

2. Why Does Kafka Use Partitions?

Kafka uses partitions to improve performance, scalability and reliability. Instead of storing all messages of a topic in one place, Kafka spreads them across multiple partitions, which can live on different brokers:

• More write capacity: producers write to many partitions on many brokers at once.
• More read capacity: each partition can be read by a different consumer in a group.
• No single-server limit: a topic can grow beyond the disk and network of one machine.

Without partitions, every topic would be limited to one server and one reader per application.

3. Key Features of Kafka Partitions

4. What Is Inside a Partition?

A partition is an append-only log: new messages are only ever added at the end, and existing messages are never changed. On the broker’s disk, each partition is a folder, such as order-events-0, containing:

Old segments are deleted when they pass the topic’s retention time (7 days by default) or size limit, so a partition does not grow forever. Reading messages does not remove them.

5. Simple Example of a Kafka Partition

Let’s say you have a topic called order-events with 3 partitions: Partition 0, Partition 1 and Partition 2. When an order event is produced:

• If the message has a key such as orderId = 101, Kafka always sends it to the same partition, for example Partition 1. All events for order 101 stay together and in order.
• If no key is provided, the producer spreads messages across the partitions so that each gets a fair share over time.

On the consumer side, if you have three consumers in the same consumer group, each consumer reads one partition at the same time. This makes order processing faster, and each order’s events are still processed in sequence by one consumer.

6. How Producers Choose a Partition

partition = positive( murmur2( keyBytes ) ) % numberOfPartitions

key "101" -> same hash every time -> same partition every time (for example 1)
key "205" -> a different hash     -> possibly a different partition

Because the result depends on the number of partitions, adding partitions later changes where keys go. This is why the partition count should be planned up front for keyed topics.

7. Creating a Topic with Partitions

The number of partitions is chosen when a topic is created. If a topic is created automatically, the broker uses num.partitions, which defaults to 1, so it is better to create topics explicitly.

bin/kafka-topics.sh --bootstrap-server localhost:9092 \
  --create --topic order-events \
  --partitions 6 --replication-factor 3

package com.example.kafka;

import java.util.List;
import java.util.Properties;

import org.apache.kafka.clients.admin.Admin;
import org.apache.kafka.clients.admin.AdminClientConfig;
import org.apache.kafka.clients.admin.NewTopic;

public class CreateTopic {

    public static void main(String[] args) throws Exception {
        Properties props = new Properties();
        props.put(AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");

        try (Admin admin = Admin.create(props)) {
            NewTopic topic = new NewTopic("order-events", 6, (short) 3);   // 6 partitions, 3 replicas
            admin.createTopics(List.of(topic)).all().get();
            System.out.println("Topic created");
        }
    }
}

In Spring Boot, declare a NewTopic bean and the application creates the topic at startup if it does not exist:

package com.example.kafka;

import org.apache.kafka.clients.admin.NewTopic;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.kafka.config.TopicBuilder;

@Configuration
public class KafkaTopicConfig {

    @Bean
    public NewTopic orderEventsTopic() {
        return TopicBuilder.name("order-events")
                .partitions(6)
                .replicas(3)
                .build();
    }
}

8. Sending Keyed Messages in Java

The producer below sends events for two orders. Because the order ID is the key, every event for the same order lands in the same partition. The returned RecordMetadata shows where each message went:

package com.example.kafka;

import java.util.Properties;

import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerConfig;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.clients.producer.RecordMetadata;
import org.apache.kafka.common.serialization.StringSerializer;

public class OrderEventProducer {

    public static void main(String[] args) throws Exception {
        Properties props = new Properties();
        props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
        props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
        props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class);

        String[][] events = {
                {"101", "OrderCreated"},
                {"205", "OrderCreated"},
                {"101", "PaymentReceived"},
                {"101", "OrderShipped"}
        };

        try (KafkaProducer<String, String> producer = new KafkaProducer<>(props)) {
            for (String[] event : events) {
                ProducerRecord<String, String> record =
                        new ProducerRecord<>("order-events", event[0], event[1]);   // key = order ID

                RecordMetadata meta = producer.send(record).get();
                System.out.printf("key=%s value=%s -> partition=%d offset=%d%n",
                        event[0], event[1], meta.partition(), meta.offset());
            }
        }
    }
}

key=101 value=OrderCreated    -> partition=4 offset=0
key=205 value=OrderCreated    -> partition=1 offset=0
key=101 value=PaymentReceived -> partition=4 offset=1
key=101 value=OrderShipped    -> partition=4 offset=2

Calling .get() waits for each message, which is fine for a demo. In real applications, pass a callback to send() instead, so messages are sent in batches without waiting.

9. Writing a Custom Partitioner

Sometimes the key hash is not enough. For example, you may want all orders from a very large customer to go to a dedicated partition. A custom partitioner implements the Partitioner interface:

package com.example.kafka;

import java.util.Map;

import org.apache.kafka.clients.producer.Partitioner;
import org.apache.kafka.common.Cluster;
import org.apache.kafka.common.utils.Utils;

public class VipCustomerPartitioner implements Partitioner {

    private static final String VIP_CUSTOMER = "customer-1";

    @Override
    public int partition(String topic, Object key, byte[] keyBytes,
                         Object value, byte[] valueBytes, Cluster cluster) {

        int partitionCount = cluster.partitionCountForTopic(topic);

        if (partitionCount == 1 || VIP_CUSTOMER.equals(key)) {
            return 0;                                             // reserved partition (or the only one)
        }
        if (keyBytes == null) {
            return 1 + (int) (Math.random() * (partitionCount - 1));
        }
        // other keys spread over partitions 1 .. partitionCount-1
        return 1 + Utils.toPositive(Utils.murmur2(keyBytes)) % (partitionCount - 1);
    }

    @Override
    public void configure(Map<String, ?> configs) {
    }

    @Override
    public void close() {
    }
}

partitioner.class=com.example.kafka.VipCustomerPartitioner

Use custom partitioners only when there is a clear need. The built-in behaviour is well tested and usually spreads load better.

10. Partitions and Consumer Groups

Consumers that share the same group.id form a consumer group. Kafka assigns each partition to exactly one consumer in the group, so the group shares the work without reading any message twice.

When a consumer joins, leaves or crashes, Kafka rebalances the group and moves partitions between members. Each partition continues from its last committed offset. A different consumer group, for example a reporting application, gets its own copy of every partition and reads independently.

Kafka 4.0 also offers a newer consumer group protocol (group.protocol=consumer) in which the broker calculates assignments and rebalances happen more smoothly. The partition rules above stay the same.

11. Message Ordering and Partitions

Kafka guarantees order only within a single partition, not across a whole topic:

Partition 0:  A1  A2  A3      <- always read in this order
Partition 1:  B1  B2          <- always read in this order

A consumer reading both may see:  A1  B1  A2  B2  A3   (or another mix)

12. Partition Replication: Leaders and Followers

Each partition can be copied to several brokers. The number of copies is the replication factor:

• One copy is the leader. All writes go to it, and by default all reads too.
• The other copies are followers. They continuously copy the leader’s data.
• Followers that are fully caught up form the in-sync replicas (ISR).
• If the leader’s broker fails, an in-sync follower becomes the new leader, and clients switch to it automatically.

            Broker 1       Broker 2       Broker 3
P0          LEADER         follower       follower
P1          follower       LEADER         follower
P2          follower       follower       LEADER

Kafka spreads the leaders over the brokers so that each broker handles a fair share of the traffic. Use a replication factor of 3 for important topics, with min.insync.replicas=2 and producers using acks=all. See Kafka Fault Tolerance for details.

13. How Many Partitions Should a Topic Have?

There is no single correct number, but a simple method works well:

partitions >= target throughput / throughput one consumer can handle

Example:
  target            = 30,000 messages per second
  one consumer      =  5,000 messages per second
  partitions needed = 30,000 / 5,000 = 6  ->  choose 12 to leave room for growth

Many topics work well with 3 to 12 partitions. Choose a number that covers expected growth for the next year or two, because changing it later is not free.

14. Increasing the Number of Partitions

You can add partitions to an existing topic, but you can never reduce them:

bin/kafka-topics.sh --bootstrap-server localhost:9092 \
  --alter --topic order-events --partitions 12

• Existing messages stay in their current partitions; they are not moved.
• New messages with a given key may now go to a different partition, so ordering for that key can break across the change.
• Consumers detect the new partitions automatically and the group rebalances.

For keyed topics where order matters, the safer option is often to create a new topic with more partitions and move producers and consumers to it.

15. Hot Partitions and Uneven Load

A hot partition receives much more traffic than the others, usually because one key is very common, for example one huge customer or a default key like "unknown". Its consumer falls behind while others sit idle.

16. Inspecting Partitions with the Command Line

bin/kafka-topics.sh --bootstrap-server localhost:9092 \
  --describe --topic order-events

Topic: order-events  PartitionCount: 3  ReplicationFactor: 3
  Partition: 0  Leader: 1  Replicas: 1,2,3  Isr: 1,2,3
  Partition: 1  Leader: 2  Replicas: 2,3,1  Isr: 2,3,1
  Partition: 2  Leader: 3  Replicas: 3,1,2  Isr: 3,1

Use kafka-consumer-groups.sh --describe --group <name> to see which consumer owns each partition and how far behind it is.

17. Important Things to Remember

18. Summary

A Kafka partition is a fundamental building block that allows Kafka to handle large amounts of data efficiently. Partitions divide a topic into ordered, append-only logs spread across brokers. Producers choose a partition by key, so related messages stay together and in order; consumer groups share partitions so many consumers work in parallel; and replicas with a leader and followers keep each partition available when a broker fails. Choose the partition count carefully, because it limits consumer parallelism and can only be increased.

By understanding partitions, you gain a clear picture of how Kafka achieves high performance and reliability in real-time data streaming systems.

19. Frequently Asked Questions (FAQ)

producer.send(new ProducerRecord<>("order-events", 2, "101", "OrderCreated"));

consumer.assign(List.of(new TopicPartition("order-events", 0)));