Kafka Features

Apache Kafka is a powerful platform used to handle large amounts of data in real time. It is widely used in modern applications to send, receive and process data quickly and reliably.

This guide explains Kafka’s main features in simple language: what each feature means, how Kafka achieves it, and where it helps in real projects, with formatted configuration, command-line and Java examples. If you are new to Kafka, start with Kafka Introduction.

1. What Makes Kafka Different?

Most of Kafka’s features come from one simple idea: Kafka stores messages in an append-only log. New messages are always added at the end, and stored messages are never changed. Readers keep track of their own position in the log.

 producers  --->  [ 0 ][ 1 ][ 2 ][ 3 ][ 4 ][ 5 ][ 6 ]  --->  new messages appended
                    ^              ^              ^
                    |              |              |
            analytics app     billing app    alerting app
          (reading at 0)    (reading at 3)  (reading at 6)

Because messages stay in the log after they are read, many applications can read the same data at their own speed, and data can be read again later. Splitting the log into partitions spread across servers gives Kafka its speed and scale.

2. High Throughput

Kafka can handle millions of messages per second across a cluster. This makes it ideal for applications that generate a lot of data, such as log processing, event tracking and real-time analytics.

Kafka is fast because of a few design choices:

3. Real-Time Data Streaming

Kafka allows data to be processed as soon as it is produced, usually within milliseconds. Instead of waiting for data to be collected and processed in nightly batches, applications can react to events immediately.

This is very useful for live dashboards, monitoring systems, fraud checks, recommendations and alerting applications.

4. Scalability

Kafka is designed to grow easily:

• More brokers: add servers to the cluster and move partitions onto them to handle more data.
• More partitions: split busy topics into more partitions so more consumers can work in parallel.
• More consumers: start more instances of an application; Kafka shares the partitions among them automatically.

This means Kafka can support small systems as well as very large enterprise applications without changing application code. See Kafka Partition for how partitions make this possible.

5. Fault Tolerance

Kafka keeps multiple copies of each partition on different servers. One copy is the leader; the others are followers that continuously copy it. If the leader’s server fails, a follower that is fully up to date becomes the new leader automatically, and clients switch to it.

bin/kafka-topics.sh --bootstrap-server localhost:9092 \
  --create --topic payments \
  --partitions 6 --replication-factor 3 \
  --config min.insync.replicas=2

With three copies and min.insync.replicas=2, the topic keeps working and loses no acknowledged data when one broker fails. See Kafka Fault Tolerance for details.

6. Data Durability

All messages in Kafka are stored on disk, not only kept in memory. Even if a broker restarts, the data remains and can be read again. Combined with replication, this makes Kafka reliable for critical business data.

Producers choose how safe each write must be with the acks setting:

Messages are kept until the topic’s retention limit, not forever by default; see section 12.

7. Message Ordering

Kafka keeps messages in order within a partition. This is important for use cases like account transactions or order status changes, where the sequence of events must be preserved.

To keep related messages in order, give them the same key, such as an order ID. Kafka sends all messages with the same key to the same partition. Order is not guaranteed across different partitions.

8. Multiple Consumers Support

Kafka allows multiple applications to read the same data independently. Each application uses its own consumer group and keeps its own position, so one application can process data for analytics while another uses the same data for monitoring or notifications.

Within one group, the topic’s partitions are shared among the group’s consumers, so each message is processed once per group.

9. High Availability

Kafka stays available by spreading data and leadership across several servers. Even during maintenance or hardware failures, applications can usually continue to read and write data:

• Brokers can be restarted one at a time for upgrades (a rolling restart) while the cluster keeps serving clients.
• Partition leaders move automatically when a broker goes down.
• Clients retry automatically and reconnect to the new leaders.

Availability still depends on good setup: enough brokers, replication factor 3 for important topics, and brokers placed in different racks or availability zones.

10. Decoupling of Systems

Kafka acts as a middle layer between data producers and consumers. Producers do not need to know who reads the data, and consumers do not need to know who wrote it.

Without Kafka (point-to-point):          With Kafka:

  Orders --> Billing                       Orders --+
  Orders --> Shipping                               |
  Orders --> Analytics                              v
  Orders --> Email                          [ orders topic ]
                                              |   |   |   |
  4 direct links to maintain              Billing Shipping Analytics Email

As a result, systems become more flexible and easier to upgrade. A new service can start reading existing data without any change to the producer, and a slow consumer does not slow down the producer.

11. Support for Event-Driven Architecture

Kafka is a natural fit for event-driven systems, where applications communicate by publishing events such as “OrderPlaced” or “PaymentReceived”. Other applications react to these events in real time, making the overall system faster and more responsive. See Kafka Use Cases for real examples.

12. Message Retention and Replay

Unlike many traditional message queues, Kafka does not delete messages when they are read. It keeps them for a configured time or size, and consumers can go back and read them again:

# Change retention of an existing topic to 30 days
bin/kafka-configs.sh --bootstrap-server localhost:9092 \
  --alter --entity-type topics --entity-name orders \
  --add-config retention.ms=2592000000

# Replay all retained messages for a consumer group (group must be stopped)
bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
  --group analytics --topic orders \
  --reset-offsets --to-earliest --execute

Replay is useful for fixing bugs (process the data again with corrected code), loading a new database, or testing a new application with real history. See Kafka Offset for more ways to move a consumer’s position.

13. Log Compaction

For some data, only the latest value per key matters, for example a customer’s current address. A compacted topic keeps at least the latest message for each key and removes older ones:

Before:  (cust-1, Delhi) (cust-2, Pune) (cust-1, Mumbai) (cust-2, Goa) (cust-1, Chennai)
After:                                                    (cust-2, Goa) (cust-1, Chennai)

bin/kafka-topics.sh --bootstrap-server localhost:9092 \
  --create --topic customer-addresses \
  --partitions 3 --replication-factor 3 \
  --config cleanup.policy=compact

A new application can read the compacted topic from the beginning to get the current value for every key, without reading years of history.

14. Idempotence and Exactly-Once Processing

• Idempotent producer (on by default since Kafka 3.0): if a producer retries after a network problem, the broker drops the duplicate, so each message is written once.
• Transactions: a producer can write to several topics and partitions all or nothing, and can include the consumer’s offsets in the same transaction.
• Exactly-once processing: combining the two, a read-process-write application can make each input message affect the output exactly once, for data flowing from Kafka to Kafka.

When results go to an external system such as a database, the application should also make its writes idempotent, for example by using the message key as a unique ID.

15. Integration with Kafka Connect

Kafka Connect is a framework, included with Kafka, for moving data between Kafka and other systems without writing code. You configure ready-made connectors instead:

Connect handles scaling, failures and progress tracking. See Kafka Connect to learn more.

16. Built-In Stream Processing

Kafka includes the Kafka Streams Java library for filtering, transforming, joining and aggregating data as it flows between topics. It runs inside your own application, so no separate processing cluster is needed. See Kafka Stream for an introduction.

17. Built-In Security

Security is off by default and must be configured for production. See Kafka Security for a step-by-step guide.

18. KRaft: No ZooKeeper Needed

Older Kafka versions needed a separate Apache ZooKeeper cluster to store metadata and elect a controller. Modern Kafka uses KRaft, a built-in consensus protocol, instead. KRaft has been production-ready since Kafka 3.3, and Kafka 4.0 removed ZooKeeper support completely.

• Only one system to install, secure and monitor.
• Faster startup and failover of the controller.
• Support for many more partitions per cluster.

See Kafka KRaft for details.

19. Tiered Storage

With tiered storage, brokers keep recent data on local disks and move older segments to cheaper remote storage, such as object storage. Consumers read old data transparently. This makes long retention much cheaper and lets brokers recover faster, because they hold less local data. Tiered storage became production-ready in Kafka 3.9 and needs a remote storage plugin.

20. Java Example: Kafka Features in Action

The example below shows several features together: a durable, idempotent producer with keys for ordering, and two consumers in different groups that each receive every message and start by replaying from the beginning.

package com.example.kafka;

import java.util.Properties;

import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerConfig;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.common.serialization.StringSerializer;

public class FeatureDemoProducer {

    public static void main(String[] args) {
        Properties props = new Properties();
        props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
        props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
        props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
        props.put(ProducerConfig.ACKS_CONFIG, "all");                  // durability
        props.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, true);     // no duplicates on retry
        props.put(ProducerConfig.COMPRESSION_TYPE_CONFIG, "lz4");      // throughput
        props.put(ProducerConfig.LINGER_MS_CONFIG, 10);                // batching

        try (KafkaProducer<String, String> producer = new KafkaProducer<>(props)) {
            String orderId = "ord-501";                                // same key = same partition = ordered
            producer.send(new ProducerRecord<>("orders", orderId, "OrderPlaced"));
            producer.send(new ProducerRecord<>("orders", orderId, "PaymentReceived"));
            producer.send(new ProducerRecord<>("orders", orderId, "OrderShipped"));
        }
    }
}

package com.example.kafka;

import java.time.Duration;
import java.util.List;
import java.util.Properties;

import org.apache.kafka.clients.consumer.ConsumerConfig;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.clients.consumer.KafkaConsumer;
import org.apache.kafka.common.serialization.StringDeserializer;

public class FeatureDemoConsumer {

    public static void main(String[] args) {
        String groupId = args.length > 0 ? args[0] : "billing";        // run again with "analytics"

        Properties props = new Properties();
        props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
        props.put(ConsumerConfig.GROUP_ID_CONFIG, groupId);            // independent position per group
        props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest"); // new group replays from the start
        props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
        props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);

        try (KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props)) {
            consumer.subscribe(List.of("orders"));

            while (true) {
                for (ConsumerRecord<String, String> record : consumer.poll(Duration.ofMillis(500))) {
                    System.out.printf("[%s] key=%s value=%s partition=%d offset=%d%n",
                            groupId, record.key(), record.value(),
                            record.partition(), record.offset());
                }
            }
        }
    }
}

[billing]   key=ord-501 value=OrderPlaced     partition=2 offset=0
[billing]   key=ord-501 value=PaymentReceived partition=2 offset=1
[billing]   key=ord-501 value=OrderShipped    partition=2 offset=2

[analytics] key=ord-501 value=OrderPlaced     partition=2 offset=0
[analytics] key=ord-501 value=PaymentReceived partition=2 offset=1
[analytics] key=ord-501 value=OrderShipped    partition=2 offset=2

Both groups receive all three events, in the same order, because they share one key and therefore one partition. The partition number in your output may differ.

21. Kafka Features at a Glance

22. When Kafka May Not Be the Best Fit

23. Conclusion

Apache Kafka is a reliable, scalable and high-performance platform for real-time data streaming. Its log-based design gives it high throughput, ordering per partition, durable storage with replication, independent consumer groups, and the ability to replay history. Features such as log compaction, idempotence and transactions, Kafka Connect, Kafka Streams, built-in security, KRaft and tiered storage make it a complete platform for event-driven systems.

Because of this combination of simplicity and power, Kafka has become a core component of many enterprise systems that need fast data processing, high availability and strong fault tolerance.

24. Frequently Asked Questions (FAQ)

spring:
  kafka:
    producer:
      acks: all
    consumer:
      group-id: billing