Kafka Producer: A Beginner-Friendly Guide with Examples

A Kafka producer is the part of an application that publishes messages to Apache Kafka. This guide explains how a producer works inside, which settings matter, how to send messages safely and quickly, and how to handle errors, with formatted Java, Spring Boot and Python examples.

It builds on Kafka Topic and Kafka Partition, and pairs with Kafka Consumer, the other half of every Kafka pipeline.

1. What Is a Kafka Producer?

A Kafka producer is any application that writes data into Apache Kafka. If Kafka is a storage system for streams of events, the producer is the part that puts those events in.

Apache Kafka is a distributed event streaming platform. It lets one part of a system send a continuous flow of records, and lets other parts read that flow at their own pace. Companies use it for order tracking, log collection, payment events, sensor readings and much more.

Every Kafka setup has two sides:

• Producers send (publish) messages to Kafka.
• Consumers read (subscribe to) messages from Kafka.

This tutorial focuses on the first side: the producer.

A simple way to picture it

Think of a railway parcel office. Shopkeepers drop off parcels at the counter, each labelled with a destination route. The office sorts the parcels onto the right train, and people at the other end collect them whenever they arrive at the station.

In this picture, the shopkeeper is the producer, the parcel is a message, the destination route is a topic, the parcel office is the Kafka broker, and the person collecting parcels is the consumer. The shopkeeper does not need to know who will collect the parcel or when. They just hand it over and move on. That separation is exactly what makes Kafka producers so useful.

2. Key Terms You Need First

Five words come up again and again when working with producers. Learn these and the rest of the tutorial will feel easy.

One point is worth remembering early: the key decides the partition. Messages with the same key always land in the same partition, so they stay in order. Messages with no key are spread across partitions to balance the load.

3. How a Kafka Producer Works, Step by Step

When your code calls send(), the message does not fly straight to the broker. It passes through a small pipeline inside the producer:

 your code                         producer (inside your app)                       Kafka
 ---------                         --------------------------                       -----
 send(record) -> serializer -> partitioner -> record accumulator -> sender thread -> broker
                 (to bytes)    (which         (batches per         (network I/O)    (leader of
                                partition?)    partition)                            the partition)
                                                                        <- ack: partition + offset

1. Create the record. Your application builds a ProducerRecord holding the topic name, an optional key and the value.
2. Serialize. Kafka only stores bytes. A serializer turns the key and value (for example a String or JSON object) into a byte array.
3. Choose a partition. The partitioner picks which partition the message goes to:
   • If you set a partition number yourself, that one is used.
   • If the message has a key, Kafka hashes the key, so the same key always maps to the same partition.
   • If there is no key, Kafka spreads messages across partitions in batches (the “sticky” approach).
4. Add to a batch. The message waits in an in-memory buffer called the record accumulator, grouped with other messages headed to the same partition. Sending in groups is far faster than sending one by one.
5. Send to the broker. A background thread (the sender) takes ready batches and sends them over the network to the broker that leads that partition.
6. Get an acknowledgement. The broker writes the batch and replies. On success, the producer receives the topic, partition and offset. On failure, it retries if allowed, or reports an error.

Because steps 4 to 6 happen in the background, send() returns almost instantly. Your application keeps working while Kafka handles delivery.

4. Important Producer Settings

A producer is controlled by a set of properties. You only need three to get started, but knowing the others helps you balance speed against safety.

Understanding acks

The acks setting is the most important choice for data safety:

A good rule: use acks=all with idempotence turned on unless you have a clear reason not to. Tune linger.ms and batch.size later if you need more throughput.

5. Example: A Simple Kafka Producer in Java

Let’s build a producer for a small online bookstore. Each time a customer places an order, we send an event to a topic called book-orders.

Step 1: Add the dependency

If you use Maven, add the Kafka client library to pom.xml:

<dependency>
    <groupId>org.apache.kafka</groupId>
    <artifactId>kafka-clients</artifactId>
    <version>4.0.0</version>
</dependency>

Any recent version works the same way for this example. Kafka 4.x clients need Java 11 or later.

Step 2: Create the topic

With Kafka running locally, create the topic from the terminal:

kafka-topics.sh --create --topic book-orders \
  --bootstrap-server localhost:9092 \
  --partitions 3 --replication-factor 1

Step 3: Write the producer

package com.example.kafka;

import java.util.Properties;

import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerConfig;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.common.serialization.StringSerializer;

public class BookOrderProducer {

    public static void main(String[] args) {

        // 1. Configure the producer
        Properties props = new Properties();
        props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
        props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
        props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
        props.put(ProducerConfig.ACKS_CONFIG, "all");
        props.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, "true");

        // 2. Create the producer (try-with-resources closes it automatically)
        try (KafkaProducer<String, String> producer = new KafkaProducer<>(props)) {

            String[] customers = {"cust-101", "cust-202", "cust-101"};
            String[] books = {"Clean Code", "Atomic Habits", "The Alchemist"};

            for (int i = 0; i < customers.length; i++) {
                String key = customers[i];
                String value = "{\"customer\":\"" + key + "\",\"book\":\"" + books[i] + "\"}";

                // 3. Build the record
                ProducerRecord<String, String> record =
                        new ProducerRecord<>("book-orders", key, value);

                // 4. Send it
                producer.send(record);
                System.out.println("Sent order: " + value);
            }

            // 5. Push any waiting messages out before closing
            producer.flush();
        }
    }
}

What each part does

• Properties hold the settings. We point to the broker, tell Kafka to treat key and value as plain Strings, and ask for the safest delivery (acks=all with idempotence).
• KafkaProducer<String, String> is the producer object. The two types are the key type and the value type. It is thread-safe, so one instance can be shared across your whole application.
• ProducerRecord wraps one message: the topic book-orders, the customer ID as the key, and the order details as the value.
• send() hands the record to the producer’s internal buffer and returns at once.
• flush() forces every buffered message to be sent now. close() (called automatically by try-with-resources) does the same and then frees resources. Forgetting to close is a common reason messages never arrive.

Notice that cust-101 appears twice. Because both messages share the same key, they go to the same partition, and a consumer will read them in the order they were placed.

Step 4: Check the result

Open a console consumer in another terminal to see the messages arrive:

kafka-console-consumer.sh --topic book-orders \
  --bootstrap-server localhost:9092 \
  --from-beginning --property print.key=true

cust-101    {"customer":"cust-101","book":"Clean Code"}
cust-202    {"customer":"cust-202","book":"Atomic Habits"}
cust-101    {"customer":"cust-101","book":"The Alchemist"}

The order across different keys may vary, because they can sit in different partitions.

6. Three Ways to Send a Message

The example above uses “fire and forget”: it sends and never checks the result. In real projects you usually want to know whether a message made it. Kafka gives you three styles.

Synchronous send

RecordMetadata meta = producer.send(record).get();

System.out.println("Stored in partition " + meta.partition()
        + " at offset " + meta.offset());

The .get() call blocks the thread until the broker answers. If the write fails, it throws an exception you can catch. Simple, but it sends only one message at a time, which wastes the batching described earlier.

Asynchronous send with a callback

producer.send(record, (metadata, exception) -> {
    if (exception == null) {
        System.out.printf("Order saved: partition=%d, offset=%d%n",
                metadata.partition(), metadata.offset());
    } else {
        System.err.println("Failed to send order: " + exception.getMessage());
        // for example, write to a fallback log or alert the team
    }
});

Here send() returns right away and your loop continues. When the broker replies, Kafka runs the callback. If exception is null, the send worked and metadata holds the partition and offset.

Keep callback code short and quick. It runs on the producer’s own background thread, so slow work there delays every other message.

7. Sending Java Objects as JSON

Building JSON strings by hand is error-prone. A better approach is to send Java objects and let a serializer convert them. Here is a small, reusable JSON serializer based on the Jackson library:

package com.example.kafka;

import java.math.BigDecimal;

public record BookOrder(String orderId, String customer, String book, BigDecimal price) {
}

package com.example.kafka;

import org.apache.kafka.common.errors.SerializationException;
import org.apache.kafka.common.serialization.Serializer;

import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.databind.ObjectMapper;

public class JsonValueSerializer<T> implements Serializer<T> {

    private final ObjectMapper mapper = new ObjectMapper();

    @Override
    public byte[] serialize(String topic, T data) {
        if (data == null) {
            return null;
        }
        try {
            return mapper.writeValueAsBytes(data);
        } catch (JsonProcessingException e) {
            throw new SerializationException("Could not convert value to JSON", e);
        }
    }
}

Properties props = new Properties();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, JsonValueSerializer.class);

try (KafkaProducer<String, BookOrder> producer = new KafkaProducer<>(props)) {
    BookOrder order = new BookOrder("ord-1", "cust-101", "Clean Code", new BigDecimal("29.99"));
    producer.send(new ProducerRecord<>("book-orders", order.customer(), order));
}

For larger systems, consider a schema format such as Avro, Protobuf or JSON Schema with a schema registry. It checks that every message matches an agreed structure, so producers and consumers stay compatible as the data changes.

8. Delivery Guarantees: Retries and Idempotence

Networks fail and brokers restart. The producer handles most temporary problems by retrying automatically. These settings work together:

Without idempotence, a retry can create a duplicate: the broker saved the first attempt, but the reply was lost, so the producer sends it again. With idempotence, every producer gets an ID and every message a sequence number, so the broker writes each message once and keeps the order within the partition.

Without idempotence:   send #7 -> saved, reply lost -> retry #7 -> saved again  (duplicate)
With idempotence:      send #7 -> saved, reply lost -> retry #7 -> "already have #7" (no duplicate)

Idempotence protects against duplicates from the producer’s own retries. If your application itself calls send() twice for the same order, Kafka stores two messages, so consumers should still handle repeats safely.

9. Handling Producer Errors

producer.send(record, (metadata, exception) -> {
    if (exception == null) {
        return;                                            // success
    }
    if (exception instanceof RecordTooLargeException) {
        log.error("Order {} is too large and was not sent", record.key());
    } else if (exception instanceof TimeoutException) {
        log.error("Kafka not reachable; order {} saved for later", record.key());
        failedOrders.save(record.key(), record.value());   // retry later from storage
    } else {
        log.error("Could not send order {}", record.key(), exception);
    }
});

When the callback receives an error, the producer has already given up on that message. Save it somewhere safe, such as a database table or a log file, and alert someone if failures continue.

10. Transactional Producer

Sometimes several messages must be written all together or not at all, for example an order event and a stock-update event. A transactional producer makes this possible:

Properties props = new Properties();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
props.put(ProducerConfig.TRANSACTIONAL_ID_CONFIG, "order-service-tx-1");   // stable and unique per instance

KafkaProducer<String, String> producer = new KafkaProducer<>(props);
producer.initTransactions();

try {
    producer.beginTransaction();
    producer.send(new ProducerRecord<>("book-orders", "cust-101", "{\"book\":\"Clean Code\"}"));
    producer.send(new ProducerRecord<>("stock-updates", "clean-code", "{\"change\":-1}"));
    producer.commitTransaction();                      // both messages become visible together
} catch (ProducerFencedException e) {
    producer.close();                                  // another instance took over this transactional.id
} catch (KafkaException e) {
    producer.abortTransaction();                       // neither message becomes visible
}

Consumers must use isolation.level=read_committed to see only committed transactions. Transactions add some overhead, so use them only when atomic writes are really needed.

11. Adding Headers to Messages

Headers are small key-value pairs attached to a message, separate from its key and value. They are useful for metadata such as a trace ID, the message type or its source:

ProducerRecord<String, String> record =
        new ProducerRecord<>("book-orders", "cust-101", orderJson);

record.headers()
      .add("event-type", "OrderPlaced".getBytes(StandardCharsets.UTF_8))
      .add("source", "web-shop".getBytes(StandardCharsets.UTF_8));

producer.send(record);

Consumers can read headers without parsing the message body, for example to route or filter messages by type.

12. Kafka Producer in Spring Boot

Spring Boot creates and manages the producer for you. Add the spring-kafka dependency, configure it in application.yml, and use KafkaTemplate to send messages:

<dependency>
    <groupId>org.springframework.kafka</groupId>
    <artifactId>spring-kafka</artifactId>
</dependency>

spring:
  kafka:
    bootstrap-servers: localhost:9092
    producer:
      key-serializer: org.apache.kafka.common.serialization.StringSerializer
      value-serializer: org.apache.kafka.common.serialization.StringSerializer
      acks: all
      properties:
        linger.ms: 10
        compression.type: lz4

package com.example.kafka;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.stereotype.Service;

@Service
public class BookOrderPublisher {

    private static final Logger log = LoggerFactory.getLogger(BookOrderPublisher.class);

    private final KafkaTemplate<String, String> kafkaTemplate;

    public BookOrderPublisher(KafkaTemplate<String, String> kafkaTemplate) {
        this.kafkaTemplate = kafkaTemplate;
    }

    public void publish(String customerId, String orderJson) {
        kafkaTemplate.send("book-orders", customerId, orderJson)
                .whenComplete((result, ex) -> {
                    if (ex == null) {
                        log.info("Order sent to partition {} at offset {}",
                                result.getRecordMetadata().partition(),
                                result.getRecordMetadata().offset());
                    } else {
                        log.error("Failed to send order for {}", customerId, ex);
                    }
                });
    }
}

KafkaTemplate.send() returns a CompletableFuture, so the callback style works the same way as with the plain client. Spring shares one producer for the whole application.

13. Performance Tuning

For high volumes, tune how the producer batches and compresses messages:

acks=all
enable.idempotence=true
linger.ms=20
batch.size=65536
compression.type=lz4

Measure before and after each change. Producer metrics such as record-send-rate, request-latency-avg, batch-size-avg and record-error-rate are available through JMX or producer.metrics().

14. Example: The Same Producer in Python

Kafka producers are not limited to Java. Here is the bookstore producer written with the confluent-kafka library, which wraps Kafka’s fast C client. Install it first:

pip install confluent-kafka

Then write the producer:

import json
from confluent_kafka import Producer

# 1. Configure the producer
config = {
    "bootstrap.servers": "localhost:9092",
    "acks": "all",
    "enable.idempotence": True,
}
producer = Producer(config)

# 2. Callback that reports what happened to each message
def on_delivery(err, msg):
    if err is not None:
        print(f"Delivery failed: {err}")
    else:
        print(f"Delivered to {msg.topic()} [partition {msg.partition()}] at offset {msg.offset()}")

# 3. Send a few orders
orders = [
    {"customer": "cust-101", "book": "Clean Code"},
    {"customer": "cust-202", "book": "Atomic Habits"},
    {"customer": "cust-101", "book": "The Alchemist"},
]

for order in orders:
    producer.produce(
        topic="book-orders",
        key=order["customer"],
        value=json.dumps(order),
        callback=on_delivery,
    )
    producer.poll(0)   # lets Kafka run callbacks for finished sends

# 4. Wait until every message is delivered
producer.flush()

The ideas map one-to-one with Java: a config dictionary instead of Properties, produce() instead of send(), and a delivery callback for results. The one new piece is poll(0), which gives the library a chance to trigger callbacks while you keep sending. Strings are encoded to bytes for you, so no serializer setting is needed here.

15. Best Practices and Common Mistakes

Do this

• Reuse one producer. Creating a producer opens network connections and buffers. Make one per application and share it, rather than one per message.
• Choose keys with care. Use a key whenever order matters for an entity, such as a customer ID or account number. Avoid a key that most messages share, or one partition will do all the work.
• Keep the safety settings. acks=all plus enable.idempotence=true protects against lost and duplicated messages from retries.
• Handle errors in callbacks. Log failures, send them to a backup store, or raise an alert. Silent failures are the hardest bugs to find.
• Batch and compress for volume. A small linger.ms (like 10 ms) with compression.type=lz4 can multiply throughput with little added delay.
• Use a schema for structured data. Formats like Avro, JSON Schema or Protobuf with a schema registry keep producers and consumers in agreement as data changes.
• Create topics explicitly. Choose partitions and replication factor yourself instead of relying on automatic topic creation.

Avoid this

• Not closing the producer. Messages still in the buffer are lost when the program exits. Always call close() or flush().
• Calling .get() on every send. It turns a fast, batched producer into a slow one-at-a-time sender.
• Heavy work inside callbacks. Database calls or long loops in a callback slow every other send.
• Ignoring message size. Brokers reject messages above about 1 MB by default. Store large files elsewhere and send a reference instead.
• Setting acks=0 or acks=1 for important data. Speed gains are small compared with the risk of losing messages.

16. Quick Recap

A Kafka producer is the piece of your application that publishes messages to Kafka topics. Here is everything in a few lines:

• A producer sends records (key + value) to a topic, which is split into partitions stored on brokers.
• Inside the producer, each record is serialized, assigned a partition, grouped into a batch and sent by a background thread.
• The key keeps related messages together and in order.
• acks controls safety: 0 is fastest, all is safest and the default.
• Retries and idempotence handle temporary failures without duplicates; transactions write several messages atomically.
• You can send in three styles: fire and forget, synchronous, or asynchronous with a callback. The callback style is the usual choice.
• In Spring Boot, KafkaTemplate wraps the producer for you.
• Reuse one producer, handle errors, and always flush or close before exiting.

Once you are comfortable with producers, the natural next step is learning Kafka consumers, the other half of every Kafka pipeline.

17. Frequently Asked Questions (FAQ)

kafkaTemplate.send("book-orders", customerId, orderJson)
        .whenComplete((result, ex) -> {
            if (ex != null) {
                log.error("Send failed", ex);
            }
        });