Kafka MirrorMaker

Kafka MirrorMaker is a tool provided by Apache Kafka that copies (replicates) data from one Kafka cluster to another. In simple words, it keeps topics in two different Kafka clusters in sync.

This guide explains why cross-cluster replication is needed, how MirrorMaker 2 works internally, how to configure and run it step by step, how topic names and consumer offsets are handled during failover, and how to monitor it, with formatted configuration, command-line and Java examples. It builds on Kafka Cluster and Kafka Connect.

1. What Is Kafka MirrorMaker?

Think of MirrorMaker as a bridge between two Kafka clusters. Messages produced to selected topics in the source cluster are automatically copied to the target cluster, usually within seconds.

   Cluster A (source)                          Cluster B (target)
 +--------------------+                      +--------------------+
 |  orders            |    +------------+    |  A.orders          |
 |  payments          | -> | MirrorMaker| -> |  A.payments        |
 |  consumer offsets  |    |     2      |    |  translated offsets|
 +--------------------+    +------------+    +--------------------+

MirrorMaker is commonly used for:

• Disaster recovery.
• Multi-data-center and multi-region setups.
• Migrating data between Kafka clusters.
• Sharing data across teams or regions.

2. Why Do We Need Kafka MirrorMaker?

Kafka already replicates each partition across brokers inside one cluster, which protects against a broker or disk failure. It does not protect against losing a whole cluster or data center, and a single cluster stretched across distant regions would be slow.

In real-world systems, companies often run Kafka clusters in several locations, for example one in India and another in the US. If one cluster goes down, the data should still be available in the other. MirrorMaker continuously copies messages between clusters, so both have the same data.

3. Key Features of Kafka MirrorMaker

4. MirrorMaker 1 vs MirrorMaker 2

Kafka has had two generations of the tool:

For all new projects, use MirrorMaker 2 (often written MM2). The rest of this tutorial covers MirrorMaker 2.

5. How MirrorMaker 2 Works

MirrorMaker 2 is a set of three Kafka Connect connectors that work together:

MirrorMaker 2 also creates a few internal topics:

6. Simple Example of Kafka MirrorMaker

Let’s understand this with a simple scenario:

• Source cluster: Kafka cluster A (India).
• Target cluster: Kafka cluster B (US).
• Topic: order-events.

An application produces order messages to order-events in cluster A. MirrorMaker 2 is configured to:

1. Consume messages from order-events in cluster A.
2. Produce the same messages to A.order-events in cluster B. The A. prefix shows where the data came from.
3. Translate the offsets of consumer groups reading in cluster A, so they can continue in cluster B.

Whenever a new order message is published in cluster A, it appears in cluster B shortly afterwards. If cluster A goes down, applications can read the order data from cluster B and continue close to where they stopped.

7. Running MirrorMaker 2 Step by Step

The simplest way to run MirrorMaker 2 is dedicated mode, using a script and one properties file that ship with Kafka.

Step 1: Create the configuration file

# Names (aliases) for the two clusters
clusters = A, B
A.bootstrap.servers = kafka-a1:9092,kafka-a2:9092
B.bootstrap.servers = kafka-b1:9092,kafka-b2:9092

# Enable replication from A to B
A->B.enabled = true

# Topics and consumer groups to replicate (regular expressions)
A->B.topics = order-events, payments.*
A->B.groups = .*

# Translate consumer offsets into cluster B every 60 seconds
A->B.sync.group.offsets.enabled = true

# Replication factor for the copied topics in cluster B
replication.factor = 3

# Replication factor for MirrorMaker 2 internal topics
checkpoints.topic.replication.factor = 3
heartbeats.topic.replication.factor = 3
offset-syncs.topic.replication.factor = 3
offset.storage.replication.factor = 3
status.storage.replication.factor = 3
config.storage.replication.factor = 3

# Parallel tasks for copying data
tasks.max = 4

Step 2: Start MirrorMaker 2

bin/connect-mirror-maker.sh config/mm2.properties

Run the same command on two or more machines for high availability. The instances form a group and share the work automatically.

Step 3: Check the replicated topic

# List topics in the target cluster
bin/kafka-topics.sh --bootstrap-server kafka-b1:9092 --list

# Read the mirrored messages
bin/kafka-console-consumer.sh --bootstrap-server kafka-b1:9092 \
  --topic A.order-events --from-beginning

Place MirrorMaker 2 close to the target cluster. Reading over a long network link and writing locally is more reliable than the other way round.

8. Remote Topic Names and Replication Policies

By default, MirrorMaker 2 uses the DefaultReplicationPolicy, which names each copied topic <source alias>.<topic>. These copies are called remote topics.

The prefix has two benefits: you can always see where data came from, and two-way replication cannot loop, because MirrorMaker 2 never copies A.order-events back to cluster A.

If you need identical names, for example during a one-way migration, use the IdentityReplicationPolicy:

replication.policy.class = org.apache.kafka.connect.mirror.IdentityReplicationPolicy

With identical names, MirrorMaker 2 cannot tell local and copied topics apart, so use this policy only for one-way replication.

9. Consumer Offset Translation and Failover

Offsets in cluster B are different from offsets in cluster A: the same message may be at offset 5,000 in A and offset 1,200 in B. A consumer cannot simply reuse its old offset after failing over.

MirrorMaker 2 solves this with checkpoints. It regularly records how source offsets map to target offsets and writes each group’s translated position to the target cluster. With sync.group.offsets.enabled=true, it also commits these positions directly to the target’s consumer offsets, so a consumer that switches to cluster B with the same group.id starts close to where it stopped.

1. Cluster A fails.
2. Stop consumers that were reading from cluster A.
3. Point consumers to cluster B (topic A.order-events, same group.id).
4. Consumers resume from the translated offsets.
5. A few recently processed messages may be read again.

Translation is not exact to the last message, and the newest messages may not have been copied when cluster A failed. Design consumers to be idempotent, so a few repeated messages cause no harm.

10. Working with Mirrored Topics in Java

A consumer in cluster B can read both the local topic and the remote copy with a regular-expression subscription:

package com.example.kafka;

import java.time.Duration;
import java.util.Properties;
import java.util.regex.Pattern;

import org.apache.kafka.clients.consumer.ConsumerConfig;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.clients.consumer.KafkaConsumer;
import org.apache.kafka.common.serialization.StringDeserializer;

public class MirroredOrdersConsumer {

    public static void main(String[] args) {
        Properties props = new Properties();
        props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "kafka-b1:9092");
        props.put(ConsumerConfig.GROUP_ID_CONFIG, "order-reporting");
        props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
        props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);

        try (KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props)) {
            // matches "order-events" and remote copies such as "A.order-events"
            consumer.subscribe(Pattern.compile("(.*\\.)?order-events"));

            while (true) {
                for (ConsumerRecord<String, String> record : consumer.poll(Duration.ofMillis(500))) {
                    System.out.printf("topic=%s offset=%d value=%s%n",
                            record.topic(), record.offset(), record.value());
                }
            }
        }
    }
}

To look up translated offsets yourself, for example in a failover script, use RemoteClusterUtils from the connect-mirror-client library:

<dependency>
    <groupId>org.apache.kafka</groupId>
    <artifactId>connect-mirror-client</artifactId>
    <version>4.0.0</version>
</dependency>

package com.example.kafka;

import java.time.Duration;
import java.util.Map;

import org.apache.kafka.clients.consumer.OffsetAndMetadata;
import org.apache.kafka.common.TopicPartition;
import org.apache.kafka.connect.mirror.RemoteClusterUtils;

public class TranslateOffsets {

    public static void main(String[] args) throws Exception {
        // connection settings for the TARGET cluster (B)
        Map<String, Object> targetProps = Map.of("bootstrap.servers", "kafka-b1:9092");

        Map<TopicPartition, OffsetAndMetadata> offsets =
                RemoteClusterUtils.translateOffsets(
                        targetProps,
                        "A",                    // alias of the source cluster
                        "order-service",        // consumer group
                        Duration.ofSeconds(30));

        offsets.forEach((partition, offset) ->
                System.out.println(partition + " -> " + offset.offset()));
    }
}

The returned map can be passed to consumer.commitSync(offsets), or used with seek(), before the consumer starts reading from cluster B.

11. Replication Topologies: Active-Passive and Active-Active

clusters = A, B
A.bootstrap.servers = kafka-a1:9092
B.bootstrap.servers = kafka-b1:9092

A->B.enabled = true
A->B.topics = order-events
B->A.enabled = true
B->A.topics = order-events

Here cluster B ends up with order-events (local writes) and A.order-events (copied from A), and cluster A has the mirror image. Consumers that need all orders subscribe to both topics, as in the Java pattern above.

12. Running MirrorMaker 2 on a Kafka Connect Cluster

If you already run a Kafka Connect cluster, you can deploy the MirrorMaker 2 connectors there through the Connect REST API instead of using the dedicated script:

curl -X POST http://connect:8083/connectors \
  -H "Content-Type: application/json" \
  -d '{
    "name": "mm2-source-A-to-B",
    "config": {
      "connector.class": "org.apache.kafka.connect.mirror.MirrorSourceConnector",
      "source.cluster.alias": "A",
      "target.cluster.alias": "B",
      "source.cluster.bootstrap.servers": "kafka-a1:9092",
      "target.cluster.bootstrap.servers": "kafka-b1:9092",
      "topics": "order-events",
      "tasks.max": "4"
    }
  }'

Create MirrorCheckpointConnector and MirrorHeartbeatConnector the same way. This mode gives you Connect’s REST API and existing monitoring, but needs more configuration; dedicated mode is simpler to start with.

13. Monitoring MirrorMaker 2

These metrics are exposed through JMX, so they can be collected with tools such as Prometheus and shown in dashboards. Alert when replication latency keeps rising or when heartbeats stop arriving.

14. Real-World Use Cases

15. Limitations to Know

• Asynchronous copy: if the source fails, the newest messages may not have reached the target yet.
• At-least-once by default: after a restart, some records may be copied twice. Newer Kafka versions can be configured for exactly-once copying, but it needs extra setup.
• Different offsets: offsets in the target do not match the source; consumers rely on offset translation.
• Network cost: all replicated data crosses the network between sites, which can be slow or expensive.
• Not a backup of everything: it does not copy broker settings, users or data from topics you did not select.

16. Best Practices

17. Summary

Kafka MirrorMaker is a simple and powerful tool for replicating Kafka topics from one cluster to another. MirrorMaker 2, built on Kafka Connect, is the version to use today: it copies records, topic settings and ACLs, renames copied topics to show their source and prevent loops, translates consumer offsets for failover, and reports its health through heartbeats and metrics. Configure it with a single properties file, run several instances near the target cluster, and plan for asynchronous replication by making consumers idempotent.

By copying messages in near real time, MirrorMaker helps build reliable, scalable and fault-tolerant systems, and it is an essential component for anyone running Kafka across several data centers or regions.

18. Frequently Asked Questions (FAQ)

A->B.topics = order-events, payments.*
A->B.topics.exclude = .*internal.*

A.security.protocol = SASL_SSL
A.sasl.mechanism = SCRAM-SHA-512
A.sasl.jaas.config = org.apache.kafka.common.security.scram.ScramLoginModule required \
    username="mm2" password="${file:/etc/mm2/secrets.properties:a.password}";

consumer.subscribe(Pattern.compile("(.*\\.)?order-events"));