What Is Dark Launching? Test New Code on Real Traffic, Invisibly

Dark launching is the practice of deploying a new feature or a new version of code to production and running it on real traffic, while keeping it hidden from users. The users keep getting results from the existing code, and the team quietly measures how the new code behaves: its correctness, speed and load on the system.

It answers a question that no staging environment can: "What happens when this code meets real production data and real traffic?" And it answers it before a single customer depends on the new code.

1. Why Dark Launching Exists

Imagine an e-commerce company replacing its ten-year-old tax calculation with a new tax engine. A tiny difference in rounding could mean wrong invoices for thousands of customers and trouble with the tax authorities. Unit tests cover the cases the developers thought of, but production contains millions of real carts with combinations nobody imagined.

A canary release would still show the new engine's answers to some real customers. A dark launch goes one step further: the new engine calculates tax for every real cart, but customers only ever see the old engine's answer. The team compares the two and fixes differences before switching.

The term became widely known after Facebook used it in 2008 to launch its chat feature. Before chat was visible, users' browsers quietly sent chat requests to the new backend, so the team could see how it handled real load.

2. Two Forms of Dark Launching

2.1 Hidden Feature (Feature Flag Off)

The new code is deployed but switched off for users by a feature flag, a setting checked at runtime that decides whether a code path is active. It may be switched on only for employees or test accounts. Deployment and release are now separate: code can be in production for weeks before anyone outside the team sees it.

2.2 Shadow Execution

The new code actually runs on real requests, alongside the old code, but its output is thrown away or only compared. This is also called shadow mode or shadow traffic. It can happen:

• In code: the service calls both implementations and compares results, as in Example 1.
• At the infrastructure level: a gateway or service mesh mirrors (copies) each request to a second deployment and ignores its responses. Istio's mirror setting and Nginx's mirror directive work this way.

3. How Shadow Execution Works

Four rules make shadow execution safe:

1. The user's response never depends on the new code. Its result is only compared.
2. The shadow runs asynchronously, so it can't slow down the user (Example 2).
3. The shadow has no visible side effects. It must not send emails, charge cards or write to the real tables (Example 3).
4. The shadow can be switched off instantly with a flag, without a deployment.

4. Code Examples

These JDK-only Java 21 programs show the comparison, the latency trap and the side-effect trap.

4.1 Example 1: Comparing Results in Shadow Mode

import java.math.BigDecimal;
import java.math.RoundingMode;
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;

public class ShadowCompareDemo {

    static final BigDecimal GST_RATE = new BigDecimal("0.18");

    // Current code: the result users actually see
    static BigDecimal legacyTax(BigDecimal amount) {
        return amount.multiply(GST_RATE).setScale(2, RoundingMode.HALF_UP);
    }

    // New tax engine, dark launched: it runs on real carts, but its answer is never shown
    static BigDecimal newTax(BigDecimal amount) {
        return amount.multiply(GST_RATE).setScale(2, RoundingMode.HALF_EVEN); // subtle change
    }

    record Comparison(String amount, BigDecimal legacy, BigDecimal candidate) {
        boolean matches() { return legacy.compareTo(candidate) == 0; }
    }

    public static void main(String[] args) throws Exception {
        List<String> carts = List.of("499.00", "12.25", "1299.99", "87.75", "250.00",
                "64.40", "33.75", "999.00", "18.25", "1500.50");
        List<Future<Comparison>> shadows = new ArrayList<>();

        try (ExecutorService shadowPool = Executors.newVirtualThreadPerTaskExecutor()) {
            for (String cart : carts) {
                var amount = new BigDecimal(cart);
                BigDecimal shown = legacyTax(amount); // the user gets this immediately
                shadows.add(shadowPool.submit(() -> new Comparison(cart, shown, newTax(amount))));
            }
        }

        int mismatches = 0;
        for (Future<Comparison> f : shadows) {
            Comparison c = f.get();
            if (!c.matches()) {
                mismatches++;
                System.out.printf("MISMATCH cart %8s: legacy %s, new engine %s%n", c.amount(), c.legacy(), c.candidate());
            }
        }
        System.out.printf("%d of %d carts matched; users saw only legacy results%n",
                carts.size() - mismatches, carts.size());
    }
}

MISMATCH cart    12.25: legacy 2.21, new engine 2.20
MISMATCH cart    18.25: legacy 3.29, new engine 3.28
8 of 10 carts matched; users saw only legacy results

The new engine used HALF_EVEN rounding (also called banker's rounding) instead of HALF_UP. The two only differ when the exact half lands after an even digit, like 2.205 and 3.285, so a handful of hand-written tests could easily miss it. Running on real carts found it immediately, and no customer ever saw a wrong total.

4.2 Example 2: Keeping the Shadow Off the User's Clock

import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.Semaphore;

public class ShadowLatencyDemo {

    static void sleep(long ms) {
        try {
            Thread.sleep(ms);
        } catch (InterruptedException e) {
            Thread.currentThread().interrupt();
        }
    }

    static void legacySearch() { sleep(20); }       // current code: fast
    static void newSearchEngine() { sleep(300); }   // dark-launched code: still slow

    static final Semaphore shadowSlots = new Semaphore(2); // at most 2 shadow calls at once

    static void handle(int request, boolean async, ExecutorService shadowPool, int[] skipped) {
        long start = System.nanoTime();
        legacySearch();
        if (!async) {
            newSearchEngine(); // user waits for code whose result is thrown away
        } else if (shadowSlots.tryAcquire()) {
            shadowPool.submit(() -> {
                try {
                    newSearchEngine();
                } finally {
                    shadowSlots.release();
                }
            });
        } else {
            skipped[0]++; // shadow is optional: drop it rather than queue it
        }
        long ms = (System.nanoTime() - start) / 1_000_000;
        System.out.printf("  request %d: user waited %s%n", request, ms < 150 ? "~20 ms" : "~320 ms");
    }

    public static void main(String[] args) {
        for (boolean async : new boolean[] {false, true}) {
            System.out.println(async ? "Shadow call in background:" : "Shadow call inside the request:");
            int[] skipped = {0};
            try (ExecutorService shadowPool = Executors.newVirtualThreadPerTaskExecutor()) {
                for (int r = 1; r <= (async ? 5 : 2); r++) {
                    handle(r, async, shadowPool, skipped);
                }
            }
            if (async) {
                System.out.println("  shadow calls skipped because 2 were already running: " + skipped[0]);
            }
        }
    }
}

Shadow call inside the request:
  request 1: user waited ~320 ms
  request 2: user waited ~320 ms
Shadow call in background:
  request 1: user waited ~20 ms
  request 2: user waited ~20 ms
  request 3: user waited ~20 ms
  request 4: user waited ~20 ms
  request 5: user waited ~20 ms
  shadow calls skipped because 2 were already running: 3

Calling the new engine inside the request made every user wait for code whose answer was thrown away. Running it in the background kept the user's wait at the normal level.

The second lesson is less obvious: shadow work must be limited. Here a semaphore allows only two shadow calls at a time, and extra ones are dropped instead of queued. This is the bulkhead pattern applied to dark launching. Otherwise, a slow new engine builds up a growing queue of background work that eats memory and CPU from the real requests.

4.3 Example 3: Side Effects Are Not Invisible

import java.util.ArrayList;
import java.util.List;

public class SideEffectDemo {

    interface EmailSender { void send(String to, String body); }

    // Real sender: the customer receives every email that goes through here
    static class SmtpSender implements EmailSender {
        final List<String> inbox = new ArrayList<>();
        public void send(String to, String body) { inbox.add(to + " <- " + body); }
    }

    // Dry-run sender: records what would have been sent, sends nothing
    static class DryRunSender implements EmailSender {
        final List<String> wouldSend = new ArrayList<>();
        public void send(String to, String body) { wouldSend.add(to + " <- " + body); }
    }

    record OrderPlaced(String orderId, String email) { }

    static void legacyNotifier(OrderPlaced e, EmailSender sender) {
        sender.send(e.email(), "Order " + e.orderId() + " confirmed");
    }

    static void newNotifierV2(OrderPlaced e, EmailSender sender) { // dark-launched rewrite
        sender.send(e.email(), "Your order " + e.orderId() + " is confirmed!");
    }

    public static void main(String[] args) {
        var event = new OrderPlaced("ORD-3301", "riya@example.com");

        System.out.println("Dark launch wired to the real sender:");
        var smtp = new SmtpSender();
        legacyNotifier(event, smtp);
        newNotifierV2(event, smtp); // the "invisible" code is not invisible
        smtp.inbox.forEach(m -> System.out.println("  INBOX " + m));

        System.out.println("Dark launch wired to a dry-run sender:");
        var realSmtp = new SmtpSender();
        var dryRun = new DryRunSender();
        legacyNotifier(event, realSmtp);
        newNotifierV2(event, dryRun);
        realSmtp.inbox.forEach(m -> System.out.println("  INBOX " + m));
        dryRun.wouldSend.forEach(m -> System.out.println("  RECORDED for comparison " + m));
    }
}

Dark launch wired to the real sender:
  INBOX riya@example.com <- Order ORD-3301 confirmed
  INBOX riya@example.com <- Your order ORD-3301 is confirmed!
Dark launch wired to a dry-run sender:
  INBOX riya@example.com <- Order ORD-3301 confirmed
  RECORDED for comparison riya@example.com <- Your order ORD-3301 is confirmed!

The first run shows how a dark launch can stop being dark: the new notifier used the real email sender, and the customer got two confirmation emails. In the second run, the new code received a dry-run sender that only records what it would have sent, so the team can compare messages safely. Apply the same idea to payments, database writes and outgoing API calls.

5. Comparison Table: Dark Launch vs Other Release Techniques

These techniques work well in sequence: dark launch to prove correctness, canary to roll out safely, and an A/B test if the business impact needs measuring.

6. Common Mistakes and Pitfalls

1. Side effects leaking out. Any write, message or external call from the shadow path reaches the real world. Route them to dry-run implementations, a separate shadow database, or sandbox endpoints.

2. Doubling load on shared dependencies. Mirroring every request to a new service also doubles its calls to databases, caches and third-party APIs, which may hit rate limits or raise costs. Mirror a sample, or stub the downstream calls.

3. Comparing things that are always different. Timestamps, generated IDs and list order can differ even when both paths are correct. Normalise them before comparing, or the mismatch report fills with noise and real bugs get missed.

4. Infrastructure mirrors are fire-and-forget. With Istio mirroring, responses from the shadow are discarded, so nothing compares them for you. Istio also appends -shadow to the Host header of mirrored requests, which can break host-based routing or logging in the shadow service. Plan how you will collect and compare its results.

5. Forgetting to clean up. Shadow code and flags left in place after launch become confusing dead code. Remove them as soon as the new path is fully live.

7. Best Practices

• Decide what "correct" means before you start: which fields must match exactly and which are allowed to differ.
• Run the shadow asynchronously, with limits and timeouts, and drop shadow work under pressure.
• Make side effects impossible, not just unlikely: inject dry-run versions of senders, writers and clients.
• Start with a sample of traffic, then increase it while watching resource usage.
• Track the mismatch rate as a metric and group mismatches by cause.
• Keep a kill switch for the shadow path that works without a deployment.
• Protect personal data: mismatch logs and mirrored traffic contain real user data and need the same care as production data.

8. Summary

• Dark launching runs new code in production, on real traffic, without users seeing its results.
• It comes in two forms: hidden features behind flags, and shadow execution that runs new code alongside old code.
• Shadow results are compared with the live results to find bugs that tests missed, like the rounding difference in Example 1.
• The shadow must be asynchronous, resource-limited and free of side effects.
• Mirroring at the gateway or mesh level discards responses, so you still need a way to compare.
• Dark launching pairs well with canary releases and A/B tests in a full release process.

Used carefully, dark launching lets you meet production reality before your users do.

Dark Launching: Interview Questions

Common interview questions on Dark Launching, from basic to scenario-based, with short answers.

http:
- route:
  - destination: { host: tax-engine, subset: v1 }
  mirror: { host: tax-engine, subset: v2 }
  mirrorPercentage: { value: 10.0 }