What Is Canary Deployment? Safer Releases One Step at a Time

Canary deployment is a release strategy where a new version of a service is given to a small share of real traffic first, for example 5%, while the stable version keeps serving everyone else. If the new version's metrics stay healthy, its share is increased step by step until it serves 100%; if not, traffic is sent back to the stable version and only a few users were affected.

The name comes from coal mining. Miners carried canaries underground because the birds reacted to toxic gas before people did. In software, the canary version is the early warning: it meets real users first, so problems show up while their impact is still small.

1. Why Canary Deployment Exists

Picture a payments team releasing a new version of its Checkout service on a Friday afternoon. Every test passed in staging. But staging never has real traffic: real card types, real browsers, odd user behaviour, and the real load on the database.

With a full release, any hidden bug reaches 100% of customers at once. With blue-green deployment, rollback is fast, but everyone still switches at the same moment, so everyone feels the bug until someone notices.

A canary changes the question from "does it work?" to "how much damage can it do before we find out?". If only 5% of checkout requests reach the new version and it fails, 95% of customers never notice, and the rollback is automatic.

2. How Canary Deployment Works

The process has four parts:

1. Deploy without traffic. The new version (the canary) starts next to the current version (the stable version) but receives nothing yet.
2. Shift a small share. A traffic router sends, say, 5% of requests to the canary. The router can be a Kubernetes Service, an ingress controller, an API gateway or a service mesh.
3. Analyse. For a fixed period, metrics from the canary are compared with the stable version: error rate, latency, CPU, memory and business numbers such as successful payments.
4. Promote or roll back. If the canary is healthy, increase its share (25%, 50%, 100%). If any check fails, send 0% to the canary and investigate.

An important difference from blue-green: during a canary, two versions serve real users at the same time, often for hours. They share the same database, message topics and caches, so they must be compatible with each other. Example 3 shows why this matters.

3. Deciding Pass or Fail: Canary Analysis

A canary is only as good as the decision at each step. Canary analysis is the process of making that decision, ideally automatically. Good analysis follows three rules:

• Compare with a baseline, not with the past. Yesterday's error rate is a poor reference, because traffic and dependencies change. Compare the canary with stable pods serving traffic during the same minutes. Some teams even start a fresh "baseline" copy of the old version, so that both sides have just started and have equally cold caches.
• Wait for enough data. One error in 40 requests is 2.5%, which looks alarming but may be pure chance. Set a minimum number of requests before judging.
• Check more than errors. Latency, saturation and business metrics catch problems that error counts miss.

Tools such as Argo Rollouts and Flagger run these checks automatically against Prometheus or other monitoring systems and roll back without human action.

4. Code Examples

These JDK-only Java 21 programs act out the logic that canary tools and routers apply inside.

4.1 Example 1: A Naive Gate vs a Smart Gate

import java.util.List;

public class CanaryGateDemo {

    // One analysis window, as a monitoring system (e.g. Prometheus) would report it
    record Stats(int requests, int errors, int p99Ms) {
        double errorRate() { return 100.0 * errors / requests; }
    }

    record Window(String label, Stats canary, Stats baseline) { }

    static final int MIN_REQUESTS = 200;

    // Naive gate: fixed error threshold, no matter how little data there is
    static String naiveGate(Window w) {
        return w.canary().errorRate() <= 1.0 ? "PASS" : "ROLLBACK";
    }

    // Better gate: enough data first, then compare against a baseline running at the same time
    static String smartGate(Window w) {
        Stats c = w.canary(), b = w.baseline();
        if (c.requests() < MIN_REQUESTS) {
            return "WAIT (" + c.requests() + " requests, need " + MIN_REQUESTS + ")";
        }
        if (c.errorRate() > b.errorRate() + 0.5) {
            return "ROLLBACK (errors " + fmt(c.errorRate()) + " vs " + fmt(b.errorRate()) + ")";
        }
        if (c.p99Ms() > b.p99Ms() * 1.2) {
            return "ROLLBACK (p99 " + c.p99Ms() + " ms vs " + b.p99Ms() + " ms)";
        }
        return "PASS";
    }

    static String fmt(double pct) {
        return String.format("%.2f%%", pct);
    }

    public static void main(String[] args) {
        List<Window> windows = List.of(
                new Window("5%  minute 1 ", new Stats(40, 1, 180), new Stats(760, 6, 175)),
                new Window("5%  minutes 1-6", new Stats(240, 2, 182), new Stats(4_560, 36, 176)),
                new Window("25% minutes 1-5", new Stats(1_250, 9, 190), new Stats(3_750, 30, 178)),
                new Window("50% minutes 1-5", new Stats(2_500, 21, 410), new Stats(2_500, 20, 181)));

        System.out.printf("%-16s %-7s %-12s %s%n", "Window", "Canary", "Naive gate", "Smart gate");
        for (Window w : windows) {
            System.out.printf("%-16s %-7s %-12s %s%n", w.label(), fmt(w.canary().errorRate()),
                    naiveGate(w), smartGate(w));
        }
    }
}

Window           Canary  Naive gate   Smart gate
5%  minute 1     2.50%   ROLLBACK     WAIT (40 requests, need 200)
5%  minutes 1-6  0.83%   PASS         PASS
25% minutes 1-5  0.72%   PASS         PASS
50% minutes 1-5  0.84%   PASS         ROLLBACK (p99 410 ms vs 181 ms)

The naive gate made two opposite mistakes. In minute one it rolled back a healthy release, because 1 error in 40 requests looked like 2.5%. At 50% traffic it approved a release whose p99 latency had more than doubled, because it only looked at errors.

The smart gate waited for enough data, then compared with the baseline and caught the latency problem. That problem only appeared at 50%, when the canary's database connection pool started to run short under real load. This is exactly why canaries move in several steps, instead of jumping from 5% straight to 100%.

4.2 Example 2: Why "1 of 10 Pods" Is Not Always 10% of Traffic

The simplest canary in Kubernetes is to run 9 stable pods and 1 canary pod behind the same Service. Many tutorials say this gives 10% of traffic. This program shows why that is only roughly true.

import java.util.Random;

public class ConnectionSkewDemo {

    public static void main(String[] args) {
        int stablePods = 9, canaryPods = 1, pods = stablePods + canaryPods;
        var random = new Random(11); // fixed seed so the output is repeatable

        // 20 calling pods, each with a pool of 2 keep-alive connections
        int clients = 20, connsPerClient = 2;
        boolean[] connToCanary = new boolean[clients * connsPerClient];
        int canaryConns = 0;
        for (int i = 0; i < connToCanary.length; i++) {
            connToCanary[i] = random.nextInt(pods) >= stablePods; // pod chosen once, when the connection opens
            if (connToCanary[i]) canaryConns++;
        }

        // Traffic is uneven: client 0 is a busy batch job sending as much as all the others together
        long total = 0, toCanaryPerConnection = 0, toCanaryPerRequest = 0;
        for (int c = 0; c < clients; c++) {
            int requests = c == 0 ? 19_000 : 1_000;
            for (int r = 0; r < requests; r++) {
                int conn = c * connsPerClient + r % connsPerClient; // requests reuse the open connections
                total++;
                if (connToCanary[conn]) toCanaryPerConnection++;
                if (random.nextInt(pods) >= stablePods) toCanaryPerRequest++; // L7: pod chosen per request
            }
        }

        System.out.printf("Pods: %d stable + %d canary -> expected share %.0f%%%n",
                stablePods, canaryPods, 100.0 * canaryPods / pods);
        System.out.printf("Connections to canary: %d of %d%n", canaryConns, connToCanary.length);
        System.out.printf("Requests to canary, balanced per connection: %.1f%%%n", 100.0 * toCanaryPerConnection / total);
        System.out.printf("Requests to canary, balanced per request   : %.1f%%%n", 100.0 * toCanaryPerRequest / total);
        System.out.printf("Smallest canary with replica ratios: 1 of %d pods = %.0f%%; 5%% needs 1 canary + 19 stable%n",
                pods, 100.0 / pods);
    }
}

Pods: 9 stable + 1 canary -> expected share 10%
Connections to canary: 4 of 40
Requests to canary, balanced per connection: 5.3%
Requests to canary, balanced per request   : 10.2%
Smallest canary with replica ratios: 1 of 10 pods = 10%; 5% needs 1 canary + 19 stable

A Kubernetes Service (through kube-proxy) chooses a pod when a connection opens, not for every request. Java HTTP clients reuse connections (keep-alive), so all requests on one connection go to the same pod. Here the canary got exactly 4 of the 40 connections, yet only 5.3% of the requests, because the busy batch job's connections happened to land on stable pods. If they had landed on the canary, it could have received a quarter of all traffic or more.

Balancing per request, as an ingress controller, gateway or service mesh does, gives about 10% as planned. Replica ratios also limit your choices: with 10 pods, the smallest canary is 10%.

4.3 Example 3: Two Versions Living Together

During a canary, the new version produces data that the old version reads, and the other way round.

import java.util.LinkedHashMap;
import java.util.Map;
import java.util.Set;

public class MixedVersionDemo {

    static final Set<String> V1_FIELDS = Set.of("orderId", "amount");

    // Strict reader: like a plain Jackson 2 ObjectMapper, which fails on unknown fields by default
    static String strictV1(Map<String, Object> event) {
        for (String field : event.keySet()) {
            if (!V1_FIELDS.contains(field)) {
                return "FAILED: Unrecognized field \"" + field + "\"";
            }
        }
        return "processed " + event.get("orderId");
    }

    // Tolerant reader: ignores fields it doesn't know
    static String tolerantV1(Map<String, Object> event) {
        return "processed " + event.get("orderId");
    }

    // Canary v2 reads a new field, with a safe default when an old producer omits it
    static String tolerantV2(Map<String, Object> event) {
        boolean giftWrap = (boolean) event.getOrDefault("giftWrap", false);
        return "processed " + event.get("orderId") + (giftWrap ? " with gift wrap" : " without gift wrap");
    }

    static Map<String, Object> event(String orderId, Boolean giftWrap) {
        var e = new LinkedHashMap<String, Object>();
        e.put("orderId", orderId);
        e.put("amount", 1299);
        if (giftWrap != null) e.put("giftWrap", giftWrap); // only v2 producers send this
        return e;
    }

    public static void main(String[] args) {
        var fromCanaryV2 = event("ORD-881", true);
        var fromStableV1 = event("ORD-882", null);

        System.out.println("v2 event " + fromCanaryV2);
        System.out.println("  stable v1, strict reader  : " + strictV1(fromCanaryV2));
        System.out.println("  stable v1, tolerant reader: " + tolerantV1(fromCanaryV2));
        System.out.println("v1 event " + fromStableV1);
        System.out.println("  canary v2, with default   : " + tolerantV2(fromStableV1));
    }
}

v2 event {orderId=ORD-881, amount=1299, giftWrap=true}
  stable v1, strict reader  : FAILED: Unrecognized field "giftWrap"
  stable v1, tolerant reader: processed ORD-881
v1 event {orderId=ORD-882, amount=1299}
  canary v2, with default   : processed ORD-882 without gift wrap

The canary added a giftWrap field to its events. A strict reader in the stable version rejected them, so the canary broke the stable version, which the canary's own metrics would never show. This is a real Java trap: a plain Jackson 2 ObjectMapper fails on unknown properties by default (Spring Boot's auto-configured mapper disables this). Make readers tolerant, and give new fields safe defaults, as the v2 reader does.

5. Comparison Table: Ways to Run a Canary in Kubernetes

Note that the community ingress-nginx controller, whose canary annotations were a popular choice, has been retired. New setups usually use weighted backendRefs in the Kubernetes Gateway API HTTPRoute instead.

6. Common Mistakes and Pitfalls

1. Judging too early. As Example 1 showed, a few requests give noisy numbers. Use a minimum request count, not just a timer.

2. Watching only the canary. A canary can harm the stable version through shared data, events or caches (Example 3). Watch the whole service and its consumers during the rollout.

3. Breaking changes between versions. Both versions run together for a long time. Database changes must follow expand and contract, and APIs and events must stay backward compatible.

4. Trusting replica ratios for small percentages. Connection-level balancing skews the share, and small percentages need many pods (Example 2).

5. Canary steps that are too short. A step lasting five minutes may miss the nightly batch job, a cache that expires every hour, or the evening traffic peak. Make sure the total rollout covers the patterns that matter.

6. Users flipping between versions. With per-request splitting, one user can see the new checkout page and then the old one. For user-visible changes, keep each user on one version, for example with a cookie or header set from a hash of the user ID.

7. Best Practices

• Automate the analysis and the rollback, so the decision doesn't depend on someone watching a dashboard.
• Compare canary and baseline over the same time window, with a minimum request count per step.
• Include business metrics, such as payment success rate, next to technical ones.
• Use several steps (for example 5%, 25%, 50%, 100%) so that load-related problems appear before everyone is affected.
• Start with internal users by routing employees or test accounts to the canary using a header or cookie.
• Keep every change backward compatible, including database schemas, events and APIs.
• Use per-request routing (gateway, mesh or Gateway API) when you need precise percentages.

8. Summary

• Canary deployment sends a small share of real traffic to a new version and increases it step by step while metrics stay healthy.
• A failed check sends traffic back to the stable version, limiting the damage to a small group of users.
• Good canary analysis waits for enough data, compares against a baseline from the same time, and checks latency and business metrics, not just errors.
• Kubernetes Services balance per connection, so "1 of 10 pods" does not reliably mean 10% of requests.
• Old and new versions run together, so events, APIs and schemas must be compatible in both directions.
• Tools like Argo Rollouts and Flagger automate the steps, the analysis and the rollback.

Done well, canary deployment lets you release often without betting all your users on every change.

Canary Deployment: Interview Questions

Common interview questions on Canary Deployment, from basic to scenario-based, with short answers.

strategy:
  canary:
    steps:
      - setWeight: 5
      - pause: { duration: 10m }
      - setWeight: 25
      - analysis: { templates: [ { templateName: success-rate } ] }
      - setWeight: 50
      - pause: { duration: 30m }