What Is A/B Testing? How It Works in Java Microservices

A/B testing is a controlled experiment where users are randomly split into groups, each group sees a different version of a feature, and you compare a measurable result, such as purchases, to see which version really works better. Version A is usually the current experience (the control) and version B is the change (the variant).

Instead of arguing about whether a green "Buy now" button beats a blue one, you let real user behaviour decide. For backend developers, A/B testing is not just a marketing tool: you must assign users correctly, log the right events and analyse the numbers honestly, or the "winner" can be pure luck.

1. Why A/B Testing Matters

Imagine an online bookstore. The product team believes that showing "Free delivery over ₹499" on the cart page will increase orders. They could simply release it and compare this month's sales with last month's. But sales also change because of holidays, discounts, the weather and a hundred other things. A "before vs after" comparison can't separate the effect of the banner from everything else.

An A/B test solves this by running both versions at the same time, on randomly chosen groups of similar users. Anything that affects one group, like a holiday sale, affects the other group equally. So the only systematic difference between the groups is the change you made.

2. Core Concepts of A/B Testing

2.1 Variants and Traffic Split

Each version is a variant. A typical test splits traffic 50/50 between A and B, but you can also start smaller, for example 90/10, to limit risk.

2.2 Unit of Randomisation

This is what you randomise: a user, a session, a device or an account. For most product tests it is the user, so that one person sees the same version every time. If you randomise per request, a user might see the new checkout on one click and the old one on the next.

2.3 Metrics

• The primary metric decides the test, for example "checkout conversion rate": the share of users who complete a purchase.
• Guardrail metrics must not get worse, such as page load time, error rate or refunds.

2.4 Statistical Significance

Even two identical pages will show slightly different results, just by chance. Statistical significance tells you whether the difference is bigger than chance would normally produce.

The usual tool is the p-value: the probability of seeing a difference at least this large if there were really no difference. Teams commonly treat p < 0.05 as significant, which means accepting a 5% risk of declaring a winner that isn't real. This risk is called a false positive.

2.5 Sample Size

Small tests are noisy. Before starting, you calculate how many users you need to detect the smallest improvement you care about, and you run the test until you reach that number.

3. How A/B Testing Works in a Microservices System

1. Assignment. When the Cart service builds the page, it asks a feature flag SDK which variant this user gets. A feature flag is a switch in code whose value is decided at runtime. The SDK hashes the experiment name and user ID, so the answer is the same on every server and every request, with no database lookup.
2. Exposure logging. The service records that the user actually saw variant B. Only exposed users should be counted, which is why this event is sent at the point where the variant is shown, not at login.
3. Conversion logging. Other services publish business events, such as "order placed", with the user ID.
4. Analysis. A batch or streaming job joins exposures with conversions and runs the statistical test.

The split can also happen at the infrastructure level. An API gateway or service mesh can route users to two deployed versions of a service based on a header or cookie. Code-level flags suit small UI or logic changes; routing suits completely different service versions.

4. Code Examples

All three programs use only the JDK (Java 21), so you can run them anywhere.

4.1 Example 1: Stable, Independent User Bucketing

import java.nio.charset.StandardCharsets;
import java.security.MessageDigest;
import java.security.NoSuchAlgorithmException;

public class BucketingDemo {

    static final MessageDigest SHA256;
    static {
        try {
            SHA256 = MessageDigest.getInstance("SHA-256");
        } catch (NoSuchAlgorithmException e) {
            throw new ExceptionInInitializerError(e);
        }
    }

    // Stable bucket 0..9999: same input -> same bucket, on every server, forever
    static int bucket(String key) {
        byte[] h = SHA256.digest(key.getBytes(StandardCharsets.UTF_8));
        int first4 = ((h[0] & 0xFF) << 24) | ((h[1] & 0xFF) << 16) | ((h[2] & 0xFF) << 8) | (h[3] & 0xFF);
        return Integer.remainderUnsigned(first4, 10_000);
    }

    static String variant(String experiment, String userId, boolean salted) {
        String key = salted ? experiment + ":" + userId : userId; // salt = experiment name
        return bucket(key) < 5_000 ? "A" : "B";
    }

    public static void main(String[] args) {
        System.out.println("user-1042 in checkout-button: "
                + variant("checkout-button", "user-1042", true) + " "
                + variant("checkout-button", "user-1042", true) + " "
                + variant("checkout-button", "user-1042", true)); // asked three times

        for (boolean salted : new boolean[] {false, true}) {
            int inB1 = 0, inBoth = 0;
            for (int i = 0; i < 100_000; i++) {
                String user = "user-" + i;
                boolean b1 = variant("checkout-button", user, salted).equals("B");
                boolean b2 = variant("free-delivery-banner", user, salted).equals("B");
                if (b1) inB1++;
                if (b1 && b2) inBoth++;
            }
            System.out.printf("%-9s: %,d of 100,000 users in B of experiment 1; %,d in B of BOTH%n",
                    salted ? "salted" : "unsalted", inB1, inBoth);
        }
    }
}

user-1042 in checkout-button: B B B
unsalted : 49,785 of 100,000 users in B of experiment 1; 49,785 in B of BOTH
salted   : 50,004 of 100,000 users in B of experiment 1; 25,102 in B of BOTH

Two lessons:

• Stability: user-1042 gets B every time, on any server, because SHA-256 always produces the same output for the same input. Avoid Math.random() or Object.hashCode() for assignment.
• Independence: without the experiment name in the hash (unsalted), the same users land in group B for every experiment. Then the checkout test and the banner test are tangled together: you can't tell which change caused what. With salting, about 25% of users are in B for both tests, exactly what independent 50/50 splits should give.

4.2 Example 2: Is the Difference Real?

public class SignificanceDemo {

    // Standard normal CDF using the Abramowitz-Stegun erf approximation (error < 1.5e-7)
    static double normalCdf(double z) {
        double x = Math.abs(z) / Math.sqrt(2);
        double t = 1 / (1 + 0.3275911 * x);
        double erf = 1 - t * (0.254829592 + t * (-0.284496736 + t * (1.421413741
                + t * (-1.453152027 + t * 1.061405429)))) * Math.exp(-x * x);
        return z >= 0 ? 0.5 * (1 + erf) : 0.5 * (1 - erf);
    }

    // Two-proportion z-test: is the difference in conversion rate bigger than chance?
    static void compare(String label, int usersA, int convA, int usersB, int convB) {
        double pA = (double) convA / usersA, pB = (double) convB / usersB;
        double pooled = (double) (convA + convB) / (usersA + usersB);
        double se = Math.sqrt(pooled * (1 - pooled) * (1.0 / usersA + 1.0 / usersB));
        double z = (pB - pA) / se;
        double pValue = 2 * (1 - normalCdf(Math.abs(z))); // two-sided
        System.out.printf("%s%n  A %.2f%%  B %.2f%%  lift %+.1f%%  z=%.2f  p=%.4f  -> %s%n",
                label, pA * 100, pB * 100, (pB / pA - 1) * 100, z, pValue,
                pValue < 0.05 ? "significant at 5%" : "not significant");
    }

    // Sample ratio mismatch: with a 50/50 split, are the group sizes suspiciously different?
    static void srmCheck(int usersA, int usersB) {
        double expected = (usersA + usersB) / 2.0;
        double chi2 = Math.pow(usersA - expected, 2) / expected + Math.pow(usersB - expected, 2) / expected;
        double pValue = 2 * (1 - normalCdf(Math.sqrt(chi2))); // chi-square with 1 degree of freedom
        System.out.printf("SRM check %,d vs %,d: p=%.6f -> %s%n", usersA, usersB, pValue,
                pValue < 0.001 ? "MISMATCH, do not trust this test" : "split looks healthy");
    }

    public static void main(String[] args) {
        compare("Big test (10,000 users per group):", 10_000, 520, 10_000, 585);
        compare("Small test (400 users per group):", 400, 21, 400, 23);
        srmCheck(10_000, 10_090);
        srmCheck(10_000, 10_500);
    }
}

Big test (10,000 users per group):
  A 5.20%  B 5.85%  lift +12.5%  z=2.01  p=0.0442  -> significant at 5%
Small test (400 users per group):
  A 5.25%  B 5.75%  lift +9.5%  z=0.31  p=0.7564  -> not significant
SRM check 10,000 vs 10,090: p=0.525448 -> split looks healthy
SRM check 10,000 vs 10,500: p=0.000479 -> MISMATCH, do not trust this test

The big test shows a 12.5% lift with p = 0.044, so it passes the 5% threshold. The small test shows a similar lift, but with only 400 users per group the p-value is 0.76, so the result could easily be noise.

The last two lines show a check most tutorials skip: sample ratio mismatch (SRM). With a 50/50 split, the group sizes should be close. 10,000 vs 10,500 is so unlikely by chance (p < 0.001) that something in the system is broken. Common causes are a variant that crashes before logging exposure, a bot filter that treats variants differently, or a redirect that loses some users. When SRM appears, the results can't be trusted, however good they look.

4.3 Example 3: The Peeking Problem

Many teams watch the dashboard every day and stop the test as soon as it turns "significant". This program runs 1,000 A/A tests, where both groups get the identical experience, so every "winner" is false.

import java.util.Random;

public class PeekingDemo {

    static double normalCdf(double z) {
        double x = Math.abs(z) / Math.sqrt(2);
        double t = 1 / (1 + 0.3275911 * x);
        double erf = 1 - t * (0.254829592 + t * (-0.284496736 + t * (1.421413741
                + t * (-1.453152027 + t * 1.061405429)))) * Math.exp(-x * x);
        return z >= 0 ? 0.5 * (1 + erf) : 0.5 * (1 - erf);
    }

    static double pValue(int n, int convA, int convB) {
        double pooled = (convA + convB) / (2.0 * n);
        double se = Math.sqrt(pooled * (1 - pooled) * (2.0 / n));
        if (se == 0) return 1;
        double z = ((double) convB - convA) / n / se;
        return 2 * (1 - normalCdf(Math.abs(z)));
    }

    public static void main(String[] args) {
        var random = new Random(42);    // fixed seed so the output is repeatable
        int experiments = 1_000, usersPerGroup = 10_000, peekEvery = 500;
        int winnersWhenPeeking = 0, winnersAtEndOnly = 0;

        for (int e = 0; e < experiments; e++) {
            int convA = 0, convB = 0;
            boolean declaredEarly = false;
            for (int n = 1; n <= usersPerGroup; n++) {
                if (random.nextDouble() < 0.05) convA++; // A/A test: both groups truly convert at 5%
                if (random.nextDouble() < 0.05) convB++;
                if (n % peekEvery == 0 && !declaredEarly && pValue(n, convA, convB) < 0.05) {
                    declaredEarly = true; // someone checks the dashboard and stops the test
                }
            }
            if (declaredEarly) winnersWhenPeeking++;
            if (pValue(usersPerGroup, convA, convB) < 0.05) winnersAtEndOnly++;
        }
        System.out.printf("A/A tests with NO real difference: %,d%n", experiments);
        System.out.printf("False winners, checked once at the end : %d (%.1f%%)%n",
                winnersAtEndOnly, 100.0 * winnersAtEndOnly / experiments);
        System.out.printf("False winners, peeking every %d users : %d (%.1f%%)%n",
                peekEvery, winnersWhenPeeking, 100.0 * winnersWhenPeeking / experiments);
    }
}

A/A tests with NO real difference: 1,000
False winners, checked once at the end : 49 (4.9%)
False winners, peeking every 500 users : 248 (24.8%)

Checking once at the planned end gives about the promised 5% false winners. Checking every 500 users and stopping at the first p < 0.05 gives about 25%, five times more. Each peek is another chance for random noise to cross the line. Either fix the sample size in advance and look only at the end, or use a method designed for continuous monitoring, such as a sequential test.

5. Comparison Table: A/B Testing vs Canary vs Blue-Green

A canary asks "does it break?", while an A/B test asks "is it better?". Many teams run a canary first, then an A/B test on the safe new version.

6. Common Mistakes and Pitfalls

1. Stopping early because it looks good. As Example 3 showed, peeking turns a 5% false-positive rate into about 25%.

2. Ignoring sample ratio mismatch. Always check that the group sizes match the planned split before looking at results.

3. Counting users who never saw the change. If a banner appears only on the cart page but you assign everyone at login, most users in group B never saw it. This dilutes the effect and wastes traffic. Log exposure where the variant is shown.

4. Testing too many metrics. Check 20 metrics at p < 0.05 and, on average, one will "win" by chance. Choose one primary metric before the test starts.

5. Running for only a few days. Behaviour changes between weekdays and weekends, and new designs can get extra clicks just because they are new (the novelty effect). Run tests for at least one or two full weeks.

6. Assignment that changes between services. If the Cart service and the Order service each call Random to choose a variant, a user can see B in the cart and A at checkout. Use one deterministic hash, or pass the assigned variant along in a header.

7. Best Practices

• Write a test plan first: hypothesis, primary metric, guardrails, minimum detectable effect and sample size.
• Use deterministic, salted hashing on a stable ID, as in Example 1.
• Run an A/A test when you set up a new experiment system, to prove it doesn't create false differences.
• Log exposure and conversion events with the user ID, experiment name and variant.
• Check SRM and guardrails before reading the primary metric.
• Keep a kill switch so you can turn a variant off instantly without redeploying.
• Clean up finished flags. Old flags left in code become confusing dead branches.

8. Summary

• A/B testing compares two versions at the same time on randomly split users, so outside factors affect both groups equally.
• Randomise by a stable unit, usually the user, with a deterministic hash salted by experiment name.
• Log exposure where the variant is actually shown, and conversions as business events.
• Use a significance test, and decide the sample size before you start.
• Peeking and stopping early can raise false winners from 5% to around 25%.
• A sample ratio mismatch means the test is broken, no matter what the results say.
• A/B tests measure business value; canary and blue-green releases manage deployment risk.

Done properly, A/B testing turns product opinions into evidence. The engineering details in this tutorial are what make that evidence trustworthy.

A/B Testing: Interview Questions

Common interview questions on A/B Testing, from basic to scenario-based, with short answers.

double p = 0.052, delta = 0.0065;
long perGroup = Math.round(16 * p * (1 - p) / (delta * delta)); // about 18,670

Client flags = OpenFeatureAPI.getInstance().getClient();
boolean newCart = flags.getBooleanValue("new-cart-banner", false, new ImmutableContext(userId));