Best Practices for Microservices Architecture

The best practices for microservices architecture are proven rules for designing, building and running services that stay independent and reliable. They cover where to draw service boundaries, how each service owns its data, how services call each other safely, and how you observe and deploy them. Following them is what separates a working microservices system from a "distributed monolith" that has all the costs and none of the benefits.

This tutorial groups the practices into design, communication and operations, explains the reason behind each one, and shows three of them in runnable Java 21 code. Interview questions follow at the end.

1. What Are Microservices Best Practices?

A microservice is a small application that owns one business capability, such as payments or inventory, and can be deployed on its own. Microservices best practices are the habits that keep that promise true as the system grows: each service can change, deploy, scale and fail without dragging the others along. If the basics are new to you, read the monolithic vs microservices comparison first.

2. Why They Matter: The Distributed Monolith

Splitting code into services is easy. Keeping them independent is hard. Without these practices, teams end up with a distributed monolith: services that share a database, must be released together, and call each other in long chains. It is a monolith with network calls added, so it is slower and less reliable than the original.

Each practice below protects one kind of independence: of change (boundaries and data), of failure (resilience) and of operation (observability and automation).

3. Design Practices: Boundaries and Data

3.1 Draw Boundaries Around Business Capabilities

Split by what the business does (orders, payments, shipping), not by technical layer (a "database service" or a "validation service"). A good test: most new features should change one service. If every feature touches four services, the boundaries are wrong. Domain-Driven Design gives you tools to find these boundaries.

3.2 Give Every Service Its Own Database

A service's tables are private. Other services read its data through its API or its events, never by querying its tables. This is the database per service pattern. A shared schema means no team can rename a column without coordinating with every other team.

3.3 Prefer Events Over Long Call Chains

If a request needs five services to answer one after another, it works only when all five are up. Their availabilities multiply: five services at 99.9% give about 99.5% for the request. When an immediate answer is not needed, publish an event instead, as described in event-driven architecture.

3.4 Plan for Transactions That Span Services

One database transaction cannot cover two services' databases. Use a Saga: a series of local steps, each with a compensating action that undoes it if a later step fails.

4. Communication and Resilience Practices

• Set a timeout on every remote call. Many Java HTTP clients wait forever by default, and one hung call can block a thread for good. See the timeout pattern.
• Retry only temporary failures, such as 503, with exponential backoff and jitter, a random part of the wait that stops thousands of clients from retrying at the same moment.
• Make write operations idempotent, meaning that repeating them has the same effect as doing them once. Retries are only safe when this is true.
• Stop calling a failing service with a circuit breaker, and isolate resources with the bulkhead pattern.
• Keep APIs backward compatible. Add fields, never rename or remove them in place. Consumers should ignore fields they do not know, so providers can add new ones safely.

5. Operations Practices

• Make every request traceable. Give each request a correlation ID, pass it to every service it touches, and include it in every log line. Add distributed tracing and metrics such as request rate, error rate and latency.
• Expose health checks. Spring Boot Actuator provides /actuator/health, plus separate liveness and readiness probes for Kubernetes.
• Automate builds and deployments. Each service needs its own pipeline, so it can be released without waiting for others.
• Keep configuration outside the code, in environment variables or a config server, so one build runs in every environment.
• Secure service-to-service traffic with tokens or mutual TLS. A request is not trusted just because it comes from inside the network.

6. Best Practices for Microservices Architecture in Java

These programs use only the JDK on Java 21, so the idea is visible without framework code. In production you would use libraries such as Resilience4j for retries and Spring for the outbox.

6.1 Idempotency Keys for Safe Retries

The client's payment call times out, but the payment may have succeeded. It retries with the same idempotency key, and the service returns the stored result instead of charging again:

import java.math.BigDecimal;
import java.util.Map;
import java.util.concurrent.ConcurrentHashMap;

public class IdempotentPaymentHandler {

    record Payment(String paymentId, BigDecimal amount) {}

    // Idempotency key -> the result of the first request with that key
    static final Map<String, Payment> processed = new ConcurrentHashMap<>();
    static int cardCharges = 0;

    static String pay(String idempotencyKey, BigDecimal amount) {
        Payment payment = processed.computeIfAbsent(idempotencyKey, key -> {
            cardCharges++;                                   // the real side effect runs only once per key
            return new Payment("PAY-" + cardCharges, amount);
        });
        if (payment.amount().compareTo(amount) != 0) {       // same key, different request: a client bug
            return "422 key " + idempotencyKey + " was already used for " + payment.amount();
        }
        return "200 " + payment;
    }

    public static void main(String[] args) {
        BigDecimal amount = new BigDecimal("2499.00");

        // The client's first call timed out, so it retries with the SAME key
        System.out.println("Attempt 1: " + pay("ord-1001", amount));
        System.out.println("Retry 1:   " + pay("ord-1001", amount));
        System.out.println("Retry 2:   " + pay("ord-1001", amount));
        System.out.println("Wrong use: " + pay("ord-1001", new BigDecimal("99.00")));
        System.out.println("New order: " + pay("ord-1002", new BigDecimal("650.00")));

        System.out.println("Card charged " + cardCharges + " times for 5 requests");
    }
}

Attempt 1: 200 Payment[paymentId=PAY-1, amount=2499.00]
Retry 1:   200 Payment[paymentId=PAY-1, amount=2499.00]
Retry 2:   200 Payment[paymentId=PAY-1, amount=2499.00]
Wrong use: 422 key ord-1001 was already used for 2499.00
New order: 200 Payment[paymentId=PAY-2, amount=650.00]
Card charged 2 times for 5 requests

The edge case on the "Wrong use" line is easy to miss: the same key with a different amount is a client bug, and returning the old result would hide it. In a real service the key goes in a database column with a unique constraint, so two instances of the service cannot both charge.

6.2 Retries with Exponential Backoff and Jitter

The program retries 503 responses with growing, randomised waits, gives up at once on 400, and stops at the attempt limit:

import java.util.Random;
import java.util.Set;

public class RetryWithBackoff {

    static final Set<Integer> RETRYABLE = Set.of(502, 503, 504);   // temporary server problems only
    static final Random random = new Random(7);                     // fixed seed so the output repeats

    // responses[i] is the HTTP status the remote service returns on attempt i + 1
    static String callWithRetry(String name, int[] responses, int maxAttempts) throws InterruptedException {
        long baseDelayMs = 100;
        for (int attempt = 1; ; attempt++) {
            int status = responses[Math.min(attempt - 1, responses.length - 1)];
            if (status == 200) {
                return name + ": attempt " + attempt + " -> 200 OK";
            }
            if (!RETRYABLE.contains(status) || attempt == maxAttempts) {
                return name + ": attempt " + attempt + " -> " + status + ", giving up";
            }
            long maxDelay = baseDelayMs * (1L << (attempt - 1));    // 100, 200, 400 ms ...
            long delay = random.nextLong(maxDelay + 1);             // full jitter: anything from 0 to maxDelay
            System.out.println("  " + name + ": attempt " + attempt + " -> " + status
                    + ", waiting " + delay + " ms (max " + maxDelay + ")");
            Thread.sleep(delay);
        }
    }

    public static void main(String[] args) throws InterruptedException {
        System.out.println(callWithRetry("inventory", new int[] {503, 503, 200}, 4));
        System.out.println(callWithRetry("pricing", new int[] {400}, 4));      // a bad request never gets better
        System.out.println(callWithRetry("shipping", new int[] {503}, 3));     // keeps failing: stop at the limit
    }
}

  inventory: attempt 1 -> 503, waiting 78 ms (max 100)
  inventory: attempt 2 -> 503, waiting 190 ms (max 200)
inventory: attempt 3 -> 200 OK
pricing: attempt 1 -> 400, giving up
  shipping: attempt 1 -> 503, waiting 69 ms (max 100)
  shipping: attempt 2 -> 503, waiting 105 ms (max 200)
shipping: attempt 3 -> 503, giving up

A 400 Bad Request will fail the same way every time, so retrying it only adds load.

6.3 The Transactional Outbox: Saving Data and Publishing Events Safely

A common bug is the dual write: the service saves an order, then publishes an OrderPlaced event. If it crashes between the two, the order exists but no other service ever hears about it. The transactional outbox fixes this by saving the event in the service's own database, in the same transaction as the order. A separate relay publishes it later:

import java.util.ArrayList;
import java.util.HashSet;
import java.util.List;
import java.util.Set;

public class TransactionalOutbox {

    record Event(String eventId, String type, String orderId) {}

    // The Order Service's own database: the orders table and the outbox table
    static final List<String> orders = new ArrayList<>();
    static final List<Event> outbox = new ArrayList<>();

    static boolean brokerUp = false;
    static final List<Event> broker = new ArrayList<>();

    static void placeOrder(String orderId) {
        // One local transaction: both rows are saved, or neither is
        orders.add(orderId);
        outbox.add(new Event("evt-" + orderId, "OrderPlaced", orderId));
        System.out.println("Saved " + orderId + " and its event in one transaction");
    }

    // A background relay publishes outbox rows; a row is removed only after the broker accepts it
    static void relay() {
        if (!brokerUp) {
            System.out.println("Relay: broker down, " + outbox.size() + " event(s) wait safely in the outbox");
            return;
        }
        broker.addAll(outbox);
        System.out.println("Relay: published " + outbox.size() + " event(s)");
        outbox.clear();
    }

    public static void main(String[] args) {
        placeOrder("ORD-1001");
        placeOrder("ORD-1002");
        relay();

        brokerUp = true;
        relay();
        broker.add(broker.getFirst());   // the relay crashed before clearing once: the same event is sent twice

        // Delivery is at-least-once, so the consumer skips event ids it has already handled
        Set<String> handled = new HashSet<>();
        for (Event event : broker) {
            String action = handled.add(event.eventId()) ? "handled" : "skipped duplicate";
            System.out.println("Inventory Service " + action + ": " + event);
        }
    }
}

Saved ORD-1001 and its event in one transaction
Saved ORD-1002 and its event in one transaction
Relay: broker down, 2 event(s) wait safely in the outbox
Relay: published 2 event(s)
Inventory Service handled: Event[eventId=evt-ORD-1001, type=OrderPlaced, orderId=ORD-1001]
Inventory Service handled: Event[eventId=evt-ORD-1002, type=OrderPlaced, orderId=ORD-1002]
Inventory Service skipped duplicate: Event[eventId=evt-ORD-1001, type=OrderPlaced, orderId=ORD-1001]

What happens internally: if the relay crashes after publishing but before marking the row as sent, it publishes the same event again on restart. Delivery is therefore at least once, and consumers must skip duplicates by event ID, as the Inventory Service does here.

7. Good vs Poor Practice: Comparison

8. Common Mistakes and Pitfalls

• Starting with microservices. Splitting before the domain is understood puts boundaries in the wrong places. A well-structured monolith is often the better first step.
• Services that are too small. One service per table or class creates chatty services and long call chains.
• Shared libraries with business logic. A shared "common-model" JAR forces every service to upgrade together. Share technical utilities, not domain classes.
• Breaking API changes without versions. Renaming a field breaks every consumer at the moment you deploy.
• Observability added last. The first production incident without correlation IDs turns into hours of searching logs on several servers.

9. Where to Start: A Priority Order

You do not need everything on day one. A sensible order for a new system is:

1. Correct boundaries and a database per service, because these are the hardest to change later.
2. Timeouts on every remote call.
3. Correlation IDs, central logs and health checks.
4. Idempotent writes, then retries and circuit breakers.
5. The outbox and Sagas, once services exchange events and multi-step workflows appear.

10. Summary

• Microservices best practices protect independence of change, failure and operation.
• Split by business capability, give each service its own database, and keep synchronous call chains short.
• Put a timeout on every call, retry only temporary errors at one layer, and make writes idempotent.
• Use the transactional outbox to avoid dual writes, and expect at-least-once delivery.
• Following the best practices for microservices architecture, starting with boundaries, data and observability, keeps your system from becoming a distributed monolith.

Microservices Best Practices: Interview Questions

Common interview questions on microservices best practices, from basic to scenario-based, with short answers.