00:00

Batch Processing vs Stream Processing

Last updated:

An online store produces a constant flow of data: orders, page views, payments, stock changes. Some questions about that data can wait until tomorrow, such as "what were yesterday's total sales?". Others need an answer right now, such as "is this card being used fraudulently?". These two needs lead to two styles of data processing: batch and stream.

This tutorial compares the two styles, explains windows and delivery guarantees that make streaming work, and shows how Lambda and Kappa architectures combine them. It connects to Event-Driven Architecture and Kafka vs RabbitMQ.

1. What Are Batch and Stream Processing?

Batch processing collects data over a period, stores it, and then processes the whole set in one run, usually on a schedule. The input is bounded: the job knows where the data starts and ends. Tools include Apache Spark, Hadoop MapReduce and scheduled SQL jobs.

Stream processing handles data continuously, event by event or in small groups, as soon as it arrives. The input is unbounded: it never ends, so results are produced incrementally. Tools include Apache Flink, Kafka Streams and Spark Structured Streaming, which processes data in small micro-batches.

2. Batch and Stream Pipelines

Batch processing and stream processing pipelines Top: in batch processing, order events are first stored in a data lake, a scheduled job processes the whole day at 02:00, and a report is ready the next morning. Bottom: in stream processing, the same events flow into a Kafka topic, a stream processor handles each event as it arrives, and a live dashboard updates within seconds. BATCH · PROCESS LATER, ALL AT ONCE Order events Data lakestored files Nightly jobruns at 02:00 Daily reporthours later STREAM · PROCESS EACH EVENT NOW Order events Kafka topicevent log Stream processoralways running Live dashboardseconds
Batch processing stores data first and processes it later in large chunks; stream processing handles each event as it arrives, so results are ready in seconds.

The same events can feed both pipelines. Many companies stream data into Kafka once and then use it both for real-time processing and for loading the data lake used by batch jobs.

3. How Each Style Processes Data

A batch job

  1. Events are written to storage all day, for example as files in object storage.
  2. At 02:00 a scheduler starts the job, which reads the full day of data.
  3. The job aggregates, joins and cleans the data in parallel across a cluster.
  4. It writes the results, such as a sales report, and then stops until the next run.

A stream processor

  1. Each order event is published to a Kafka topic as it happens.
  2. A stream processor reads events continuously and keeps running state, such as revenue per minute.
  3. It emits updated results whenever a window closes or an event changes the state.
  4. It regularly checkpoints its state, so after a failure it resumes from the last checkpoint instead of starting again.

Between the two sits micro-batching, used by Spark Structured Streaming: events are processed in very small batches, by default each one starting as soon as the previous one finishes, which gives near-real-time results with a batch-style programming model.

4. Windows, Time and Delivery Guarantees

Because a stream never ends, aggregations must be computed over windows: finite slices of the stream. The three main window types answer different questions.

Tumbling windows

Consecutive windows of a fixed length that never overlap, such as each calendar minute.

Tumbling windows Events on a three-minute timeline are grouped into fixed one-minute windows that do not overlap: 0:00 to 1:00 holds four events, 1:00 to 2:00 holds two events, and 2:00 to 3:00 holds five events. 4 events 2 events 5 events 0:00 1:00 2:00 3:00
Fixed-size, non-overlapping windows: every event belongs to exactly one window. Typical use: "orders per minute".

Hopping (sliding) windows

Windows of a fixed length that start at a smaller interval, so they overlap. Some tools, including Kafka Streams, call these hopping windows and use "sliding" for a slightly different, event-driven variant.

Hopping (sliding) windows Two-minute windows start every minute, so they overlap: one window covers 0:00 to 2:00 and the next covers 1:00 to 3:00. Events between 1:00 and 2:00 are counted in both windows. window 1: 0:00 – 2:00 window 2: 1:00 – 3:00 0:00 1:00 2:00 3:00
Windows of a fixed size that advance by a smaller step, so they overlap and one event can fall into several windows. Typical use: "orders in the last 2 minutes, updated every minute".

Session windows

Windows defined by activity rather than the clock. A session stays open while events keep arriving and closes after a gap of inactivity.

Session windows Events are grouped by activity. A burst of four events forms session 1, a pause longer than the 30-second gap ends it, two events form session 2, and after another long pause five events form session 3. session 1 session 2 session 3 0:00 1:00 2:00 3:00
Windows that open with activity and close after a gap of inactivity (here 30 seconds), so their length varies. Typical use: user browsing sessions.

Event time and late data

Events carry the time they happened (event time), which can differ from when they arrive (processing time). Stream processors use watermarks to decide how long to wait for late events before closing a window.

Delivery guarantees

At-most-once may lose events; at-least-once guarantees delivery but can repeat an event after a retry; exactly-once makes each event affect the results once, using checkpoints and transactional writes. Flink and Kafka Streams both offer exactly-once processing within their supported sources and sinks.

Lambda architecture

Runs a batch layer and a speed layer side by side and merges their results.

Lambda architecture Incoming data is sent to two paths. The batch layer reprocesses all historical data periodically for accurate results, and the speed layer processes recent data in real time. A serving layer merges both views to answer queries. Incomingdata Batch layerall history, hourly Speed layerrecent data, seconds Serving layermerges both views
Two paths for the same data: a batch layer for accurate history and a speed layer for fresh results, merged at query time. Powerful, but the logic is written twice.

Kappa architecture

Uses only a streaming path and reprocesses history by replaying a retained event log.

Kappa architecture All data is written to one durable event log such as Kafka. A single stream processor produces the results, and to recompute history the log is replayed from the beginning through a new version of the same processor. Incomingdata Event logKafka, retained Streamprocessor Views replay to reprocess
One streaming path only: data lands in a replayable log, and history is recomputed by replaying the log through the stream processor. Simpler than Lambda, since the logic exists once.

5. Advantages and Disadvantages

Batch processing

  • Advantages: simple to reason about, efficient for huge volumes, easy to rerun, and cheap because clusters run only when needed.
  • Disadvantages: results are hours old, and a failed nightly job delays everything that depends on it.

Stream processing

  • Advantages: results in seconds, continuous insight, and immediate reactions such as alerts or fraud blocks.
  • Disadvantages: always-on infrastructure, harder handling of state, late events and exactly-once behaviour, and more complex testing.

6. Batch vs Stream Processing Comparison

AspectBatch processingStream processing
InputBounded data setUnbounded, continuous events
LatencyMinutes to hoursMilliseconds to seconds
ProcessingWhole data set per runEach event or micro-batch as it arrives
StateRecomputed each runKept continuously and checkpointed
Typical toolsApache Spark, Hadoop MapReduce, SQL jobsApache Flink, Kafka Streams, Spark Structured Streaming
Typical resultsReports, billing, model trainingAlerts, dashboards, fraud detection
AspectLambdaKappa
Processing pathsBatch layer and speed layerStreaming only
Code to maintainTwo versions of the logicOne version
Reprocessing historyRerun the batch layerReplay the event log

7. Real-World Use Cases

  • Batch: end-of-day billing, payroll, monthly reports, data-warehouse loads and machine-learning model training.
  • Stream: fraud detection on card payments, live delivery tracking, monitoring and alerting, and real-time recommendations.
  • Both: an e-commerce platform streams orders for live dashboards and stock alerts, and runs nightly batch jobs for accurate finance reports.
  • Microservices: services publish events to Kafka; stream processors build read models and analytics without querying the services directly.

8. Best Practices

  • Start from the latency requirement: choose streaming only when results must be fresh within seconds or minutes.
  • Design for replay: keep raw events in a durable log or data lake so results can be recomputed after bugs.
  • Make consumers idempotent: even with exactly-once processing inside the framework, external side effects can repeat.
  • Use event time and watermarks for correct results when events arrive late or out of order.
  • Monitor lag: consumer lag in a stream, and run time and failures in batch jobs, are the key health signals.
  • Secure the data: encrypt data in transit and at rest, and mask personal data before it reaches analytics systems.

9. Summary

  • Batch processing handles bounded data on a schedule; stream processing handles unbounded data continuously.
  • Batch is simpler and cheaper; streaming gives results in seconds.
  • Streams are aggregated over tumbling, hopping or session windows.
  • Watermarks handle late events, and checkpoints support exactly-once results.
  • Lambda combines batch and stream paths; Kappa uses a replayable stream only.

Next, see how services coordinate long-running business transactions with the Saga Pattern.