Batch Processing vs Stream Processing
An online store produces a constant flow of data: orders, page views, payments, stock changes. Some questions about that data can wait until tomorrow, such as "what were yesterday's total sales?". Others need an answer right now, such as "is this card being used fraudulently?". These two needs lead to two styles of data processing: batch and stream.
This tutorial compares the two styles, explains windows and delivery guarantees that make streaming work, and shows how Lambda and Kappa architectures combine them. It connects to Event-Driven Architecture and Kafka vs RabbitMQ.
1. What Are Batch and Stream Processing?
Batch processing collects data over a period, stores it, and then processes the whole set in one run, usually on a schedule. The input is bounded: the job knows where the data starts and ends. Tools include Apache Spark, Hadoop MapReduce and scheduled SQL jobs.
Stream processing handles data continuously, event by event or in small groups, as soon as it arrives. The input is unbounded: it never ends, so results are produced incrementally. Tools include Apache Flink, Kafka Streams and Spark Structured Streaming, which processes data in small micro-batches.
2. Batch and Stream Pipelines
The same events can feed both pipelines. Many companies stream data into Kafka once and then use it both for real-time processing and for loading the data lake used by batch jobs.
3. How Each Style Processes Data
A batch job
- Events are written to storage all day, for example as files in object storage.
- At 02:00 a scheduler starts the job, which reads the full day of data.
- The job aggregates, joins and cleans the data in parallel across a cluster.
- It writes the results, such as a sales report, and then stops until the next run.
A stream processor
- Each order event is published to a Kafka topic as it happens.
- A stream processor reads events continuously and keeps running state, such as revenue per minute.
- It emits updated results whenever a window closes or an event changes the state.
- It regularly checkpoints its state, so after a failure it resumes from the last checkpoint instead of starting again.
Between the two sits micro-batching, used by Spark Structured Streaming: events are processed in very small batches, by default each one starting as soon as the previous one finishes, which gives near-real-time results with a batch-style programming model.
4. Windows, Time and Delivery Guarantees
Because a stream never ends, aggregations must be computed over windows: finite slices of the stream. The three main window types answer different questions.
Tumbling windows
Consecutive windows of a fixed length that never overlap, such as each calendar minute.
Hopping (sliding) windows
Windows of a fixed length that start at a smaller interval, so they overlap. Some tools, including Kafka Streams, call these hopping windows and use "sliding" for a slightly different, event-driven variant.
Session windows
Windows defined by activity rather than the clock. A session stays open while events keep arriving and closes after a gap of inactivity.
Event time and late data
Events carry the time they happened (event time), which can differ from when they arrive (processing time). Stream processors use watermarks to decide how long to wait for late events before closing a window.
Delivery guarantees
At-most-once may lose events; at-least-once guarantees delivery but can repeat an event after a retry; exactly-once makes each event affect the results once, using checkpoints and transactional writes. Flink and Kafka Streams both offer exactly-once processing within their supported sources and sinks.
Lambda architecture
Runs a batch layer and a speed layer side by side and merges their results.
Kappa architecture
Uses only a streaming path and reprocesses history by replaying a retained event log.
5. Advantages and Disadvantages
Batch processing
- Advantages: simple to reason about, efficient for huge volumes, easy to rerun, and cheap because clusters run only when needed.
- Disadvantages: results are hours old, and a failed nightly job delays everything that depends on it.
Stream processing
- Advantages: results in seconds, continuous insight, and immediate reactions such as alerts or fraud blocks.
- Disadvantages: always-on infrastructure, harder handling of state, late events and exactly-once behaviour, and more complex testing.
6. Batch vs Stream Processing Comparison
| Aspect | Batch processing | Stream processing |
|---|---|---|
| Input | Bounded data set | Unbounded, continuous events |
| Latency | Minutes to hours | Milliseconds to seconds |
| Processing | Whole data set per run | Each event or micro-batch as it arrives |
| State | Recomputed each run | Kept continuously and checkpointed |
| Typical tools | Apache Spark, Hadoop MapReduce, SQL jobs | Apache Flink, Kafka Streams, Spark Structured Streaming |
| Typical results | Reports, billing, model training | Alerts, dashboards, fraud detection |
| Aspect | Lambda | Kappa |
|---|---|---|
| Processing paths | Batch layer and speed layer | Streaming only |
| Code to maintain | Two versions of the logic | One version |
| Reprocessing history | Rerun the batch layer | Replay the event log |
7. Real-World Use Cases
- Batch: end-of-day billing, payroll, monthly reports, data-warehouse loads and machine-learning model training.
- Stream: fraud detection on card payments, live delivery tracking, monitoring and alerting, and real-time recommendations.
- Both: an e-commerce platform streams orders for live dashboards and stock alerts, and runs nightly batch jobs for accurate finance reports.
- Microservices: services publish events to Kafka; stream processors build read models and analytics without querying the services directly.
8. Best Practices
- Start from the latency requirement: choose streaming only when results must be fresh within seconds or minutes.
- Design for replay: keep raw events in a durable log or data lake so results can be recomputed after bugs.
- Make consumers idempotent: even with exactly-once processing inside the framework, external side effects can repeat.
- Use event time and watermarks for correct results when events arrive late or out of order.
- Monitor lag: consumer lag in a stream, and run time and failures in batch jobs, are the key health signals.
- Secure the data: encrypt data in transit and at rest, and mask personal data before it reaches analytics systems.
9. Summary
- Batch processing handles bounded data on a schedule; stream processing handles unbounded data continuously.
- Batch is simpler and cheaper; streaming gives results in seconds.
- Streams are aggregated over tumbling, hopping or session windows.
- Watermarks handle late events, and checkpoints support exactly-once results.
- Lambda combines batch and stream paths; Kappa uses a replayable stream only.
Next, see how services coordinate long-running business transactions with the Saga Pattern.
Batch vs Stream Processing: Top Interview Questions
These are the batch and stream processing questions most often asked in system design interviews, with short answers you can explain in your own words.
Q1. What is the main difference between batch and stream processing?
Batch processing works on a bounded, stored data set in scheduled runs, so results arrive minutes or hours later. Stream processing works on unbounded data as it arrives and produces results within seconds.
Q2. What is the difference between tumbling, hopping and session windows?
Tumbling windows have a fixed size and do not overlap. Hopping windows have a fixed size but start more often than their length, so they overlap. Session windows have no fixed size; they close after a period of inactivity.
Q3. What is a watermark in stream processing?
A watermark is the processor's estimate of how far event time has progressed. When the watermark passes the end of a window, the window is considered complete and its result is emitted; later events are treated as late data.
Q4. How is exactly-once processing achieved?
By combining replayable sources, periodic checkpoints of operator state and source offsets, and transactional or idempotent writes to sinks. After a failure, the job restores the last checkpoint and replays events from the matching offsets, so results are not counted twice.
Q5. Lambda or Kappa: which would you choose?
Kappa is usually preferred today because it keeps one code path and reprocesses history by replaying a retained log. Lambda can still make sense when the batch path relies on tools or data that the streaming system cannot handle.