What you will learn
- When streaming beats batch and when it does not
- How windows, watermarks, and late data shape results
- The build order from ingestion to live aggregates
- Why exactly-once semantics are harder than they sound
What this project is
Batch analytics answers what happened yesterday. Streaming answers what is happening now, while there is still time to act: a payment spike, a failing deployment, a surge in signups from one campaign. That speed changes the product, because the same events that once fed reports can now trigger alerts and actions.
Speed costs complexity. Events arrive out of order, late, duplicated, or not at all, and a streaming system must decide what counts, when a window closes, and how to correct results when late data arrives. Those decisions are the platform, more than any single tool choice.
Core features
- Event ingestion with durable logs, schema checks, and replay from any point.
- Stream processing with windows, aggregations, and joins against reference data.
- Watermark and late-data handling with stated rules for corrections.
- Live aggregate stores serving dashboards and alerts at low latency.
- Exactly-once or clearly documented at-least-once delivery per pipeline, never assumed.
- Replay and backfill, so a fixed pipeline can recompute history correctly.
How it is built
- Ingestion first. A durable event log with schemas and replay, before any processing exists.
- One live aggregate second, proving windowing and latency on a real use case.
- More pipelines third, reusing ingestion, processing, and storage patterns.
- Alerting and actions fourth, once aggregates are trusted.
- Scale and governance last. Partitioning, quotas, and ownership for many producers.
Technology considerations
The event log is the foundation: durable, ordered, replayable storage that every downstream job reads from. Processing should be stateless where possible, with state kept in managed stores that survive restarts. Time semantics deserve explicit design, since event time and processing time diverge the moment a device goes offline. Streaming costs scale with event volume and retention; confirm with a quote.
- A durable, replayable event log as the ingestion layer.
- Stream processors with explicit window and watermark rules.
- Low-latency aggregate stores for dashboards and alerts.
- Schema registry and compatibility checks on every event type.
Challenges to plan for
- Late and out-of-order events. Watermarks and correction rules must be stated, or results quietly shift after publication.
- Duplicate delivery. At-least-once systems need idempotent writes, or counts inflate.
- State growth. Unbounded windows and joins grow state forever, so retention and compaction rules matter.
- Debugging live pipelines. A wrong aggregate needs replay and per-event tracing, not log guessing.
A worked example
A food delivery company watches live order volume during a major sporting event, when a sudden surge in one city could mean either genuine demand or a failing payment provider. Its streaming platform ingests order events into a durable log, aggregates orders per city in five-minute windows, and serves the live counts to an operations dashboard and to an alert that fires when a city's error rate crosses a threshold.
Mid-event, the alert fires for one city: orders are up, but payment failures are up faster. The operations team sees from the live aggregates that failures cluster on one payment method, contacts the provider, and reroutes that method within twenty minutes, while the rest of the country keeps ordering. Late events from phones that briefly lost connectivity arrive minutes later and are handled by the platform's watermark rules, correcting the affected windows with a note rather than silently shifting published numbers. A nightly batch would have reported the incident the next morning, long after the revenue was lost.
Who it is for, and its limits
This fits teams acting on live data: fraud, operations, product usage, and monitoring, where minutes matter.
It does not fit reporting that stakeholders read daily, where batch is simpler, cheaper, and easier to correct. It also does not fix event quality; bad events stream as faithfully as good ones.
Frequently asked questions
When is streaming worth it over batch?
When a decision or alert loses value within minutes, such as fraud blocking or incident response. Daily reporting does not need it.
What is a watermark?
A signal that event data up to a certain time is probably complete, letting windows close and results publish, with late data handled by stated rules.
What does exactly-once mean in streaming?
That each event affects results once, even across failures and retries. It requires idempotent sinks and careful state handling.
Can streaming results be corrected later?
Yes, with late-data rules and replay: reprocess the affected window and publish corrected aggregates with a note.
How long should events be retained?
Long enough for replay and debugging, balanced against storage cost. Retention policy belongs in the platform design, not as an afterthought.
Why not just run batch jobs more often?
Because some decisions lose value within minutes, such as blocking a failing payment method. Frequent batch still lags, and streaming adds complexity only where that speed pays for itself.