"Real-time" is not a product you buy. It is a latency target: how old the data on the screen may be before the decision it supports becomes wrong. A dispatch team may need seconds. A monthly management report does not. Defining that target is the first architectural decision, and it determines most of the cost.
This explainer describes the components of a real-time analytics architecture, how they fit together, and where the approach is more trouble than it is worth.
Define the freshness requirement
Write the requirement as a sentence with a number: "the dispatch dashboard must show the status of every order within two minutes of the warehouse scan". Then add the tolerance for late or missing events and the consequence of showing stale data.
- Operational: seconds to a minute, used to act on a customer, stock or delivery event.
- Near real-time: minutes, used for queues, alerts and daily operational decisions.
- Analytical: hours or overnight, used for planning, reconciliation and reporting.
A single dashboard can mix all three. Label each metric with its freshness, or users will assume everything is live.
Batch, micro-batch and streaming
Batch processing runs on a schedule and handles a defined window of data. It is simple, cheap and easy to reprocess. Micro-batch runs every few seconds or minutes and is often enough for operational dashboards. Streaming processes each event as it arrives, with continuous state and low latency.
Streaming is not automatically better. It introduces always-on infrastructure, ordering problems, late events and harder debugging. Choose the simplest model that meets the freshness requirement, and keep batch for history, reconciliation and restatement.
The architecture in six components
1. Event sources and capture
Events come from application code, webhooks, database change streams and device or warehouse scans. Each event needs a stable identifier, a timestamp from the source, a type, a schema version and a partition key. Capture should be durable before the producer is acknowledged.
2. Transport
A log-based broker such as Kafka, or a managed queue, holds events until consumers process them. The choice affects ordering, replay, retention and operational effort. Decide the ordering guarantee per entity — usually per order or per customer — rather than globally.
3. Processing
Stream processing validates, deduplicates, enriches and aggregates events. Windowing decides how late events are handled. A watermark defines when a window is considered complete. State stores keep running totals and last-known status.
4. Storage
Different questions need different stores. An operational store serves current order state; an analytical store or lakehouse keeps history for trends; a time-series store handles metrics. Duplicating data is normal, but each copy needs an owner and a freshness expectation.
5. Serving
Dashboards, APIs and alerts read from a serving layer, often with a cache. The serving layer should expose freshness and last-update time, not pretend the data is instantaneous. Access control belongs here as much as in the source systems.
6. Observability and governance
You need to know when events stop arriving, when lag grows, when duplicates spike and when a schema changes. A schema registry, lineage, data quality checks, retention rules and an on-call owner turn a pipeline into a service.
A worked example: order-to-dispatch
A distributor wants a live view of orders from confirmation to dispatch. The order system emits an event when an order is confirmed, the warehouse emits a pick event and a dispatch event, and the courier emits a delivery event through a webhook.
Each event carries an order identifier and an event identifier. The processor deduplicates on the event identifier, keeps the latest status per order and calculates time in each stage. A late warehouse event is accepted within a defined window and triggers a correction to the dashboard. If no event arrives for an order that should have been picked, a separate alert fires.
The dashboard shows current status, ageing by stage and a count of orders with missing events. The finance team still uses a nightly batch job for revenue reconciliation, because the live view is not the ledger. That separation is deliberate: one system answers "where is the order now?", the other answers "what do we book?".
When real-time is the wrong choice
- The decision is made weekly or monthly, so overnight data is sufficient.
- The event volume is low and a scheduled query finishes in seconds.
- The source system cannot emit reliable events, and building that reliability is not funded.
- Nobody will act on the alert within the freshness window.
- The team has no on-call capacity to operate an always-on pipeline.
In these cases, a batch pipeline with a clear schedule and a data quality report delivers more value at lower risk.
Failure modes to design for
- Duplicate events: retries and at-least-once delivery create repeats; deduplicate with a stable key.
- Late events: define lateness tolerance and a correction path.
- Schema drift: version every event and reject incompatible changes at the producer.
- Backpressure: a slow consumer must not silently lose events; monitor lag and scale or buffer.
- Silent failure: the dashboard looks fine because no events arrived; alert on absence, not only errors.
- Cost creep: always-on compute and retained event history grow without a retention decision.
A design sequence
- Write the freshness requirement and the decision it supports.
- Inventory the events, their owners and their current reliability.
- Choose batch, micro-batch or streaming for each metric, not for the whole platform.
- Build one thin slice from source to screen, including duplicate and late-event handling.
- Test failure: stop a producer, replay a batch, send a duplicate and change a schema.
- Document freshness, ownership and the correction process before expanding scope.
Metrics for the pipeline
- End-to-end latency from event creation to dashboard display.
- Completeness: events received versus events expected, by source.
- Duplicate and rejected event counts.
- Consumer lag and the age of the oldest unprocessed event.
- Reconciliation differences between the live view and the ledger.
- Cost per million events and the share of idle capacity.
Where this fits in our work
If the reporting requirement is real but the data foundations are not ready, start with Custom Software and Integrations. The pricing page explains how discovery and data engineering are scoped. For the reporting layer above it, read how a business KPI dashboard is put together.
Sources
- Apache Kafka documentation, accessed 17 November 2026.
- Debezium documentation, change data capture, accessed 17 November 2026.
- Google, The Site Reliability Workbook, accessed 17 November 2026.
- dbt documentation, accessed 17 November 2026.
Responses (0)
No comments yet. Add the first one below.
Leave a response
Your email address will not be published. Required fields are marked *.