Real-time is a promise about the age of data. A dispatch team may need a two-minute view of every order. A finance reconciliation may need an overnight close. Buying a streaming platform before agreeing the number is how projects acquire complexity they cannot operate. This guide is written for the buyer who has to turn a vague request for live dashboards into a scope, a shortlist and an acceptance test.
Write the requirement as one sentence
A useful requirement sounds like this: "The dispatch dashboard must show the status of every order within two minutes of the warehouse scan, including events that arrive up to ten minutes late." It contains the decision, the freshness target and the tolerance for late data. Add the consequence of showing stale or wrong data, the history the team needs and the people who will read it.
- Decision: what someone does differently when the number is fresh.
- Freshness: the maximum age of data at the moment of use, not the speed of the pipeline in a demo.
- Late and missing events: how long the system waits, what it does with a late event and how it signals a gap.
- Correctness: whether duplicate, out-of-order or restated events are acceptable.
- History: how far back users can query and how quickly an old period can be rebuilt.
- Consumers: dashboards, APIs, alerts, downstream systems and their separate latency needs.
- Access and residency: who can see which rows, and where the data may live.
What belongs in the RFP
- Sources and event types, with average and peak volume.
- Event time versus processing time, time zone handling and clock assumptions.
- Ordering guarantees per entity, deduplication keys and the acceptable duplicate rate.
- Schema ownership, versioning, compatibility rules and the process for a breaking change.
- Backfill and replay: how a new metric is computed over history and how a correction is published.
- Query patterns and concurrency, including the busiest dashboard and the largest export.
- Retention by layer: hot serving data, historical storage and archive.
- Security, access control, audit and data residency.
- Cost at current volume, at double and at ten times, with backfill included.
- The support boundary: what the vendor operates, what your team operates and who is called at 2am.
Build, buy or managed
Build when latency and control are core to the product and a platform team can own the pipeline. Buy when the pattern is standard, time matters and the vendor's operating model fits your team. Managed when you need the outcome without owning on-call, upgrades and capacity planning. Hybrid is common: managed transport, in-house processing and a warehouse for history.
Five architectures you can buy
| Architecture | Typical latency | Build effort | Flexibility | Cost shape | Fits when |
|---|---|---|---|---|---|
| Warehouse micro-batch | Minutes | Low | Medium | Compute and storage | Analytical dashboards and frequent refreshes are enough |
| Managed streaming plus serving store | Seconds | Medium | High | Ingest, compute and storage | Operational dashboards with a small platform team |
| Event bus plus stream processing | Seconds or less | High | Highest | Always-on compute and operations | Complex event logic, replay and multiple consumers |
| Operational database plus change capture | Sub-second to seconds | Medium | Medium | Database, compute and connectors | Current-state views close to the transaction system |
| Custom end-to-end | As designed | Highest | Highest | Everything, including people | The pipeline is a differentiated product |
Vendor categories and what to ask
Cloud platforms offer managed queues, stream processing and storage in one account. Ask how the pricing behaves at peak, how backfills are charged and whether the serving layer is included. Specialist streaming vendors offer deeper guarantees and tooling. Ask for evidence of replay, schema evolution and multi-region behaviour, not a slide. Warehouse vendors increasingly support micro-batch and streaming ingestion. Ask about latency under concurrency, not in a benchmark. Analytics databases are strong serving layers but not complete pipelines. Systems integrators can deliver the whole scope, but the operating knowledge must transfer to your team.
Scorecard
| Criterion | Weight | Evidence to request |
|---|---|---|
| Freshness proof | 20 | A production-like test at your peak volume, with measured percentiles |
| Correctness and replay | 15 | Duplicate, late and out-of-order events handled in a live demonstration |
| Schema evolution | 10 | A breaking change and a new field handled without a full rebuild |
| Observability | 10 | Lag, missing events, failed sinks and a runbook for each |
| Security and residency | 10 | Access model, audit trail, encryption and data location |
| Integration | 10 | Named connectors and an estimate for the sources you actually use |
| Cost predictability | 10 | A quote at current, double and ten times volume, including backfill |
| Support | 10 | Response times, escalation path and who owns the pipeline |
| Exit | 5 | Open formats, export and a tested migration path |
Score each criterion from zero to three and multiply by the weight. Attach the evidence to the score.
Pricing questions
- What is charged: ingestion, compute hours, storage, egress, sinks, queries or seats?
- Is there a minimum commit, and how does overage work?
- How is a backfill charged when a pipeline is rebuilt over a year of history?
- What does retention cost at thirty days, one year and seven years?
- Are connectors, private networking and support included or separate?
- Which currency applies, and who carries the exchange-rate risk?
- What happens to the price when volume doubles or a second region is added?
Model the total at current volume and at the volume you expect after a successful campaign.
Pilot and acceptance tests
Run a four to six week pilot on one decision and one source. Write the acceptance tests before the pilot starts, then run them with the vendor watching:
- Send the same event twice and confirm the output is counted once.
- Send an event after the allowed lateness window and confirm the system signals a correction or a gap.
- Add a field, then change its type, and confirm the consumers still work.
- Rebuild one metric over three months of history and compare it with the source.
- Burst to peak volume and measure freshness, lag and error rate.
- Disable a sink and confirm the pipeline recovers without silent data loss.
- Restart a processing node and confirm state, offsets and duplicates behave as documented.
- Check that a user without permission cannot read restricted rows.
Contract and exit
Own the schema and the event definitions even if the vendor hosts the pipeline. Require an open export format, a documented retention and deletion process, and a transition period at the end of the contract. If the vendor supplies the serving layer, agree what happens to dashboards and queries on exit. Price protection and a clear support boundary are part of the architecture, not procurement decoration.
Red flags
- A latency claim from a demo with no production volume or concurrency test.
- No replay or backfill path, or a backfill priced as a separate project after signature.
- A proprietary event format with no export.
- "Exactly once" claimed without an explanation of duplicates, retries and sink behaviour.
- Cost quoted only at current volume, with no peak or backfill scenario.
- No named owner for pipeline monitoring and on-call.
- Dashboards that hide freshness, so users assume every number is live.
If you need the pipeline, serving layer and operating routine built together, start with Business Automation. The pricing page shows how a data pipeline engagement is scoped. For the component-level view, read the real-time analytics architecture explainer.
Sources
- Apache Software Foundation, Apache Kafka documentation, accessed 17 November 2026.
- Apache Software Foundation, Apache Flink documentation, accessed 17 November 2026.
- Amazon Web Services, Analytics Lens, Well-Architected Framework, accessed 17 November 2026.
Responses (0)
No comments yet. Add the first one below.
Leave a response
Your email address will not be published. Required fields are marked *.