Skip to content
EncryptTechBD
Home Industries Our Work Blog Pricing About Contact Discuss Your Project
বাংলা (BN)
Data & Analytics 6 min read

Real-time analytics architecture: a buying guide

Buy real-time analytics on evidence: freshness requirements, architecture options, vendor scorecard, acceptance tests and exit terms.

E
EncryptTechBD · 0 comments
Tablet displaying a stock market graph
Photo: Burak Kebapci / StockSnap

Real-time is a promise about the age of data. A dispatch team may need a two-minute view of every order. A finance reconciliation may need an overnight close. Buying a streaming platform before agreeing the number is how projects acquire complexity they cannot operate. This guide is written for the buyer who has to turn a vague request for live dashboards into a scope, a shortlist and an acceptance test.

Write the requirement as one sentence

A useful requirement sounds like this: "The dispatch dashboard must show the status of every order within two minutes of the warehouse scan, including events that arrive up to ten minutes late." It contains the decision, the freshness target and the tolerance for late data. Add the consequence of showing stale or wrong data, the history the team needs and the people who will read it.

  • Decision: what someone does differently when the number is fresh.
  • Freshness: the maximum age of data at the moment of use, not the speed of the pipeline in a demo.
  • Late and missing events: how long the system waits, what it does with a late event and how it signals a gap.
  • Correctness: whether duplicate, out-of-order or restated events are acceptable.
  • History: how far back users can query and how quickly an old period can be rebuilt.
  • Consumers: dashboards, APIs, alerts, downstream systems and their separate latency needs.
  • Access and residency: who can see which rows, and where the data may live.

What belongs in the RFP

  • Sources and event types, with average and peak volume.
  • Event time versus processing time, time zone handling and clock assumptions.
  • Ordering guarantees per entity, deduplication keys and the acceptable duplicate rate.
  • Schema ownership, versioning, compatibility rules and the process for a breaking change.
  • Backfill and replay: how a new metric is computed over history and how a correction is published.
  • Query patterns and concurrency, including the busiest dashboard and the largest export.
  • Retention by layer: hot serving data, historical storage and archive.
  • Security, access control, audit and data residency.
  • Cost at current volume, at double and at ten times, with backfill included.
  • The support boundary: what the vendor operates, what your team operates and who is called at 2am.

Build, buy or managed

Build when latency and control are core to the product and a platform team can own the pipeline. Buy when the pattern is standard, time matters and the vendor's operating model fits your team. Managed when you need the outcome without owning on-call, upgrades and capacity planning. Hybrid is common: managed transport, in-house processing and a warehouse for history.

Five architectures you can buy

ArchitectureTypical latencyBuild effortFlexibilityCost shapeFits when
Warehouse micro-batchMinutesLowMediumCompute and storageAnalytical dashboards and frequent refreshes are enough
Managed streaming plus serving storeSecondsMediumHighIngest, compute and storageOperational dashboards with a small platform team
Event bus plus stream processingSeconds or lessHighHighestAlways-on compute and operationsComplex event logic, replay and multiple consumers
Operational database plus change captureSub-second to secondsMediumMediumDatabase, compute and connectorsCurrent-state views close to the transaction system
Custom end-to-endAs designedHighestHighestEverything, including peopleThe pipeline is a differentiated product

Vendor categories and what to ask

Cloud platforms offer managed queues, stream processing and storage in one account. Ask how the pricing behaves at peak, how backfills are charged and whether the serving layer is included. Specialist streaming vendors offer deeper guarantees and tooling. Ask for evidence of replay, schema evolution and multi-region behaviour, not a slide. Warehouse vendors increasingly support micro-batch and streaming ingestion. Ask about latency under concurrency, not in a benchmark. Analytics databases are strong serving layers but not complete pipelines. Systems integrators can deliver the whole scope, but the operating knowledge must transfer to your team.

Scorecard

CriterionWeightEvidence to request
Freshness proof20A production-like test at your peak volume, with measured percentiles
Correctness and replay15Duplicate, late and out-of-order events handled in a live demonstration
Schema evolution10A breaking change and a new field handled without a full rebuild
Observability10Lag, missing events, failed sinks and a runbook for each
Security and residency10Access model, audit trail, encryption and data location
Integration10Named connectors and an estimate for the sources you actually use
Cost predictability10A quote at current, double and ten times volume, including backfill
Support10Response times, escalation path and who owns the pipeline
Exit5Open formats, export and a tested migration path

Score each criterion from zero to three and multiply by the weight. Attach the evidence to the score.

Pricing questions

  • What is charged: ingestion, compute hours, storage, egress, sinks, queries or seats?
  • Is there a minimum commit, and how does overage work?
  • How is a backfill charged when a pipeline is rebuilt over a year of history?
  • What does retention cost at thirty days, one year and seven years?
  • Are connectors, private networking and support included or separate?
  • Which currency applies, and who carries the exchange-rate risk?
  • What happens to the price when volume doubles or a second region is added?

Model the total at current volume and at the volume you expect after a successful campaign.

Pilot and acceptance tests

Run a four to six week pilot on one decision and one source. Write the acceptance tests before the pilot starts, then run them with the vendor watching:

  1. Send the same event twice and confirm the output is counted once.
  2. Send an event after the allowed lateness window and confirm the system signals a correction or a gap.
  3. Add a field, then change its type, and confirm the consumers still work.
  4. Rebuild one metric over three months of history and compare it with the source.
  5. Burst to peak volume and measure freshness, lag and error rate.
  6. Disable a sink and confirm the pipeline recovers without silent data loss.
  7. Restart a processing node and confirm state, offsets and duplicates behave as documented.
  8. Check that a user without permission cannot read restricted rows.

Contract and exit

Own the schema and the event definitions even if the vendor hosts the pipeline. Require an open export format, a documented retention and deletion process, and a transition period at the end of the contract. If the vendor supplies the serving layer, agree what happens to dashboards and queries on exit. Price protection and a clear support boundary are part of the architecture, not procurement decoration.

Red flags

  • A latency claim from a demo with no production volume or concurrency test.
  • No replay or backfill path, or a backfill priced as a separate project after signature.
  • A proprietary event format with no export.
  • "Exactly once" claimed without an explanation of duplicates, retries and sink behaviour.
  • Cost quoted only at current volume, with no peak or backfill scenario.
  • No named owner for pipeline monitoring and on-call.
  • Dashboards that hide freshness, so users assume every number is live.

If you need the pipeline, serving layer and operating routine built together, start with Business Automation. The pricing page shows how a data pipeline engagement is scoped. For the component-level view, read the real-time analytics architecture explainer.

Sources

Services that deliver this

Want this implemented properly?

Pricing →

Every engagement starts with a written scope. If a requirement is outside it, we quote it separately before starting.

Responses (0)

No comments yet. Add the first one below.

Leave a response

Your email address will not be published. Required fields are marked *.

Keep reading

More field notes