Skip to content
EncryptTechBD
Home Industries Our Work Blog Pricing About Contact Discuss Your Project
বাংলা (BN)
Data & Analytics 6 min read

API data integration architecture: a practical reference

How a reliable API integration is architected: contracts, authentication, retries, idempotency, reconciliation, observability and failure handling.

E
EncryptTechBD · 0 comments
Tablet showing analytics reports beside printed charts
Photo: WDnet Studio / StockSnap

An integration is a product with users, failure modes and an owner. It is not a script that copies records from one system to another until someone notices the numbers disagree. The architecture exists to make those disagreements visible, recoverable and rare.

This explainer describes a reference architecture for API-based data integration. It is for the manager or analyst who must review a design, ask sensible questions and decide whether the connection is robust enough for daily operations.

The integration flow

  1. Detect a change. The source system emits an event, exposes a webhook, or is polled on a schedule for records modified since the last run.
  2. Validate the payload. Required fields, data types, identifiers and business rules are checked before anything is written elsewhere.
  3. Transform to a shared model. Source-specific names and codes are mapped to a canonical customer, order, invoice or product representation.
  4. Route. The message is sent to one or more destinations, or parked in a queue when a destination is unavailable.
  5. Deliver with an idempotency key. The destination can recognise a repeated request and avoid creating a duplicate record.
  6. Acknowledge. The destination returns a clear success or failure response, including a machine-readable error code.
  7. Reconcile. A scheduled comparison checks counts, totals and statuses between systems, independent of the message path.
  8. Alert and replay. Failed messages go to a dead-letter queue with enough context to fix and replay them without guesswork.

Five integration styles

  • Request and response: one system calls another and waits. Simple, but brittle when the destination is slow or unavailable.
  • Webhook: the source calls your endpoint when something changes. Efficient, but you need signature verification, retries and a public endpoint that stays available.
  • Event stream: changes are published to a broker and consumed by several systems. Good for scale and decoupling, but it adds operational complexity.
  • Batch file or scheduled export: periodic exchange of CSV or structured files. Underrated for reconciliation and historical loads; poor for near-real-time needs.
  • Shared database: two applications read the same tables. Fast to build and expensive to change; it couples releases and bypasses business rules. Treat it as a legacy pattern, not a target.

Layers of an integration architecture

Lay the design out in layers so responsibilities are clear.

  • Source and destination adapters: one per system, handling its authentication, pagination, rate limits and quirks. The rest of the architecture should not know these details.
  • API gateway or ingress: terminates TLS, verifies signatures, applies rate limits and routes requests to the right service.
  • Orchestration service: owns the workflow, retries, ordering and error policy. It should be stateless where possible, with state stored in a durable queue or database.
  • Canonical model: the shared definitions for entities and statuses, plus mapping tables for source-specific codes.
  • Queue and dead-letter store: buffers bursts, isolates failures and preserves messages for replay.
  • Observability: structured logs, metrics, traces and business-level reconciliation reports.
  • Secrets and configuration: credentials in a managed secret store, rotated on a schedule, never in code or a spreadsheet.

Contracts and error behaviour

Write the contract before writing the connector. An OpenAPI description is useful even when the counterparty does not publish one, because it forces decisions about fields, required values and error responses.

Use HTTP status codes as intended, and return a problem document with a stable code, a human-readable message and a correlation identifier. RFC 9457 defines a standard shape for this; adopting it makes errors easier to log and support. Pagination, filtering and rate limits should be documented, not discovered in production.

Version explicitly. Adding an optional field is usually safe; renaming a field, changing a type or altering a status meaning is not. Publish a deprecation window and test the old and new versions in parallel before switching.

Authentication and authorisation

For server-to-server calls, OAuth 2.0 client credentials with narrowly scoped permissions is a common baseline. Prefer short-lived tokens, store client secrets in a secret manager, and rotate them without a deployment. Where both sides support it, mutual TLS adds a strong second factor.

Authorisation is not just "can this system call the API". It is "can this system read this customer's invoice". Apply the same data-access rules as the user-facing application, and log who accessed what. An integration account with broad administrator rights is a single compromise away from a full data export.

Reliability patterns that earn their place

  • Idempotency keys: the sender generates a unique key per logical operation; the receiver stores it and returns the original result on a repeat.
  • Retries with exponential backoff and jitter: retry transient failures, not validation errors. Cap attempts and move to a dead-letter queue.
  • Timeouts: set them on every call, below the caller's own timeout, so failures surface quickly.
  • Ordering: do not assume messages arrive in order. Use version numbers or status transitions to reject stale updates.
  • Circuit breaking: stop calling a failing destination for a cooling-off period instead of adding load.
  • Replay tooling: a documented way to reprocess a failed message after the cause is fixed.

"Exactly once" is usually a marketing phrase. Design for at-least-once delivery plus idempotent processing, then verify with reconciliation.

Reconciliation and observability

Message-level success does not prove business correctness. Add a daily check: orders created versus orders received, invoice totals versus source totals, stock movements versus adjustments. Report the differences, not just the count of successful calls.

Use a correlation identifier that travels from the first request to the final write. Log the source, destination, record identifier, outcome and duration. Alert on lag, error rate and reconciliation variance, and route alerts to a person who can act. A dashboard nobody watches is decoration.

A worked example: order to invoice

An online store publishes an order-created webhook. The integration verifies the signature, validates the payload, maps the customer and product identifiers, and creates an invoice in the accounting system using the order number as the idempotency key. If accounting is unavailable, the message waits in a queue and retries. If the customer record is missing, the message goes to a dead-letter queue with the reason.

Each night, a reconciliation job compares order totals, invoice totals and payment status. A mismatch creates a ticket with the order numbers attached; that report catches more real problems than API monitoring.

When not to build an API integration

If the process is broken, connecting the two systems will automate the breakage. Fix the ownership, the status definitions and the approval rules first. If volumes are low and a daily export is sufficient, a scheduled file exchange with validation may be cheaper and easier to support. If the counterparty has no documented API, do not build around an unofficial endpoint; agree on a supported method or a manual checkpoint.

Integration engineering sits within Custom Software & Integrations. The pricing page explains how discovery, build and support are separated. For the API design side, read the REST API design checklist.

Sources

Services that deliver this

Want this implemented properly?

Pricing →

Every engagement starts with a written scope. If a requirement is outside it, we quote it separately before starting.

Responses (0)

No comments yet. Add the first one below.

Leave a response

Your email address will not be published. Required fields are marked *.

Keep reading

More field notes