Skip to content
EncryptTechBD
Home Industries Our Work Blog Pricing About Contact Discuss Your Project
বাংলা (BN)
Software Engineering 6 min read

Event-driven architecture problems and fixes

Diagnose common event-driven architecture problems in business systems: duplicates, ordering, lag, schema drift, poison messages and replay side effects.

E
EncryptTechBD · 0 comments
Developer working at dual monitors in an office
Photo: Christina Morillo / StockSnap

An event-driven system looks clean on a diagram: one service publishes an order event, another updates stock, a third sends a notification and a dashboard reflects the change. In production, the same order is processed twice, the notification arrives before the order is saved, and a consumer falls hours behind without anyone noticing.

These problems are normal. Event-driven architecture trades a simple request-response flow for decoupling, and that trade introduces failure modes the team must design for. This guide lists the symptoms, likely causes, fixes and a diagnostic sequence for business systems such as order management, payments and inventory. For a business in Bangladesh, these failures often surface first in order processing and stock updates, where a small team cannot watch every queue.

How the system is supposed to work

A producer writes an event to a broker. One or more consumers read it, update their own state and acknowledge it. Most brokers provide at-least-once delivery, which means duplicates are possible. Ordering is usually guaranteed only within a partition or a key, not across the whole topic. The design must therefore assume duplicates, out-of-order delivery and temporary failure.

Problem 1: the same event is processed twice

Symptoms: duplicate invoices, double stock deductions, two confirmation emails, duplicate loyalty points.

Causes: a consumer processed the event but crashed before acknowledging it; a retry policy resends after a timeout; a producer publishes on every save instead of on a state change; two consumers write to the same record without coordination.

Fixes: make consumers idempotent. Store a processed-event identifier and reject repeats, or use an upsert keyed by the business identifier. Give each event a stable ID at the producer. For financial records, use a unique constraint rather than a read-then-write check, which fails under concurrency.

Problem 2: events arrive out of order

Symptoms: a shipment notice before the order confirmation, a cancellation processed before the original order, a status that flips back to an earlier value.

Causes: multiple partitions, retries with delays, parallel consumers, or events published from different services without a shared sequence.

Fixes: choose a partition key that matches the business entity, such as the order ID, so related events stay in order. Include a version or sequence number and reject stale updates. Design consumers to be state-aware: apply a transition only if it is valid from the current state. Where ordering cannot be guaranteed, make the workflow tolerant of late events.

Problem 3: consumers fall behind

Symptoms: the dashboard is stale, notifications arrive late, stock updates lag after a promotion, consumer lag grows steadily.

Causes: a slow downstream API, a database lock, insufficient consumer capacity, a large batch of backfill events, or a retry storm caused by one bad message.

Fixes: measure lag per consumer and alert on a threshold that matters to the business, not just a technical number. Scale consumers or partitions, batch where appropriate and add backpressure. Isolate slow downstream calls with timeouts and circuit breakers. If a backlog is expected, tell the business and prioritise the events that affect customers.

Problem 4: a poison message blocks the queue

Symptoms: one event fails repeatedly, the consumer stops making progress, logs fill with the same error.

Causes: malformed data, a missing reference record, a schema change the consumer does not understand, or a bug triggered by one edge case.

Fixes: retry a limited number of times with backoff, then move the event to a dead-letter queue with the original payload and error context. Alert the owner, provide a way to inspect and replay, and never let a single message stop the whole consumer. Fix the data or the code before replaying.

Problem 5: schema drift breaks consumers

Symptoms: a consumer fails after a producer deploy, a field is missing, a date format changes, a numeric value arrives as a string.

Causes: no contract between producer and consumer, no compatibility check in the pipeline, or a shared database schema treated as an event contract.

Fixes: define event schemas explicitly and keep them in a registry or a versioned repository. Use backward-compatible changes: add optional fields, do not rename or remove fields without a migration plan. Run contract tests in the producer pipeline. Record the schema version in every event and monitor for rejected versions.

Problem 6: events are lost between the database and the broker

Symptoms: an order exists in the database but no downstream event was published; a payment is captured but the fulfilment system never hears about it.

Causes: the application writes to the database and publishes to the broker as two separate operations. If one succeeds and the other fails, the system is inconsistent.

Fixes: use the transactional outbox pattern: write the event to an outbox table in the same database transaction, then publish from the outbox with a relay. Alternatively, use change data capture from the database log. Avoid publishing inside a long transaction without an outbox, and make the relay idempotent.

Problem 7: replaying events causes side effects

Symptoms: a replay sends old emails, reopens closed orders, duplicates payments or overwrites newer data.

Causes: consumers perform external actions without checking whether the event was already applied, or the replay includes events outside the intended window.

Fixes: separate event handling from external side effects. Mark notifications and payments as actions that require a current-state check. Replay into a staging environment first, use a time window and a dry-run mode, and record the replay range and operator.

Diagnostic sequence

  1. Confirm the symptom with a specific business case, not a general report.
  2. Find the event ID and trace it across producer, broker, consumer and downstream systems.
  3. Check consumer lag and the dead-letter queue before changing any code.
  4. Compare the event schema with the consumer's expected version.
  5. Check recent deploys, configuration changes and downstream incidents.
  6. Reproduce in staging with the same payload and the same consumer version.
  7. Fix the cause, add a regression test and replay only the affected range.
  8. Record the incident and update the alert threshold or runbook.

Common mistakes

  • Assuming exactly-once delivery from the broker.
  • Using a random partition key and expecting global ordering.
  • Hiding retries and failures in logs nobody reads.
  • Letting each consumer define its own interpretation of an event.
  • Replaying a full topic to fix one bad hour of data.

Metrics

Track consumer lag, duplicate-processing rate, dead-letter depth, event-to-consumer latency, schema rejection count and replay duration. Add a business measure such as orders stuck in a pending state, because technical metrics alone can look healthy while customers wait.

Where this fits in our work

Event-driven design is an integration decision as much as a software decision. If you need help reviewing or building the flow, start with Custom Software & Integrations and review the engagement options on our pricing page. For the testing side of the same problem, read API integration testing problems and fixes.

Sources

Services that deliver this

Want this implemented properly?

Pricing →

Every engagement starts with a written scope. If a requirement is outside it, we quote it separately before starting.

Responses (0)

No comments yet. Add the first one below.

Leave a response

Your email address will not be published. Required fields are marked *.

Keep reading

More field notes