An event-driven system looks clean on a diagram: one service publishes an order event, another updates stock, a third sends a notification and a dashboard reflects the change. In production, the same order is processed twice, the notification arrives before the order is saved, and a consumer falls hours behind without anyone noticing.
These problems are normal. Event-driven architecture trades a simple request-response flow for decoupling, and that trade introduces failure modes the team must design for. This guide lists the symptoms, likely causes, fixes and a diagnostic sequence for business systems such as order management, payments and inventory. For a business in Bangladesh, these failures often surface first in order processing and stock updates, where a small team cannot watch every queue.
How the system is supposed to work
A producer writes an event to a broker. One or more consumers read it, update their own state and acknowledge it. Most brokers provide at-least-once delivery, which means duplicates are possible. Ordering is usually guaranteed only within a partition or a key, not across the whole topic. The design must therefore assume duplicates, out-of-order delivery and temporary failure.
Problem 1: the same event is processed twice
Symptoms: duplicate invoices, double stock deductions, two confirmation emails, duplicate loyalty points.
Causes: a consumer processed the event but crashed before acknowledging it; a retry policy resends after a timeout; a producer publishes on every save instead of on a state change; two consumers write to the same record without coordination.
Fixes: make consumers idempotent. Store a processed-event identifier and reject repeats, or use an upsert keyed by the business identifier. Give each event a stable ID at the producer. For financial records, use a unique constraint rather than a read-then-write check, which fails under concurrency.
Problem 2: events arrive out of order
Symptoms: a shipment notice before the order confirmation, a cancellation processed before the original order, a status that flips back to an earlier value.
Causes: multiple partitions, retries with delays, parallel consumers, or events published from different services without a shared sequence.
Fixes: choose a partition key that matches the business entity, such as the order ID, so related events stay in order. Include a version or sequence number and reject stale updates. Design consumers to be state-aware: apply a transition only if it is valid from the current state. Where ordering cannot be guaranteed, make the workflow tolerant of late events.
Problem 3: consumers fall behind
Symptoms: the dashboard is stale, notifications arrive late, stock updates lag after a promotion, consumer lag grows steadily.
Causes: a slow downstream API, a database lock, insufficient consumer capacity, a large batch of backfill events, or a retry storm caused by one bad message.
Fixes: measure lag per consumer and alert on a threshold that matters to the business, not just a technical number. Scale consumers or partitions, batch where appropriate and add backpressure. Isolate slow downstream calls with timeouts and circuit breakers. If a backlog is expected, tell the business and prioritise the events that affect customers.
Problem 4: a poison message blocks the queue
Symptoms: one event fails repeatedly, the consumer stops making progress, logs fill with the same error.
Causes: malformed data, a missing reference record, a schema change the consumer does not understand, or a bug triggered by one edge case.
Fixes: retry a limited number of times with backoff, then move the event to a dead-letter queue with the original payload and error context. Alert the owner, provide a way to inspect and replay, and never let a single message stop the whole consumer. Fix the data or the code before replaying.
Problem 5: schema drift breaks consumers
Symptoms: a consumer fails after a producer deploy, a field is missing, a date format changes, a numeric value arrives as a string.
Causes: no contract between producer and consumer, no compatibility check in the pipeline, or a shared database schema treated as an event contract.
Fixes: define event schemas explicitly and keep them in a registry or a versioned repository. Use backward-compatible changes: add optional fields, do not rename or remove fields without a migration plan. Run contract tests in the producer pipeline. Record the schema version in every event and monitor for rejected versions.
Problem 6: events are lost between the database and the broker
Symptoms: an order exists in the database but no downstream event was published; a payment is captured but the fulfilment system never hears about it.
Causes: the application writes to the database and publishes to the broker as two separate operations. If one succeeds and the other fails, the system is inconsistent.
Fixes: use the transactional outbox pattern: write the event to an outbox table in the same database transaction, then publish from the outbox with a relay. Alternatively, use change data capture from the database log. Avoid publishing inside a long transaction without an outbox, and make the relay idempotent.
Problem 7: replaying events causes side effects
Symptoms: a replay sends old emails, reopens closed orders, duplicates payments or overwrites newer data.
Causes: consumers perform external actions without checking whether the event was already applied, or the replay includes events outside the intended window.
Fixes: separate event handling from external side effects. Mark notifications and payments as actions that require a current-state check. Replay into a staging environment first, use a time window and a dry-run mode, and record the replay range and operator.
Diagnostic sequence
- Confirm the symptom with a specific business case, not a general report.
- Find the event ID and trace it across producer, broker, consumer and downstream systems.
- Check consumer lag and the dead-letter queue before changing any code.
- Compare the event schema with the consumer's expected version.
- Check recent deploys, configuration changes and downstream incidents.
- Reproduce in staging with the same payload and the same consumer version.
- Fix the cause, add a regression test and replay only the affected range.
- Record the incident and update the alert threshold or runbook.
Common mistakes
- Assuming exactly-once delivery from the broker.
- Using a random partition key and expecting global ordering.
- Hiding retries and failures in logs nobody reads.
- Letting each consumer define its own interpretation of an event.
- Replaying a full topic to fix one bad hour of data.
Metrics
Track consumer lag, duplicate-processing rate, dead-letter depth, event-to-consumer latency, schema rejection count and replay duration. Add a business measure such as orders stuck in a pending state, because technical metrics alone can look healthy while customers wait.
Where this fits in our work
Event-driven design is an integration decision as much as a software decision. If you need help reviewing or building the flow, start with Custom Software & Integrations and review the engagement options on our pricing page. For the testing side of the same problem, read API integration testing problems and fixes.
Sources
- Confluent, Event-driven architecture, accessed 17 November 2026.
- Amazon Web Services, Event-driven architecture, accessed 17 November 2026.
- Enterprise Integration Patterns, accessed 17 November 2026.
- CloudEvents specification, accessed 17 November 2026.
- AsyncAPI documentation, accessed 17 November 2026.
Responses (0)
No comments yet. Add the first one below.
Leave a response
Your email address will not be published. Required fields are marked *.