Skip to content
EncryptTechBD
Home Industries Our Work Blog Pricing About Contact Discuss Your Project
বাংলা (BN)
Payments & Reconciliation 6 min read

Payment webhook failures: symptoms, causes and fixes

A diagnostic guide for payments that confirm at the provider but not in your system, duplicate ledger entries, out-of-order events and callbacks that never arrive.

E
EncryptTechBD · 0 comments
Vintage cash register in a banking setting
Photo: vedanti / StockSnap

The customer has a confirmation SMS from the provider. Your order screen still says pending. Support asks the technical team, the technical team asks the provider, and the customer asks for a refund. By the time anyone finds the log line, three more orders have failed the same way.

Webhook problems are rarely mysterious once they are separated into layers. The event either was not sent, did not arrive, arrived but was rejected, or arrived and was processed incorrectly. Each layer has different evidence and a different fix. This guide works through the symptoms in the order a support team is likely to see them.

Start with the four layers

  • Provider: was the event generated, and does the provider show a delivery attempt?
  • Transport: DNS, TLS, firewall, proxy or rate limiting between provider and endpoint.
  • Endpoint: signature verification, request parsing, timeouts and response codes.
  • Ledger: idempotency, ordering and the business rules that decide the final state.

Ask which layer failed before changing code. A fix applied at the wrong layer usually creates a second defect.

Symptom, likely cause and first check

SymptomLikely causeFirst check
Paid at provider, pending in your systemCallback rejected, timed out, or never sentProvider event log and endpoint response codes
One payment, two orders or two creditsDuplicate delivery without an idempotency keyEvent identifier stored with the ledger row
Order marked paid after a refundEvents processed out of orderEvent timestamps versus processing timestamps
Every callback fails after a releaseSignature secret changed or body parsed before verificationRaw body handling and secret version
Callbacks stop during a campaignEndpoint saturation, rate limits or WAF rulesLatency, queue depth and blocked requests
Refund succeeds at provider, not in reportsRefund event not handled, or handled as a new paymentEvent type coverage and state transitions

1. Paid at the provider, pending in your system

Likely cause: the callback was rejected by your endpoint, the response took too long, or the provider never sent it because the URL was wrong or unreachable.

Fix: compare the provider event log with your access log for the same minute. If the provider shows a non-2xx response, read the response body. If there is no request at all, test the endpoint from outside your network. Then add a scheduled status poll that asks the provider for the current state of any payment pending beyond its stated window.

Prevention: acknowledge quickly, do the slow work asynchronously, and never let a downstream timeout turn into a failed acknowledgement. Keep a reconciliation job that does not depend on callbacks at all.

2. Duplicate orders or double credits

Likely cause: the same event was delivered more than once, or the customer retried and both attempts were treated as new orders.

Fix: store the provider event identifier with a unique constraint before applying any business effect. Make the handler idempotent so a replay returns the existing result rather than creating a second ledger entry. Separate payment attempt references from order references.

Prevention: test duplicate delivery deliberately. Send the same event twice in staging and confirm one order, one ledger line and one customer notification.

3. Events arrive out of order

Likely cause: retries and parallel delivery mean an earlier event can arrive after a later one. Your handler applies whatever arrives last.

Fix: store the provider event time and a monotonic sequence where available. Apply transitions only if they move the state forward, and record rejected transitions for review. Treat "paid" and "refunded" as states with allowed paths, not as independent flags.

Prevention: draw the state machine and forbid shortcuts. A refund should not be able to exist without a captured payment, and a settlement should not be able to arrive before its payment.

4. Signature verification fails

Likely cause: the signing secret was rotated on one side only, the body was parsed and re-serialised before verification, or a proxy altered the payload.

Fix: verify against the raw request body, support two active secrets during rotation, and log the key version rather than the secret. Check that your framework is not normalising JSON before the signature check.

Prevention: write a rotation runbook with an overlap window and a test event after each step. Alert on a sudden rise in signature failures, which is often the first sign of a deployment mistake.

5. Callbacks never arrive

Likely cause: wrong URL, expired certificate, DNS change, firewall rule, or the provider disabled the endpoint after repeated failures.

Fix: test from an external network, inspect the certificate chain, confirm the registered URL in the provider dashboard, and check whether the endpoint was automatically disabled. Re-enable only after the cause is fixed.

Prevention: monitor the endpoint from outside your infrastructure, keep certificate expiry in the operations calendar, and own the provider dashboard account with more than one person able to access it.

6. Volume spikes break processing

Likely cause: the handler does synchronous work, the queue has no backpressure, or a database lock turns a burst into a queue of timeouts.

Fix: put a queue between receipt and processing, set a dead-letter path, and load test at a multiple of the largest expected campaign. Measure the age of the oldest unprocessed event rather than only the error rate.

Prevention: agree a processing lag target with support and alert before it is breached. Slow is recoverable; silent is not.

The diagnostic sequence

  1. Get the provider transaction reference and find the event in the provider log.
  2. Match the event to your access log by timestamp and request identifier.
  3. Read the exact response your endpoint returned, including the body.
  4. Check whether the event exists in your store, and whether it was applied once or more.
  5. Compare the resulting ledger state with the provider statement.
  6. Reproduce the event in staging with the same payload before changing production.

Metrics worth alerting on

  • Age of the oldest unprocessed payment event.
  • Payments pending beyond the provider's stated window.
  • Duplicate event deliveries and how many were suppressed.
  • Signature failures by key version.
  • Unmatched settlement lines at daily close.

Where our commerce team fits

We build and repair payment integrations, event handling and daily reconciliation under Commerce Systems. Scope and commercial terms are set out on the pricing page. For the budgeting view of this work, read payment webhook reliability cost in Bangladesh.

Sources

Services that deliver this

Want this implemented properly?

Pricing →

Every engagement starts with a written scope. If a requirement is outside it, we quote it separately before starting.

Responses (0)

No comments yet. Add the first one below.

Leave a response

Your email address will not be published. Required fields are marked *.

Keep reading

More field notes