When a site is down and the monitoring dashboard still shows green, the tool is not lying on purpose. It answers the question it was configured to ask, which is usually narrower than the one the business cares about. A request to the home page confirms that a server responds. It does not confirm that a customer can log in, search for a product or complete a payment.
This guide is for the manager who already pays for uptime monitoring and wants to know why it keeps missing real incidents. It covers the recurring failure patterns, the steps that identify them, and the changes that stop the same alert being ignored next month.
What the monitor is measuring, and what it is not
Uptime monitoring is usually one of three things. A synthetic check requests a URL on a schedule and tests the status code or a string in the response. Real user monitoring records what actual browsers experience. Log and metric monitoring watches server-side signals such as error rates and response time. Each sees a different layer, and most false-green incidents come from a team owning only the first layer.
Before changing tools, list every check you have, what it requests and what it asserts. A check that accepts any HTTP 200 response will pass on a styled error page, a maintenance notice or a cached copy of yesterday's content. That weakness explains many "the monitor said it was up" conversations.
Symptom: the status page is green during a visible outage
Likely causes. The check requests the home page, which is static and served from cache while checkout is broken. It follows a redirect to a login screen that returns 200. It runs every five or ten minutes, so a short outage falls between checks. Or it only checks the web server, not the database, payment provider or background queue the page depends on.
Fix. Replace page checks with a small number of transaction checks that follow the revenue path: open the catalogue, search for a product, add to basket, reach the payment step, confirm the order callback. Assert a content marker that only appears when the page is complete, not just a status code. Where a real payment is impossible to test safely, test the provider handshake and the application's own order state.
Prevention. Keep one synthetic journey per critical user path, and review the list whenever a feature is launched. If a path matters enough to staff a support queue for it, it matters enough to monitor it.
Symptom: alerts arrive after customers complain
Likely causes. The check interval is longer than the promise you make to customers. Confirmation rules wait for three consecutive failures. Alerts go to an email inbox that one person checks. There is no measurement of latency, so a site that takes twelve seconds per page technically responds.
Fix. Measure time to detect and time to acknowledge from the last three incidents. Shorten the interval for the checkout path only, and keep noisy background checks slower. Send high-severity alerts to a channel with a named on-call owner, and keep a secondary route for the case where the first one fails. Add response-time thresholds to the checks.
Prevention. Write a detection target in plain language, such as "a failed checkout is noticed within five minutes during business hours", and test it with a controlled failure once a quarter.
Symptom: an alert storm follows every release
Likely causes. Checks expect the previous version's markup. The application restarts and cold caches make the first requests slow. A health endpoint reports failure because a non-critical dependency is temporarily unavailable. Deployments happen without a maintenance window or a silence rule.
Fix. Separate liveness from readiness, and classify dependencies as critical or degradable. Put a short, automatic silence on checks during a deployment, with an expiry so a failed deploy cannot leave monitoring muted. Update check assertions in the same pull request as the markup change.
Prevention. Treat monitoring configuration as part of the release, not as a separate administrative task. A release that changes the checkout page and not the checkout check is incomplete.
Symptom: the check passes locally and fails for customers abroad
Likely causes. A single probe location. DNS answers that differ by region. A CDN edge node serving stale content. An international route that is slow or filtering traffic while the local connection is fine.
Fix. Run the critical checks from at least two regions that represent your actual customers, including one outside Bangladesh if you sell abroad. Compare DNS responses and response times by region, then trace the failing path. If the site is only monitored from the same network as the server, the monitor is testing the office, not the market.
Prevention. Choose probe locations from customer data, not from convenience. Review them when you enter a new market or change CDN configuration.
Symptom: alerts fire and nobody acts
Likely causes. No named owner. No severity definitions, so every alert looks equally urgent. No runbook, so the person who receives it has to investigate from scratch. A history of false alarms has trained the team to wait.
Fix. Assign one owner per critical journey, define three severity levels with response expectations, and link each alert to a short runbook: what to check first, who to call, how to roll back. Review every alert that fired in the past week and delete or retune the ones that did not require action.
Prevention. Hold a short weekly review of alerts, incidents and near misses. The output should be a change to a check, a runbook or an owner, not a meeting note.
A diagnostic sequence for this week
- List the five journeys that generate revenue or legal obligation.
- Map every existing check to a journey, and mark the journeys with no check.
- Pull the last quarter's incidents and compare their start times with alert timestamps.
- Run one controlled failure in staging or a low-traffic window and record how long detection took.
- Confirm that domain and TLS expiry, queue depth and backup completion are monitored, not just web pages.
- Interview the alert recipient about their first action. If the answer is to open the dashboard, the runbook is missing.
- Write a one-page gap list with an owner and a date for each fix.
Design the monitor around the customer journey
Monitor outcomes rather than machines. A server can be healthy while the business is offline. Pair infrastructure checks with journey checks, publish an honest status page, and review coverage whenever the product changes.
We design monitoring and escalation as part of Managed Technology & Cybersecurity, alongside backup testing and incident response. The pricing page explains how a monitoring retainer is scoped. For the escalation side of the same problem, read how SME incident response changes year by year.
Sources
- NIST, SP 800-34 Rev. 1, Contingency Planning Guide for Federal Information Systems, accessed 17 November 2026.
- Google, Site Reliability Engineering, Chapter 6: Monitoring Distributed Systems, accessed 17 November 2026.
- IETF, RFC 9110: HTTP Semantics, accessed 17 November 2026.
- MDN Web Docs, HTTP response status codes, accessed 17 November 2026.
Responses (0)
No comments yet. Add the first one below.
Leave a response
Your email address will not be published. Required fields are marked *.