Loading live status…
Loading live status…
Ten mechanisms that take down large sites — config pushes, BGP withdrawals, expired certificates, split-brain databases — and, for each one, what it looks like from outside while it is happening.
Almost nothing large goes down because a server caught fire.
The failures that take out a big site are nearly always a small, correct-looking change meeting a system that was already carrying more implicit dependencies than anyone had written down. The interesting part is not that this happens. It is that each mechanism has a signature — a particular shape it makes from outside, before anybody has published anything.
That outside view is what isuptime spends all day looking at: 102 providers' status pages, read on a schedule. Here are ten causes that account for most of the big ones, and for each, what you can actually see while it is happening.
The single most common cause of a global outage at a company that is otherwise very good at this.
Config is not code. It skips the canary, the staged rollout, the soak time — because it is "just a setting". Then it reaches every edge node in ninety seconds.
Two textbook cases. Cloudflare in July 2019 deployed a WAF rule containing a regular expression that backtracked catastrophically; CPU on every machine in the network went to 100% and stayed there for about half an hour. Fastly in June 2021 had a latent bug sitting in a build for nearly a month, and one customer changing their own settings was enough to trigger it across roughly 85% of the network. Reddit, Shopify, the BBC and large parts of the news web went dark for about 49 minutes.
Signature from outside: near-instant, near-total, and geographically indiscriminate. It also gets acknowledged fast — this class of failure is unambiguous and self-inflicted, so the status page usually goes red within minutes. If a huge site fails everywhere at once and the provider posts quickly, you are almost certainly watching a config push.
BGP is how networks announce "traffic for these addresses comes to me". It is famously trusting, and a bad announcement propagates globally in under a minute.
The Facebook outage of October 2021 is the reference case. A command issued during a routine capacity audit withdrew the BGP routes to Facebook's DNS servers. Those servers are built to withdraw themselves when they cannot reach the data centres behind them — a sane safety behaviour that, in this case, meant Facebook, Instagram and WhatsApp stopped existing as far as the internet's routing tables were concerned. It lasted about six hours. Engineers could not fix it remotely because the remote-access tooling was behind the same routes, and reportedly could not get into the buildings quickly either, because the badge readers were too.
Signature from outside: the site does not respond slowly, it responds not at all — no TLS handshake, no timeout-after-30-seconds, just nowhere to send packets. And critically, the company's own status page often goes with it, because it was hosted behind the same infrastructure. A provider whose status page is unreachable is not telling you nothing. It is telling you something quite specific.
DNS failures are unusually confusing because everything breaks simultaneously across companies with no relationship to each other.
When Dyn was hit by the Mirai botnet in October 2016, Twitter, Reddit, Spotify, Netflix and GitHub all became unreachable for parts of a day. None of them had a problem. Their servers were healthy, their deploys were fine, their dashboards were green. They just could not be found.
Signature from outside: a cluster of unrelated big names failing at the same moment, while each individual provider reports operational — because from where they are sitting, they are operational. If you are seeing five unconnected services fail at once, stop looking at the services and start looking at what sits in front of all of them.
Certificates expire at a precise second, and the failure is total and instantaneous.
An expired certificate inside Ericsson software took mobile data offline for O2 in the UK and SoftBank in Japan in December 2018 — tens of millions of people, for most of a day. Microsoft Teams went down in February 2020 for the same category of reason.
What makes this one nasty is not the technology, it is the org chart. The certificate is usually owned by nobody in particular, on a system nobody has touched in eighteen months, with an expiry date in a calendar that left with an employee.
Signature from outside: a hard failure that starts on a suspiciously round timestamp, and — this is the tell — is often slow to be acknowledged, because the first reports look like individual client problems. Users see a browser security warning and assume it is their machine. It can take a while for a provider to accept that the call is coming from inside the house.
If you run in us-east-1, you do not have one dependency on AWS. You have several hundred, and some of them are between AWS services rather than between you and AWS.
November 2020: adding capacity to Kinesis pushed the operating system thread count past a limit, and the failure took out services that merely used Kinesis internally — including, awkwardly, parts of AWS's own status tooling. December 2021: an automated scaling action produced unexpected client behaviour that congested an internal network, and the impact ran for hours across an enormous surface of dependent services.
Signature from outside: dozens of unrelated products degrade within a few minutes of each other, all of them coincidentally American, and none of the individual companies can tell you much. AWS is also known to be conservative and slow in updating its public feed, so this is the case where the public status page most lags reality. Check the Personal Health Dashboard, which is account-scoped and routinely shows impact the public feed never mentions.
Failover is dangerous precisely because it is automatic. The system is designed to act decisively on incomplete information, and occasionally it acts decisively on wrong information.
GitHub's October 2018 incident is the one everybody should read. Planned maintenance on failing optical equipment caused 43 seconds of connectivity loss between two data centres. Automation did what it was told and promoted a database on the other coast. Those 43 seconds produced writes on both sides that could not be automatically reconciled, and GitHub ran degraded for more than twenty-four hours while humans untangled it.
Signature from outside: the weird one. Partial and asymmetric — reads succeed while writes fail, or the site loads but nothing you do sticks. Duration wildly out of proportion to the triggering event. Status pages tend to sit on "degraded" for a long, long time, because the service genuinely is partly working, and it is not clear when it will stop being partly working.
This is the cause that turns a two-minute blip into a two-hour outage.
Something fails briefly. Every client retries. Well-behaved clients retry with backoff; the rest retry immediately, and in aggregate they deliver more load to a recovering system than it was carrying when it was healthy. It falls over again. Now there are more clients waiting to retry. Caches are cold, connection pools are exhausted, and every attempt to bring the service back is met with the entire internet's pent-up demand arriving in one wave.
Signature from outside: recovery that keeps failing. The status page flips to "monitoring", then back to "identified", then back again. If you see a provider appear to recover two or three times before it holds, you are watching a herd being brought back under control.
Your application is fine. Your servers are fine. But your login flow calls an identity provider, your checkout calls a payment processor, your frontend pulls a script from a CDN, and your feature flags come from somewhere else entirely.
Any one of those can take you down while every dashboard you own stays green. Worse, the failure often looks like your bug, because it surfaces inside your product with your branding around it.
Signature from outside: the most disorienting case of all — your service is broken and every status page you check is operational, including your own. The move is to work outward from the failing user action and enumerate every third party it touches. Usually it is the one you would not have listed if asked.
Less dramatic than the rest and much more frequent. It is on this list because its shape is different from everything above.
Modern deploys are gradual. Code goes to a canary, then 1%, then 10%. So a bad deploy does not fail everything at once — it fails a fraction of requests, or all requests for a fraction of users, in a way that is easy to mistake for flakiness.
Signature from outside: intermittent and user-dependent. Reproducible for you, invisible to your colleague, working again if you hard-refresh onto a different backend. And this class very often never appears on a status page at all, because it was caught and rolled back before it crossed whatever internal threshold triggers an incident declaration. Some meaningful share of "it was just being weird earlier" is this.
The S3 outage of February 2017 was an engineer running an established playbook to debug a billing system, who mistyped one parameter and removed far more capacity than intended. Two subsystems needed a full restart — something that had not been done at that scale in years — and a large fraction of the web that stores its images on S3 was broken for about four hours.
The lesson usually drawn is "be careful with commands". The better one is that a system where a single typo can remove a critical subsystem, and where the recovery path has not been exercised at scale in years, was already fragile. The typo just found it.
Signature from outside: it happens during an announced maintenance window, which means it is frequently mislabelled as maintenance for the first stretch — both by the provider and by aggregators reading its feed. A maintenance window that runs well past its stated end is worth treating with suspicion.
Put together, these are less a taxonomy of faults than a set of questions worth asking in the first two minutes:
The uncomfortable thread running through all ten is that the status page is the last thing to know. Every one of these mechanisms is visible to users before it is visible to the company's incident process, because that process starts with a human deciding something is worth declaring. We have written before about why a status page is not a measurement, and this is the practical consequence: the gap between "broken" and "acknowledged" is where you actually live during an outage.
What an aggregator can do is narrow the second question. Seeing 102 providers on one screen will not tell you why something broke, but it will tell you very quickly whether the problem is one company's or everybody's — and that single distinction is usually the difference between rolling back your deploy and going to make coffee while somebody else fixes theirs.
If you want the detail on how we read those feeds and what we do and do not trust in them, that is written up in how isuptime works.