Want to understand the Microsoft outage? Watch Top Gun

Much virtual ink has been spilled on this Friday the 13th concerning the global Microsoft outage that has seen Outlook, Teams and many other business-critical applications laid low for the better part of a day.

Talking heads have graced our TV screens and airwaves, and their common wisdom has been that we shouldn’t put all of our eggs in one basket. Enterprises, one analyst told CIO, “have priced Microsoft 365 as a licence and never as a single point of failure”. Redundancy is surely the oldest principle in reliable system engineering. It says that reliable systems require more than one copy of each critical component, so that no single point of failure can bring them down.

This outage showed that Microsoft’s systems together represent a single point of failure for many businesses. But there is an irony here. Microsoft’s systems are about as redundant as systems get, and the machinery that manages all of that redundancy seems to be right at the centre of what went wrong.

As this post’s intentionally facetious title suggests, to understand why this failure might have lasted so long we’d do well to think of Top Gun. Specifically, the scene that had me tear up every time I watched it as a kid1: the one in which Maverick’s F-14 Tomcat flies through Iceman’s jet wash2, loses both engines, and enters an unrecoverable flat spin.

So what does that have to do with today’s outage?

The Microsoft Outage as a Failure of a Complex Control System

The exact cause is still unknown, but Microsoft’s status updates tell us quite a lot about the shape of the failure. I’m no site reliability engineer, so what follows is an educated guess pieced together from those updates.

When you open Outlook, your request doesn’t go straight to the computer holding your mailbox. It first lands on Microsoft’s routing services, whose job is to pass it on to one of many interchangeable backend servers that do the actual work. To decide where to send it, the routing services keep checking that each server is still responding properly. In other words, the routing services perform periodic health checks of the backend servers. A backend server that becomes unhealthy is taken out of rotation, and the routing services instead send requests to the remaining healthy ones. This job is known as load balancing, and that pool of interchangeable servers is exactly the redundancy that is supposed to make Microsoft’s services reliable.

This makes the whole thing a (massively distributed) control system with a feedback loop. A thermostat is the classic example of a control system: it measures the temperature, switches the heater on or off, and that changes the temperature it measures next. Microsoft’s routing services do the same thing, except that what they measure is the health of the servers, and what they control is how much work each server gets.

So what happened? Servers started failing their health checks and being taken out of rotation. Microsoft later said that the extra load came from “a large number of servers being unexpectedly unavailable, which is shifting load onto the remaining infrastructure”, rather than from any increase in user traffic. It also reported a surge of traffic from its routing services to its backend servers, due in part to “traffic rebalancing actions” it had taken itself.

Now we can see how the redundancy might have made matters worse. When some servers are taken out of rotation, the remaining ones get more requests, possibly overloading them so that they fail their health checks too, which shrinks the pool of healthy servers further. This is a classic recipe for a cascading failure, driven by the very machinery that is meant to help avoid failure. Microsoft’s own updates point the same way, as do several experts, though Microsoft hasn’t yet said what started it.

Stuck in a Flat Spin

I’m no control theory person either. But even some basic intuition suggests that a control system, once pushed into a bad state, can get stuck there, with its own feedback loop holding it in place3. Maverick’s unrecoverable flat spin is a good picture of this. Once the spin develops, it sustains itself: the aerodynamics of the spinning aircraft keep it stalled and rotating, long after the jet wash that started it has passed.

A similar thing seems to have happened to Microsoft. Its engineers restarted affected servers and shifted traffic elsewhere, which gave only brief relief. The servers they returned to service apparently became unresponsive again as soon as they received traffic. Microsoft even began rolling back a network configuration change that coincided with the start of the outage. Whatever originally pushed the system into its flat spin, the spin itself was self-sustaining.

How Do You Get Out of a Flat Spin?

For a system like Microsoft’s, the answer seems to be to take away enough of the load that the feedback loop breaks, and then carefully guide the system back towards its good state, a little at a time, without letting it tip back into the bad state. That is more or less what Microsoft ended up doing. It began temporarily limiting which types of traffic can reach the affected infrastructure and recovering servers “in an isolated manner”, watching each portion of infrastructure for stability before expanding more broadly. Recovery has, in Microsoft’s words, “progressed in phases”, with parts of Asia-Pacific among the first to be restored. Indeed, sadly, my own work Outlook is now back up.

One might like to think of the recovery as akin to the hand of a god reaching down and arresting the flat spin of the doomed F-14. In reality, Microsoft likely had thousands of F-14s in flat spins, each one’s spin feeding the spins of its neighbours. Stopping them meant a long, careful procedure of recovering them one plane at a time.

This analogy is nothing if not imperfect. But it does help to explain why the outage has taken so long to fix.

I’ll update this post once Microsoft publishes its post-incident report. In the meantime, if you’re looking for something to do while your Outlook is down, you know which film to rewatch.


  1. Let’s be honest: as a middle-aged man I tear up way more easily these days than I ever did as a kid. ↩︎

  2. You better believe there’s something entirely too satisfying about throwing around fighter-jet jargon like I have any idea what “jet wash” is. ↩︎

  3. Researchers who study large computing systems call failures like this metastable failures. Some trigger pushes the system into a bad state, and a feedback loop keeps it there even after the trigger has gone away. Microsoft’s outage looks a lot like one, though we won’t know for sure until its post-incident report. It could yet turn out to be something far more boring. ↩︎