Microsoft has now traced the global Microsoft 365 outage that started November 12th to a hidden bug in a fix it shipped in July. The outage hit Exchange Online email hardest.
The fix was only for Microsoft’s own network gateway, the servers that sit in front of many Microsoft 365 services. Under certain loads, the bug left tasks inside an internal Windows process waiting their turn for the same locks. That used up the servers’ CPU and memory until they stopped responding. In a post on X sharing the preliminary report, Microsoft’s Charles Lamanna said the bug doesn’t affect other Windows PCs or servers.
Why Microsoft’s tests missed the bug
The fix ran worldwide for more than three months without issues. Microsoft said the bug gets worse on newer machines with more cores. But its load tests mostly ran on weaker hardware with fewer cores.
Two events at once set off the bug
About 15 hours before the outage, Teams switched from Microsoft’s other network gateway onto the “primary” one, which also serves Exchange Online. Microsoft called this kind of switch a normal part of operations, but it raised the load on the primary gateway.
At the same time, nearly 14% of the gateway’s server capacity was out because of an unrelated issue the day before. That alone had no impact, since there was enough spare capacity. Together, though, the extra load and the lost capacity set off the July bug during peak US traffic hours.
How the outage spread
Requests then piled up until servers stopped responding. The systems built to keep Microsoft 365 running moved users onto working servers, so the same load landed on fewer machines. In less than two hours, the failure had jumped to servers across the fleet. At the lowest point, more than 80% of them had stopped responding.
Why email was down the longest
Teams, OneDrive, SharePoint, and Office apps moved off the faulty gateway, so they came back sooner. Teams chats and meetings even worked for most of the outage. But Exchange Online can’t move off this gateway because of its scale, so email stayed down the longest.
Why the fix took so long
Because the bug came from a change made months earlier, Microsoft’s usual rollback tools couldn’t quickly undo it. Instead, the company built an update that put the code back to its earlier, working state. It then updated servers in big batches. Some covered 50% or more of a region at once.
The report is still only preliminary. Microsoft’s final review, with more details, is due November 20th.



