More than 80 percent of servers behind Microsoft's networking gateway were unresponsive at the peak of last week's global outage affecting Outlook and other Microsoft 365 services. Charles Lamanna, Microsoft's Executive Vice President of Copilot, Agents and Platform, detailed the factors that led to the incident in a post on X.
According to the company, three primary factors caused the outage of the networking gateway. The first was a latent software defect introduced by a fix prepared specifically for Microsoft 365 networking gateway servers in July. The change had been in place worldwide for more than three months without incident. Load testing failed to catch the problem because tests focused on weaker hardware, whereas the defect manifested worse on newer machines with more processing cores.
The second factor occurred about 15 hours before the outage, when Microsoft Teams shifted from an alternative gateway to the primary gateway that also supports Exchange Online. This operational shift increased the load and altered traffic patterns. Thirdly, nearly 14 percent of the gateway's server capacity was out of service at the time due to an unrelated issue from the previous day.
The combination of higher load and reduced capacity caused the gateway to hit the July bug, building up requests until servers stopped responding during peak traffic in the United States. While traffic management systems normally route users to healthy endpoints to maintain service resilience, the redirected traffic increased the likelihood of the same problem occurring on remaining endpoints. This triggered a cascading failure from server to server, rendering more than 80 percent of servers unresponsive in less than two hours.
Safeguards that normally separate different parts of the gateway failed to stop the failure due to a separate issue, allowing the impact to spread globally. Services like Teams, OneDrive, SharePoint, and Office apps recovered earlier by moving off the faulty gateway, but Exchange Online could not do so. Recovery took longer because returning servers were immediately overwhelmed by waiting traffic.
Engineers eventually developed an update to return the code to its previous state by taking groups of servers out of service, applying the change, and restarting them. A final post-incident review will be provided later this week.
The disruption, which began late on Thursday, affected Indian airlines, financial services firms, and businesses. Domestic carriers IndiGo, Air India, and Akasa Air reported disruptions to email-based customer support, though flight operations remained unaffected. Financial services firms also faced difficulties accessing research reports and conducting client interactions. At its peak, the outage generated over 1.3 lakh reports of Outlook problems on Downdetector across 72 to 74 countries.
"This incident highlights a critical operational lesson for modern technology enterprises: legacy software defects can remain dormant for months before manifesting under specific load conditions and traffic shifts. For businesses heavily reliant on cloud infrastructure, maintaining robust redundancy and thorough testing across diverse hardware environments is essential to mitigate the risk of cascading failures. Enterprises must implement resilient architectures that can handle unexpected capacity bottlenecks without compromising core communication systems." — Dr. Shishir Gupta, Founder & CEO, StartupLanes
Recent StartupLanes Articles
Browse through our 30 latest publications on venture capital, startups, and angel investing.