Network Infrastructure - East US - Mitigated

Incident
November 22, 3:23pm EST

Network Infrastructure - East US - Mitigated

Status: closed
Start: November 22, 3:22pm EST
End: November 22, 3:23pm EST
Duration: 1 minute
Affected Components:
Cloud Providers Azure Network Infrastructure
Update

November 22, 3:22pm EST

November 22, 3:22pm EST

Summary of Impact: At 18:29 UTC, 20:04 UTC, and 20:24 UTC on 22 Nov 2023 for periods of 1 minute, 15 minutes, and 5 minutes respectively, resources in a single availability zone in East US may have seen intermittent network connection failures, delays, packet loss when reaching out to services across the region. However, the retries would have been successful. 


Preliminary Root Cause: While conducting an urgent break-fix repair on some network capacity in a single availability zone of East US, live traffic was impacted. Traffic was re-routed to alternate spans with sufficient capacity. The region is now stable. Customers can continue to operate workloads in the impacted Availability Zone. 


Mitigation: We have moved traffic to healthy optical fiber spans in the region. The issue is mitigated.


Next Steps: We will investigate why this impact occurred to establish the full root cause and prevent future occurrences. Stay informed about Azure service issues by creating custom service health alerts: https://aka.ms/ash-videos for video tutorials and https://aka.ms/ash-alerts for how-to documentation.

Resolved

November 22, 3:23pm EST

November 22, 3:23pm EST

Resolved