On Jan 25th 2025 between 16:06 and 16:12 UTC, customers may have experienced a slight period of increased error rates and latencies.
The issue was caused by two of our PlanetScale database shards experiencing issues simultaneously. When these shards became unhealthy, workers across multiple services became blocked while waiting on operations to complete, effectively reducing the Messenger and Mobile Messenger fleets ability to serve requests, resulting in increased error rates and latency issues.
While we do not have yet have a full understanding of the root cause of the issues experienced by these shards, our investigation with PlanetScale is ongoing, and we will update this root cause analysis when we have a deeper understanding of the issue and when we identify what actions we will be taking to protect our services from similar issues going forward.