Degraded performance on one application server
Affected: applications on a single serverDuration: 4 hours 7 minutesLess than a day after the previous incident
Customer data lost: none
Timeline (UTC)
| Time | |
|---|---|
| 09:50 | Storage fault begins affecting one server's ability to write data. Applications there start responding slowly. |
| 12:00 | Same fault confirmed as the one that affected a different server the previous day. Reads unaffected. Other servers unaffected. |
| 12:30 | One account moved to a healthy server as a precaution. |
| 13:00 | A replacement server ordered from a second hosting provider, and brought into service. |
| 13:57 | Write performance on the affected server returns to normal, with no change made to it. |
| 14:09–15:28 | Largest account moved to the new provider's server, and its data verified. |
Applications on the affected server responded slowly while the fault lasted. Accounts moved during the response were interrupted briefly — one for about five minutes, the largest for about eight.
Cause
The same storage fault as the previous day's incident, on a different server at the same hosting provider. Writes slowed from about one millisecond to over thirty seconds while reads stayed normal.
The server carried the applications moved off the previous day's failed server, so the same accounts were affected twice in two days. That is the part of this we most regret.
Data
No customer data lost. Database contents verified after each move. Off-site backups continued uninterrupted throughout.