Incident history
Every xhostd incident that affected customers, newest first, with its timeline, its cause, and what happened to customer data.
Degraded performance on one application server
Affected: applications on a single serverDuration: 54 minutesLess than a day after the previous incident
Customer data lost: none
Timeline (UTC)
| Time | |
|---|---|
| 05:30 | Storage fault begins on the same server affected the previous day. Applications there start responding slowly. |
| 06:07 | Move of every remaining account to a second hosting provider begins. |
| 06:24 | Write performance returns to normal, with no change made to the server and most accounts still on it. |
| 06:37–07:32 | Remaining accounts moved in sequence, as planned. |
| 07:32 | All moved accounts running on the second provider. |
Each account was interrupted briefly while moving — most under a minute.
Cause
The third occurrence of one storage fault at one hosting provider, across two of their servers, in three days. Writes slowed from about one millisecond to over twenty seconds while reads stayed normal.
The fault cleared on its own partway through the move, with most accounts still on the affected server and nothing changed on it. That rules out our own software and our own workload as the cause: removing accounts did not fix it, and keeping them did not prevent the recovery.
Confirmed against an identical server at the same provider, built the same day from the same image, running the same software, unaffected throughout all three occurrences.
Data
No customer data lost. Database contents verified after each move, including a record count of 32,991,066 rows matched exactly for the largest account. Off-site backups continued uninterrupted throughout.
Degraded performance on one application server
Affected: applications on a single serverDuration: 4 hours 7 minutesLess than a day after the previous incident
Customer data lost: none
Timeline (UTC)
| Time | |
|---|---|
| 09:50 | Storage fault begins affecting one server's ability to write data. Applications there start responding slowly. |
| 12:00 | Same fault confirmed as the one that affected a different server the previous day. Reads unaffected. Other servers unaffected. |
| 12:30 | One account moved to a healthy server as a precaution. |
| 13:00 | A replacement server ordered from a second hosting provider, and brought into service. |
| 13:57 | Write performance on the affected server returns to normal, with no change made to it. |
| 14:09–15:28 | Largest account moved to the new provider's server, and its data verified. |
Applications on the affected server responded slowly while the fault lasted. Accounts moved during the response were interrupted briefly — one for about five minutes, the largest for about eight.
Cause
The same storage fault as the previous day's incident, on a different server at the same hosting provider. Writes slowed from about one millisecond to over thirty seconds while reads stayed normal.
The server carried the applications moved off the previous day's failed server, so the same accounts were affected twice in two days. That is the part of this we most regret.
Data
No customer data lost. Database contents verified after each move. Off-site backups continued uninterrupted throughout.
Degraded performance on one application server
Affected: applications on a single server · dashboard, sign-in, git and deployments (5 minutes, all customers)Duration: 4 hours 43 minutes
Customer data lost: none
Timeline (UTC)
| Time | |
|---|---|
| 07:25 | Storage fault begins affecting one server's ability to write data. Applications there start responding slowly. |
| 08:00 | Cause identified as a storage-level fault on that server. Reads unaffected. Other servers unaffected. |
| 09:00–09:05 | Control-plane restart, required for the fix. Dashboard, sign-in, git and deployments unavailable to all customers. Running applications unaffected. |
| 09:09 | Move of affected applications to a replacement server begins. |
| 09:41 | Largest affected account serving from the replacement server. |
| 10:26 | That account's data migration complete and verified. |
| 10:37–11:26 | Remaining accounts moved in sequence. |
| 12:08 | All affected applications running on the replacement server. Resolved. |
Each account was briefly interrupted while moving — most under a minute, the largest for about 30 minutes.
Cause
Storage fault at one of our hosting providers degraded a single server's write path. Writes slowed from roughly one millisecond to over 40 seconds. Reads stayed normal, which is why some applications kept partly working while unable to save data reliably.
Confirmed as an infrastructure fault rather than a software fault by comparison against an identical server running the same software, unaffected throughout. Provider investigation remains open.
Data
No customer data lost. Database contents verified against independent backup records after each move. Off-site backups continued uninterrupted throughout, so a recovery point was available at all times.