Degraded performance on one application server

Customer data lost: none

Timeline (UTC)

Time
07:25	Storage fault begins affecting one server's ability to write data. Applications there start responding slowly.
08:00	Cause identified as a storage-level fault on that server. Reads unaffected. Other servers unaffected.
09:00–09:05	Control-plane restart, required for the fix. Dashboard, sign-in, git and deployments unavailable to all customers. Running applications unaffected.
09:09	Move of affected applications to a replacement server begins.
09:41	Largest affected account serving from the replacement server.
10:26	That account's data migration complete and verified.
10:37–11:26	Remaining accounts moved in sequence.
12:08	All affected applications running on the replacement server. Resolved.

Each account was briefly interrupted while moving — most under a minute, the largest for about 30 minutes.

Cause

Storage fault at one of our hosting providers degraded a single server's write path. Writes slowed from roughly one millisecond to over 40 seconds. Reads stayed normal, which is why some applications kept partly working while unable to save data reliably.

Confirmed as an infrastructure fault rather than a software fault by comparison against an identical server running the same software, unaffected throughout. Provider investigation remains open.

Data

No customer data lost. Database contents verified against independent backup records after each move. Off-site backups continued uninterrupted throughout, so a recovery point was available at all times.