Degraded performance on one application server

Customer data lost: none

Timeline (UTC)

Time
05:30	Storage fault begins on the same server affected the previous day. Applications there start responding slowly.
06:07	Move of every remaining account to a second hosting provider begins.
06:24	Write performance returns to normal, with no change made to the server and most accounts still on it.
06:37–07:32	Remaining accounts moved in sequence, as planned.
07:32	All moved accounts running on the second provider.

Each account was interrupted briefly while moving — most under a minute.

Cause

The third occurrence of one storage fault at one hosting provider, across two of their servers, in three days. Writes slowed from about one millisecond to over twenty seconds while reads stayed normal.

The fault cleared on its own partway through the move, with most accounts still on the affected server and nothing changed on it. That rules out our own software and our own workload as the cause: removing accounts did not fix it, and keeping them did not prevent the recovery.

Confirmed against an identical server at the same provider, built the same day from the same image, running the same software, unaffected throughout all three occurrences.

Data

No customer data lost. Database contents verified after each move, including a record count of 32,991,066 rows matched exactly for the largest account. Off-site backups continued uninterrupted throughout.