We are providing an update regarding an unforeseen data loss incident that has impacted some of our servers. We want to assure you that this incident is not the result of any security breach or unauthorized access to our systems.
Our team is diligently working to identify the underlying cause of this issue and is focused on recovering the affected data wherever possible.
If your server has been affected, we are committed to keeping you informed with detailed updates and instructions on the subsequent steps. This includes appropriate compensation measures once we have fully resolved the situation.
We sincerely regret any inconvenience this situation may be causing and appreciate your continued patience as we dedicate our efforts to resolving this matter efficiently.
Understanding the Server Data Loss Incident
It is important to reiterate that this data loss is not due to any external security threats. Our initial investigations indicate an internal issue within our storage infrastructure as the root cause. We understand the critical nature of your data and are treating this incident with the highest priority. Our technical teams are working around the clock to restore server functionality and data integrity.
Server Restoration Timeline and Progress
We are making steady progress in restoring services. Below is a timeline of our efforts to keep you informed about the server restoration process:
February 28th:
- 1:05 PM EST: The data restoration rate has been significantly increased, and we are continuing restorations as quickly as possible.
- 6:20 AM EST: Rate limitations on our restoration process have been lifted, allowing us to accelerate concurrent restorations.
- 12:05 AM EST: Automatic unlocking of servers is now implemented upon successful restoration completion.
- 12:00 AM EST: Data restoration across all affected servers remains ongoing.
February 27th:
- 10:00 PM EST: Restoration processes are continuing. We are collaborating with our storage provider to further enhance the restoration rate.
- 8:58 PM EST: Restorations are progressing at a consistent pace.
- 7:08 PM EST: Server restorations are proceeding as quickly as possible.
- 5:33 PM EST: Restoration efforts have been successfully expanded to additional regions. Successfully restored servers are having their staff locks automatically removed.
- 4:28 PM EST: Larger restoration batches have commenced following adjustments to address storage provider errors.
- 3:21 PM EST: Data restoration across all servers is in progress.
- 2:37 PM EST: We are experiencing elevated error rates from our storage providers and are actively working on solutions while continuing restorations.
- 1:48 PM EST: Data restoration across all servers is ongoing.
- 12:30 PM EST: We are actively working to restore all server data. Servers are automatically unlocked upon successful restoration.
- 11:17 AM EST: Initial test restoration batch was successful. We are now implementing this process across all servers in stages.
- 10:08 AM EST: Data restoration has begun for affected servers.
- 9:45 AM EST: Emails have been dispatched to all customers who are affected by this incident.
- 8:57 AM EST: Data restoration efforts have commenced for all affected servers.
- 8:35 AM EST: A comprehensive list of all affected servers has been compiled.
- 8:15 AM EST: We were alerted to the issue causing data loss across multiple servers.
Frequently Asked Questions (FAQ)
Q: When will server locks be removed?
A: Server locks are automatically removed as soon as the server restoration process is successfully completed.
Q: When will services be fully restored?
A: We are unable to provide specific Estimated Times of Arrival (ETAs) for individual nodes or servers. However, we are working tirelessly to restore all services as quickly as possible and will continue to provide updates on our progress.
We understand that server status and uptime are critical for your operations. We are committed to transparency and will continue to update you regularly on the restoration progress. Please monitor this page for further announcements and detailed information.
