View Full Version : SC, LC, TD, FF Emergency Maintenance
Solclaim, Leafcull, Thistledown, and Frostfell are being taken down for emergency database server maintenance. I do not currently have an ETA on when they will be back up, please watch this thread for further updates.
Our engineers are currently performing diagnostics on the database servers, this process can take up to four hours. I will post further status updates as soon as I have them, no later than 11AM PDT / 3PM EDT / 7PM UTC.
This is the current situation:
The database farm for Solclaim, Leafcull, Thistledown and Frostfell has experienced a major systems failure.
Right now we are working on recovering the last good data from the servers. If that is successful, characters on these worlds will be saved as they were at 7AM EDT today. If it is not, they will be rolled back to approximately 7AM EDT yesterday -- Sunday July 25.
We should know within the next two hours whether the recovery is successful and thus how long the rollback will be.
Once we have finished this process and know how long the rollback will be, we will begin bringing the worlds back up. I'll be able to post more information on how long this will take once the data recovery is complete.
We apologize for this unintended downtime, and we are working to resolve this issue as quickly as possible.
Good news. The data restore was successful. Characters on the four worlds will be rolled back to their state at between 6AM EDT and 7AM EDT this morning. The possible rollback to Sunday morning appears to have been successfully averted, and worlds are on their way back up now.
We will in the near future need to take the worlds down, possibly one at a time, to do further hardware maintenance on the database servers. We'll be announcing these downtimes in advance.
Status update. FF and LC are having trouble starting with current data, and we are receiving reports of items lost from houses on SC. It may be necessary to revert to the Sunday morning backup after all. It may be possible to leave TD up, we are waiting to see. Will post a further update as soon as I have one.
FF and LC are up and I'm not aware of any reports of lost house items. TD appears to be stable. We are current watching SC very closely.
Right now we are watching SC for any additional reports, and we will have to make a difficult decision. We can either revert all of SC to the Sunday morning backup, inconveniencing everyone, or we can leave it as-is, inconveniencing those players who lost housing items.
Currently we are leaning towards the latter but we are going to be paying close attention through the evening. If you are on Solclaim and have lost items from your house please report it using the Urgent Assistance button.
The database server cluster has suffered another failure, all four worlds are currently down.
I'm waiting for a status update from operations, as soon as I have one I'll post in this thread.
vBulletin® v3.7.1, Copyright ©2000-2010, Jelsoft Enterprises Ltd.