Disconnects and lag post-event

[Ibn]

As the worlds come back up, we're seeing a number of reports of disconnects and inability to connect to the servers. We are investigating these reports.

I must note that the Internet as a whole may be experiencing increased latency due to several email viruses that are being resent at a very high volume. I am not saying that this is the cause -- as I said, we are investigating -- but it may be a factor.

[Ibn]

Regarding the second part -- my IT folks are telling me that parts of the 'net are currently down to about 40% of their total throughput, with the rest being taken up by the new viruses.

We've just cycled the server lobbies again to try and resolve the issue with some of them still being unavailable.

[Ibn]

We are taking down all the worlds while we investigate this issue.

All the worlds were backed up prior to the event, so if necessary we can restore from those backups.

Watch this thread for further updates.

[Ibn]

No new status at the moment -- we are still investigating the issues.

[Ibn]

We are continuing to investigate. Due to the severity of the issue, we cannot in good faith bring the worlds back online until it is resolved.

One rumor that I'd like to address -- I've seen some folks wonder if this was caused by player actions or a malicious user. While we'd love to be able to hold a hacker responsible for this, currently this does appear to be a bug in our code.

[Ibn]

We believe we have found the bug that was causing this issue. We are at this very moment working on resolving it and getting the updated code to the worlds.

It will be unfortunately necessary to roll the worlds back to the snapshot taken immediately after they came down for the event this morning.

What does this mean? You will keep everything that you had when the worlds came down for the event. However, everything that was done between the worlds coming back up, and going down again, will be lost.

We apologize for this extended patch-day downtime and we thank you for your patience.

[Ibn]

The database restores are proceeding as we continue building the fixed server executable. No ETA at the moment, but I am working on getting one.

[Ibn]

OK, we are sending the operations team the updated server, they are also still working on completing the restore from backup.

Currently we expect the servers to be back up between 7PM and 8PM PST (10PM and 11PM EST).

We apologize again for this extended downtime and thank you for being patient with us.

[Ibn]

It is possible that we may be back up earlier than I indicated in my previous email. Keep your fingers crossed.

[Ibn]

Lobbies should be coming up now!