HUGE UPDATE
Most wikis that were only the original db141 have now been brought back online. The db141 hostname has been retired due to it's history, and is now replaced with db142.
A few of the original ones have not yet been able to be restored, and wikis that had been recreated after the initial outage are also still not back online yet. We are still working at this process to bring wikis back online fully.
Wikis created (initial creation) after November 25 will not be possible to be restored. However, we have backups for all public wikis beginning with A-O, and can restore those from backups.
If you requested a recreation after the initial outage, it will take a bit more time to get those wikis back online. The version that will eventually be restored will be the pre November 16 version. If you wish to have the wiki reset, or imported from personal backups, or if it is one we have backups from, you may later request a reset of that wiki, to this state.
Also not that around 20 original wikis were (so-far) unable to be restored due to varying degrees of issues, and tables missing/corruption. We are not certain in the ability to later restore these wikis. The updated list of affected wikis may be read here.
We will try our best to get us fully operational again. Thank you for bearing with us during this time, and for all the patience.
I would also like to note that in light of these issues, some additional plans have been put in place, such as active automated wiki backups again (stored externally, off-site). See Backups#General backup Schedules. Additionally, an internal SRE policy has been drafted and should become public soon, to prevent the issues that have occurred during the previous maintenance window. It formalizes our policy regarding server maintenance, and requires backups be 100% complete and stored off the affected servers before any maintenance can be conducted.
We are very sorry for all the issues that has occurred over the past month, and hope to never have an issue like this again, and backups are in place.
If you encounter certain database exceptions on any of the now-restored wikis, please let us know. Thank you!
Update: Wikis originally recreated are now being brought back online with original data. No data after recreation. If a backup exists, they may be imported. Or you may later request a reset.
Comments
Post a Comment