From the Author:2022/On the state of db141
Originally published on: 2022-11-30, 23:38
Updated: 2022-11-30, 23:38
Shortcut: FTA:20221130
...straight from the Miraheze team themselves; further details in subsequent edits, because we're closing in on the UTC deadline and I have niblings to look after. (Text republished under CC-BY-SA 4.0.)
Cloud14 issues
The cloud server (cloud14) which hosts one of our database, db141, experienced a disk issue. As a result, a small number of wikis hosted on db141 are unavailable. We have reinstalled the affected server on new disks and are working to recover the data from the affected disks. Earliest ETA of these wikis being back online is early next week. We deeply apologise for the inconvenience but rest assured we're working diligently to have this issue fixed ASAP.
- LATEST UPDATE
- 4AM (UTC), Tuesday, Nov. 29 - The affected disks have been shipped to Owen as of November 24th. We are still in the process of determining how to recover the data and if it is even feasible by our means. The previous update has been amended to reflect the fact that we have not yet involved a professional data recovery service as it may be prohibitively expensive to do so.
- 2AM (UTC), Monday, Nov. 21 - We have reinstalled cloud14 and have began re-provisioning servers affected by the disk issue. Mail and IRC bots are now functional. We are working on re-provisioning servers for MediaWiki which should improve loading speeds. We are in the process of sending the disks containing db141 to Owen to review the physical disks and determine how to proceed with professional data recovery and the earliest ETA we can provide for when wikis may be back online is early next week.
- FAQ
- What happened?
A cloud server (cloud14) hosting one of our database, db141, ran into disk issues. As a result, the database cannot be accessed and some services hosted by the cloud server have been knocked offline. We have reinstalled the affected cloud server on new disks and are working to restore affected services.
- Who is affected?
Only wikis on db141. Affected wikis display an error saying "Wiki temporarily unavailable." Most wikis on Miraheze are fine.
- When will this be fixed?
While cloud14 has been reinstalled, we will have to send the affected disks to professional data recovery. The earliest ETA for having wikis restored is potentially early next week.
- Is data loss involved?
We are unsure. It may be possible that the disks are not actually faulty but rather that the RAID controller is which would mean your data is safe, or it's possible the actual disks have gone bad. If it is the latter, that would indicate we received a bad batch of SSDs from the manufacturer.
- What other services are affected?
At this moment, the only user-facing affected server is MediaWiki due to some servers being knocked offline. We are working to provision new MediaWiki servers which should fix loading.
- What is the plan for now?
We have reinstalled the affected cloud server on new disks. Most affected services (excluding wikis) are fully functional once again. We are going to send the affected disks to a professional data recovery service to see what can be done. While costly, we thank each one of our donors who has supported us along the way. If all goes well, the earliest estimate for affected wikis coming back online is early next week.
Our number one priority at this moment is restoring wikis. About 500 open public wikis are affected by this so we understand this has certainly caused an impact for many of Miraheze's users. Rest assured we have not forgotten about those wikis. Every one of our 5,500+ wikis is important so we are working very hard to restore these wiki's data and bring them back online. We are so grateful that for the patience our users have had before this unprecedented issue. We will be posting updates here so please stay tuned. If you have any questions, please join us on our Discord. Thank you. Miraheze Site Reliability Engineering 00:00, 20 November 2022 (UTC)
Obligatory Shimajiro recap, because permanent feature. As we're running out of time—and ideas—this portrait of the tiger-cub mascot in puppet form is all I could find from the latest tweets.
Due to interference between MediaWiki:Edittools and the Form-namespace infrastructure as of late (T9882—reported to both Yaron Koren and Phabricator; further investigation/diagnosis pending), recent editions of FTA have been composed on the traditional MW editbox until further notice. Once this bug is resolved, I'll remind you right here.
And that wraps it up right here, folks; be there soon as I take care of the agenda and the maps, report on our brand-new Dell Inspiron (while we're on that system), see about my side of an art trade, and try to get my PayPal credentials going—if only to cut as many ties with them as I can per my superior's insistence. Until next we meet, take care, stay safe/connected, keep exploring, tapal, see you in the bestsellers, we'll meet you back home...and watch your tails.
P.S. I'm finally getting my shots soon—a year late. See you after my mandatory rest.
Remembering Elizabeth II; in solidarity with those affected by Ian and Nicole.
To our next ten years...
(3808)
Characters (c) Benesse Corporation/Shimajiro; English version licensed by WildBrain.
- No comments yet. To leave one, please select the "Discussion" tab and follow the general talk rules above the edit box.