So we installed the MindTouch software upgrade last night, and as many of you already know, it didn’t go well. The site became incredibly unstable, despite the update working very well on our staging server. While MindTouch investigates why this happened, we’ve rolled MDC back to MindTouch 9.08.3.
As an unfortunate side effect, we had to roll back our database to its state as of about 7 AM PDT yesterday, so most edits applied yesterday were lost. We did back up the database as it stood when we took the site down to do the rollback, so we can retrieve any major edits on request, but you may want to check on the status of any pages you edited yesterday. If you made major edits that were lost, please let me know and I’ll try to get them pulled from the backup database. But this is non-trivial, so please only ask for sizable edits.
Obviously I’m really disappointed that this happened, especially after how carefully — and lengthily — we tested and debugged the upgrade process before applying it to the live site. Clearly there’s some key difference between the staging server and the live MDC that we aren’t seeing, but is causing the software to behave radically differently.
It’s possible that it’s simply the usage. Simulated load doesn’t necessarily precisely duplicate real usage. Analysis is underway already. I’ll blog further details as they’re available.