Obviously, MDC is a mess right now. We’re making headway on understanding the problem — or, at least, understanding what we need to understand.

For some reason, a specific process on the two servers that host MDC is racing out of control; within seconds of startup, it’s chewing up essentially every single cycle of processor time. We don’t yet know why.

We have a plan of attack for hopefully learning why this is happening. It also so happens that this plan may help alleviate the problem somewhat while we continue to analyze the situation.

There are three key problems at hand, all happening at once:

• We’re running MindTouch 9.08.3 at the moment, and need to upgrade to 9.12.2. That version offers performance and stability improvements.
• We had hoped that upgrading to MindTouch 9.12.2 would resolve our problems; however, it appears that we have some sort of configuration problem, which is not just the cause of our problems on 9.08.3, but are actually somehow exacerbated by the upgrade to 9.12.2, resulting in a completely unusable system post-upgrade.
• So we need to figure out what this configuration issue is and resolve it, then do the upgrade again; in theory, this should resolve our problems fairly well.

This is all easier said than done. As far as we — and the folks at MindTouch — can tell, our configuration is generally okay. There are some settings we previously did not have correct, but even after fixing them, performance is still woefully unacceptable.

So the next step is to set up a third machine in addition to the two we currently have running MDC. It will have some additional profiling tools installed, and some percentage of our traffic will be directed to that machine for analysis.

Some of the details of this plan are subject to change, and I’m intentionally being a little vague since we’re still sorting out the specifics and I’m not entirely clear on some of the details yet. We’re having more meetings over the next couple of days to finalize the plan and get things rolling.

We’ve finished rebuilding MDC’s search index, and it’s now searchable again. End of line.

After fending off many emails and marking many bugs as duplicates of bug 559648, I figured it’s time to post on my blog explaining that I’m aware that search is currently not working on the Mozilla Developer Center, and what the situation is.

As I described in my post on April 14th, our attempt at upgrading MDC to MindTouch 9.12.2 failed pretty spectacularly. We’re still trying to figure out — with help from MindTouch — exactly what went wrong. So far we don’t really know, although the general consensus is some configuration setting or settings were not correct after the upgrade, which included several major infrastructural changes that might be sensitive to minor configuration issues.

After several unsuccessful attempts to fix the problem, we rolled back to the previous version of the MindTouch software and reverted our database, which had been updated with new schema information as part of the upgrade process.

We missed a step during the rollback; it didn’t occur to us that the Lucene index also needed to be rolled back. So after restarting the site after rolling back the site, we wound up with an older version of MindTouch trying to use a version of the Lucene index built for a newer version of the database. The result: searches are totally busted.

IT should be restoring the Lucene index file from backup sometime today, which should get search back online.

However, because there have been edits to MDC’s content since that index was last updated, we’ll have to do a full rebuild of the index at some point to pick up the changes. I don’t know yet when that will be done, since I’ll need to coordinate with IT to pick a time of day during which MDC is relatively idle, since the indexing process hits the database pretty hard and really bogs things down if there are also a lot of users trying to access the site.

To sum up: we know it’s broken, and we have a plan to fix it, which is in work.

IT has scheduled our upgrade from MindTouch 9.08.3 to MindTouch 9.12.2 for tonight at 4:30 PM Pacific Daylight Time. It shouldn’t be an overly complicated upgrade, and we’ve done it already on our staging server without trouble. I’ve blogged details of what this upgrade gets us (and will hopefully get us) previously, so I’ll leave it at that.

Ironically, this had been scheduled for tonight until just an instant before I hit the “Publish” button on this blog post; then the revised scheduling came in. Timing is, as always, everything.

Edit: This is back on for tonight.