After fending off many emails and marking many bugs as duplicates of bug 559648, I figured it’s time to post on my blog explaining that I’m aware that search is currently not working on the Mozilla Developer Center, and what the situation is.
As I described in my post on April 14th, our attempt at upgrading MDC to MindTouch 9.12.2 failed pretty spectacularly. We’re still trying to figure out — with help from MindTouch — exactly what went wrong. So far we don’t really know, although the general consensus is some configuration setting or settings were not correct after the upgrade, which included several major infrastructural changes that might be sensitive to minor configuration issues.
After several unsuccessful attempts to fix the problem, we rolled back to the previous version of the MindTouch software and reverted our database, which had been updated with new schema information as part of the upgrade process.
We missed a step during the rollback; it didn’t occur to us that the Lucene index also needed to be rolled back. So after restarting the site after rolling back the site, we wound up with an older version of MindTouch trying to use a version of the Lucene index built for a newer version of the database. The result: searches are totally busted.
IT should be restoring the Lucene index file from backup sometime today, which should get search back online.
However, because there have been edits to MDC’s content since that index was last updated, we’ll have to do a full rebuild of the index at some point to pick up the changes. I don’t know yet when that will be done, since I’ll need to coordinate with IT to pick a time of day during which MDC is relatively idle, since the indexing process hits the database pretty hard and really bogs things down if there are also a lot of users trying to access the site.
To sum up: we know it’s broken, and we have a plan to fix it, which is in work.