Apr 232010

Obviously, MDC is a mess right now. We’re making headway on understanding the problem — or, at least, understanding what we need to understand.

For some reason, a specific process on the two servers that host MDC is racing out of control; within seconds of startup, it’s chewing up essentially every single cycle of processor time. We don’t yet know why.

We have a plan of attack for hopefully learning why this is happening. It also so happens that this plan may help alleviate the problem somewhat while we continue to analyze the situation.

There are three key problems at hand, all happening at once:

  • We’re running MindTouch 9.08.3 at the moment, and need to upgrade to 9.12.2. That version offers performance and stability improvements.
  • We had hoped that upgrading to MindTouch 9.12.2 would resolve our problems; however, it appears that we have some sort of configuration problem, which is not just the cause of our problems on 9.08.3, but are actually somehow exacerbated by the upgrade to 9.12.2, resulting in a completely unusable system post-upgrade.
  • So we need to figure out what this configuration issue is and resolve it, then do the upgrade again; in theory, this should resolve our problems fairly well.

This is all easier said than done. As far as we — and the folks at MindTouch — can tell, our configuration is generally okay. There are some settings we previously did not have correct, but even after fixing them, performance is still woefully unacceptable.

So the next step is to set up a third machine in addition to the two we currently have running MDC. It will have some additional profiling tools installed, and some percentage of our traffic will be directed to that machine for analysis.

Some of the details of this plan are subject to change, and I’m intentionally being a little vague since we’re still sorting out the specifics and I’m not entirely clear on some of the details yet. We’re having more meetings over the next couple of days to finalize the plan and get things rolling.

 Posted by at 12:10 AM
Apr 212010

As you may have heard, we’re working building the new Mozilla Developer Network, which will unify all our developer support and documentation services under one umbrella. This will make finding the resources you need easier than ever.

While work is ongoing on designing the Mozilla Developer Network web presence, we’ve taken an initial step to help make it easier to find all this content by redesigning the Mozilla Developer Center‘s main page to help route people to the right sites.

The new design features a tabbed interface giving you quick access to information about web development, add-ons, mobile, Mozilla applications, and Mozilla Labs. Sidebars offer quick access to popular content including our HTML and CSS references, documentation about add-on development, and so forth.  And at the bottom of the page, you’ll find links to developer tools and other popular documentation resources.

Hopefully this will help make it easier for newcomers to find what they want, and should even be handy for long-time members of our community. I hope you like it, and I look forward to seeing the redesigned site as we put it together over the coming weeks and months.

 Posted by at 1:49 PM
Apr 162010

As I’ve previously blogged about, I’ve written a Firefox extension that lets you easily add images from the Web to your iPhoto on Mac OS X. This serves both as a handy utility and as a nice demonstration of how to use js-ctypes to access native libraries from JavaScript without having to write any C glue code.

I’ve also written a rather lengthy article on MDC about how it works. Hopefully this will be helpful!

I’m now getting started on writing the more extensive documentation for js-ctypes, including a full reference for its API.

If it’s not obvious, I think js-ctypes is one of the coolest things ever.

 Posted by at 3:02 PM
Apr 162010

After fending off many emails and marking many bugs as duplicates of bug 559648, I figured it’s time to post on my blog explaining that I’m aware that search is currently not working on the Mozilla Developer Center, and what the situation is.

As I described in my post on April 14th, our attempt at upgrading MDC to MindTouch 9.12.2 failed pretty spectacularly. We’re still trying to figure out — with help from MindTouch — exactly what went wrong. So far we don’t really know, although the general consensus is some configuration setting or settings were not correct after the upgrade, which included several major infrastructural changes that might be sensitive to minor configuration issues.

After several unsuccessful attempts to fix the problem, we rolled back to the previous version of the MindTouch software and reverted our database, which had been updated with new schema information as part of the upgrade process.

We missed a step during the rollback; it didn’t occur to us that the Lucene index also needed to be rolled back. So after restarting the site after rolling back the site, we wound up with an older version of MindTouch trying to use a version of the Lucene index built for a newer version of the database. The result: searches are totally busted.

IT should be restoring the Lucene index file from backup sometime today, which should get search back online.

However, because there have been edits to MDC’s content since that index was last updated, we’ll have to do a full rebuild of the index at some point to pick up the changes. I don’t know yet when that will be done, since I’ll need to coordinate with IT to pick a time of day during which MDC is relatively idle, since the indexing process hits the database pretty hard and really bogs things down if there are also a lot of users trying to access the site.

To sum up: we know it’s broken, and we have a plan to fix it, which is in work.

 Posted by at 12:08 PM
Apr 142010

So we installed the MindTouch software upgrade last night, and as many of you already know, it didn’t go well. The site became incredibly unstable, despite the update working very well on our staging server. While MindTouch investigates why this happened, we’ve rolled MDC back to MindTouch 9.08.3.

As an unfortunate side effect, we had to roll back our database to its state as of about 7 AM PDT yesterday, so most edits applied yesterday were lost. We did back up the database as it stood when we took the site down to do the rollback, so we can retrieve any major edits on request, but you may want to check on the status of any pages you edited yesterday. If you made major edits that were lost, please let me know and I’ll try to get them pulled from the backup database. But this is non-trivial, so please only ask for sizable edits.

Obviously I’m really disappointed that this happened, especially after how carefully — and lengthily — we tested and debugged the upgrade process before applying it to the live site. Clearly there’s some key difference between the staging server and the live MDC that we aren’t seeing, but is causing the software to behave radically differently.

It’s possible that it’s simply the usage. Simulated load doesn’t necessarily precisely duplicate real usage. Analysis is underway already. I’ll blog further details as they’re available.

 Posted by at 1:45 PM
Apr 132010

IT has scheduled our upgrade from MindTouch 9.08.3 to MindTouch 9.12.2 for tonight at 4:30 PM Pacific Daylight Time. It shouldn’t be an overly complicated upgrade, and we’ve done it already on our staging server without trouble. I’ve blogged details of what this upgrade gets us (and will hopefully get us) previously, so I’ll leave it at that.

Ironically, this had been scheduled for tonight until just an instant before I hit the “Publish” button on this blog post; then the revised scheduling came in. Timing is, as always, everything.

Edit: This is back on for tonight.

 Posted by at 4:12 PM
Apr 122010

I’m back at the office now for the first time since my surgery. Today will be a little complicated due to certain family obligations during the day, but at least I’m really working!


The big task for this week, other than getting caught up (which shouldn’t be too hard since I generally was able to triage most of my email and did handle MDC editing tasks while I was out), is to start the overhaul of the js-ctypes documentation now that I have a pretty good handle on the changes there.

We’ve scheduled — finally — our upgrade to MindTouch 2009 9.12.2, to take place during tomorrow’s maintenance window. We have hopes that this will further improve reliability and performance. It does on our staging server, at least under certain situations, but it’s hard to gauge how that will carry over to the live site, which obviously gets much more traffic and is more powerful hardware.


I’ve started playing with the code for the iPhone Weave client a bit. I think I’m going to try to build a simple web browser around the code, so I can do quick surfing to look at stuff using my Weave bookmarks and history without having to tap links to swap over to Safari to do it. At the moment, I have the existing client building but it’s crashing on me. Not sure, yet, why that’s happening. Looks like I may have pulled the repository in a state of transition, so I’ll try again tonight.

Weave has turned into something I really can’t live without, and it’s driving me nuts not having it on my iPhone.

This will become more of a problem once I have my iPad; I placed my order for a 3G iPad last night after finally having a chance to try one out yesterday afternoon. I foresee the iPad becoming my primary method of surfing and handling email when I’m at home, just because of how convenient it is to have with me. So getting a good Weave client together is pretty important to me.

 Posted by at 10:19 AM
Apr 102010

I pushed myself to get back to work a little too fast, I think. Worked much of yesterday and by the end of the day was hurting pretty badly, so I took today off and spent almost all of it either sleeping or reclining in a comfortable chair watching TV.

I’ll take it easy through the weekend and will try again to get back to work on Monday.

Apparently just because the surgery only took 20 minutes doesn’t mean I don’t need some time to recover.

 Posted by at 2:13 AM