MindTouch released MindTouch 2010 “Olympic” today. This has a lot of features they added specifically as a result of conversions I had with them earlier this year, and makes a lot of user experience improvements.

Take a look at the release notes.

Given that we’re planning to switch to a new wiki system in the first or second quarter of 2011, I’d like to get your opinions on whether or not we should install a major software update like this one right now.

On the plus side, it does give us some nice new features and usability improvements both for editing and maintaining, as well as supposedly substantial search improvements.

On the minus side, we’ve had problems with major MindTouch upgrades in the past. In addition, some of the exciting new features are ones that we won’t have with the first version of the new MDN wiki system our development team is building for us. I’d hate for people to get used to new features only to have them go away a few months later.

So what do you all think? I could go either way on this one — I’d like to have the new features but worry about how it would impact the community. So let me know how you feel either by email or commenting here.

On Friday, we re-enabled caching for the Mozilla Developer Center in Europe. If you’re in Europe, you should now be seeing much better performance when browsing Mozilla’s documentation.

In general, performance is better than it was a few weeks ago. We do see occasional lag, sometimes worse than others, but the site is definitely working better than it was. Work is ongoing to improve things further.

Among other things, we’re testing enabling a memcache server for caching query results, which would let us cache query results across requests, further improving performance. I don’t know when this testing will be completed.

Looking even farther into the future, MindTouch’s upcoming Olympic release has a bunch of improvements that will hopefully benefit us. This update is due later this summer, and includes:

• CKEditor 3 is the default editor; this editor is faster and will have a much improved toolbar.
• Users will be able to rate content thumbs-up or thumbs-down; this will help make quality content float to the top, and alert writers to material that needs work.
• The article tagging interface will be more efficient.
• User pages will be “dashboards” with mini-apps, notepad space, and user activity reports.
• A new “page layout wizard” will be available; we’ll be able to create page templates that users can choose among to pre-layout content. For example, when documenting an interface, you can get a page with the basic hierarchy of the page, including the default tables for attributes, constants, and methods, already in place.
• Improved search, with smarter page ranking, various search result sort options, and more.
• Improved “curation” features, to help identify potentially out-of-date content.

Should be a nice update. We’ll keep an eye on it.

We had a brief meeting with the folks at MindTouch yesterday. As usual, I thought I’d share the highlights of what’s going on with you.

Stability

We’re definitely more stable than we were, although we are still having occasional errors. We don’t yet know why, but I’m looking into that and will post again once there’s something to report.

RSS feeds

MindTouch has implemented some new caching support for RSS feeds that they’ve given us a patch for, which we’ll be installing soon. At the same time, we’ll be upgrading to MindTouch 9.12.3, which is apparently arriving shortly. We’re actually already using most of the fixes in that update (there’s only one that we don’t already have).

Caching

We also plan, sometime next week, to bump the caching currently in use from “request caching”, which only helps a little bit, to using memcache to cache more usefully. This will also let all our hosts share cached data, instead of each maintaining their own cache.

All of these changes are going to be tested and staged before being deployed, so I don’t have exact dates yet on which you’ll see them take effect.

Performance status

Performance is better than it has been in some time, although it’s clearly got more room for improvement. We’re still chasing stuff related to the RSS feeds right now, but once that’s done, we’ll do another round of profiling to see if we can figure out where we stand.

Hopefully we’ll be able to start re-enabling some of the feeds we’ve shut off after that.

One thing we did already this week is fix an error in our robots.txt file; all spiders were being told to ignore the entire site, instead of just the areas we wanted them to ignore. This hopefully won’t negatively impact performance (it certainly won’t improve it), but should get our search results on Google and the like back up to date.

Although we’ve actually been working on Gecko 1.9.3 documentation for a while now, we finally now know that these changes will be in Firefox 4. As such, I’ve now set up the Firefox 4 for developers page on the Mozilla Developer Center site; this page will provide links to all Firefox 4 developer documentation.

A few articles have already been written. There’s lots left to do, of course, but we’re making progress. If you see something you’d like to write about, feel free to contribute!

As always, the content of that page is in flux while the feature set of Firefox 4 continues to be adjusted. In addition, it’s obviously too early to be sure exactly what articles will be produced for all the topics at hand, so there are a number of placeholders.

If you’ve contributed code to Gecko or Firefox recently, you may be hearing from me or one of the documentation contributors soon, to gather information about your area of expertise as documentation comes together.

As I mentioned earlier today, we’ve disabled (for now) all of the RSS feeds on MDC except the site-wide “recent changes” feed.

Even though feeds were being cached, and only refreshed on demand, spiders and bots were causing “on demand” to mean “every time they expired”. And since every page on the site has multiple feeds, and every user has a feed of their contributions, something like 360,000 feeds were being queued for regeneration, meaning that there were pretty much always feeds being regenerated, chewing up lots of CPU.

Now, instead of each host constantly having something like 25+ tasks ongoing, never less, each host is averaging more like 3 ongoing tasks at any given time, because they’re not piling up while the system is busy handling RSS feed generation.

Work is ongoing on coming up with a fix for the performance of the feed generation code, so that the other feeds can be re-enabled, but at least we have the main one still.

In the meantime, performance seems quite reasonable. Please let me know if you run into any performance issues, so I can get them looked into.

In an attempt to make the Mozilla Developer Center generally usable, we’re going to disable most RSS feeds offered by the site, with the exception of the site-wide “recent changes” feed, which we use for monitoring the site for maintenance purposes.

Hopefully this won’t be too huge an inconvenience, and hopefully it will also help performance enough to make the site usable while we pursue a real fix for our feed-related performance problems.

I don’t know yet when this change will occur, but it should be soon.

We’ve been running on MindTouch 9.12.2 now for about 12 hours, and things seem stable. As long as the RSS feed generators aren’t running, the site is quite fast, but as soon as the feed generator starts working to build a feed whose cache has expired, the CPU gets bogged down pretty mightily.

This is alleviated somewhat by the fact that we’re now running MDC on three hosts instead of two. That said, each individual box still bogs down when any feeds are updating, and there are lots of different feeds. So even though each feed is set to expire after an hour, and only gets rebuilt when someone loads it after it’s expired, it still causes a major performance drain when the feeds do rebuild.

Anyway, we have a third box in the pool now, which will help some, and I’m talking actively with MindTouch (literally while writing this blog post) about ways to improve the RSS feed impact to our performance while they work toward rewriting the feed generation code.

One idea I have, which they’re going to do some testing on today, is to set up a fourth box in the pool, and direct all — and only — feed traffic to it, then disable the feed generators on the other three hosts. That way, the boxes serving up the site content never get hammered by the feed generation, while a dedicated machine can use all its CPU to handle the feeds.

It’s a bit of a cheesy hack, but I think it will work. I’ll blog further as our research continues!

In the meantime, I hope to re-enable editing on MDC very shortly; I want to confirm one last thing with IT before I do so. I’ll post again when editing is switched back on. Thanks for your patience!

We’ve largely finished the MDC upgrade process. A few notes about where things stand right now:

• Our two existing hosts are up and running, using a slightly out of date search index.
• Our third, new and more powerful host is currently outside the pool building the updated search index.
• When that new box finishes building the index, we’ll copy it to the other two boxes, then add the new host to the pool.
• Editing will remain disabled until the morning or early afternoon, so we can monitor the status of the site for a while to ensure things are stable before allowing edits; this way, if by some chance a revert becomes necessary, we don’t lose anything.

There’s a patch we’ll be installing, hopefully sometime Wednesday, that will address a glitch we ran into for a while under a specific circumstance.

Right now, things are performing nicely though. We’ll be keeping an eye on it for a while longer tonight, and tomorrow we’ll see where we stand.

As I’ve mentioned before, we’ve found that the code that generates RSS feeds is pretty sluggish, and chews up a ton of CPU. This is something we’re talking to MindTouch about addressing sooner rather than later. Right now, the feeds are being cached for an hour, so they won’t update as often as they used to. But this should help the site remain reasonably peppy. Time will tell!

For now, Jeremy in IT, Pete from MindTouch, and I are hanging around keeping an eye on things for a while to see if things stay stable. If they do for a while, we’ll sleep. If things are still looking good when I wake up, I’ll re-enable editing.

I thought I’d add a few notes on what’s up with MDC since my last report.

Upgrade Ahoy

First, the upgrade from MindTouch 9.08.3 to 9.12.2 is still scheduled for Tuesday, but has been bumped from 4 PM Pacific Daylight Time to 7 PM Pacific Daylight Time, in order to better accommodate the schedules of certain people that need to be available during the upgrade. So MDC will be (hopefully briefly) offline around that time while we perform that upgrade.

Performance

I also have some additional news regarding our work on performance issues.

I’ve revised a few of our templates — particularly some of those that generate sidebars in the XUL Reference — to use the MindTouch web caching service. This means that the templates now cache the HTML they generate for up to an hour before regenerating them.

This appears to have noticeably improved performance while browsing the XUL Reference. These templates are very large and call multiple other templates; by caching their results, we save many, many dozens of template executions for each page rendered in the XUL Reference.

Also, some of the testing MindTouch did determined that the mediawiki extension, which is used in templates that were built using MindTouch’s automated MediaWiki to MindTouch conversion utility, can be very slow. We have several dozen templates still using this extension. That needs to change.

So I’m going through and rewriting every template and page that uses this extension. This will take some time. Also, I’m only doing it for English; if you work on documentation for another language, you probably will want to try to do the same for pages and templates for your language.

You can download a copy of the list of pages that use the mediawiki template to take a look at it. Some of them have already been fixed, but most still need going over. It’s possible that some of these templates are entirely obsolete, too. Feel free to help fix these!

Note: I’ve fixed the corrupt UTF-8 in the list now.

The past couple of weeks of testing and investigation have revealed some interesting things about the Mozilla Developer Center and how we use the MindTouch software on it. I thought I’d share some of what we’ve learned, and what’s being done about it.

RSS Feeds

The MindTouch software responds to each request for an RSS feed by scouring the database and generating the results on the fly. For small, low-traffic sites, which have few RSS users, that works okay. But MDC has a lot of users, many of whom watch RSS feeds. Generating the results for an RSS feed hit takes time and a lot of database hits.

MindTouch has developed a patch, which we’ve been testing, to enable caching of RSS feeds. The feeds will be cached for some amount of time — the exact amount of time is still being experimented with, but we’ll be able to change later. While this will reduce the granularity of updates, it will dramatically reduce load on the site.

Templates

We make extensive use of sometimes complex templates on MDC. For the uninitiated, templates are embeddable snippets that can include script code to generate content on the fly. We have templates that are very simple, generating a quick string based on a parameter, but we also have some that are very involved, doing lookups of potential target pages to generate valid links depending on the availability of different destinations.

Some of our pages use a lot of templates. For example, let’s consider the XUL prefpane element‘s document. It directly uses 37 templates, each of which is in turn using others. As a result, on the current MDC site, this page requires — wait for it — over 11,000 database queries to render.

Let me say that again.

Over 11,000 database queries. Now, it’s important to note here that our database server doesn’t care. It’s hardly noticing any load at all. The problem appears to be in the overhead required to handle setting up and issuing the queries, and waiting for the results.

However, with just a handful of pages being loaded at once, the number of connections to the database server can get quite large. And the site is currently not configured to allow more than 50 at a time. So only a few pages need to be getting loaded at once, and you’ll start getting “unable to access database” errors. This is also why the site was crashing; when the available connections ran out, the site would start stacking up pending queries at a rapid pace, until finally the site would just die.

Obviously, this is totally uncool. I don’t think MindTouch ever realized just how much use people would get out of templates.

So what’s being done about it? A few things.

First, MindTouch has created a patch for us that caches the results of database queries for a few minutes. This reduces the number of queries required to render that page by a couple of orders of magnitude.

Second, we’ll be increasing the number of connections to the database server that can be open at once, by a sizable amount.

Third, we’ll be enabling caching of page diffs. I’m not sure how much win we’ll get from this, but every little bit will help.

Now, to be honest, I’m not sure how much of a performance gain we’ll get out of these changes, but it should dramatically improve reliability, at least.

There is another thing we’re investigating, but it may or may not be implemented due to complexities in terms of site management.

MindTouch offers a caching service that allows templates to cache data for re-use later. Using this would require rewriting a lot of our templates, and possibly even significant revisions to the pages that use them. We’re looking at how difficult this would be, as well how much performance gain we might get from doing it.

Basically, what it comes down to, is this: processing templates is taking a surprising amount of CPU time, and we use a lot of them, resulting in the server becoming overloaded very quickly.

Upgrading to 9.12.2 — Again

We’re planning to attempt this upgrade again on Tuesday, May 18th, at 4 PM Pacific Daylight Time. This time, we’ve done a substantial amount of testing around the areas that failed on us last time, and MindTouch personnel will be available to help if anything does go wrong.

The failure of the last upgrade was caused by the database connection pool configuration issue I mentioned above. We have confidence that the upgrade will work better this time.

More Metal

We’ll be increasing the number of hosts driving MDC. I don’t know what the schedule is for this. Obviously this will help too.

In Summary

To be clear: I don’t expect MDC to suddenly become amazingly fast and responsive next Tuesday evening after the upgrade and all these tweaks are put into place. I do expect that it will become more stable, however, and performance will hopefully be better to some extent.

There’s ongoing work to be done to further improve performance, and it’s too early to say exactly when that will all happen.

I know this has all been frustrating. Trust me, I’m vastly more annoyed by it all than you are. Please be nice to me in your comments. I don’t really feel like being yelled at anymore about this.