May 272010
 

We had a brief meeting with the folks at MindTouch yesterday. As usual, I thought I’d share the highlights of what’s going on with you.

Stability

We’re definitely more stable than we were, although we are still having occasional errors. We don’t yet know why, but I’m looking into that and will post again once there’s something to report.

RSS feeds

MindTouch has implemented some new caching support for RSS feeds that they’ve given us a patch for, which we’ll be installing soon. At the same time, we’ll be upgrading to MindTouch 9.12.3, which is apparently arriving shortly. We’re actually already using most of the fixes in that update (there’s only one that we don’t already have).

Caching

We also plan, sometime next week, to bump the caching currently in use from “request caching”, which only helps a little bit, to using memcache to cache more usefully. This will also let all our hosts share cached data, instead of each maintaining their own cache.

All of these changes are going to be tested and staged before being deployed, so I don’t have exact dates yet on which you’ll see them take effect.

Performance status

Performance is better than it has been in some time, although it’s clearly got more room for improvement. We’re still chasing stuff related to the RSS feeds right now, but once that’s done, we’ll do another round of profiling to see if we can figure out where we stand.

Hopefully we’ll be able to start re-enabling some of the feeds we’ve shut off after that.

One thing we did already this week is fix an error in our robots.txt file; all spiders were being told to ignore the entire site, instead of just the areas we wanted them to ignore. This hopefully won’t negatively impact performance (it certainly won’t improve it), but should get our search results on Google and the like back up to date.

 Posted by at 12:57 PM
May 262010
 

Although we’ve actually been working on Gecko 1.9.3 documentation for a while now, we finally now know that these changes will be in Firefox 4. As such, I’ve now set up the Firefox 4 for developers page on the Mozilla Developer Center site; this page will provide links to all Firefox 4 developer documentation.

A few articles have already been written. There’s lots left to do, of course, but we’re making progress. If you see something you’d like to write about, feel free to contribute!

As always, the content of that page is in flux while the feature set of Firefox 4 continues to be adjusted. In addition, it’s obviously too early to be sure exactly what articles will be produced for all the topics at hand, so there are a number of placeholders.

If you’ve contributed code to Gecko or Firefox recently, you may be hearing from me or one of the documentation contributors soon, to gather information about your area of expertise as documentation comes together.

 Posted by at 6:02 PM
May 202010
 

As I mentioned earlier today, we’ve disabled (for now) all of the RSS feeds on MDC except the site-wide “recent changes” feed.

Even though feeds were being cached, and only refreshed on demand, spiders and bots were causing “on demand” to mean “every time they expired”. And since every page on the site has multiple feeds, and every user has a feed of their contributions, something like 360,000 feeds were being queued for regeneration, meaning that there were pretty much always feeds being regenerated, chewing up lots of CPU.

Now, instead of each host constantly having something like 25+ tasks ongoing, never less, each host is averaging more like 3 ongoing tasks at any given time, because they’re not piling up while the system is busy handling RSS feed generation.

Work is ongoing on coming up with a fix for the performance of the feed generation code, so that the other feeds can be re-enabled, but at least we have the main one still.

In the meantime, performance seems quite reasonable. Please let me know if you run into any performance issues, so I can get them looked into.

 Posted by at 5:30 PM
May 202010
 

In an attempt to make the Mozilla Developer Center generally usable, we’re going to disable most RSS feeds offered by the site, with the exception of the site-wide “recent changes” feed, which we use for monitoring the site for maintenance purposes.

Hopefully this won’t be too huge an inconvenience, and hopefully it will also help performance enough to make the site usable while we pursue a real fix for our feed-related performance problems.

I don’t know yet when this change will occur, but it should be soon.

 Posted by at 11:06 AM
May 192010
 

We’ve been running on MindTouch 9.12.2 now for about 12 hours, and things seem stable. As long as the RSS feed generators aren’t running, the site is quite fast, but as soon as the feed generator starts working to build a feed whose cache has expired, the CPU gets bogged down pretty mightily.

This is alleviated somewhat by the fact that we’re now running MDC on three hosts instead of two. That said, each individual box still bogs down when any feeds are updating, and there are lots of different feeds. So even though each feed is set to expire after an hour, and only gets rebuilt when someone loads it after it’s expired, it still causes a major performance drain when the feeds do rebuild.

Anyway, we have a third box in the pool now, which will help some, and I’m talking actively with MindTouch (literally while writing this blog post) about ways to improve the RSS feed impact to our performance while they work toward rewriting the feed generation code.

One idea I have, which they’re going to do some testing on today, is to set up a fourth box in the pool, and direct all — and only — feed traffic to it, then disable the feed generators on the other three hosts. That way, the boxes serving up the site content never get hammered by the feed generation, while a dedicated machine can use all its CPU to handle the feeds.

It’s a bit of a cheesy hack, but I think it will work. I’ll blog further as our research continues!

In the meantime, I hope to re-enable editing on MDC very shortly; I want to confirm one last thing with IT before I do so. I’ll post again when editing is switched back on. Thanks for your patience!

 Posted by at 3:13 PM
May 192010
 

We’ve largely finished the MDC upgrade process. A few notes about where things stand right now:

  • Our two existing hosts are up and running, using a slightly out of date search index.
  • Our third, new and more powerful host is currently outside the pool building the updated search index.
  • When that new box finishes building the index, we’ll copy it to the other two boxes, then add the new host to the pool.
  • Editing will remain disabled until the morning or early afternoon, so we can monitor the status of the site for a while to ensure things are stable before allowing edits; this way, if by some chance a revert becomes necessary, we don’t lose anything.

There’s a patch we’ll be installing, hopefully sometime Wednesday, that will address a glitch we ran into for a while under a specific circumstance.

Right now, things are performing nicely though. We’ll be keeping an eye on it for a while longer tonight, and tomorrow we’ll see where we stand.

As I’ve mentioned before, we’ve found that the code that generates RSS feeds is pretty sluggish, and chews up a ton of CPU. This is something we’re talking to MindTouch about addressing sooner rather than later. Right now, the feeds are being cached for an hour, so they won’t update as often as they used to. But this should help the site remain reasonably peppy. Time will tell!

For now, Jeremy in IT, Pete from MindTouch, and I are hanging around keeping an eye on things for a while to see if things stay stable. If they do for a while, we’ll sleep. If things are still looking good when I wake up, I’ll re-enable editing.

 Posted by at 2:11 AM
May 172010
 

I thought I’d add a few notes on what’s up with MDC since my last report.

Upgrade Ahoy

First, the upgrade from MindTouch 9.08.3 to 9.12.2 is still scheduled for Tuesday, but has been bumped from 4 PM Pacific Daylight Time to 7 PM Pacific Daylight Time, in order to better accommodate the schedules of certain people that need to be available during the upgrade. So MDC will be (hopefully briefly) offline around that time while we perform that upgrade.

Performance

I also have some additional news regarding our work on performance issues.

I’ve revised a few of our templates — particularly some of those that generate sidebars in the XUL Reference — to use the MindTouch web caching service. This means that the templates now cache the HTML they generate for up to an hour before regenerating them.

This appears to have noticeably improved performance while browsing the XUL Reference. These templates are very large and call multiple other templates; by caching their results, we save many, many dozens of template executions for each page rendered in the XUL Reference.

Also, some of the testing MindTouch did determined that the mediawiki extension, which is used in templates that were built using MindTouch’s automated MediaWiki to MindTouch conversion utility, can be very slow. We have several dozen templates still using this extension. That needs to change.

So I’m going through and rewriting every template and page that uses this extension. This will take some time. Also, I’m only doing it for English; if you work on documentation for another language, you probably will want to try to do the same for pages and templates for your language.

You can download a copy of the list of pages that use the mediawiki template to take a look at it. Some of them have already been fixed, but most still need going over. It’s possible that some of these templates are entirely obsolete, too. Feel free to help fix these!

Note: I’ve fixed the corrupt UTF-8 in the list now.

 Posted by at 12:56 PM
May 112010
 

The past couple of weeks of testing and investigation have revealed some interesting things about the Mozilla Developer Center and how we use the MindTouch software on it. I thought I’d share some of what we’ve learned, and what’s being done about it.

RSS Feeds

The MindTouch software responds to each request for an RSS feed by scouring the database and generating the results on the fly. For small, low-traffic sites, which have few RSS users, that works okay. But MDC has a lot of users, many of whom watch RSS feeds. Generating the results for an RSS feed hit takes time and a lot of database hits.

MindTouch has developed a patch, which we’ve been testing, to enable caching of RSS feeds. The feeds will be cached for some amount of time — the exact amount of time is still being experimented with, but we’ll be able to change later. While this will reduce the granularity of updates, it will dramatically reduce load on the site.

Templates

We make extensive use of sometimes complex templates on MDC. For the uninitiated, templates are embeddable snippets that can include script code to generate content on the fly. We have templates that are very simple, generating a quick string based on a parameter, but we also have some that are very involved, doing lookups of potential target pages to generate valid links depending on the availability of different destinations.

Some of our pages use a lot of templates. For example, let’s consider the XUL prefpane element‘s document. It directly uses 37 templates, each of which is in turn using others. As a result, on the current MDC site, this page requires — wait for it — over 11,000 database queries to render.

Let me say that again.

Over 11,000 database queries. Now, it’s important to note here that our database server doesn’t care. It’s hardly noticing any load at all. The problem appears to be in the overhead required to handle setting up and issuing the queries, and waiting for the results.

However, with just a handful of pages being loaded at once, the number of connections to the database server can get quite large. And the site is currently not configured to allow more than 50 at a time. So only a few pages need to be getting loaded at once, and you’ll start getting “unable to access database” errors. This is also why the site was crashing; when the available connections ran out, the site would start stacking up pending queries at a rapid pace, until finally the site would just die.

Obviously, this is totally uncool. I don’t think MindTouch ever realized just how much use people would get out of templates.

So what’s being done about it? A few things.

First, MindTouch has created a patch for us that caches the results of database queries for a few minutes. This reduces the number of queries required to render that page by a couple of orders of magnitude.

Second, we’ll be increasing the number of connections to the database server that can be open at once, by a sizable amount.

Third, we’ll be enabling caching of page diffs. I’m not sure how much win we’ll get from this, but every little bit will help.

Now, to be honest, I’m not sure how much of a performance gain we’ll get out of these changes, but it should dramatically improve reliability, at least.

There is another thing we’re investigating, but it may or may not be implemented due to complexities in terms of site management.

MindTouch offers a caching service that allows templates to cache data for re-use later. Using this would require rewriting a lot of our templates, and possibly even significant revisions to the pages that use them. We’re looking at how difficult this would be, as well how much performance gain we might get from doing it.

Basically, what it comes down to, is this: processing templates is taking a surprising amount of CPU time, and we use a lot of them, resulting in the server becoming overloaded very quickly.

Upgrading to 9.12.2 — Again

We’re planning to attempt this upgrade again on Tuesday, May 18th, at 4 PM Pacific Daylight Time. This time, we’ve done a substantial amount of testing around the areas that failed on us last time, and MindTouch personnel will be available to help if anything does go wrong.

The failure of the last upgrade was caused by the database connection pool configuration issue I mentioned above. We have confidence that the upgrade will work better this time.

More Metal

We’ll be increasing the number of hosts driving MDC. I don’t know what the schedule is for this. Obviously this will help too.

In Summary

To be clear: I don’t expect MDC to suddenly become amazingly fast and responsive next Tuesday evening after the upgrade and all these tweaks are put into place. I do expect that it will become more stable, however, and performance will hopefully be better to some extent.

There’s ongoing work to be done to further improve performance, and it’s too early to say exactly when that will all happen.

I know this has all been frustrating. Trust me, I’m vastly more annoyed by it all than you are. Please be nice to me in your comments. I don’t really feel like being yelled at anymore about this. :)

 Posted by at 12:10 PM
May 032010
 

I’ve been using it some more, and I have additional thoughts. Go figure!

Back To My Mac

One thing that would be insanely useful for me would be support for accessing Back To My Mac services using my iPad. Prime among these would be accessing my Mac via Screen Sharing. Yes, I’m aware there are VNC clients available, but VNC uses more bandwidth and is slower than the Apple Remote Desktop style screen sharing used by BTMM. I would love to be able to quickly slip over to my Mac and do something like email myself a file I forgot or something like that.

Apple’s Extra Apps

Apple has released some extra iPhone apps over the years. I own two of these: Texas Hold ‘Em and the iDisk app. Neither of these has been updated for iPad yet, which is annoying.

Keyboard Dock

I like this thing. I bought it on a whim, but I predict I’ll actually use it pretty often. I like that waking up the iPad using the keyboard doesn’t require the “Slide to unlock” part.

Printing

It’s amazing how fast you get so used to doing stuff on the iPad that the lack of printing support from arbitrary apps becomes a problem. For example, last night I placed an order online on it, forgetting that I needed to be able to print my receipt. Now I’m stuck. Apple needs to add a printing API, or at a bare minimum a way to email a PDF of the page you’re looking at in Safari. Just emailing links to the desktop doesn’t always cut it.

 Posted by at 6:04 PM
May 022010
 

I got my iPad with 3G a couple of days ago, and I have some initial thoughts.

First off, by and large, this is possibly the most amazing device I’ve ever used. It’s beautiful, it’s fun to use, and it seems perfectly suited to replace my laptop for those basic tasks that I used to whip it out for: email, general web surfing, quick blogging, etc.

Obviously I’ll be keeping the MacBook Pro for heavier-duty tasks.

However, I thought I’d sum up a few specific things that occur to me as I use it.

Safari

Browsing the web works pretty well in Safari, for basic surfing tasks. It feels quite natural, by and large, and pages render cleanly and quickly.

However, I do have one specific nit to pick: when you switch between tabs, about 90% of the time, Safari reloads the tab you switch into. This, for my needs, is a serious drawback.

One of the major tasks I plan(ned) to use my iPad for is basic maintenance of the Mozilla Developer Center wiki. That is, reading the recently changed articles list in Google Reader and opening up articles that need re-editing in new tabs to tweak them or proofread them, and also occasionally opening up the user ban panel to ban users.

This is all technically possible, except for this common usage scenario:

  1. A user creates an account.
  2. The first thing they do is post spam.
  3. Now I need to both delete the spam and ban the user.
  4. So I click the link in the RSS feed to open the article to delete it.
  5. Now I switch back to the Google Reader tab so I can click the “ban user” button.
  6. But the page reloads, and that item in my Google Reader tab, now having been read, is no longer available, so I’ve lost the “ban user” button.

Frustrating. There are workarounds, but they’re tedious and require significant changes to my workflow over how I do things on the desktop.

I fail to understand why Safari insists on reloading tabs on switch. Especially when it doesn’t do it every single time. I’d very much like it to stop, please.

Mail

Mail is good but not awesome on the iPad. To be fair, doing mail “awesome” is a difficult task, one which I don’t think anyone has really achieved yet.

My main gripe about Mail on the iPad is that it insists on opening messages automatically. This same thing frustrates me on desktop clients. Having to turn off a “preview” mode on the desktop is annoying enough; having it be the only way to fly on the iPad is annoying. I only want to see the content of a message when I explicitly tap on it. Until then, I’d rather see a blank panel next to it. Seeing the content of messages automatically makes it too easily to accidentally mark a message as read before I’m actually ready to read it, let alone act on its contents.

I’m pleased to know that unified inbox support is on its way in iPhone OS 4. I hope other improvements are coming as well.

iBooks

iBooks is the hotness. That’s all I have to say about that so far. I’ve not spent enormous amounts of time reading yet, so I don’t have more comments than that yet.

Third-party Apps

Another gripe, although not an overly important one, is that many of the apps I use on a daily basis (often many, many times a day) haven’t yet got iPad versions, or have iPad versions that aren’t quite ready for prime time.

High on the list is the Facebook app, which hasn’t been updated to use all that screen space yet. Yes, you can do Facebook in Safari, but I find I actually prefer the iPhone app’s experience over the browser experience. Curious to see what Facebook will do on iPad.

I also would like a good Google Reader-syncing RSS reader. On iPhone, I’m a huge fan of Byline. I bought “My Times” yesterday for a couple dollars, but it crashes pretty often. According to the developer, an update is winding its way through the approval process. However, I prefer the Byline UI and suspect that an iPad version of that would be nicer.

I love Rooms for IRC on iPhone, but it doesn’t have an iPad version yet. I bought a copy of LimeChat, which works well, but in landscape mode, it insists on having two sidebars, one for your channels, and one for the users in the channel, and there’s no way to hide either or both of them, meaning you’re stuck with the chat squeezed in between them. I’d like this to let me use more of my screen for actual content. The user and channel lists could easily be drop-downs.

I’m a long-time Twittelator user on the iPhone, and the iPad version was one of my first purchases. By and large I like it, but IMHO it uses too much of the screen for borders and cute effects and not enough for content. And in landscape mode, it makes the friends timeline very narrow, using most of the screen for a box that lets you look at various other views, such as @mentions. I don’t use any of those views very often. I’d much rather be able to have my friends timeline use the majority of the screen.

 Posted by at 5:55 PM