May 202010

As I mentioned earlier today, we’ve disabled (for now) all of the RSS feeds on MDC except the site-wide “recent changes” feed.

Even though feeds were being cached, and only refreshed on demand, spiders and bots were causing “on demand” to mean “every time they expired”. And since every page on the site has multiple feeds, and every user has a feed of their contributions, something like 360,000 feeds were being queued for regeneration, meaning that there were pretty much always feeds being regenerated, chewing up lots of CPU.

Now, instead of each host constantly having something like 25+ tasks ongoing, never less, each host is averaging more like 3 ongoing tasks at any given time, because they’re not piling up while the system is busy handling RSS feed generation.

Work is ongoing on coming up with a fix for the performance of the feed generation code, so that the other feeds can be re-enabled, but at least we have the main one still.

In the meantime, performance seems quite reasonable. Please let me know if you run into any performance issues, so I can get them looked into.

 Posted by at 5:30 PM

  6 Responses to “More on the RSS feed performance issue on MDC”

  1. Would a robots.txt excluding all feeds help?

  2. Yes; that’s one of the things we’re going to be doing.

  3. Oh, and it’s actually over 850,000 RSS feeds. Eep!

  4. Please let me know if you run into any performance issues

    What should be included in such report? It certainly feels slow here (Russia), but I don’t know if it’s expected performance for now. I’m talking about 3-10 seconds here for a full page load. E.g. on when logged in (although I’m constantly logged out for some reason and it still isn’t fast).

    Oh and I just got this error, reloading fixed it:
    Site settings could not be loaded

    We were unable to locate the API to request site settings. Please see below for debugging information.

  5. Same answer than Nickolay for me, from France, 3~8 seconds for every full page load, i’m not logged.

    And sorry to spam the subject, but what about a downloable static version of MDC, periodically updated (even rarely)? I’m sure I’m not the only one who will find it *really* usefull.

  6. Nickolay: Looking into it. Looks like the mono daemon is crashing occasionally, which may be what you’re seeing. We’re debugging.