As I mentioned earlier today, we’ve disabled (for now) all of the RSS feeds on MDC except the site-wide “recent changes” feed.
Even though feeds were being cached, and only refreshed on demand, spiders and bots were causing “on demand” to mean “every time they expired”. And since every page on the site has multiple feeds, and every user has a feed of their contributions, something like 360,000 feeds were being queued for regeneration, meaning that there were pretty much always feeds being regenerated, chewing up lots of CPU.
Now, instead of each host constantly having something like 25+ tasks ongoing, never less, each host is averaging more like 3 ongoing tasks at any given time, because they’re not piling up while the system is busy handling RSS feed generation.
Work is ongoing on coming up with a fix for the performance of the feed generation code, so that the other feeds can be re-enabled, but at least we have the main one still.
In the meantime, performance seems quite reasonable. Please let me know if you run into any performance issues, so I can get them looked into.