We’ve been running on MindTouch 9.12.2 now for about 12 hours, and things seem stable. As long as the RSS feed generators aren’t running, the site is quite fast, but as soon as the feed generator starts working to build a feed whose cache has expired, the CPU gets bogged down pretty mightily.
This is alleviated somewhat by the fact that we’re now running MDC on three hosts instead of two. That said, each individual box still bogs down when any feeds are updating, and there are lots of different feeds. So even though each feed is set to expire after an hour, and only gets rebuilt when someone loads it after it’s expired, it still causes a major performance drain when the feeds do rebuild.
Anyway, we have a third box in the pool now, which will help some, and I’m talking actively with MindTouch (literally while writing this blog post) about ways to improve the RSS feed impact to our performance while they work toward rewriting the feed generation code.
One idea I have, which they’re going to do some testing on today, is to set up a fourth box in the pool, and direct all — and only — feed traffic to it, then disable the feed generators on the other three hosts. That way, the boxes serving up the site content never get hammered by the feed generation, while a dedicated machine can use all its CPU to handle the feeds.
It’s a bit of a cheesy hack, but I think it will work. I’ll blog further as our research continues!
In the meantime, I hope to re-enable editing on MDC very shortly; I want to confirm one last thing with IT before I do so. I’ll post again when editing is switched back on. Thanks for your patience!