Distributed Workers
Posted by Joel Fri, 04 Aug 2006 03:15:00 GMT
Yesterday was stressful at work because of stability problems. I ended up getting rid of Lightty and Pen and going back to Pound balancing four Mongrel processes. Furthermore I had to disable session management in Pound and increase the timeout (Pen worked just as well really, but Pound has virtual hosting, making Lightty largely unnecessary).
The problem is that our site has to resize images frequently and when it does one processor is going to get eaten up. It employs a simple caching system so that when people click on a size to download, it will see if it exists and if not resize the photo for that size. This means the customer has to wait anywhere between 3 seconds and a minute, depending on the size requested, the size of the source image, and the traffic on the server. The 15 second default timeout on Pound resulted in 500 errors when a request took longer than 15 seconds (either because it was crunching an image, or was waiting in a line behind a request that was). So I increased that.
We have two processors. Pen and Pound (with a Session set) will try to send you back to the same server you used last time. This means that if your server is presently spending 30 seconds crunching an image, you wait 30 seconds. In Pen you can solve this by putting it on a more traditional round robin load balancing, so it will always use the available processor, and in Pound you can remove the Session attribute for the same round robin functionality.
However, if two people crunch images simultaneously, everyone gets to wait until one of them finishes. It's not ideal. And while one solution is to scale out and get fifty processors, another solution is to just approach resizing images more intelligently.
And that's how I found BackgrounDRb. A pretty cool system that will shift stuff off to another server which you can then run at a low priority. I can just queue up photos to resize in there and have it resize them all immediately after they are accepted into the system, not when a customer requests it. Meanwhile the web servers can easily ask this server whether or not it's done working on stuff. I think this is going to become a huge part of scaling sites. Why get new hardware when you can just increase the efficiency of your software? More info on the basic concept at Mongrel's site.
I've been messing with it so far with pretty good results. Responsive mailing list community.
I was confused by this because I thought requests would just share processors and multitask happily but it seems like they monopolize it. I tried running good ol' Lightty and FastCGI and got the same two maxed-out request limit. And my processors have hyperthreading, so I thought it would be four. I wonder if there's something I'm missing.
This seems like a basic CS issue. Is there a point to having more than two Mongrel web servers running if I only have two processors available? Why doesn't hyperthreading grant me four processors? Apache always spawned lots of processes, FastCGI always does too; so why would it do that if the simultaneous requests were limited by the processors? I need to do more research.
