node.to

Updated 2009 (original post February 1, 2008 at 3:38 am): I’ve been working for RescueTime for a year now

So, my good friend Brian Fioca of RescueTime calls me with load concerns. Their public beta has become very popular among users and has been generating traffic of around 80-180 open rails app requests in any given second. As we discussed concerns, some ideas emerge. The first obvious things were addressed, such as database optimizations. However, the problem remained: one of the 3 servers was fully loaded and the others were not.

The type of load generated was diverse: some simple static content, including images, css, javascript, etc., a Ruby on Rails driven UI for analytics and profile management, and Rails driven web service API. The API accepted streams of XML data from thousands of clients on a continuous basis.

RescueTime is a time management tool. It gathers lots of data from a client you install, and then you can perform analytics on this data. The data will help you define how you want to spend your time, give you feedback, and much more. This gives you an idea of the activity on the servers: some small simple requests, constant streaming XML input, and sometimes intensive analysis.

Their orignial model was pretty straightforward:

• Server 1: Apache reverse proxy for single front door using mod_proxy_balancer with a rewrite rule hack for serving static resources; rails served by some mongrel instances, for app ui
• Server 2: rails served by some mongrel instances for data collection from client app
• Server 3: mysql server primary database

Server 1 was moderately loaded, Server 2 was saturated, and Server 3 was lightly loaded (after some major optimizations).

Step 1

Goal: eliminate Rewrite rules in reverse proxy

Before Changes

It seems that there is a recipe floating around in connection to Capistrano that suggests using rewrite rules that use file tests to determine what mod_proxy should do with an inbound request. In a busy system, this incurs an unnecessary load on the system: every single inbound request can trigger multiple calls to the filesystem. Specifically, in this model Server 1 would follow this order of checks: first, it checks for the existance of the Capistrano generated maintenance page, in which case it would serve it; next it would serve static files if the path to the inbound request existed on the filesystem, then serve it, else pass it on to proxy. This is for any and every request.

The Problems

Thus most requests generated 2 file system hits before being proxied. Not good. Furthemore, the static rule is implied, not explicit. For a configuration management perspective, it’s always easier to maintain and debug when you have strict mappings for resources. The system maintenance trick is nifty, but not scalable or even very logical, because it interacts with rails to be generated. What if you rails install is really broken? It also means you have to have rails on the reverse proxy for it to work. Additionally, setting Server 2’s mongrels to maintenance mode would not trigger this. It’s almost always better for operational controls like maintenance screens to be maintained independently and logically above the apps they are relevant too. That doesn’t prevent them from being dynamic.

The Fix

Remove all the rewrite rules.

Solving the statics problem: We define a new localhost only vhost in the proxy server config that serves the static files like a normal web server. The reverse proxy uses explicit ProxyPass rules to map static directories to this vhost. Benefits: simplicity and clarity of config, no extra file system hit. Also, as this system scales, the static vhost can be moved to independent systems, placed in an AthFarm, and load balanced. Only thing to change in proxy is to remove the vhost, add the AthFarm, and remap the ProxyPass to the farm. This also adds the ability to do invisible maintenance on the static systems using the administrative features of mod_athena.

Solving the maintenance page problem: Using mod_athena’s “AthOfflineURL” in global and/or farm context, you can administratively manage your servers directly in the reverse proxy. If the farm or total system is flagged offline, any affected inbound request (only reqs mapped to farm in former, all reqs in latter case) while be returned the url stored in the directive. This can point to a local or remote url, and is handled as normal proxy request. You can also break up your servers in multiple farms so that you can restart group by group. If our changes are only to the app code and do not affect the mysql server, we can leave API farm up while updating UI farm, and vice versa.

Step 2:

Goal: shuffle mongrels distribution and optimize load balancing with mod_athena

Before Changes:

The mongrels are memory intensive. Due to ruby’s lack of support for kernel threads, all ruby’s user threads can only take advantage of 1 core of a system. To maximize resource use, we needed multiple mongrels per server, but only as many as could be supported by memory and CPU available. Initially, there were an equal number on Server 1 and 2, none on 3. The old system used mod_proxy_balancer, which offered request count and round robin balancing options; round robin was in use. The database server was periodically busy, so it was undesireble to blindly send requests there.

The Problems:

Uneven load distribution and irregular performance. Server 3 underutilized much of the time. Initial layout had application UI rails requests all going, round robin, to multiple mongrels on Server 1. Server 1 was only about 1/2 saturated, but suffered from periodic swapping due to memory load, and thus poor performance. Server 2 was very saturated with the incoming data streams delivered round robin to multiple mongrels on it.

The Fix:

Tune mongrel count to memory resources to guarantee no swapping. Server 1 had too many, the count was reduced. Server 2 had more available memory and an extra CPU core, count was retained. And due to features of mod_athena, we were able to bring up a few mongrels on Server 3, despite it also running mysql.

Rearrange mongrel distribution in farms. The idea of retaining a UI farm and an API (inbound POST data stream) farm was retained, for administrative control and metrics detail. UI requests were mostly light weight, and API requests heavier. There was some developer concern about mixing UI and API requests, so no particular mongrels were declared in both farms (although this is possible). So, all but 1 of the mongrels on Server 1 were put in UI farm, the remaining one in API. All mongrels in Server 2 were in API farm. And the new mongrels on Server 3 were also in the API farm. As such, an API request could use physical resources on any of the systems. However, no individual mongrel would handle both.

Load balance using real load statistics. We used the ath_agent.sh tool from the mod_athena package to report CPU utilization stats back to the load balancing reverse proxy from all 3 servers. The load balancer was then configured to load balance to mongrels based on the real CPU load for the particular server they were on. Additionally, since obviously multiple mongrels on a given server share the same CPU resources, using CPU to balance among them was inadequate. So a further tuning to the algorithm was enabled. This is the AthAlgoHitAdds feature. It allows you to increment a particular statistic field by an arbitrary value per request. When applied to CPU, this has the pratical affect of adding simulated load to that particular mongrel. This is very important bequest the CPU load updates are point in time, and on a set frequency (5 seconds in this case). This feature prevents flooding servers with batches of requests. If tuned appropriately (we used .1, it’s accepts float number type), this will evenly pass load among all 3 servers, and when all servers and/or mongrels are equally loaded, a round-robin behavior emerges from the tuned setting.

Because we use real load to determine server desirability in load balancing, we are able to now use Server 3. When mysql load kicks up, so will its reported CPU load, and it will cease to be chosen in load balancing decisions.

Result: Efficiency + Performance + Flexibility

This setup allows us to administratively manage any particular mongrel with no end-user affects. It also demonstrates extremely even load distribution across all the assets. The end result was a faster, more reliable complete web application, with a configuration that is easy to scale further. Next steps would be dedicated reverse proxy load balancer, moving all rails and static content to separate systems, and adding a rails agent for mod_athena that gives it more statistics to use in load balancing.

Fundamental Lessons:

Load balancing is tricky. Round robin and open connection count algorithms work well in general for static content and simple web applications. More complicated web applications, especially ones thay may also have asynchronous capabilities (backround threads), demand more complicated algorithms, even more complicated than the one described here.

Use all your assets, it matters more when you have less, and the VC’s will love your potential margins!