Production Tools

Production Tools

This page is for information on the tools that are needed but don't currently exist to help run the site.

pbadm

A general Perlbal administration tool. This is useful for issuing commands to all of the Perlbals, doing checks on status, etc. Here's a rough series of requirements...

• Must load $ENV{LJHOME}/etc/config-private.pl which defines this hash:

   # list of perlbals, mostly used for admin tools
   %LJ::PERLBAL_SERVERS = (
       lb01 => '10.176.64.125:60000',
       lb02 => '10.176.64.126:60000',
   );

This is how the script should know the active Perlbals. The key of the hash is the name of the machine running Perlbal, the value is a host:port for that given Perlbal's management port.

• Support "check" command

This command will connect to each of the Perlbals and run the "states" command. Here's an example of what it looks like:

[root @ dfw-admin01 - ~] -> echo states | nc -q 1 lb01 60000
Perlbal::BackendHTTP bored 1
Perlbal::BackendHTTP wait_res 1
Perlbal::ClientHTTPBase persist_wait 82
Perlbal::ClientHTTPBase reading_headers 1
Perlbal::ClientProxy wait_backend 1
Perlbal::ClientProxy wait_res 1
Perlbal::ClientProxy xfer_res 1
.

This tells you the thing (class) e.g. Perlbal::BackendHTTP, the state e.g. bored, and the count. Note that each class can have many states... anyway, most of the values are not that important, but we need a few of them to be displayed. Here's the output I'm looking for...

http://pics.livejournal.com/xb95/pic/0005dr7q

Yeah, I had a screenshot sitting around, heh. Anyway, the idea is that you run this tool in check mode, and it starts looping every few seconds and printing out status which it gets from running 'state' on each of the Perlbals. This output is then formatted in some useful way.

In one output section:

[lb01 - 150, 2344]

The first item is the machine name from the hash. The second item is the sum of (Perlbal::BackendHTTP xfer_res + Perlbal::BackendHTTP wait_res) which is the number of backend requests in-flight. The third item is just Perlbal::ClientProxy wait_backend.