Production Puppet

This page describes the steps that the Puppet configuration management system should be responsible for.

Setup

Setup DW user account and add it to sudo.

adduser dw
echo "dw ALL=NOPASSWD: ALL" >> /etc/sudoers

Configure /etc/apt/sources.list to use our cache:

deb http://dw-lb01:9999/ubuntu/ hardy main restricted universe
deb-src http://dw-lb01:9999/ubuntu/ hardy main restricted universe

deb http://dw-lb01:9999/ubuntu/ hardy-updates main restricted universe
deb-src http://dw-lb01:9999/ubuntu/ hardy-updates main restricted universe

deb http://dw-lb01:9999/ubuntu hardy-security main restricted universe
deb-src http://dw-lb01:9999/ubuntu hardy-security main restricted universe

Let's do a system update and upgrade. This may not be something we can easily do in puppet? Or is it? I only want to do this from time to time - i.e., only on initial install, and only manually. (I don't want to automatically update packages every night, e.g.)

apt-get update
apt-get dist-upgrade

Now install packages:

vim-perl screen mercurial subversion dh-make-perl apache2-mpm-prefork
libgtop2-dev libapache2-mod-perl2 libapache2-request-perl
libcaptcha-recaptcha-perl libclass-accessor-perl libclass-autouse-perl
libclass-data-inheritable-perl libclass-trigger-perl libcompress-zlib-perl
libcrypt-dh-perl libdbd-mysql-perl libdbi-perl libdatetime-perl
libdigest-hmac-perl libmd5-perl libdigest-sha1-perl libgd-gd2-perl
libgd-graph-perl libgnupg-interface-perl libhtml-parser-perl
libhtml-tagset-perl libhtml-template-perl libio-stringy-perl
perlmagick libimage-size-perl libwww-perl libwww-perl libmime-lite-perl
libmime-perl libmailtools-perl libmail-gnupg-perl libmath-bigint-gmp-perl
libnet-dns-perl libproc-process-perl librpc-xml-perl libsoap-lite-perl
libstring-crc32-perl libtext-vcard-perl liburi-fetch-perl liburi-perl
libunicode-maputf8-perl libxml-atom-perl libxml-rss-perl libxml-simple-perl
mysql-client

Next up, we need to configure Apache. This only needs to be done on webservers. On non-webservers, we need to DISABLE "apache2" from starting at boot time.

$ cat /etc/apache2/conf.d/dw.conf
User dw
Group dw
UseCanonicalName off

StartServers 3
MaxSpareServers 5
MinSpareServers 1
MaxClients 2
MaxRequestsPerChild 200

DocumentRoot /home/dw/current/htdocs
PerlSetEnv  LJHOME /home/dw/current
PerlPassEnv LJHOME
PerlRequire  /home/dw/current/cgi-bin/modperl.pl

There's some more Apache setup that is required to get rid of the default site and enable the request library:

sudo a2dissite default
sudo a2enmod apreq

Last thing is to configure the network. We need to add the following lines in /etc/network/interfaces:

# ensure we load our rules 
pre-up iptables-restore < /etc/iptables.up.rules

Then we need this file created in /etc/iptables.up.rules:

*filter

# Allows all loopback (lo0) traffic and drop all 
# traffic to 127/8 that doesn't use lo0
-A INPUT -i lo -j ACCEPT
-A INPUT -i ! lo -d 127.0.0.0/8 -j REJECT

# Accept inbound private traffic from one of our servers
-A INPUT -s 172.19.1.5 -j ACCEPT
-A INPUT -s 172.19.1.84 -j ACCEPT
-A INPUT -s 172.19.1.85 -j ACCEPT
-A INPUT -s 172.19.1.86 -j ACCEPT
-A INPUT -s 172.19.1.87 -j ACCEPT
-A INPUT -s 172.19.1.88 -j ACCEPT
-A INPUT -s 172.19.1.89 -j ACCEPT
-A INPUT -s 172.19.1.90 -j ACCEPT
-A INPUT -s 172.19.1.91 -j ACCEPT

# Accepts all established inbound connections
-A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT

# Allows all outbound traffic
-A OUTPUT -j ACCEPT

# Reject all other inbound - default deny unless explicitly allowed policy
-A INPUT -j REJECT
-A FORWARD -j REJECT

COMMIT

One thing to note is that this file will get changed each time I add a new server. Also, every time it's pushed out, you have to run the iptables-restore command mentioned above. Puppet should handle that.

The source code for the site needs to be copied, but that's something I can do with rsync. Assuming the dw user is setup from the first step, then all I have to do is rsync the code and start Apache. Assuming the rest of this is all good.

Setup Round Two

This section contains things I didn't include in the first section, because I forgot about them! But now that I'm setting up the new production cluster, I have to remember them.

other packages

System: sysstat dnsutils netcat telnet strace libio-aio-perl bc

Also, there are packages that I had to build for the system. They're in our private repository, but I'll move them to dw-ops so puppet has access to them.

Memcache: memcached

Mail: postfix

Database: mysql-server

Perlbal: lighttpd

snmpd

There's a simple /etc/snmpd/snmpd.conf that needs to be sent out. This configuration includes a snmp v2 community writable password, so we may have to figure out some way of having a private config.

Also, /etc/default/snmpd needs to be updated to change 127.0.0.1 to the IP address of the internal interface. (Otherwise snmpd won't ever listen to our requests for help...)

Then make sure snmpd is restarted.

Perlbal server

The perlbal machines (dfw-lb01, dfw-lb02) need to have heartbeat setup... /etc/heartbeat/{ha.cf,haresources,authkeys} and also be up and running. (Don't worry about this one yet. I'm still debating the best way to setup the frontend. LVS? Heartbeat?)

iptables exceptions

Some machines need to have exceptions on various ports. Uh, lbs need port 80 and 443, the mailserver needs port 25...

Downtime Server

We run a very simple webserver (lighttpd) that just shows a downtime page when we have that enabled. Downtime servers run on the perlbals, but in the future I hope to have perlbal do the downtime stuff itself.

Admin Server

We also run an Apache2 instance that gives us access to cacti, nagios, and other monitoring tools. This runs on the admin server.

Memcache Server

Need to edit /etc/memcached.conf to use 450MB of RAM (although this might change depending on the server) plus listen on the internal interface IP.