Difference between revisions of "Dev Getting Started"

Revision as of 07:01, 30 July 2008

Welcome to Development

This document is designed to give you a rough guide to setting up the DW code on your server. There is, currently, a high expectation of "tech savvy." If you do not know what sudo is or how to get a root console, then you probably are going to get pretty lost in this.

Doesn't mean you shouldn't try, but forewarned is forearmed!

Help can be found on IRC (#dw on irc.zhzh.org) or on the mailing lists.

The original LJ Server documentation is here. You might also find the LJCode wiki useful.

You can find information on how to keep your code updated on Dev Maintenance, information on programming style on Dev Programming Guidelines, information on submitting patches on Dev Patches.

You might also like Setting up Dreamwidth on Linode.

Requirements

Here is what you'll want for a development environment. Sure, you can probably get away with less or different, but that's not what we're going to support. The DW code is known to work with the following. But we'd love to hear of your results using something else.

Linux based server

We are presently using Ubuntu. This may stay or become Debian or Centos at some point, that's undecided. But for now, Ubuntu is the supported distro. If you use anything else, you will need to be familiar enough with the differences to work out any different steps on your own. (But we'd love to hear success stories and get documentation for other ones!)

Notes for other OSes:

• Running Dreamwidth on a Mac OS X system

64-bit (x86_64 typically, NOT i686/i386)

This is something you can probably get away without doing, but you'll be unable to test some things and others might be a bit wonky. Notably, you need the 64 bit version of Perl in order for the pack/unpack operations to work for memcache.

But if you absolutely do not have a 64 bit system, then you might still be okay. Don't use memcache and don't try to use more than 30 friend groups and you shouldn't run into any big issues.

If you're not sure whether you have a 64 bit system or not, check out the collection of tips on this page: http://www.stata.com/products/64bit.html

512mb RAM

I have gotten away with less. Depends on how much traffic you expect and what else you are running on the box. Also, if you intend on running TheSchwartz workers or not. If you do not, and you only run the web site code, then it will probably work on 256.

Dedicated!

The LJ/DW code is pretty heavy, and doesn't like other things running. You should expect that if you are not that technical, you will need a fairly blank server to run it on. But if you are technical you can do some pretty convoluted setups... (This one is more of a recommendation and not a hard requirement.)

root

If you do not have root on the box, you are in for a rough road. Sure, the code doesn't need root privileges to run. But there are a lot of libraries that need to be installed on the system. If you are using a shared development environment where the owner has installed the modules for you, then you should be fine.

MySQL

You will need a MySQL database. It can be local or remote, it doesn't matter really. As long as you have the ability to create/drop/alter tables in a directory, that's what matters.

Suitable hosting

These hosts are reported to have suitable plans for running Dreamwidth code:

• Linode -- 64-bit available by request, not the default; used by afuna and exor674. Instructions for setting up on this host from a pre-built image (from exor674) are here.
• Slicehost -- used by xb95

Preparation

Okay, for the sake of argument, let's say you have a machine (virtual or otherwise) that matches the above requirements. Getting the DW code setup is fairly straightforward.

User Account

Create a user account that will be used for the code. This can be your own personal account if you wish, but it is recommended that you setup a new user that is only going to be used for running the code. That way if you have any insecure code running (i.e., a bad page, or a patch you're working on doesn't have security implemented yet) you don't have to worry about someone getting access to your personal files.

Setup User Account

I'm assuming that you use the bash shell. If you do not, then this section does not apply. You will have to find the shell specific way of setting environment variables.

For bash, you should look in the home directory of the user account you just created for a file named .bashrc or .bash_profile or maybe even .profile. One of these files will have some commands in it (alias commands usually). You will want to add one more line to this file:

   export LJHOME=/home/dw

Or whatever your user account's home directory is. Yes, the variable is named LJHOME. We'll live with it for now.  ;-)

Test this. Log out of your user account and log back in, then type:

   echo $LJHOME

You should see /home/dw or whatever you set it to. If you don't, then you didn't get the export line in the right place. (Hey, I'm not here to teach basic sysadmin stuff. You should know how to set environment variables.)

Prep Packages

Next, you will need two package management systems installed: subversion and mercurial. This is typically done on an Ubuntu system with:

   apt-get install mercurial subversion

You will need to be root to run that command.

Get and Run Bootstrap

Now, log in as the user account you created in the first step. We have put together a bootstrap script. You can obtain this file with your favorite tool (wget, curl, etc):

   wget http://hg.dwscoalition.org/dw-free/raw-file/e4be2e864b95/bin/bootstrap.pl

The next step is fairly automatic. Assuming that your environment variable is good to go, you should be able to just do this:

   perl bootstrap.pl

The script will start doing some work. It should start out by saying 'seems we need to start at the beginning', which is what you expect. Then you will sit around for half an hour while it checks out various packages and repositories from various locations.

When it's done, it will say so. It will also say to delete the bootstrap script. Do a quick ls in $LJHOME to see if you have a bunch of new directories like cvs, bml, cgi-bin, htdocs, etc. If you do, huzzah! You now have the code checked out.

If you don't, figure out what step didn't work, and try again. All else fails, hit the mailing list or find someone on IRC.

dw-nonfree

If you are doing development on Dreamwidth custom files (what few there are), then you will need to do another step. This is fairly manual, the bootstrap script does not handle this process for you. But assuming that you did the above steps, you can get dw-nonfree code like this:

   cd $LJHOME/cvs
   hg clone http://hg.dwscoalition.org/dw-nonfree dw-nonfree
   cp dw-nonfree/cvs/multicvs-local.conf .
   cd $LJHOME
   bin/cvsreport.pl -s

You should see a number of files get copied around.

Database Setup

I'm assuming that you have a local MySQL installation. I'm assuming that you have the root login on this database, or know how to get a root shell to the MySQL console. If you don't, you'll have to figure out how to create a database in your setup.

But assuming you do have root, this is how you can easily create your development database:

   create database dw_dev_1;
   grant all on dw_dev_1.* to 'dw_dev_user'@'localhost' identified by 'somePassword';

Of course, you might want to pick an appropriate username/database/password instead of using the crap I made up here.

ljconfig.pl Setup

Next you need to configure the site configuration script. This is probably the most tricky part of the whole process, since there are so many things you can tweak. (Note to self: get the DW ljconfig cleaned up and posted...)

   cd $LJHOME
   cp doc/ljconfig.pl.txt etc/ljconfig.pl
   vim etc/ljconfig.pl

Or use your editor of choice, of course. Do a search for the phrase 'CHANGE THIS' which is in three places. You will want to change at least your $DOMAIN and the %DBINFO structure. It should be fairly self-explanatory.

Note that %DBINFO contains the authentication credentials you chose in the database setup step above!

checkconfig.pl

Now, you need to see if everything is working. If you've setup the files as indicated above, run this command:

   cd $LJHOME
   bin/checkconfig.pl --no=ljconfig

It should take a few minutes to run and then tell you that you are missing modules. This is where you need some systems specific knowledge for your system. You will need to install whatever modules it's missing - I can't really help you there at this time. (But if you get well and truly stuck, find someone on IRC or the mailing list!)

If all goes well, then take heart, you're almost done!

Install Data

There are a few commands you can now run to install the database. Just run these and watch for errors.

   cd $LJHOME
   bin/upgrading/update-db.pl -r -p --innodb
   bin/upgrading/update-db.pl -r --cluster=all --innodb
   bin/upgrading/texttool.pl load
   bin/upgrading/make_system.pl

(That step will ask you for a password for the system account. You can change it later by logging in as system, so just give it something for now.)

   bin/upgrading/copy-emailpass-out-of-user

And that's it for the code!

Configure Apache

This is the Apache 2 configuration running on the Dreamwidth staging site. I put this in a file named "stage" in /etc/apache2/conf.d, and then I removed the "default" file from /etc/apache2/sites-enabled.

   User dw
   Group dw
   UseCanonicalName off
   
   StartServers 1
   MaxSpareServers 2
   MinSpareServers 1
   
   DocumentRoot /home/dw/htdocs
   
   PerlSetEnv  LJHOME /home/dw
   PerlPassEnv LJHOME
   
   PerlRequire  /home/dw/cgi-bin/modperl.pl

That's about it... nice and simple.