Difference between revisions of "Dreamwidth Scratch Installation"

From Dreamwidth Notes
Jump to: navigation, search
(New page: These are the instructions to install a Dreamwidth system from scratch, assuming you are running Ubuntu. See Setting up Dreamwidth on Linode for instructions on setting up a system to...)
 
(Further setup: DNS and email)
 
(192 intermediate revisions by 25 users not shown)
Line 1: Line 1:
These are the instructions to install a Dreamwidth system from scratch, assuming you are running Ubuntu.  See [[Setting up Dreamwidth on Linode]] for instructions on setting up a system to the point where you can log in and start this.
 
  
If you have difficulties or problems or suggestions for installation procedures, please report them to [http://www.dwscoalition.org/show_bug.cgi?id=61 Bug 61].
+
Firstly, you'll need to set up a Linux system -- here are the [[Suggested Server Requirements]].  These instructions assume Ubuntu 16.04.  Later versions (as of December 18, 2017) may or may not work, see https://github.com/dreamwidth/dw-free/issues/2248. Please let us know if you run into difficulties--we want to update this if people are having problems!  We recommend coming into the #dreamwidth-dev channel on [[IRC]].
  
== Bringing the system up to date ==
+
After you set up, you can keep updated with the instructions on [[Dev Maintenance]].
  
Before you start installing anything else, you might want to make sure your system is up to date:
+
If you have difficulties or problems or suggestions for installation procedures, please report them to <dwuser>dw_dev</dwuser>.  Also look at [[Production Setup: Webserver]].
  
  apt-get update
+
== Command convention ==
  apt-get upgrade
+
 
 +
In this walkthrough, commands to be run as a regular user are prefixed with <tt>$</tt>. Commands to be run as the super-user are prefixed with <tt>#</tt>. If you are not the super-user, you can run single commands like this:
 +
 
 +
$ sudo <command goes here>
 +
 
 +
Alternately, you can spawn a super-user shell using:
 +
 
 +
$ sudo -i
 +
# <command goes here>
 +
 
 +
When done with the super-user shell, exit the shell via:
 +
 
 +
# exit
 +
 
 +
== Configuring the OS and environment==
 +
 
 +
=== Bringing the system up to date ===
 +
 
 +
Before you start installing anything else, you might want to make sure your system is up to date.  You should update the packages on your system:
 +
 
 +
  # apt-get update
 +
  # apt-get upgrade
  
 
If you want man pages on your system:  
 
If you want man pages on your system:  
  
  apt-get install man
+
  # apt-get install man
  
== Installing necessary packages ==
+
=== Installing necessary packages ===
  
This will install necessary packages you'll need to run Dreamwidth. You'll want to be root when running these commands.
+
This will install necessary packages you'll need to run Dreamwidth (and some optional ones). You'll want to be root when running these commands.
  
  apt-get install mercurial subversion apache2-mpm-prefork \
+
  # apt-get install git apache2 apache2-bin apache2-data apache2-utils \
 
   libapache2-mod-perl2 libapache2-mod-apreq2 libapache2-request-perl \
 
   libapache2-mod-perl2 libapache2-mod-apreq2 libapache2-request-perl \
   mysql-server wget unzip links
+
   mysql-server wget unzip links vim libclass-autouse-perl libdatetime-perl \
apt-get install libclass-autouse-perl libcompress-zlib-perl \
+
   libcache-memcached-perl libhash-multivalue-perl libgd-gd2-perl \
   libdatetime-perl libdigest-sha1-perl libgd-gd2-perl libhtml-template-perl \
+
  libhtml-template-perl libwww-perl libmime-lite-perl \
  libwww-perl libmime-lite-perl libnet-dns-perl liburi-perl libxml-simple-perl
+
  liburi-perl libxml-simple-perl libclass-accessor-perl \
apt-get install libclass-accessor-perl libclass-data-inheritable-perl \
+
  libclass-data-inheritable-perl libclass-trigger-perl libcrypt-dh-perl \
  libclass-trigger-perl libcrypt-dh-perl libmath-bigint-gmp-perl \
+
  libmath-bigint-gmp-perl liburi-fetch-perl libgd-graph-perl \
  liburi-fetch-perl libgd-graph-perl libgnupg-interface-perl
+
  libgnupg-interface-perl libmail-gnupg-perl perlmagick \
apt-get install libmail-gnupg-perl perlmagick libproc-processtable-perl \
+
  libproc-processtable-perl libsoap-lite-perl librpc-xml-perl \
  libsoap-lite-perl librpc-xml-perl libstring-crc32-perl libtext-vcard-perl \
+
  libstring-crc32-perl libxml-atom-perl libxml-rss-perl \
  libxml-atom-perl libxml-rss-perl
+
  libimage-size-perl libunicode-maputf8-perl libgtop2-dev build-essential \
apt-get install libimage-size-perl libunicode-maputf8-perl
+
  libnet-openid-consumer-perl libnet-openid-server-perl libyaml-perl \
apt-get install libgtop2-dev build-essential
+
  libcaptcha-recaptcha-perl libdbd-sqlite3-perl libtest-simple-perl \
perl -MCPAN -e 'install GTop' # this will guide you through setting up CPAN
+
  libtemplate-perl libterm-readkey-perl libmime-base64-urlsafe-perl \
perl -MCPAN -e 'install Unicode::CheckUTF8'
+
  gcc libtest-most-perl libgearman-client-perl libfile-find-rule-perl \
perl -MCPAN -e 'install Captcha::reCAPTCHA'
+
  libbusiness-creditcard-perl liblwpx-paranoidagent-perl libtheschwartz-perl \
perl -MCPAN -e 'install IP::Country::Fast'
+
  libfile-type-perl libjson-perl ruby libdbd-mysql-perl libdanga-socket-perl \
 +
  libio-aio-perl libsys-syscall-perl liblog-log4perl-perl libtext-markdown-perl \
 +
  libimage-exiftool-perl libnet-oauth-perl libnet-smtps-perl libxmlrpc-lite-perl
 +
 
 +
This will download about 106MB of files and use around 511MB of disk space.
 +
 
 +
Check whether these packages have actually installed.  Later on in the process, if you find you are having inexplicable problems, try installing these again.
  
You will also want to install and configure Postfix:
+
You will also want to install and configure Postfix so your DW can send out email:
  
  apt-get install postfix
+
  # apt-get install postfix
  
 
If you choose not to configure on the install, you can do so later with:
 
If you choose not to configure on the install, you can do so later with:
  
  dpkg-reconfigure postfix
+
  # dpkg-reconfigure postfix
  
== Setting up the DW user account ==
+
=== Configure Apache 2 ===
  
Create a user account that will be used for the code. This can be your own personal account if you wish, but it is recommended that you setup a new user that is only going to be used for running the code. That way if you have any insecure code running (i.e., a bad page, or a patch you're working on doesn't have security implemented yet) you don't have to worry about someone getting access to your personal files.
+
[http://perl.apache.org/docs/2.0/user/install/install.html#Prerequisites Check] whether your system is using threaded MPMs or pre-fork. You will need the latter. <em>Note</em>: if you ran the above command to install Apache, then you need to run these commands to switch to the pre-fork model:
  
To set up the user account:
+
# a2dismod mpm_event
 +
# a2enmod mpm_prefork
 +
# service apache2 restart
  
apt-get install vim
+
=== Install Perl modules with CPAN ===
sudo visudo
+
  
uncomment line: %sudo ALL=NOPASSWD: ALL
+
You will also want to install some perl libraries with CPAN.  Defaults during CPAN's setup should be okay.
  
Add the user:
+
Before installing, check if your CPAN shell has <tt>make</tt> at the correct location.  Run
 +
 
 +
# which make
 +
/usr/bin/make
 +
 
 +
then compare the results with running:
 +
 
 +
# cpan
 +
...
 +
Would you like to configure as much as possible automatically? [yes] <PRESS ENTER>
 +
...
 +
cpan[1]> o conf make
 +
    make              [/usr/bin/make]
 +
...
 +
cpan[2]> exit
 +
 
 +
The value printed should match that from the <tt>which make</tt> command you previously ran.
 +
 
 +
To install the perl libraries, first set up CPAN:
 +
 
 +
# cpan Bundle::CPAN
 +
 
 +
When prompted, press Enter to exit tests. Then, run the following command to install the rest of the necessary libraries:
 +
 
 +
# cpan GTop Digest::SHA1 Unicode::CheckUTF8 MogileFS::Client \
 +
  TheSchwartz::Worker::SendEmail LWP::UserAgent::Paranoid \
 +
  Mozilla::CA List::Util Paws::S3 Net::DNS Text::Fuzzy
 +
 
 +
(Net::DNS is required to be installed via CPAN because the version included with current versions of Ubuntu is 0.81, which is too old for LWP::UserAgent::Paranoid to work properly.)
 +
 
 +
Note that the MogileFS::Client is required even if you are not configuring MogileFS.
 +
 
 +
=== Set Timezone to UTC ===
 +
 
 +
Dreamwidth will complain if you aren't running in UTC.  If you're not configured for UTC already, you can reset it (on Ubuntu/Debian) using
 +
 
 +
# dpkg-reconfigure tzdata
 +
 
 +
and choose Etc/UTC.
 +
 
 +
=== Configuring MySQL/MariaDB ===
 +
 
 +
Newer versions of MySQL and/or MariaDB have imposed stricter requirements than the current Dreamwidth SQL definitions can comply with. These restrictions must be relaxed, or the initial database population scripts will fail.
 +
 
 +
To do this, run the following 4 commands:
 +
 
 +
$ sudo -i
 +
# <nowiki>echo "sql_mode = ''" >> /etc/mysql/mysql.conf.d/mysqld.cnf</nowiki>
 +
# service mysql restart
 +
# exit
 +
 
 +
=== Setting up the DW user account ===
 +
 
 +
{{Note|text=If you still find yourself unable to make sudo commands even after <code>usermod</code>, please see [[Making your DW user a sudo account]] for alternatives.}} Create a user account that will be used for the code.  This can be your own personal account if you wish, but it is recommended that you setup a new user that is only going to be used for running the code.  That way if you have any insecure code running (i.e., a bad page, or a patch you're working on doesn't have security implemented yet) you don't have to worry about someone getting access to your personal files.
 +
 
 +
To set up the user account:
  
  adduser <i>username</i>
+
  # adduser <i>username</i>
  
 
Fill out the user's info and put them in the sudo group:
 
Fill out the user's info and put them in the sudo group:
  
  usermod -a -G sudo <i>username</i>
+
  # usermod -a -G sudo <i>username</i>
  
 
So, to use <code>dw</code> as a username:
 
So, to use <code>dw</code> as a username:
  
  adduser dw
+
  # adduser dw
  usermod -a -G sudo dw
+
  # usermod -a -G sudo dw
  
 
Now, we will get the $LJHOME variable set.  Log into your user:
 
Now, we will get the $LJHOME variable set.  Log into your user:
  
  su - dw
+
  # su - dw
  
 
We will assume that you use the bash shell.  If you do not, then this section does not apply.  You will have to find the shell specific way of setting environment variables.
 
We will assume that you use the bash shell.  If you do not, then this section does not apply.  You will have to find the shell specific way of setting environment variables.
  
For bash, you should look in the home directory of the user account you just created for a file named .bashrc or .bash_profile or maybe even .profile. One of these files will have some commands in it (alias commands usually).  You will want to add one more line to this file:
+
For bash, you should look in the home directory of the user account you just created for a file <tt>.profile</tt>. You will want to add one more line to this file:
  
  export LJHOME=/home/dw
+
  export LJHOME=/home/dw/dw
  
Or whatever your user account's home directory is.  Yes, the variable is named LJHOME.  We'll live with it for now.  ;-)
+
Or wherever it is that you want the root of your source code install to be.  Yes, the variable is named LJHOME.  We'll live with it for now.  ;-)
  
Test this.  Log out of your user account and log back in, then type:
+
Test this.  Log out of the dw user account and log back in, then type:
  
 
  echo $LJHOME
 
  echo $LJHOME
  
You should see /home/dw or whatever you set it to.  If you don't, then you didn't get the export line in the right place. (Hey, I'm not here to teach basic sysadmin stuff.  You should know how to set environment variables.)
+
You should see <tt>/home/dw/dw</tt> or whatever you set it to.  If you don't, then something has gone awry -- maybe the export line isn't in the right place..?
  
== Downloading the Dreamwidth code ==
+
== Get the Dreamwidth code ==
  
You should do this as the user you want to run Dreamwidth as.  If you are root, you can switch with:
+
=== Configure git ===
  
su - dw
+
Pick up a name and email address you're willing to share with the world (or at least that part of it that pays attention to GitHub and Dreamwidth). Then run the following 2 commands, replacing NAME and EMAIL with what you chose:
  
We have put together a bootstrap script that will download all of the codeYou can obtain this file with your favorite tool (wget, curl, etc):
+
$ git config --global user.name "NAME"
 +
  $ git config --global user.email "EMAIL"
  
wget http://hg.dwscoalition.org/dw-free/raw-file/e4be2e864b95/bin/bootstrap.pl
+
Optionally, make git use your preferred editor for commit messages (and a few other things). First find the path to your favorite editor (replace EDITOR with the name of your preferred editor, eg nano or vi):
  
The next step is fairly automatic. Assuming that your $LJHOME environment variable is good to go, you should be able to just do this:
+
  $ which EDITOR
  
perl bootstrap.pl
+
This will print something like: <code>/usr/bin/vi</code> (if your editor of choice is vi). Then, run the following command, replacing /PATH/TO/EDITOR with whatever was output:
  
The script will start doing some work.  It should start out by saying 'seems we need to start at the beginning', which is what you expect. Then you will sit around for half an hour while it checks out various packages and repositories from various locations.
+
  $ git config --global core.editor "/PATH/TO/EDITOR"
  
When it's done, it will say so.  It will also say to delete the bootstrap script. 
+
=== Set up GitHub ===
  
rm bootstrap.pl
+
First, create an account on Github. Then, you will need to fork these two repositories:
  
Do a quick ls in $LJHOME to see if you have a bunch of new directories like cvs, bml, cgi-bin, htdocs, etc.  If you do, huzzah!  You now have the code checked out.
+
* https://github.com/dreamwidth/dw-free
 +
* https://github.com/dreamwidth/dw-nonfree
  
If you don't, figure out what step didn't work, and try again.  All else fails, hit the mailing list or find someone on IRC.
+
Go to each of their pages, and click the "Fork" button. This sets up your own copy of Dreamwidth's code which you will use to make and submit your changes.
  
=== dw-nonfree ===
+
Here's a quick overview of how the repositories will work together:
  
If you are doing development on Dreamwidth custom files (what few there are), then you will need to do another step. This is fairly manual, the bootstrap script does not handle this process for you.  But assuming that you did the above steps, you can get dw-nonfree code like this:
+
* dw-free is the main repository and goes into $LJHOME
 +
* other repositories will go into $LJHOME/ext
 +
* personal config files go into $LJHOME/ext/local
 +
* All code from dw-free ($LJHOME) and the additional repos under $LJHOME/ext are automatically live when you start your server. There's no need to run any additional syncing steps anymore.
  
cd $LJHOME/cvs
+
=== Set up dw-free ===
hg clone http://hg.dwscoalition.org/dw-nonfree dw-nonfree
+
cp dw-nonfree/cvs/multicvs-local.conf .
+
cd $LJHOME
+
bin/cvsreport.pl -s
+
  
You should see a number of files get copied around.
+
Replace the USERNAME sections of the URL with your Github username.  This will make your repository an authenticated version that can push changes back to Github, as opposed to only being able to pull them.
 +
 
 +
First, clone a copy of the repository onto your machine:
 +
 
 +
$ cd ~/
 +
$ git clone https://USERNAME@github.com/USERNAME/dw-free.git $LJHOME
 +
$ cd $LJHOME
 +
 +
Now, let's make it aware of the dreamwidth repository so we can grab updates later:
 +
 
 +
$ git remote add dreamwidth https://github.com/dreamwidth/dw-free
 +
$ git fetch dreamwidth
 +
 +
Finally, let's set up to sync up with the main repository's master branch:
 +
 
 +
$ git branch -u dreamwidth/master master
 +
 
 +
=== Set up dw-nonfree ===
 +
 
 +
{{ Warn|text=For dev servers only. Clone sites should not use dw-nonfree because it contains Dreamwidth-specific branding and files.}}
 +
 
 +
$ cd $LJHOME/ext
 +
$ git clone https://USERNAME@github.com/USERNAME/dw-nonfree.git
 +
$ cd dw-nonfree
 +
$ git remote add dreamwidth https://github.com/dreamwidth/dw-nonfree
 +
$ git fetch dreamwidth
 +
$ git branch -u dreamwidth/master master
 +
$ cd ../..
  
 
== Database setup ==
 
== Database setup ==
Line 127: Line 237:
 
You should have a local MySQL installation and know the root login to the database.  To sign into MySQL as root:
 
You should have a local MySQL installation and know the root login to the database.  To sign into MySQL as root:
  
  mysql -u root -p   
+
  $ mysql -u root -p   
  
These MySQL commands wil create your development database:
+
These MySQL commands will create your development database:
  
  create database dw_dev_1;
+
  mysql> create database dw;
  grant all on dw_dev_1.* to 'dw_dev_user'@'localhost' identified by 'somePassword';
+
  mysql> grant all on dw.* to 'dw'@'localhost' identified by 'somePassword';
quit;
+
  
You might want to pick a more appropriate database username/database name/password.
+
That last line is made up of a few different sections:
  
== Editing ljconfig.pl ==
+
* '''grant all on dw.*''' - this tells MySQL to grant all permissions to the "dw" database which you created in the first line.
 +
* '''to 'dw'@'localhost'''' - this tells MySQL to grant the above permissions to a username called "dw" connecting from "localhost" (which means the same computer as the one MySQL is on). You can change "dw" to another username if you like. '''NB:''' this is 'dw'@'localhost', not 'dw@localhost' - either cut and paste, or type carefully!
 +
* '''identified by 'somePassword' ''' - since this user doesn't exist yet, we can tell MySQL to create it by adding this. This tells MySQL that the "dw" user will use the password "somePassword" to connect. You probably want to change this password!
  
Next you need to configure the site configuration script.  This is probably the most tricky part of the whole process, since there are so many things you can tweak.  (Note to self: get the DW ljconfig cleaned up and posted...)
+
While you're here, you probably want to go ahead and create a database for TheSchwartz, too, unless you really know that you won't need it:
  
  cd $LJHOME
+
  mysql> create database dw_schwartz;
  cp doc/ljconfig.pl.txt etc/ljconfig.pl
+
  mysql> grant all on dw_schwartz.* to 'dw'@'localhost';
vim etc/ljconfig.pl
+
  
Or use your editor of choice, of course.  Do a search for the phrase 'CHANGE THIS' which is in three places.  You will want to change at least your $DOMAIN and the %DBINFO structure.  It should be fairly self-explanatory.
+
(We're leaving out the "identified by" part this time because the user already exists.)
  
Note that %DBINFO contains the authentication credentials you chose in the database setup step above!
+
When you're done, exit out of MySQL:
 +
 
 +
mysql> quit;
 +
 
 +
== Editing the config files ==
 +
 
 +
Next you need to configure the site configuration scripts.  This is probably the most tricky part of the whole process, since there are so many things you can tweak. However, you can get by with just tweaking etc/config-local.pl and etc/config-private.pl for now.
 +
 
 +
Copy over your config files, and tell them to take priority over config files in any of the repos:
 +
 
 +
$ cd $LJHOME
 +
$ mkdir -p ext/local/etc
 +
$ echo "highest"  > ext/local/.dir_scope
 +
$ cp etc/config-local.pl.example ext/local/etc/config-local.pl
 +
$ vim ext/local/etc/config-local.pl
 +
 
 +
Or use your editor of choice, of course, instead of vim.  Do a search for the phrase 'CHANGE THIS' which occurs once, before the block of human readable text you should use to describe your development installation.
 +
 
 +
Note that the <var>$IS_DEV_SERVER</var> flag is set to 1 in the template given. Be sure to set this to 0 for a production site, as there are big security issues involved with leaving the flag set to 1.
 +
 
 +
Below the 'CHANGE THIS' section (unless this is already present somewhere), add:
 +
 
 +
    $STATDOCS = "$HOME/build/static"
 +
 
 +
Next, you need to change your local variables in the <tt>config-private.pl</tt> file. This is where your passwords are configured, as well as many of the variables which define your domain. Do a search for the phrase 'CHANGETHIS'(note the slightly different spacing). You will want to change at least your <var>$DOMAIN</var> and the <var>%DBINFO</var> structure.
 +
 
 +
$ cd $LJHOME
 +
$ cp etc/config-private.pl.example ext/local/etc/config-private.pl
 +
$ chmod go-rwx ext/local/etc/config-private.pl
 +
$ vim ext/local/etc/config-private.pl
 +
 
 +
In this file, uncomment the lines in the <var>%DBINFO</var> blocks (if they are commented out), and place your database password in quotes on the "pass => ," line, before the comma. For example:
 +
 
 +
    %DBINFO = (
 +
        master => {
 +
            pass => 'mypassword',
 +
        },
 +
    );
 +
 
 +
Note that there are two DBINFO blocks; get them both.
 +
 
 +
If you created the database for TheSchwartz earlier, you'll also want to configure it:
 +
 
 +
    # Schwartz DB configuration
 +
    @THESCHWARTZ_DBS = (
 +
            {
 +
                dsn => 'dbi:mysql:dw_schwartz;host=localhost',
 +
                user => 'dw',
 +
                pass => '',    # CHANGETHIS
 +
            },
 +
        );
 +
 
 +
Near the bottom of the file is the line you will need to change to tell your server which domain to use.
 +
 
 +
    package DW::PRIVATE;
 +
   
 +
    $DOMAIN = "ljsite.com";
 +
 
 +
<div style="padding:1em; background-color:#fee;">'''A note about $DOMAIN:''' If you are running a local server for development purposes and don't need or want a hostname, you may be tempted to just set $DOMAIN to localhost, but the server won't function correctly without a domain name.  One possible workaround for this is to set it to a nonexistent domain (the default ljsite.com is fine) and add a line in your <code>/etc/hosts</code> file that resolves that domain name to your localhost IP (either <tt>127.0.0.1</tt> or <tt>::1</tt>.)  Another workaround is to [http://wiki.dreamwidth.net/wiki/index.php/Subdomain_setup#Local_development_via_dnsmasq install and configure dnsmasq].</div>
 +
 
 +
<div style="padding:1em; background-color:#fee;">'''SSL:''' If you're running a development server and don't want to fuss with SSL, put this in your local config:
 +
 
 +
    $PROTOCOL = "http";
 +
</div>
 +
 
 +
The base configuration file also needs to be copied over, although for most purposes you should not need to change very much in this file. If you forget to make a local copy of this file, you may get a null caps error when trying to populate the database tables.
 +
 
 +
$ cd $LJHOME
 +
$ cp etc/config.pl.example ext/local/etc/config.pl
 +
$ vim ext/local/etc/config.pl
 +
 
 +
Note that from here on, when any documentation says to edit <tt>etc/config*</tt>, you'll likely want to edit <tt>ext/local/etc/config-*</tt> instead.
  
 
== Make sure things are working with checkconfig.pl ==
 
== Make sure things are working with checkconfig.pl ==
  
Now, you need to see if everything is working.  If you've setup the files as indicated above, run this command:
+
Now, you need to see if everything is working.   
  
cd $LJHOME
+
If you've setup the files as indicated above, run this command:
bin/checkconfig.pl --no=ljconfig
+
  
It should take a few minutes to run and then tell you that you are missing modules. This is where you need some systems specific knowledge for your system.  You will need to install whatever modules it's missing; hopefully with these instructions, none will be.  If you get well and truly stuck, find someone on IRC or the mailing list.
+
$ cd $LJHOME
 +
$ bin/checkconfig.pl
 +
 
 +
If you installed everything given at the top of this page, you should find you have no missing modules. Congratulations! If you don't have all the modules, this is where you need some systems specific knowledge for your system.  You will need to install whatever modules are missing.  If you get well and truly stuck, find someone on IRC, the mailing list, or make a post to <dwcomm>dw_dev</dwcomm>.
  
 
== Populate database with initial data ==
 
== Populate database with initial data ==
Line 162: Line 345:
 
There are a few commands you can now run to install the database.  Just run these and watch for errors.
 
There are a few commands you can now run to install the database.  Just run these and watch for errors.
  
<b>Note:</b>  If make_system.pl says it can't give the system user admin privileges, something has gone wrong with your database population, even if there were no errors.
+
<b>Note:</b>  If make_system.pl says it can't give the system user admin [[privileges]], something has gone wrong with your database population, even if there were no errors.
  
  $LJHOME/bin/upgrading/update-db.pl -r --innodb
+
  $ $LJHOME/bin/upgrading/update-db.pl -r --innodb
  $LJHOME/bin/upgrading/update-db.pl -r --cluster=all --innodb
+
  $ $LJHOME/bin/upgrading/update-db.pl -r --innodb # at least for now we have to run this twice
  $LJHOME/bin/upgrading/make_system.pl
+
  $ $LJHOME/bin/upgrading/update-db.pl -r --cluster=all --innodb
$LJHOME/bin/upgrading/update-db.pl -p
+
  $ $LJHOME/bin/upgrading/update-db.pl -p
  $LJHOME/bin/upgrading/texttool.pl load
+
  
(That step will ask you for a password for the system account.  You can change it later by logging in as system, so just give it something for now.)
+
If you created your database for TheSchwartz earlier, you'll also want to populate it with its default tables:
  
  bin/upgrading/copy-emailpass-out-of-user
+
  $ mysql -u dw -p dw_schwartz < /usr/share/doc/libtheschwartz-perl/schema.sql
 +
 
 +
 
 +
Create a system account:
 +
 
 +
$ $LJHOME/bin/upgrading/make_system.pl
 +
 
 +
(That step will ask you for a password for the [[System account]].  You can change it later by logging in as system, so just give it something for now.)
 +
 
 +
And load in the translation strings:
 +
 
 +
$ $LJHOME/bin/upgrading/texttool.pl load
 +
 
 +
== Compile Foundation stylesheets and static files ==
 +
 
 +
Follow the [[SCSS]] instructions for installing compass and sass:
 +
 
 +
$ gem install sass --version 3.4.25
 +
$ gem install compass --version 1.0.3
 +
 
 +
Install Java, which will download about 55.9MB and take up 334MB of disk space:
 +
 
 +
$ sudo apt-get install default-jre
 +
 
 +
Then compile the stylesheets and static files for dw-free:
 +
 
 +
$ $LJHOME/bin/build-static.sh
 +
 
 +
You will see a warning for the compressor not being present, ignore it.
  
 
== Configure Apache ==
 
== Configure Apache ==
  
This is the Apache 2 configuration running on the Dreamwidth staging siteI put this in a file named "stage" in /etc/apache2/conf.d, and then I removed the "default" file from /etc/apache2/sites-enabled.
+
This step will need to be done as the root userFollow the instructions below for your version of Apache, and make sure to do <code> service apache2 restart</code> when you are done for your config changes to take effect.
  
User dw
+
=== Apache 2.4 configuration ===
Group dw
+
 
UseCanonicalName off
+
This is the configuration you need for Ubuntu 16.04 and newer.
 +
 
 +
Add the contents of [https://gist.github.com/kareila/9b1f62cfe4eab2a8bb70 this config file] (name it something like <code>dw.conf</code>), change the hostname, port, and directory if necessary and add to <code>/etc/apache2/sites-available</code>.
 +
 
 +
Create the apache log directory referenced in the configuration file:
 +
 
 +
  $ mkdir /home/dw/apache_logs
 
   
 
   
StartServers 1
+
To activate the site run:
MaxSpareServers 2
+
MinSpareServers 1
+
   
+
DocumentRoot /home/dw/htdocs
+
   
+
PerlSetEnv  LJHOME /home/dw
+
PerlPassEnv LJHOME
+
   
+
PerlRequire  /home/dw/cgi-bin/modperl.pl
+
  
You might also have to enable the Perl Apache Request module:
+
  $ sudo a2ensite dw
  
a2enmod apreq
+
(Replacing dw with whatever you named your config file before the .conf bit)
 +
 
 +
Then disable the default site:
 +
 
 +
  $ sudo a2dissite 000-default
 +
 
 +
And activate the Perl Requests and Perl modules:
 +
 
 +
  $ sudo a2enmod apreq2
 +
  $ sudo a2enmod perl
 +
 
 +
Also activate the <code>mpm_prefork</code> module - you may see an error about needing to disable <code>mpm_event</code> first, in which case do the following:
 +
 
 +
  $ sudo a2dismod mpm_event
 +
  $ sudo a2enmod mpm_prefork
 +
 
 +
Finally, restart apache:
 +
 
 +
  $ sudo service apache2 restart
 +
 
 +
Some errors may come up here, if they do, don't be disheartened! In nearly all cases Apache provides enough information to fix these.
 +
 
 +
=== Apache 2.2 configuration ===
 +
 
 +
[https://github.com/dreamwidth/dw-free/issues/923 Apache 2.2 is no longer supported by Dreamwidth code].
  
 
== Now what? ==
 
== Now what? ==
  
The list of things that need doing are in [http://www.dwscoalition.org/buglist.cgi?query_format=advanced&short_desc_type=allwordssubstr&short_desc=&long_desc_type=substring&long_desc=&bug_file_loc_type=allwordssubstr&bug_file_loc=&keywords_type=allwords&keywords=&bug_status=NEW&bug_status=ASSIGNED&bug_status=REOPENED&emailassigned_to1=1&emailtype1=substring&email1=&emailassigned_to2=1&emailreporter2=1&emailqa_contact2=1&emailcc2=1&emailtype2=substring&email2=&bugidtype=include&bug_id=&chfie Bugzilla].  You can keep your code update with the instructions in [[Dev Maintenance]].  Instructions on submitting patches are in [[Dev Patches]].
+
=== Have a look at your new DW instance ===
 +
 
 +
Congratulations!  You now have a working (though minimal) Dreamwidth install.  If you point your web browser at your server, you should see a bare-looking welcome page.
 +
 
 +
=== Further setup ===
 +
 
 +
There are lots of other articles on setting up and customizing your DW install in the [[:Category: Dreamwidth Installation|DW Installation]] category, including:
 +
 
 +
* [[TheSchwartz Setup]] - TheSchwartz is needed for a handful of features - notably comment posting
 +
* [[MogileFS setup]]
 +
* [[Subdomain setup]]
 +
* [[DNS and email setup]]
 +
* [[Statistics setup]]
 +
* [[Generating documentation]]
 +
* [[Allow users to register]]
 +
* Setting up [[Search]]
 +
* [[Dev Testing]] -- setting up the test db and running the tests
 +
 
 +
There is a list of wanted how-tos at [[Installation Wanted How-To]], in case you need to add something to the list.
 +
 
 +
=== Starting development ===
 +
 
 +
If you haven't already, see [[Dev Getting Started]].
  
 
[[Category: Dreamwidth Installation]]
 
[[Category: Dreamwidth Installation]]

Latest revision as of 00:49, 30 January 2024

Firstly, you'll need to set up a Linux system -- here are the Suggested Server Requirements. These instructions assume Ubuntu 16.04. Later versions (as of December 18, 2017) may or may not work, see https://github.com/dreamwidth/dw-free/issues/2248. Please let us know if you run into difficulties--we want to update this if people are having problems! We recommend coming into the #dreamwidth-dev channel on IRC.

After you set up, you can keep updated with the instructions on Dev Maintenance.

If you have difficulties or problems or suggestions for installation procedures, please report them to [info]dw_dev. Also look at Production Setup: Webserver.

Command convention

In this walkthrough, commands to be run as a regular user are prefixed with $. Commands to be run as the super-user are prefixed with #. If you are not the super-user, you can run single commands like this:

$ sudo <command goes here>

Alternately, you can spawn a super-user shell using:

$ sudo -i
# <command goes here>

When done with the super-user shell, exit the shell via:

# exit

Configuring the OS and environment

Bringing the system up to date

Before you start installing anything else, you might want to make sure your system is up to date. You should update the packages on your system:

# apt-get update
# apt-get upgrade

If you want man pages on your system:

# apt-get install man

Installing necessary packages

This will install necessary packages you'll need to run Dreamwidth (and some optional ones). You'll want to be root when running these commands.

# apt-get install git apache2 apache2-bin apache2-data apache2-utils \
  libapache2-mod-perl2 libapache2-mod-apreq2 libapache2-request-perl \
  mysql-server wget unzip links vim libclass-autouse-perl libdatetime-perl \
  libcache-memcached-perl libhash-multivalue-perl libgd-gd2-perl \
  libhtml-template-perl libwww-perl libmime-lite-perl \
  liburi-perl libxml-simple-perl libclass-accessor-perl \
  libclass-data-inheritable-perl libclass-trigger-perl libcrypt-dh-perl \
  libmath-bigint-gmp-perl liburi-fetch-perl libgd-graph-perl \
  libgnupg-interface-perl libmail-gnupg-perl perlmagick \
  libproc-processtable-perl libsoap-lite-perl librpc-xml-perl \
  libstring-crc32-perl libxml-atom-perl libxml-rss-perl \
  libimage-size-perl libunicode-maputf8-perl libgtop2-dev build-essential \
  libnet-openid-consumer-perl libnet-openid-server-perl libyaml-perl \
  libcaptcha-recaptcha-perl libdbd-sqlite3-perl libtest-simple-perl \
  libtemplate-perl libterm-readkey-perl libmime-base64-urlsafe-perl \
  gcc libtest-most-perl libgearman-client-perl libfile-find-rule-perl \
  libbusiness-creditcard-perl liblwpx-paranoidagent-perl libtheschwartz-perl \
  libfile-type-perl libjson-perl ruby libdbd-mysql-perl libdanga-socket-perl \
  libio-aio-perl libsys-syscall-perl liblog-log4perl-perl libtext-markdown-perl \
  libimage-exiftool-perl libnet-oauth-perl libnet-smtps-perl libxmlrpc-lite-perl

This will download about 106MB of files and use around 511MB of disk space.

Check whether these packages have actually installed. Later on in the process, if you find you are having inexplicable problems, try installing these again.

You will also want to install and configure Postfix so your DW can send out email:

# apt-get install postfix

If you choose not to configure on the install, you can do so later with:

# dpkg-reconfigure postfix

Configure Apache 2

Check whether your system is using threaded MPMs or pre-fork. You will need the latter. Note: if you ran the above command to install Apache, then you need to run these commands to switch to the pre-fork model:

# a2dismod mpm_event
# a2enmod mpm_prefork
# service apache2 restart

Install Perl modules with CPAN

You will also want to install some perl libraries with CPAN. Defaults during CPAN's setup should be okay.

Before installing, check if your CPAN shell has make at the correct location. Run

# which make
/usr/bin/make

then compare the results with running:

# cpan 
...
Would you like to configure as much as possible automatically? [yes] <PRESS ENTER>
...
cpan[1]> o conf make
    make              [/usr/bin/make]
...
cpan[2]> exit

The value printed should match that from the which make command you previously ran.

To install the perl libraries, first set up CPAN:

# cpan Bundle::CPAN

When prompted, press Enter to exit tests. Then, run the following command to install the rest of the necessary libraries:

# cpan GTop Digest::SHA1 Unicode::CheckUTF8 MogileFS::Client \
  TheSchwartz::Worker::SendEmail LWP::UserAgent::Paranoid \
  Mozilla::CA List::Util Paws::S3 Net::DNS Text::Fuzzy

(Net::DNS is required to be installed via CPAN because the version included with current versions of Ubuntu is 0.81, which is too old for LWP::UserAgent::Paranoid to work properly.)

Note that the MogileFS::Client is required even if you are not configuring MogileFS.

Set Timezone to UTC

Dreamwidth will complain if you aren't running in UTC. If you're not configured for UTC already, you can reset it (on Ubuntu/Debian) using

# dpkg-reconfigure tzdata

and choose Etc/UTC.

Configuring MySQL/MariaDB

Newer versions of MySQL and/or MariaDB have imposed stricter requirements than the current Dreamwidth SQL definitions can comply with. These restrictions must be relaxed, or the initial database population scripts will fail.

To do this, run the following 4 commands:

$ sudo -i
# echo "sql_mode = ''" >> /etc/mysql/mysql.conf.d/mysqld.cnf
# service mysql restart
# exit

Setting up the DW user account

Note: If you still find yourself unable to make sudo commands even after usermod, please see Making your DW user a sudo account for alternatives.
Create a user account that will be used for the code. This can be your own personal account if you wish, but it is recommended that you setup a new user that is only going to be used for running the code. That way if you have any insecure code running (i.e., a bad page, or a patch you're working on doesn't have security implemented yet) you don't have to worry about someone getting access to your personal files.

To set up the user account:

# adduser username

Fill out the user's info and put them in the sudo group:

# usermod -a -G sudo username

So, to use dw as a username:

# adduser dw
# usermod -a -G sudo dw

Now, we will get the $LJHOME variable set. Log into your user:

# su - dw

We will assume that you use the bash shell. If you do not, then this section does not apply. You will have to find the shell specific way of setting environment variables.

For bash, you should look in the home directory of the user account you just created for a file .profile. You will want to add one more line to this file:

export LJHOME=/home/dw/dw

Or wherever it is that you want the root of your source code install to be. Yes, the variable is named LJHOME. We'll live with it for now.  ;-)

Test this. Log out of the dw user account and log back in, then type:

echo $LJHOME

You should see /home/dw/dw or whatever you set it to. If you don't, then something has gone awry -- maybe the export line isn't in the right place..?

Get the Dreamwidth code

Configure git

Pick up a name and email address you're willing to share with the world (or at least that part of it that pays attention to GitHub and Dreamwidth). Then run the following 2 commands, replacing NAME and EMAIL with what you chose:

$ git config --global user.name "NAME"
$ git config --global user.email "EMAIL"

Optionally, make git use your preferred editor for commit messages (and a few other things). First find the path to your favorite editor (replace EDITOR with the name of your preferred editor, eg nano or vi):

$ which EDITOR

This will print something like: /usr/bin/vi (if your editor of choice is vi). Then, run the following command, replacing /PATH/TO/EDITOR with whatever was output:

$ git config --global core.editor "/PATH/TO/EDITOR"

Set up GitHub

First, create an account on Github. Then, you will need to fork these two repositories:

Go to each of their pages, and click the "Fork" button. This sets up your own copy of Dreamwidth's code which you will use to make and submit your changes.

Here's a quick overview of how the repositories will work together:

  • dw-free is the main repository and goes into $LJHOME
  • other repositories will go into $LJHOME/ext
  • personal config files go into $LJHOME/ext/local
  • All code from dw-free ($LJHOME) and the additional repos under $LJHOME/ext are automatically live when you start your server. There's no need to run any additional syncing steps anymore.

Set up dw-free

Replace the USERNAME sections of the URL with your Github username. This will make your repository an authenticated version that can push changes back to Github, as opposed to only being able to pull them.

First, clone a copy of the repository onto your machine:

$ cd ~/
$ git clone https://USERNAME@github.com/USERNAME/dw-free.git $LJHOME
$ cd $LJHOME

Now, let's make it aware of the dreamwidth repository so we can grab updates later:

$ git remote add dreamwidth https://github.com/dreamwidth/dw-free
$ git fetch dreamwidth

Finally, let's set up to sync up with the main repository's master branch:

$ git branch -u dreamwidth/master master

Set up dw-nonfree

Warning: For dev servers only. Clone sites should not use dw-nonfree because it contains Dreamwidth-specific branding and files.
$ cd $LJHOME/ext
$ git clone https://USERNAME@github.com/USERNAME/dw-nonfree.git
$ cd dw-nonfree
$ git remote add dreamwidth https://github.com/dreamwidth/dw-nonfree
$ git fetch dreamwidth
$ git branch -u dreamwidth/master master
$ cd ../..

Database setup

You should have a local MySQL installation and know the root login to the database. To sign into MySQL as root:

$ mysql -u root -p  

These MySQL commands will create your development database:

mysql> create database dw;
mysql> grant all on dw.* to 'dw'@'localhost' identified by 'somePassword';

That last line is made up of a few different sections:

  • grant all on dw.* - this tells MySQL to grant all permissions to the "dw" database which you created in the first line.
  • to 'dw'@'localhost' - this tells MySQL to grant the above permissions to a username called "dw" connecting from "localhost" (which means the same computer as the one MySQL is on). You can change "dw" to another username if you like. NB: this is 'dw'@'localhost', not 'dw@localhost' - either cut and paste, or type carefully!
  • identified by 'somePassword' - since this user doesn't exist yet, we can tell MySQL to create it by adding this. This tells MySQL that the "dw" user will use the password "somePassword" to connect. You probably want to change this password!

While you're here, you probably want to go ahead and create a database for TheSchwartz, too, unless you really know that you won't need it:

mysql> create database dw_schwartz;
mysql> grant all on dw_schwartz.* to 'dw'@'localhost';

(We're leaving out the "identified by" part this time because the user already exists.)

When you're done, exit out of MySQL:

mysql> quit;

Editing the config files

Next you need to configure the site configuration scripts. This is probably the most tricky part of the whole process, since there are so many things you can tweak. However, you can get by with just tweaking etc/config-local.pl and etc/config-private.pl for now.

Copy over your config files, and tell them to take priority over config files in any of the repos:

$ cd $LJHOME
$ mkdir -p ext/local/etc
$ echo "highest"  > ext/local/.dir_scope
$ cp etc/config-local.pl.example ext/local/etc/config-local.pl
$ vim ext/local/etc/config-local.pl

Or use your editor of choice, of course, instead of vim. Do a search for the phrase 'CHANGE THIS' which occurs once, before the block of human readable text you should use to describe your development installation.

Note that the $IS_DEV_SERVER flag is set to 1 in the template given. Be sure to set this to 0 for a production site, as there are big security issues involved with leaving the flag set to 1.

Below the 'CHANGE THIS' section (unless this is already present somewhere), add:

   $STATDOCS = "$HOME/build/static"

Next, you need to change your local variables in the config-private.pl file. This is where your passwords are configured, as well as many of the variables which define your domain. Do a search for the phrase 'CHANGETHIS'(note the slightly different spacing). You will want to change at least your $DOMAIN and the %DBINFO structure.

$ cd $LJHOME
$ cp etc/config-private.pl.example ext/local/etc/config-private.pl
$ chmod go-rwx ext/local/etc/config-private.pl
$ vim ext/local/etc/config-private.pl

In this file, uncomment the lines in the %DBINFO blocks (if they are commented out), and place your database password in quotes on the "pass => ," line, before the comma. For example:

   %DBINFO = (
       master => {
           pass => 'mypassword',
       },
   );

Note that there are two DBINFO blocks; get them both.

If you created the database for TheSchwartz earlier, you'll also want to configure it:

   # Schwartz DB configuration
   @THESCHWARTZ_DBS = (
           {
               dsn => 'dbi:mysql:dw_schwartz;host=localhost',
               user => 'dw',
               pass => ,     # CHANGETHIS
           },
       );

Near the bottom of the file is the line you will need to change to tell your server which domain to use.

   package DW::PRIVATE;
   
   $DOMAIN = "ljsite.com";
A note about $DOMAIN: If you are running a local server for development purposes and don't need or want a hostname, you may be tempted to just set $DOMAIN to localhost, but the server won't function correctly without a domain name. One possible workaround for this is to set it to a nonexistent domain (the default ljsite.com is fine) and add a line in your /etc/hosts file that resolves that domain name to your localhost IP (either 127.0.0.1 or ::1.) Another workaround is to install and configure dnsmasq.
SSL: If you're running a development server and don't want to fuss with SSL, put this in your local config:
   $PROTOCOL = "http";

The base configuration file also needs to be copied over, although for most purposes you should not need to change very much in this file. If you forget to make a local copy of this file, you may get a null caps error when trying to populate the database tables.

$ cd $LJHOME
$ cp etc/config.pl.example ext/local/etc/config.pl
$ vim ext/local/etc/config.pl

Note that from here on, when any documentation says to edit etc/config*, you'll likely want to edit ext/local/etc/config-* instead.

Make sure things are working with checkconfig.pl

Now, you need to see if everything is working.

If you've setup the files as indicated above, run this command:

$ cd $LJHOME
$ bin/checkconfig.pl

If you installed everything given at the top of this page, you should find you have no missing modules. Congratulations! If you don't have all the modules, this is where you need some systems specific knowledge for your system. You will need to install whatever modules are missing. If you get well and truly stuck, find someone on IRC, the mailing list, or make a post to [info]dw_dev.

Populate database with initial data

There are a few commands you can now run to install the database. Just run these and watch for errors.

Note: If make_system.pl says it can't give the system user admin privileges, something has gone wrong with your database population, even if there were no errors.

$ $LJHOME/bin/upgrading/update-db.pl -r --innodb
$ $LJHOME/bin/upgrading/update-db.pl -r --innodb # at least for now we have to run this twice
$ $LJHOME/bin/upgrading/update-db.pl -r --cluster=all --innodb
$ $LJHOME/bin/upgrading/update-db.pl -p

If you created your database for TheSchwartz earlier, you'll also want to populate it with its default tables:

$ mysql -u dw -p dw_schwartz < /usr/share/doc/libtheschwartz-perl/schema.sql


Create a system account:

$ $LJHOME/bin/upgrading/make_system.pl

(That step will ask you for a password for the System account. You can change it later by logging in as system, so just give it something for now.)

And load in the translation strings:

$ $LJHOME/bin/upgrading/texttool.pl load

Compile Foundation stylesheets and static files

Follow the SCSS instructions for installing compass and sass:

$ gem install sass --version 3.4.25
$ gem install compass --version 1.0.3

Install Java, which will download about 55.9MB and take up 334MB of disk space:

$ sudo apt-get install default-jre

Then compile the stylesheets and static files for dw-free:

$ $LJHOME/bin/build-static.sh

You will see a warning for the compressor not being present, ignore it.

Configure Apache

This step will need to be done as the root user. Follow the instructions below for your version of Apache, and make sure to do service apache2 restart when you are done for your config changes to take effect.

Apache 2.4 configuration

This is the configuration you need for Ubuntu 16.04 and newer.

Add the contents of this config file (name it something like dw.conf), change the hostname, port, and directory if necessary and add to /etc/apache2/sites-available.

Create the apache log directory referenced in the configuration file:

 $ mkdir /home/dw/apache_logs

To activate the site run:

 $ sudo a2ensite dw

(Replacing dw with whatever you named your config file before the .conf bit)

Then disable the default site:

 $ sudo a2dissite 000-default

And activate the Perl Requests and Perl modules:

 $ sudo a2enmod apreq2
 $ sudo a2enmod perl

Also activate the mpm_prefork module - you may see an error about needing to disable mpm_event first, in which case do the following:

 $ sudo a2dismod mpm_event
 $ sudo a2enmod mpm_prefork

Finally, restart apache:

 $ sudo service apache2 restart

Some errors may come up here, if they do, don't be disheartened! In nearly all cases Apache provides enough information to fix these.

Apache 2.2 configuration

Apache 2.2 is no longer supported by Dreamwidth code.

Now what?

Have a look at your new DW instance

Congratulations! You now have a working (though minimal) Dreamwidth install. If you point your web browser at your server, you should see a bare-looking welcome page.

Further setup

There are lots of other articles on setting up and customizing your DW install in the DW Installation category, including:

There is a list of wanted how-tos at Installation Wanted How-To, in case you need to add something to the list.

Starting development

If you haven't already, see Dev Getting Started.