Difference between revisions of "Dreamwidth Scratch Installation"

From Dreamwidth Notes
Jump to: navigation, search
(Installing necessary packages: removed subversion (we don't use it any more!), added libmime-base64-urlsafe-perl (which checkconfig.pl otherwise complains about))
(Further setup: DNS and email)
 
(47 intermediate revisions by 9 users not shown)
Line 1: Line 1:
  
Firstly, you'll need to set up a Linux system -- here are the [[Suggested Server Requirements]].  These instructions assume Ubuntu 12.04 or up.  Please let us know if you run into difficulties--we want to update this if people are having problems!  We recommend coming into the #dreamwidth-dev channel on [[IRC]].
+
Firstly, you'll need to set up a Linux system -- here are the [[Suggested Server Requirements]].  These instructions assume Ubuntu 16.04.  Later versions (as of December 18, 2017) may or may not work, see https://github.com/dreamwidth/dw-free/issues/2248.  Please let us know if you run into difficulties--we want to update this if people are having problems!  We recommend coming into the #dreamwidth-dev channel on [[IRC]].
  
 
After you set up, you can keep updated with the instructions on [[Dev Maintenance]].
 
After you set up, you can keep updated with the instructions on [[Dev Maintenance]].
  
If you have difficulties or problems or suggestions for installation procedures, please report them to [http://bugs.dwscoalition.org/show_bug.cgi?id=61 Bug 61].  Also look at [[Production Setup: Webserver]].  
+
If you have difficulties or problems or suggestions for installation procedures, please report them to <dwuser>dw_dev</dwuser>.  Also look at [[Production Setup: Webserver]].
  
== Bringing the system up to date ==
+
== Command convention ==
  
Before you start installing anything else, you might want to make sure your system is up to date. You should update the packages on your system (either in a root shell (sudo bash) or preceding these commands with 'sudo'):
+
In this walkthrough, commands to be run as a regular user are prefixed with <tt>$</tt>. Commands to be run as the super-user are prefixed with <tt>#</tt>. If you are not the super-user, you can run single commands like this:
  
  apt-get update
+
  $ sudo <command goes here>
  apt-get upgrade
+
 
 +
Alternately, you can spawn a super-user shell using:
 +
 
 +
$ sudo -i
 +
# <command goes here>
 +
 
 +
When done with the super-user shell, exit the shell via:
 +
 
 +
# exit
 +
 
 +
== Configuring the OS and environment==
 +
 
 +
=== Bringing the system up to date ===
 +
 
 +
Before you start installing anything else, you might want to make sure your system is up to date.  You should update the packages on your system:
 +
 
 +
# apt-get update
 +
  # apt-get upgrade
  
 
If you want man pages on your system:  
 
If you want man pages on your system:  
  
  apt-get install man
+
  # apt-get install man
  
== Installing necessary packages ==
+
=== Installing necessary packages ===
  
 
This will install necessary packages you'll need to run Dreamwidth (and some optional ones). You'll want to be root when running these commands.
 
This will install necessary packages you'll need to run Dreamwidth (and some optional ones). You'll want to be root when running these commands.
  
  apt-get install git-core apache2-mpm-prefork \
+
  # apt-get install git apache2 apache2-bin apache2-data apache2-utils \
 
   libapache2-mod-perl2 libapache2-mod-apreq2 libapache2-request-perl \
 
   libapache2-mod-perl2 libapache2-mod-apreq2 libapache2-request-perl \
   mysql-server wget unzip links vim libclass-autouse-perl \
+
   mysql-server wget unzip links vim libclass-autouse-perl libdatetime-perl \
   libdatetime-perl libcache-memcached-perl libhash-multivalue-perl \
+
   libcache-memcached-perl libhash-multivalue-perl libgd-gd2-perl \
  libgd-gd2-perl libhtml-template-perl libwww-perl libmime-lite-perl \
+
  libhtml-template-perl libwww-perl libmime-lite-perl \
   libnet-dns-perl liburi-perl libxml-simple-perl libclass-accessor-perl \
+
   liburi-perl libxml-simple-perl libclass-accessor-perl \
 
   libclass-data-inheritable-perl libclass-trigger-perl libcrypt-dh-perl \
 
   libclass-data-inheritable-perl libclass-trigger-perl libcrypt-dh-perl \
 
   libmath-bigint-gmp-perl liburi-fetch-perl libgd-graph-perl \
 
   libmath-bigint-gmp-perl liburi-fetch-perl libgd-graph-perl \
 
   libgnupg-interface-perl libmail-gnupg-perl perlmagick \
 
   libgnupg-interface-perl libmail-gnupg-perl perlmagick \
 
   libproc-processtable-perl libsoap-lite-perl librpc-xml-perl \
 
   libproc-processtable-perl libsoap-lite-perl librpc-xml-perl \
   libstring-crc32-perl libtext-vcard-perl libxml-atom-perl libxml-rss-perl \
+
   libstring-crc32-perl libxml-atom-perl libxml-rss-perl \
 
   libimage-size-perl libunicode-maputf8-perl libgtop2-dev build-essential \
 
   libimage-size-perl libunicode-maputf8-perl libgtop2-dev build-essential \
 
   libnet-openid-consumer-perl libnet-openid-server-perl libyaml-perl \
 
   libnet-openid-consumer-perl libnet-openid-server-perl libyaml-perl \
 
   libcaptcha-recaptcha-perl libdbd-sqlite3-perl libtest-simple-perl \
 
   libcaptcha-recaptcha-perl libdbd-sqlite3-perl libtest-simple-perl \
 
   libtemplate-perl libterm-readkey-perl libmime-base64-urlsafe-perl \
 
   libtemplate-perl libterm-readkey-perl libmime-base64-urlsafe-perl \
   gcc libtest-most-perl libgearman-client-perl \
+
   gcc libtest-most-perl libgearman-client-perl libfile-find-rule-perl \
   libbusiness-creditcard-perl liblwpx-paranoidagent-perl \
+
   libbusiness-creditcard-perl liblwpx-paranoidagent-perl libtheschwartz-perl \
  libtheschwartz-perl libfile-type-perl libjson-perl
+
  libfile-type-perl libjson-perl ruby libdbd-mysql-perl libdanga-socket-perl \
 +
  libio-aio-perl libsys-syscall-perl liblog-log4perl-perl libtext-markdown-perl \
 +
  libimage-exiftool-perl libnet-oauth-perl libnet-smtps-perl libxmlrpc-lite-perl
  
This will download about 113MB of files and use around 355MB of disk space.
+
This will download about 106MB of files and use around 511MB of disk space.
  
 
Check whether these packages have actually installed.  Later on in the process, if you find you are having inexplicable problems, try installing these again.
 
Check whether these packages have actually installed.  Later on in the process, if you find you are having inexplicable problems, try installing these again.
  
 +
You will also want to install and configure Postfix so your DW can send out email:
  
=== Then ===
+
# apt-get install postfix
  
[http://perl.apache.org/docs/2.0/user/install/install.html#Prerequisites Check] whether your system is using threaded MPMs or pre-fork. You will need the latter.  (Note: if you ran the above command to install Apache, then you have the pre-fork version installed.)
+
If you choose not to configure on the install, you can do so later with:
  
If you have weird errors saying things like "Package mercurial is not available, but is referred to by another package.", try editing <tt>/etc/apt/sources.list</tt> and uncommenting the other repositories and then doing an <tt>apt-get</tt> update. Ran into this problem on a fresh install of <tt>jaunty</tt> on Linode.
+
# dpkg-reconfigure postfix
 +
 
 +
=== Configure Apache 2 ===
 +
 
 +
[http://perl.apache.org/docs/2.0/user/install/install.html#Prerequisites Check] whether your system is using threaded MPMs or pre-fork. You will need the latter. <em>Note</em>: if you ran the above command to install Apache, then you need to run these commands to switch to the pre-fork model:
 +
 
 +
# a2dismod mpm_event
 +
# a2enmod mpm_prefork
 +
# service apache2 restart
 +
 
 +
=== Install Perl modules with CPAN ===
  
 
You will also want to install some perl libraries with CPAN.  Defaults during CPAN's setup should be okay.
 
You will also want to install some perl libraries with CPAN.  Defaults during CPAN's setup should be okay.
  
Before installing, check if your CPAN shell has 'make' at the correct location.  Run <tt>which make</tt>, then compare the results with running <tt>o conf make</tt> at the CPAN shell (to run the CPAN shell, type <tt>cpan</tt>.)
+
Before installing, check if your CPAN shell has <tt>make</tt> at the correct location. Run
  
The perl libraries are:
+
  # which make
   
+
  /usr/bin/make
cpan Bundle::CPAN  # this will guide you through setting up CPAN
+
cpan GTop 
+
cpan Digest::SHA1
+
cpan Unicode::CheckUTF8
+
cpan MogileFS::Client  # this is necessary even if you don't use MogileFS
+
cpan TheSchwartz::Worker::SendEmail
+
cpan TheSchwartz::Worker::PubSubHubbubPublish
+
cpan Digest::SHA256
+
cpan Text::Markdown
+
cpan Image::ExifTool
+
  cpan Net::OAuth
+
  
See [http://wiki.dwscoalition.org/notes/Bundle::CPAN Bundle::CPAN] if you are having trouble installing Bundle::CPAN
+
then compare the results with running:
  
You will also want to install and configure Postfix so your DW can send out email:
+
# cpan
 +
...
 +
Would you like to configure as much as possible automatically? [yes] <PRESS ENTER>
 +
...
 +
cpan[1]> o conf make
 +
    make              [/usr/bin/make]
 +
...
 +
cpan[2]> exit
  
apt-get install postfix
+
The value printed should match that from the <tt>which make</tt> command you previously ran.
  
If you choose not to configure on the install, you can do so later with:
+
To install the perl libraries, first set up CPAN:
  
  dpkg-reconfigure postfix
+
  # cpan Bundle::CPAN
  
== Set Timezone to UTC ==
+
When prompted, press Enter to exit tests. Then, run the following command to install the rest of the necessary libraries:
 +
 
 +
# cpan GTop Digest::SHA1 Unicode::CheckUTF8 MogileFS::Client \
 +
  TheSchwartz::Worker::SendEmail LWP::UserAgent::Paranoid \
 +
  Mozilla::CA List::Util Paws::S3 Net::DNS Text::Fuzzy
 +
 
 +
(Net::DNS is required to be installed via CPAN because the version included with current versions of Ubuntu is 0.81, which is too old for LWP::UserAgent::Paranoid to work properly.)
 +
 
 +
Note that the MogileFS::Client is required even if you are not configuring MogileFS.
 +
 
 +
=== Set Timezone to UTC ===
  
 
Dreamwidth will complain if you aren't running in UTC.  If you're not configured for UTC already, you can reset it (on Ubuntu/Debian) using
 
Dreamwidth will complain if you aren't running in UTC.  If you're not configured for UTC already, you can reset it (on Ubuntu/Debian) using
  
  dpkg-reconfigure tzdata
+
  # dpkg-reconfigure tzdata
  
 
and choose Etc/UTC.
 
and choose Etc/UTC.
  
== Setting up the DW user account ==
+
=== Configuring MySQL/MariaDB ===
 +
 
 +
Newer versions of MySQL and/or MariaDB have imposed stricter requirements than the current Dreamwidth SQL definitions can comply with. These restrictions must be relaxed, or the initial database population scripts will fail.
 +
 
 +
To do this, run the following 4 commands:
 +
 
 +
$ sudo -i
 +
# <nowiki>echo "sql_mode = ''" >> /etc/mysql/mysql.conf.d/mysqld.cnf</nowiki>
 +
# service mysql restart
 +
# exit
 +
 
 +
=== Setting up the DW user account ===
  
 
{{Note|text=If you still find yourself unable to make sudo commands even after <code>usermod</code>, please see [[Making your DW user a sudo account]] for alternatives.}} Create a user account that will be used for the code.  This can be your own personal account if you wish, but it is recommended that you setup a new user that is only going to be used for running the code.  That way if you have any insecure code running (i.e., a bad page, or a patch you're working on doesn't have security implemented yet) you don't have to worry about someone getting access to your personal files.
 
{{Note|text=If you still find yourself unable to make sudo commands even after <code>usermod</code>, please see [[Making your DW user a sudo account]] for alternatives.}} Create a user account that will be used for the code.  This can be your own personal account if you wish, but it is recommended that you setup a new user that is only going to be used for running the code.  That way if you have any insecure code running (i.e., a bad page, or a patch you're working on doesn't have security implemented yet) you don't have to worry about someone getting access to your personal files.
Line 93: Line 140:
 
To set up the user account:
 
To set up the user account:
  
  adduser <i>username</i>
+
  # adduser <i>username</i>
  
 
Fill out the user's info and put them in the sudo group:
 
Fill out the user's info and put them in the sudo group:
  
  usermod -a -G sudo <i>username</i>
+
  # usermod -a -G sudo <i>username</i>
  
 
So, to use <code>dw</code> as a username:
 
So, to use <code>dw</code> as a username:
  
  adduser dw
+
  # adduser dw
  usermod -a -G sudo dw
+
  # usermod -a -G sudo dw
  
 
Now, we will get the $LJHOME variable set.  Log into your user:
 
Now, we will get the $LJHOME variable set.  Log into your user:
  
  su - dw
+
  # su - dw
  
 
We will assume that you use the bash shell.  If you do not, then this section does not apply.  You will have to find the shell specific way of setting environment variables.
 
We will assume that you use the bash shell.  If you do not, then this section does not apply.  You will have to find the shell specific way of setting environment variables.
  
For bash, you should look in the home directory of the user account you just created for a file .profile. You will want to add one more line to this file:
+
For bash, you should look in the home directory of the user account you just created for a file <tt>.profile</tt>. You will want to add one more line to this file:
  
 
  export LJHOME=/home/dw/dw
 
  export LJHOME=/home/dw/dw
Line 123: Line 170:
  
 
== Get the Dreamwidth code ==
 
== Get the Dreamwidth code ==
 +
 +
=== Configure git ===
 +
 +
Pick up a name and email address you're willing to share with the world (or at least that part of it that pays attention to GitHub and Dreamwidth). Then run the following 2 commands, replacing NAME and EMAIL with what you chose:
 +
 +
$ git config --global user.name "NAME"
 +
$ git config --global user.email "EMAIL"
 +
 +
Optionally, make git use your preferred editor for commit messages (and a few other things). First find the path to your favorite editor (replace EDITOR with the name of your preferred editor, eg nano or vi):
 +
 +
$ which EDITOR
 +
 +
This will print something like: <code>/usr/bin/vi</code> (if your editor of choice is vi). Then, run the following command, replacing /PATH/TO/EDITOR with whatever was output:
 +
 +
$ git config --global core.editor "/PATH/TO/EDITOR"
  
 
=== Set up GitHub ===
 
=== Set up GitHub ===
Line 131: Line 193:
 
* https://github.com/dreamwidth/dw-nonfree
 
* https://github.com/dreamwidth/dw-nonfree
  
Go to each of their pages, and click the "Fork" button. This sets up your own copy of Dreamwidth's code which use to make and submit your changes.
+
Go to each of their pages, and click the "Fork" button. This sets up your own copy of Dreamwidth's code which you will use to make and submit your changes.
  
 
Here's a quick overview of how the repositories will work together:
 
Here's a quick overview of how the repositories will work together:
Line 144: Line 206:
 
Replace the USERNAME sections of the URL with your Github username.  This will make your repository an authenticated version that can push changes back to Github, as opposed to only being able to pull them.
 
Replace the USERNAME sections of the URL with your Github username.  This will make your repository an authenticated version that can push changes back to Github, as opposed to only being able to pull them.
  
# clone a copy of the repository onto your machine
+
First, clone a copy of the repository onto your machine:
  cd ~/
+
 
  git clone https://USERNAME@github.com/USERNAME/dw-free.git $LJHOME
+
  $ cd ~/
  cd $LJHOME
+
  $ git clone https://USERNAME@github.com/USERNAME/dw-free.git $LJHOME
 +
  $ cd $LJHOME
 
   
 
   
# and let's make it aware of the dreamwidth repository so we can grab updates later
+
Now, let's make it aware of the dreamwidth repository so we can grab updates later:
  git remote add dreamwidth https://github.com/dreamwidth/dw-free
+
 
  git fetch dreamwidth
+
  $ git remote add dreamwidth https://github.com/dreamwidth/dw-free
  # we also want to sync up with the main branches
+
  $ git fetch dreamwidth
  git branch --set-upstream develop dreamwidth/develop
+
   
 +
Finally, let's set up to sync up with the main repository's master branch:
 +
 
 +
  $ git branch -u dreamwidth/master master
  
 
=== Set up dw-nonfree ===
 
=== Set up dw-nonfree ===
  
{{ Warn|text=For dev servers only. Clone sites should not use dw-nonfree because it contains Dreamwidth-specific branding and files}}
+
{{ Warn|text=For dev servers only. Clone sites should not use dw-nonfree because it contains Dreamwidth-specific branding and files.}}
  
  cd $LJHOME/ext
+
  $ cd $LJHOME/ext
  git clone https://USERNAME@github.com/USERNAME/dw-nonfree.git
+
  $ git clone https://USERNAME@github.com/USERNAME/dw-nonfree.git
  cd dw-nonfree
+
  $ cd dw-nonfree
  git remote add dreamwidth https://github.com/dreamwidth/dw-nonfree
+
  $ git remote add dreamwidth https://github.com/dreamwidth/dw-nonfree
  git fetch dreamwidth
+
  $ git fetch dreamwidth
  git branch --set-upstream develop dreamwidth/develop
+
  $ git branch -u dreamwidth/master master
  cd ..
+
  $ cd ../..
  
 
== Database setup ==
 
== Database setup ==
Line 171: Line 237:
 
You should have a local MySQL installation and know the root login to the database.  To sign into MySQL as root:
 
You should have a local MySQL installation and know the root login to the database.  To sign into MySQL as root:
  
  mysql -u root -p   
+
  $ mysql -u root -p   
  
These MySQL commands wil create your development database:
+
These MySQL commands will create your development database:
  
  create database dw;
+
  mysql> create database dw;
  grant all on dw.* to 'dw'@'localhost' identified by 'somePassword';
+
  mysql> grant all on dw.* to 'dw'@'localhost' identified by 'somePassword';
  
You might want to pick a more appropriate database username/database name/password.
+
That last line is made up of a few different sections:
 +
 
 +
* '''grant all on dw.*''' - this tells MySQL to grant all permissions to the "dw" database which you created in the first line.
 +
* '''to 'dw'@'localhost'''' - this tells MySQL to grant the above permissions to a username called "dw" connecting from "localhost" (which means the same computer as the one MySQL is on). You can change "dw" to another username if you like. '''NB:''' this is 'dw'@'localhost', not 'dw@localhost' - either cut and paste, or type carefully!
 +
* '''identified by 'somePassword' ''' - since this user doesn't exist yet, we can tell MySQL to create it by adding this. This tells MySQL that the "dw" user will use the password "somePassword" to connect. You probably want to change this password!
  
 
While you're here, you probably want to go ahead and create a database for TheSchwartz, too, unless you really know that you won't need it:
 
While you're here, you probably want to go ahead and create a database for TheSchwartz, too, unless you really know that you won't need it:
  
  create database dw_schwartz;
+
  mysql> create database dw_schwartz;
  grant all on dw_schwartz.* to 'dw'@'localhost' identified by 'somePassword';
+
  mysql> grant all on dw_schwartz.* to 'dw'@'localhost';
 +
 
 +
(We're leaving out the "identified by" part this time because the user already exists.)
  
When you're done, exit out of mysql:
+
When you're done, exit out of MySQL:
  
  quit;
+
  mysql> quit;
  
 
== Editing the config files ==
 
== Editing the config files ==
  
Next you need to configure the site configuration scripts.  This is probably the most tricky part of the whole process, since there are so many things you can tweak. However, you can get by with just tweaking etc/config-local.pl and etc/config-private.pl for now:
+
Next you need to configure the site configuration scripts.  This is probably the most tricky part of the whole process, since there are so many things you can tweak. However, you can get by with just tweaking etc/config-local.pl and etc/config-private.pl for now.
  
## copy over your config files, and tell them to take priority
+
Copy over your config files, and tell them to take priority over config files in any of the repos:
## over config files in any of the repos
+
cd $LJHOME
+
mkdir -p ext/local/etc
+
echo "highest"  > ext/local/.dir_scope
+
cp doc/config-local.pl.txt ext/local/etc/config-local.pl
+
vim ext/local/etc/config-local.pl
+
  
Or use your editor of choice, of course.  Do a search for the phrase 'CHANGE THIS' which occurs once, before the block of human readable text you should use to describe your development installation.
+
$ cd $LJHOME
 +
$ mkdir -p ext/local/etc
 +
$ echo "highest"  > ext/local/.dir_scope
 +
$ cp etc/config-local.pl.example ext/local/etc/config-local.pl
 +
$ vim ext/local/etc/config-local.pl
 +
 
 +
Or use your editor of choice, of course, instead of vim.  Do a search for the phrase 'CHANGE THIS' which occurs once, before the block of human readable text you should use to describe your development installation.
  
 
Note that the <var>$IS_DEV_SERVER</var> flag is set to 1 in the template given. Be sure to set this to 0 for a production site, as there are big security issues involved with leaving the flag set to 1.  
 
Note that the <var>$IS_DEV_SERVER</var> flag is set to 1 in the template given. Be sure to set this to 0 for a production site, as there are big security issues involved with leaving the flag set to 1.  
 +
 +
Below the 'CHANGE THIS' section (unless this is already present somewhere), add:
 +
 +
    $STATDOCS = "$HOME/build/static"
  
 
Next, you need to change your local variables in the <tt>config-private.pl</tt> file. This is where your passwords are configured, as well as many of the variables which define your domain. Do a search for the phrase 'CHANGETHIS'(note the slightly different spacing). You will want to change at least your <var>$DOMAIN</var> and the <var>%DBINFO</var> structure.
 
Next, you need to change your local variables in the <tt>config-private.pl</tt> file. This is where your passwords are configured, as well as many of the variables which define your domain. Do a search for the phrase 'CHANGETHIS'(note the slightly different spacing). You will want to change at least your <var>$DOMAIN</var> and the <var>%DBINFO</var> structure.
  
  cd $LJHOME
+
  $ cd $LJHOME
  cp doc/config-private.pl.txt ext/local/etc/config-private.pl
+
  $ cp etc/config-private.pl.example ext/local/etc/config-private.pl
  chmod go-rwx ext/local/etc/config-private.pl
+
  $ chmod go-rwx ext/local/etc/config-private.pl
  vim ext/local/etc/config-private.pl
+
  $ vim ext/local/etc/config-private.pl
  
 
In this file, uncomment the lines in the <var>%DBINFO</var> blocks (if they are commented out), and place your database password in quotes on the "pass => ," line, before the comma. For example:
 
In this file, uncomment the lines in the <var>%DBINFO</var> blocks (if they are commented out), and place your database password in quotes on the "pass => ," line, before the comma. For example:
Line 233: Line 309:
 
         );
 
         );
  
The base configuration file is under source control, and is already in <tt>etc/config.pl</tt>. You should not need to change anything very much in this, but you might have to change some stuff. If you find you do have to touch this, copy it over first, then you can edit it:
+
Near the bottom of the file is the line you will need to change to tell your server which domain to use.
  
cd $LJHOME
+
    package DW::PRIVATE;
  cp etc/config.pl ext/local/etc/config.pl
+
   
  vim ext/local/etc/config.pl
+
    $DOMAIN = "ljsite.com";
 +
 
 +
<div style="padding:1em; background-color:#fee;">'''A note about $DOMAIN:''' If you are running a local server for development purposes and don't need or want a hostname, you may be tempted to just set $DOMAIN to localhost, but the server won't function correctly without a domain name. One possible workaround for this is to set it to a nonexistent domain (the default ljsite.com is fine) and add a line in your <code>/etc/hosts</code> file that resolves that domain name to your localhost IP (either <tt>127.0.0.1</tt> or <tt>::1</tt>.) Another workaround is to [http://wiki.dreamwidth.net/wiki/index.php/Subdomain_setup#Local_development_via_dnsmasq install and configure dnsmasq].</div>
 +
 
 +
<div style="padding:1em; background-color:#fee;">'''SSL:''' If you're running a development server and don't want to fuss with SSL, put this in your local config:
 +
 
 +
    $PROTOCOL = "http";
 +
</div>
 +
 
 +
The base configuration file also needs to be copied over, although for most purposes you should not need to change very much in this file. If you forget to make a local copy of this file, you may get a null caps error when trying to populate the database tables.
  
 +
$ cd $LJHOME
 +
$ cp etc/config.pl.example ext/local/etc/config.pl
 +
$ vim ext/local/etc/config.pl
  
Note that from here on, when any documentation says to edit etc/config*, you'll likely want to edit ext/local/etc/config-* instead.
+
Note that from here on, when any documentation says to edit <tt>etc/config*</tt>, you'll likely want to edit <tt>ext/local/etc/config-*</tt> instead.
  
 
== Make sure things are working with checkconfig.pl ==
 
== Make sure things are working with checkconfig.pl ==
Line 248: Line 336:
 
If you've setup the files as indicated above, run this command:
 
If you've setup the files as indicated above, run this command:
  
  cd $LJHOME
+
  $ cd $LJHOME
  bin/checkconfig.pl --no=ljconfig
+
  $ bin/checkconfig.pl
  
 
If you installed everything given at the top of this page, you should find you have no missing modules. Congratulations! If you don't have all the modules, this is where you need some systems specific knowledge for your system.  You will need to install whatever modules are missing.  If you get well and truly stuck, find someone on IRC, the mailing list, or make a post to <dwcomm>dw_dev</dwcomm>.
 
If you installed everything given at the top of this page, you should find you have no missing modules. Congratulations! If you don't have all the modules, this is where you need some systems specific knowledge for your system.  You will need to install whatever modules are missing.  If you get well and truly stuck, find someone on IRC, the mailing list, or make a post to <dwcomm>dw_dev</dwcomm>.
Line 259: Line 347:
 
<b>Note:</b>  If make_system.pl says it can't give the system user admin [[privileges]], something has gone wrong with your database population, even if there were no errors.
 
<b>Note:</b>  If make_system.pl says it can't give the system user admin [[privileges]], something has gone wrong with your database population, even if there were no errors.
  
  $LJHOME/bin/upgrading/update-db.pl -r --innodb
+
  $ $LJHOME/bin/upgrading/update-db.pl -r --innodb
  $LJHOME/bin/upgrading/update-db.pl -r --innodb # at least for now we have to run this twice
+
  $ $LJHOME/bin/upgrading/update-db.pl -r --innodb # at least for now we have to run this twice
  $LJHOME/bin/upgrading/update-db.pl -r --cluster=all --innodb
+
  $ $LJHOME/bin/upgrading/update-db.pl -r --cluster=all --innodb
  $LJHOME/bin/upgrading/update-db.pl -p
+
  $ $LJHOME/bin/upgrading/update-db.pl -p
  
 
If you created your database for TheSchwartz earlier, you'll also want to populate it with its default tables:
 
If you created your database for TheSchwartz earlier, you'll also want to populate it with its default tables:
  
  mysql -u dw -p dw_schwartz < /usr/share/doc/libtheschwartz-perl/schema.sql
+
  $ mysql -u dw -p dw_schwartz < /usr/share/doc/libtheschwartz-perl/schema.sql
  
  
 
Create a system account:
 
Create a system account:
  
  $LJHOME/bin/upgrading/make_system.pl
+
  $ $LJHOME/bin/upgrading/make_system.pl
  
 
(That step will ask you for a password for the [[System account]].  You can change it later by logging in as system, so just give it something for now.)
 
(That step will ask you for a password for the [[System account]].  You can change it later by logging in as system, so just give it something for now.)
Line 277: Line 365:
 
And load in the translation strings:
 
And load in the translation strings:
  
  $LJHOME/bin/upgrading/texttool.pl load
+
  $ $LJHOME/bin/upgrading/texttool.pl load
 +
 
 +
== Compile Foundation stylesheets and static files ==
 +
 
 +
Follow the [[SCSS]] instructions for installing compass and sass:
 +
 
 +
$ gem install sass --version 3.4.25
 +
$ gem install compass --version 1.0.3
 +
 
 +
Install Java, which will download about 55.9MB and take up 334MB of disk space:
 +
 
 +
$ sudo apt-get install default-jre
 +
 
 +
Then compile the stylesheets and static files for dw-free:
 +
 
 +
$ $LJHOME/bin/build-static.sh
 +
 
 +
You will see a warning for the compressor not being present, ignore it.
  
 
== Configure Apache ==
 
== Configure Apache ==
  
This step will need to be done as the root user.  Below is the Apache 2 configuration running on the Dreamwidth staging site.  Put this in a file named "stage" in /etc/apache2/conf.d:
+
This step will need to be done as the root user.  Follow the instructions below for your version of Apache, and make sure to do <code> service apache2 restart</code> when you are done for your config changes to take effect.
  
User dw
+
=== Apache 2.4 configuration ===
Group dw
+
 
UseCanonicalName off
+
This is the configuration you need for Ubuntu 16.04 and newer.
 +
 
 +
Add the contents of [https://gist.github.com/kareila/9b1f62cfe4eab2a8bb70 this config file] (name it something like <code>dw.conf</code>), change the hostname, port, and directory if necessary and add to <code>/etc/apache2/sites-available</code>.
 +
 
 +
Create the apache log directory referenced in the configuration file:
 +
 
 +
  $ mkdir /home/dw/apache_logs
 
   
 
   
StartServers 1
+
To activate the site run:
MaxSpareServers 2
+
 
MinSpareServers 1
+
  $ sudo a2ensite dw
   
+
 
DocumentRoot /home/dw/dw/htdocs
+
(Replacing dw with whatever you named your config file before the .conf bit)
   
+
PerlSetEnv  LJHOME /home/dw/dw
+
PerlPassEnv LJHOME
+
   
+
PerlRequire  /home/dw/dw/cgi-bin/modperl.pl
+
  
 
Then disable the default site:
 
Then disable the default site:
  
a2dissite default
+
  $ sudo a2dissite 000-default
 +
 
 +
And activate the Perl Requests and Perl modules:
  
You might also have to enable the Perl Apache Request module:
+
  $ sudo a2enmod apreq2
 +
  $ sudo a2enmod perl
  
a2enmod apreq
+
Also activate the <code>mpm_prefork</code> module - you may see an error about needing to disable <code>mpm_event</code> first, in which case do the following:
  
If you don't want this warning:
+
  $ sudo a2dismod mpm_event
 +
  $ sudo a2enmod mpm_prefork
  
[Thu Jan 15 01:46:54 2009] [warn] NameVirtualHost *:80 has no VirtualHosts
+
Finally, restart apache:
  ... waiting [Thu Jan 15 01:46:55 2009] [warn] NameVirtualHost *:80 has no VirtualHosts
+
  ...done.
+
  
Then use <code>vim /etc/apache2/ports.conf</code> and put a # in front of the NameVirtualHost line:
+
  $ sudo service apache2 restart
  
# NameVirtualHost *:80
+
Some errors may come up here, if they do, don't be disheartened! In nearly all cases Apache provides enough information to fix these.
  
Restart the server:
+
=== Apache 2.2 configuration ===
  
/etc/init.d/apache2 restart
+
[https://github.com/dreamwidth/dw-free/issues/923 Apache 2.2 is no longer supported by Dreamwidth code].
  
 
== Now what? ==
 
== Now what? ==
Line 331: Line 439:
  
 
* [[TheSchwartz Setup]] - TheSchwartz is needed for a handful of features - notably comment posting
 
* [[TheSchwartz Setup]] - TheSchwartz is needed for a handful of features - notably comment posting
 +
* [[MogileFS setup]]
 
* [[Subdomain setup]]
 
* [[Subdomain setup]]
 +
* [[DNS and email setup]]
 
* [[Statistics setup]]
 
* [[Statistics setup]]
 
* [[Generating documentation]]
 
* [[Generating documentation]]
 
* [[Allow users to register]]
 
* [[Allow users to register]]
 
* Setting up [[Search]]
 
* Setting up [[Search]]
 +
* [[Dev Testing]] -- setting up the test db and running the tests
  
 
There is a list of wanted how-tos at [[Installation Wanted How-To]], in case you need to add something to the list.
 
There is a list of wanted how-tos at [[Installation Wanted How-To]], in case you need to add something to the list.

Latest revision as of 00:49, 30 January 2024

Firstly, you'll need to set up a Linux system -- here are the Suggested Server Requirements. These instructions assume Ubuntu 16.04. Later versions (as of December 18, 2017) may or may not work, see https://github.com/dreamwidth/dw-free/issues/2248. Please let us know if you run into difficulties--we want to update this if people are having problems! We recommend coming into the #dreamwidth-dev channel on IRC.

After you set up, you can keep updated with the instructions on Dev Maintenance.

If you have difficulties or problems or suggestions for installation procedures, please report them to [info]dw_dev. Also look at Production Setup: Webserver.

Command convention

In this walkthrough, commands to be run as a regular user are prefixed with $. Commands to be run as the super-user are prefixed with #. If you are not the super-user, you can run single commands like this:

$ sudo <command goes here>

Alternately, you can spawn a super-user shell using:

$ sudo -i
# <command goes here>

When done with the super-user shell, exit the shell via:

# exit

Configuring the OS and environment

Bringing the system up to date

Before you start installing anything else, you might want to make sure your system is up to date. You should update the packages on your system:

# apt-get update
# apt-get upgrade

If you want man pages on your system:

# apt-get install man

Installing necessary packages

This will install necessary packages you'll need to run Dreamwidth (and some optional ones). You'll want to be root when running these commands.

# apt-get install git apache2 apache2-bin apache2-data apache2-utils \
  libapache2-mod-perl2 libapache2-mod-apreq2 libapache2-request-perl \
  mysql-server wget unzip links vim libclass-autouse-perl libdatetime-perl \
  libcache-memcached-perl libhash-multivalue-perl libgd-gd2-perl \
  libhtml-template-perl libwww-perl libmime-lite-perl \
  liburi-perl libxml-simple-perl libclass-accessor-perl \
  libclass-data-inheritable-perl libclass-trigger-perl libcrypt-dh-perl \
  libmath-bigint-gmp-perl liburi-fetch-perl libgd-graph-perl \
  libgnupg-interface-perl libmail-gnupg-perl perlmagick \
  libproc-processtable-perl libsoap-lite-perl librpc-xml-perl \
  libstring-crc32-perl libxml-atom-perl libxml-rss-perl \
  libimage-size-perl libunicode-maputf8-perl libgtop2-dev build-essential \
  libnet-openid-consumer-perl libnet-openid-server-perl libyaml-perl \
  libcaptcha-recaptcha-perl libdbd-sqlite3-perl libtest-simple-perl \
  libtemplate-perl libterm-readkey-perl libmime-base64-urlsafe-perl \
  gcc libtest-most-perl libgearman-client-perl libfile-find-rule-perl \
  libbusiness-creditcard-perl liblwpx-paranoidagent-perl libtheschwartz-perl \
  libfile-type-perl libjson-perl ruby libdbd-mysql-perl libdanga-socket-perl \
  libio-aio-perl libsys-syscall-perl liblog-log4perl-perl libtext-markdown-perl \
  libimage-exiftool-perl libnet-oauth-perl libnet-smtps-perl libxmlrpc-lite-perl

This will download about 106MB of files and use around 511MB of disk space.

Check whether these packages have actually installed. Later on in the process, if you find you are having inexplicable problems, try installing these again.

You will also want to install and configure Postfix so your DW can send out email:

# apt-get install postfix

If you choose not to configure on the install, you can do so later with:

# dpkg-reconfigure postfix

Configure Apache 2

Check whether your system is using threaded MPMs or pre-fork. You will need the latter. Note: if you ran the above command to install Apache, then you need to run these commands to switch to the pre-fork model:

# a2dismod mpm_event
# a2enmod mpm_prefork
# service apache2 restart

Install Perl modules with CPAN

You will also want to install some perl libraries with CPAN. Defaults during CPAN's setup should be okay.

Before installing, check if your CPAN shell has make at the correct location. Run

# which make
/usr/bin/make

then compare the results with running:

# cpan 
...
Would you like to configure as much as possible automatically? [yes] <PRESS ENTER>
...
cpan[1]> o conf make
    make              [/usr/bin/make]
...
cpan[2]> exit

The value printed should match that from the which make command you previously ran.

To install the perl libraries, first set up CPAN:

# cpan Bundle::CPAN

When prompted, press Enter to exit tests. Then, run the following command to install the rest of the necessary libraries:

# cpan GTop Digest::SHA1 Unicode::CheckUTF8 MogileFS::Client \
  TheSchwartz::Worker::SendEmail LWP::UserAgent::Paranoid \
  Mozilla::CA List::Util Paws::S3 Net::DNS Text::Fuzzy

(Net::DNS is required to be installed via CPAN because the version included with current versions of Ubuntu is 0.81, which is too old for LWP::UserAgent::Paranoid to work properly.)

Note that the MogileFS::Client is required even if you are not configuring MogileFS.

Set Timezone to UTC

Dreamwidth will complain if you aren't running in UTC. If you're not configured for UTC already, you can reset it (on Ubuntu/Debian) using

# dpkg-reconfigure tzdata

and choose Etc/UTC.

Configuring MySQL/MariaDB

Newer versions of MySQL and/or MariaDB have imposed stricter requirements than the current Dreamwidth SQL definitions can comply with. These restrictions must be relaxed, or the initial database population scripts will fail.

To do this, run the following 4 commands:

$ sudo -i
# echo "sql_mode = ''" >> /etc/mysql/mysql.conf.d/mysqld.cnf
# service mysql restart
# exit

Setting up the DW user account

Note: If you still find yourself unable to make sudo commands even after usermod, please see Making your DW user a sudo account for alternatives.
Create a user account that will be used for the code. This can be your own personal account if you wish, but it is recommended that you setup a new user that is only going to be used for running the code. That way if you have any insecure code running (i.e., a bad page, or a patch you're working on doesn't have security implemented yet) you don't have to worry about someone getting access to your personal files.

To set up the user account:

# adduser username

Fill out the user's info and put them in the sudo group:

# usermod -a -G sudo username

So, to use dw as a username:

# adduser dw
# usermod -a -G sudo dw

Now, we will get the $LJHOME variable set. Log into your user:

# su - dw

We will assume that you use the bash shell. If you do not, then this section does not apply. You will have to find the shell specific way of setting environment variables.

For bash, you should look in the home directory of the user account you just created for a file .profile. You will want to add one more line to this file:

export LJHOME=/home/dw/dw

Or wherever it is that you want the root of your source code install to be. Yes, the variable is named LJHOME. We'll live with it for now.  ;-)

Test this. Log out of the dw user account and log back in, then type:

echo $LJHOME

You should see /home/dw/dw or whatever you set it to. If you don't, then something has gone awry -- maybe the export line isn't in the right place..?

Get the Dreamwidth code

Configure git

Pick up a name and email address you're willing to share with the world (or at least that part of it that pays attention to GitHub and Dreamwidth). Then run the following 2 commands, replacing NAME and EMAIL with what you chose:

$ git config --global user.name "NAME"
$ git config --global user.email "EMAIL"

Optionally, make git use your preferred editor for commit messages (and a few other things). First find the path to your favorite editor (replace EDITOR with the name of your preferred editor, eg nano or vi):

$ which EDITOR

This will print something like: /usr/bin/vi (if your editor of choice is vi). Then, run the following command, replacing /PATH/TO/EDITOR with whatever was output:

$ git config --global core.editor "/PATH/TO/EDITOR"

Set up GitHub

First, create an account on Github. Then, you will need to fork these two repositories:

Go to each of their pages, and click the "Fork" button. This sets up your own copy of Dreamwidth's code which you will use to make and submit your changes.

Here's a quick overview of how the repositories will work together:

  • dw-free is the main repository and goes into $LJHOME
  • other repositories will go into $LJHOME/ext
  • personal config files go into $LJHOME/ext/local
  • All code from dw-free ($LJHOME) and the additional repos under $LJHOME/ext are automatically live when you start your server. There's no need to run any additional syncing steps anymore.

Set up dw-free

Replace the USERNAME sections of the URL with your Github username. This will make your repository an authenticated version that can push changes back to Github, as opposed to only being able to pull them.

First, clone a copy of the repository onto your machine:

$ cd ~/
$ git clone https://USERNAME@github.com/USERNAME/dw-free.git $LJHOME
$ cd $LJHOME

Now, let's make it aware of the dreamwidth repository so we can grab updates later:

$ git remote add dreamwidth https://github.com/dreamwidth/dw-free
$ git fetch dreamwidth

Finally, let's set up to sync up with the main repository's master branch:

$ git branch -u dreamwidth/master master

Set up dw-nonfree

Warning: For dev servers only. Clone sites should not use dw-nonfree because it contains Dreamwidth-specific branding and files.
$ cd $LJHOME/ext
$ git clone https://USERNAME@github.com/USERNAME/dw-nonfree.git
$ cd dw-nonfree
$ git remote add dreamwidth https://github.com/dreamwidth/dw-nonfree
$ git fetch dreamwidth
$ git branch -u dreamwidth/master master
$ cd ../..

Database setup

You should have a local MySQL installation and know the root login to the database. To sign into MySQL as root:

$ mysql -u root -p  

These MySQL commands will create your development database:

mysql> create database dw;
mysql> grant all on dw.* to 'dw'@'localhost' identified by 'somePassword';

That last line is made up of a few different sections:

  • grant all on dw.* - this tells MySQL to grant all permissions to the "dw" database which you created in the first line.
  • to 'dw'@'localhost' - this tells MySQL to grant the above permissions to a username called "dw" connecting from "localhost" (which means the same computer as the one MySQL is on). You can change "dw" to another username if you like. NB: this is 'dw'@'localhost', not 'dw@localhost' - either cut and paste, or type carefully!
  • identified by 'somePassword' - since this user doesn't exist yet, we can tell MySQL to create it by adding this. This tells MySQL that the "dw" user will use the password "somePassword" to connect. You probably want to change this password!

While you're here, you probably want to go ahead and create a database for TheSchwartz, too, unless you really know that you won't need it:

mysql> create database dw_schwartz;
mysql> grant all on dw_schwartz.* to 'dw'@'localhost';

(We're leaving out the "identified by" part this time because the user already exists.)

When you're done, exit out of MySQL:

mysql> quit;

Editing the config files

Next you need to configure the site configuration scripts. This is probably the most tricky part of the whole process, since there are so many things you can tweak. However, you can get by with just tweaking etc/config-local.pl and etc/config-private.pl for now.

Copy over your config files, and tell them to take priority over config files in any of the repos:

$ cd $LJHOME
$ mkdir -p ext/local/etc
$ echo "highest"  > ext/local/.dir_scope
$ cp etc/config-local.pl.example ext/local/etc/config-local.pl
$ vim ext/local/etc/config-local.pl

Or use your editor of choice, of course, instead of vim. Do a search for the phrase 'CHANGE THIS' which occurs once, before the block of human readable text you should use to describe your development installation.

Note that the $IS_DEV_SERVER flag is set to 1 in the template given. Be sure to set this to 0 for a production site, as there are big security issues involved with leaving the flag set to 1.

Below the 'CHANGE THIS' section (unless this is already present somewhere), add:

   $STATDOCS = "$HOME/build/static"

Next, you need to change your local variables in the config-private.pl file. This is where your passwords are configured, as well as many of the variables which define your domain. Do a search for the phrase 'CHANGETHIS'(note the slightly different spacing). You will want to change at least your $DOMAIN and the %DBINFO structure.

$ cd $LJHOME
$ cp etc/config-private.pl.example ext/local/etc/config-private.pl
$ chmod go-rwx ext/local/etc/config-private.pl
$ vim ext/local/etc/config-private.pl

In this file, uncomment the lines in the %DBINFO blocks (if they are commented out), and place your database password in quotes on the "pass => ," line, before the comma. For example:

   %DBINFO = (
       master => {
           pass => 'mypassword',
       },
   );

Note that there are two DBINFO blocks; get them both.

If you created the database for TheSchwartz earlier, you'll also want to configure it:

   # Schwartz DB configuration
   @THESCHWARTZ_DBS = (
           {
               dsn => 'dbi:mysql:dw_schwartz;host=localhost',
               user => 'dw',
               pass => ,     # CHANGETHIS
           },
       );

Near the bottom of the file is the line you will need to change to tell your server which domain to use.

   package DW::PRIVATE;
   
   $DOMAIN = "ljsite.com";
A note about $DOMAIN: If you are running a local server for development purposes and don't need or want a hostname, you may be tempted to just set $DOMAIN to localhost, but the server won't function correctly without a domain name. One possible workaround for this is to set it to a nonexistent domain (the default ljsite.com is fine) and add a line in your /etc/hosts file that resolves that domain name to your localhost IP (either 127.0.0.1 or ::1.) Another workaround is to install and configure dnsmasq.
SSL: If you're running a development server and don't want to fuss with SSL, put this in your local config:
   $PROTOCOL = "http";

The base configuration file also needs to be copied over, although for most purposes you should not need to change very much in this file. If you forget to make a local copy of this file, you may get a null caps error when trying to populate the database tables.

$ cd $LJHOME
$ cp etc/config.pl.example ext/local/etc/config.pl
$ vim ext/local/etc/config.pl

Note that from here on, when any documentation says to edit etc/config*, you'll likely want to edit ext/local/etc/config-* instead.

Make sure things are working with checkconfig.pl

Now, you need to see if everything is working.

If you've setup the files as indicated above, run this command:

$ cd $LJHOME
$ bin/checkconfig.pl

If you installed everything given at the top of this page, you should find you have no missing modules. Congratulations! If you don't have all the modules, this is where you need some systems specific knowledge for your system. You will need to install whatever modules are missing. If you get well and truly stuck, find someone on IRC, the mailing list, or make a post to [info]dw_dev.

Populate database with initial data

There are a few commands you can now run to install the database. Just run these and watch for errors.

Note: If make_system.pl says it can't give the system user admin privileges, something has gone wrong with your database population, even if there were no errors.

$ $LJHOME/bin/upgrading/update-db.pl -r --innodb
$ $LJHOME/bin/upgrading/update-db.pl -r --innodb # at least for now we have to run this twice
$ $LJHOME/bin/upgrading/update-db.pl -r --cluster=all --innodb
$ $LJHOME/bin/upgrading/update-db.pl -p

If you created your database for TheSchwartz earlier, you'll also want to populate it with its default tables:

$ mysql -u dw -p dw_schwartz < /usr/share/doc/libtheschwartz-perl/schema.sql


Create a system account:

$ $LJHOME/bin/upgrading/make_system.pl

(That step will ask you for a password for the System account. You can change it later by logging in as system, so just give it something for now.)

And load in the translation strings:

$ $LJHOME/bin/upgrading/texttool.pl load

Compile Foundation stylesheets and static files

Follow the SCSS instructions for installing compass and sass:

$ gem install sass --version 3.4.25
$ gem install compass --version 1.0.3

Install Java, which will download about 55.9MB and take up 334MB of disk space:

$ sudo apt-get install default-jre

Then compile the stylesheets and static files for dw-free:

$ $LJHOME/bin/build-static.sh

You will see a warning for the compressor not being present, ignore it.

Configure Apache

This step will need to be done as the root user. Follow the instructions below for your version of Apache, and make sure to do service apache2 restart when you are done for your config changes to take effect.

Apache 2.4 configuration

This is the configuration you need for Ubuntu 16.04 and newer.

Add the contents of this config file (name it something like dw.conf), change the hostname, port, and directory if necessary and add to /etc/apache2/sites-available.

Create the apache log directory referenced in the configuration file:

 $ mkdir /home/dw/apache_logs

To activate the site run:

 $ sudo a2ensite dw

(Replacing dw with whatever you named your config file before the .conf bit)

Then disable the default site:

 $ sudo a2dissite 000-default

And activate the Perl Requests and Perl modules:

 $ sudo a2enmod apreq2
 $ sudo a2enmod perl

Also activate the mpm_prefork module - you may see an error about needing to disable mpm_event first, in which case do the following:

 $ sudo a2dismod mpm_event
 $ sudo a2enmod mpm_prefork

Finally, restart apache:

 $ sudo service apache2 restart

Some errors may come up here, if they do, don't be disheartened! In nearly all cases Apache provides enough information to fix these.

Apache 2.2 configuration

Apache 2.2 is no longer supported by Dreamwidth code.

Now what?

Have a look at your new DW instance

Congratulations! You now have a working (though minimal) Dreamwidth install. If you point your web browser at your server, you should see a bare-looking welcome page.

Further setup

There are lots of other articles on setting up and customizing your DW install in the DW Installation category, including:

There is a list of wanted how-tos at Installation Wanted How-To, in case you need to add something to the list.

Starting development

If you haven't already, see Dev Getting Started.