MogileFS setup

Note: This page is deprecated. Dreamwidth now uses BlobStore for stuff. No new code that requires MogileFS will be accepted as of 2017.

When it wasn't deprecated, it was still a draft. Much of this information was pulled from the MogileFS Wiki combined with the INSTALL.txt instructions from the mogilefs source.

Configuring MogileFS for a Dreamwidth development installation

These are instructions for getting a simple MogileFS installation working on a single instance (likely development/testing) Dreamwidth installation. This is not a full setup document for a production, multi-server environment.

Overview

• Install the MogileFS server and prepare the database
• Configure and run MogileFS services
• Set up Dreamwidth installation to work with MogileFS

Install the MogileFS server and prepare the database

Setup for a typical solo development environment (Debian)

Build and install MogileFS Debian packages

Follow the installation instructions on the MogileFS wiki.

Create MogileFS database

You should already have MySQL installed for your Dreamwidth installation. So log in as root there and create the mogilefs user:

$ mysql -u root -p
mysql> create database mogilefs;
mysql> grant all on mogilefs.* to 'mogile'@'%' identified by 'mogilepw';
mysql> flush privileges;
mysql> quit

Then go ahead and create the schema using mogdbsetup

$ /usr/bin/mogdbsetup --yes --dbname=mogilefs --dbuser=mogile --dbpassword=mogilepw

Setup for a shared development environment (Dreamhacks)

If you are using a dreamhack, the client libraries are already installed, so you only need to download and configure the MogileFS server on your account.

The canonical server code repositories are hosted on Github. There is a simplified and combined copy of the repository available from Kareila on Github.

The remainder of this document assumes that you have checked out that repository to your dreamhack home directory, like so:

$ git clone git@github.com:kareila/mogilefs-for-dreamhacks.git ~/mogilefs

Once it is checked out, edit the two configuration files in the mogilefs/conf directory according to the instructions in comments at the top of each file. You will need to contact Mark to get permission to use three additional ports on the dreamhack machine before proceeding with the configuration.

In order for the MogileFS scripts to be able to find their libraries, you will need to update your PERL5LIB environment variable. Here's what I have added to my .bash_profile:

$ export PERL5LIB=${HOME}/mogilefs/lib:${LJHOME}/cgi-bin

That tells Perl where to find both the MogileFS modules and the DW modules.

Once the scripts are functional, you will also need to run mogdbsetup as above, but without creating a separate database first. So the command you need to execute will look something like this:

$ ~/mogilefs/mogdbsetup --dbhost=localhost --dbport=xxxxx \
--dbname=dreamhack_username --dbuser=dh_username --dbpassword=xxxxx \
--type=MySQL

(The backslashes (\) indicate one long continued line, broken up for readability.)

Again, you need to use the proper username, password and ports for your particular installation. You will receive your port designations from Mark and your database password is in dw/ext/local/etc/config-private.pl.

Before we move on, one last thing I had to do was install the Net::Netmask module into mogilefs/lib, because it was missing from the preinstalled libraries.

$ cpanm -L ~/mogilefs Net::Netmask

Without this module, mogstored will not run.

Configure and run MogileFS services

Note: This section is not required for non-dreamhacks if you've run dpkg-reconfigure as in the link above.

You will need to set up at least one tracker (mogilefsd) and one storage server (mogstored). In a production environment, we would ideally want several of each on different machines for redundancy. Keep in mind that we are describing a simple development setup.

mogilefsd

For dreamhacks, your configuration file is in ~/mogilefs/conf/mogilefsd.conf and just needs updating for corrected information. For other installations, the tracker can be configured with /etc/mogilefs/mogilefsd.conf as follows:

$ cat > /etc/mogilefs/mogilefsd.conf <<EOF
daemonize = 1
db_dsn = DBI:mysql:mogilefs
db_user = mogile
db_pass = mogilepw
listen = 127.0.0.1:7001
conf_port = 7001
default_mindevcount = 1
EOF

To start the tracker on a custom installation:

$ /usr/bin/mogilefsd

Or on a dreamhack, specify the local mogilefs directory:

$ ~/mogilefs/mogilefsd --config=${HOME}/mogilefs/conf/mogilefsd.conf

mogstored

The storage server can be configured as follows, for non-dreamhacks:

$ cat > /etc/mogilefs/mogstored.conf <<EOF
daemonize = 1
httplisten = 127.0.0.1:7500
mgmtlisten = 127.0.0.1:7501
docroot = /var/mogdata
EOF

$ mkdir /var/mogdata

Then start the storage server:

$ /usr/bin/mogstored

Users of dreamhacks should instead edit ~/mogilefs/conf/mogstored.conf, which specifies ~/.mogdata as the data store. This directory will NOT be created automatically; you must do it yourself:

$ mkdir ~/.mogdata

Then start the storage server:

$ ~/mogilefs/mogstored --config=${HOME}/mogilefs/conf/mogstored.conf --daemon

mogadm and mogtool

Next you need to pass some configuration info to the tracker with mogadm, but mogadm needs a config file too:

$ cat > /etc/mogilefs/mogilefs.conf <<EOF
trackers = 127.0.0.1:7001
EOF

Similarly for dreamhacks, except the correct file to use is ~/.mogilefs.conf and you will need to make sure the port is the same as in mogilefsd.conf.

Once mogadm knows where to find the trackers, you will need to use it to add host and device information to your mogile storage. The port number to use on the first line below in this case is the same as is listed for "httplisten" in mogstored.conf. You can use any name you like (the example below uses localhost) as long as you use the same name in both places. The mogadm utility will be either in /usr/bin/mogadm or ~/mogilefs/mogadm depending on your installation.

$ mogadm host add localhost --ip=127.0.0.1 --port=7500 --status=alive
$ mogadm device add localhost 1
$ mkdir /var/mogdata/dev1

Don't forget the last step as, again, the system will not automatically create the directory if it doesn't exist. On a dreamhack, this would be ~/.mogdata/dev1.

Once these steps are complete, you can use these mogadm commands to make sure things look healthy:

$ mogadm device list
$ mogadm check

Create a MogileFS domain and storage classes to be used by Dreamwidth:

$ mogadm domain add dreamwidth
$ mogadm class add dreamwidth temp --mindevcount=1
$ mogadm class add dreamwidth userpics --mindevcount=1
$ mogadm class add dreamwidth media --mindevcount=1
$ mogadm class add dreamwidth vgifts --mindevcount=1

NOTE: if you have defined your classes in %LJ::MOGILEFS_CONFIG as shown below, you can run update-db.pl to create the classes automatically instead of doing it manually with mogadm.

Now you should have a working MogileFS installation that you can test with mogtool:

$ cat > /etc/mogilefs/mogtool.conf <<EOF
trackers = 127.0.0.1:7001
domain = dreamwidth
class = temp
EOF

(On a dreamhack, you would use ~/.mogtool instead of /etc/mogilefs/mogtool.conf)

These sample commands test adding and deleting data:

$ mogtool inject /etc/hosts hosts
$ mogtool extract hosts -
$ mogtool delete hosts

Set up Dreamwidth installation to work with MogileFS

Add to $LJHOME/ext/local/etc/config-local.pl:

   %MOGILEFS_CONFIG = (
       hosts => [ '127.0.0.1:7001', ],
       domain => 'dreamwidth',
       classes => { 'userpics' => 1, 'media' => 1, 'vgifts' => 1, 'temp' => 1 },
   );
   $USERPIC_MOGILEFS = 1;

Restart Apache, and try uploading an icon!

Once your server has been configured to use MogileFS for storage, you will likely see errors (especially in userpics) if you run Apache without starting the MogileFS services. Remember to always run BOTH mogstored and mogilefsd.

If you want to use MogileFS with the automated test suite, you will also need to add %MOGILEFS_CONFIG to your config-test.pl. If you want, you can repeat the "mogadm domain" and "mogadm class" steps above with a different domain name in order to create a different MogileFS domain for use with the test suite.