Difference between revisions of "MogileFS setup"

From Dreamwidth Notes
Jump to: navigation, search
(Configuring MogileFS for a Dreamwidth development installation)
(Setup for a shared development environment (Dreamhacks): s/sophie/mark/ where relevant)
 
(20 intermediate revisions by 7 users not shown)
Line 1: Line 1:
Note:  This page is still a draft. Much of this information was pulled from [http://mogilefs.pbwiki.com/HowTo the MogileFS Wiki] combined with the INSTALL.txt instructions from the mogilefs source.
+
'''Note:  This page is deprecated. Dreamwidth now uses [[BlobStore]] for stuff. [http://dw-dev.dreamwidth.org/195174.html No new code that requires MogileFS will be accepted as of 2017].
 +
'''
 +
 
 +
When it wasn't deprecated, it was still a draft. Much of this information was pulled from [http://mogilefs.pbwiki.com/HowTo the MogileFS Wiki] combined with the INSTALL.txt instructions from the mogilefs source.
  
 
== Configuring MogileFS for a Dreamwidth development installation ==
 
== Configuring MogileFS for a Dreamwidth development installation ==
Line 5: Line 8:
 
These are instructions for getting a simple MogileFS installation working on a single instance (likely development/testing) Dreamwidth installation.  This is ''not'' a full setup document for a production, multi-server environment.
 
These are instructions for getting a simple MogileFS installation working on a single instance (likely development/testing) Dreamwidth installation.  This is ''not'' a full setup document for a production, multi-server environment.
  
=== Basic Steps ===
+
=== Overview ===
  
* Install prerequisites
+
* Install the MogileFS server and prepare the database
* Build and install MogileFS Debian packages
+
* Configure and run MogileFS services
* Create MogileFS Database
+
* Configure MogileFS servers
+
 
* Set up Dreamwidth installation to work with MogileFS
 
* Set up Dreamwidth installation to work with MogileFS
  
=== Install Prerequisites ===
+
== Install the MogileFS server and prepare the database ==
  
On your Dreamwidth machine:
+
=== Setup for a typical solo development environment (Debian) ===
  
$ sudo apt-get install debhelper dpkg-dev fakeroot
+
==== Build and install MogileFS Debian packages ====
  
Install prereqs for perlbal:
+
Follow the [https://code.google.com/p/mogilefs/wiki/InstallOnUbuntu installation instructions on the MogileFS wiki].
  
$ sudo apt-get install libio-aio-perl libdanga-socket-perl libnet-netmask-perl
+
==== Create MogileFS database ====
  
Install libperlbal
+
You should already have MySQL installed for your Dreamwidth installation.  So log in as root there and create the mogilefs user:
  
  $ cd $LJHOME/cvs/perlbal
+
  $ mysql -u root -p
$ dpkg-buildpackage -rfakeroot
+
  mysql> create database mogilefs;
  $ sudo dpkg -i ../libperlbal-perl_1.70-1_all.deb
+
mysql> grant all on mogilefs.* to 'mogile'@'%' identified by 'mogilepw';
 +
mysql> flush privileges;
 +
mysql> quit
  
You should have all of the other prerequisites installed as part of Dreamwidth.
+
Then go ahead and create the schema using mogdbsetup
  
=== Build and Install MogileFS Debian packages ===
+
$ /usr/bin/mogdbsetup --yes --dbname=mogilefs --dbuser=mogile --dbpassword=mogilepw
  
You should already have the source available from the scratch install.  So
+
=== Setup for a shared development environment (Dreamhacks) ===
  
  $ cd $LJHOME/cvs/mogilefs
+
If you are using a dreamhack, the client libraries are already installed, so
 +
you only need to download and configure the MogileFS server on your account.
  
Now you have to do a two-phase build and install:
+
[https://github.com/mogilefs/ The canonical server code repositories are hosted on Github].  There is a [https://github.com/kareila/mogilefs-for-dreamhacks simplified and combined copy of the repository available]
 +
from Kareila on Github.
  
  $ bin/build-all-debian.sh
+
The remainder of this document assumes that you have checked out that repository
  $ sudo dpkg -i ./api/perl/libmogilefs-perl_1.00-1_all.deb
+
to your dreamhack home directory, like so:
  $ bin/build-all-debian.sh
+
  $ sudo dpkg -i packages/mogilefsd_1.00-2_all.deb packages/mogilefs-utils_0.01-1_all.deb packages/mogstored_1.00-2_all.deb
+
  
=== Create MogileFS database ===
+
$ git clone git@github.com:kareila/mogilefs-for-dreamhacks.git ~/mogilefs
  
You should already have MySQL installed for your Dreamwidth installation.  So log in as root there and create the mogilefs user:
+
Once it is checked out, edit the two configuration files in the mogilefs/conf directory according to the instructions in comments at the top of each file.  You will need to contact Mark to get permission to use three additional ports on the dreamhack machine before proceeding with the configuration.
  
  $ mysql -u root -p
+
In order for the MogileFS scripts to be able to find their libraries, you will need to update your PERL5LIB environment variable. Here's what I have added to my .bash_profile:
mysql> create database mogilefs;
+
mysql> grant all on mogilefs.* to 'mogile'@'%' identified by 'mogilepw';
+
mysql> flush privileges;
+
mysql> quit
+
  
Then go ahead and create the schema using mogdbsetup
+
$ export PERL5LIB=${HOME}/mogilefs/lib:${LJHOME}/cgi-bin
  
$ /usr/bin/mogdbsetup --yes --dbname=mogilefs --dbuser=mogile --dbpassword=mogilepw
+
That tells Perl where to find both the MogileFS modules and the DW modules.
  
=== Configure MogileFS servers ===
+
Once the scripts are functional, you will also need to run mogdbsetup as above, but without creating a separate database first.  So the command you need to execute will look something like this:
  
You will need to set up at least one tracker (mogilefsd) and one storage server (mogstored), but ideally several of each on different machines for redundancy. Keep in mind this is a simple development setup.
+
$ ~/mogilefs/mogdbsetup --dbhost=localhost --dbport=xxxxx \
 +
--dbname=dreamhack_username --dbuser=dh_username --dbpassword=xxxxx \
 +
--type=MySQL
  
The tracker can be configured with /etc/mogilefs/mogilefsd.conf as follows:
+
(The backslashes (\) indicate one long continued line, broken up for readability.)
 +
 
 +
Again, you need to use the proper username, password and ports for your particular installation.  You will receive your port designations from Mark and your database password is in dw/ext/local/etc/config-private.pl.
 +
 
 +
Before we move on, one last thing I had to do was install the Net::Netmask module into mogilefs/lib, because it was missing from the preinstalled libraries.
 +
 
 +
$ cpanm -L ~/mogilefs Net::Netmask
 +
 
 +
Without this module, mogstored will not run.
 +
 
 +
== Configure and run MogileFS services ==
 +
 
 +
{{Note|text=This section is not required for non-dreamhacks if you've run dpkg-reconfigure as in the link above.}}
 +
 
 +
You will need to set up at least one tracker (mogilefsd) and one storage server (mogstored).  In a production environment, we would ideally want several of each on different machines for redundancy. Keep in mind that we are describing a simple development setup.
 +
 
 +
=== mogilefsd ===
 +
 
 +
For dreamhacks, your configuration file is in ~/mogilefs/conf/mogilefsd.conf and just needs updating for corrected information.  For other installations, the tracker can be configured with /etc/mogilefs/mogilefsd.conf as follows:
  
 
  $ cat > /etc/mogilefs/mogilefsd.conf <<EOF
 
  $ cat > /etc/mogilefs/mogilefsd.conf <<EOF
Line 74: Line 93:
 
  EOF
 
  EOF
  
Then start the tracker:
+
To start the tracker on a custom installation:
  
 
  $ /usr/bin/mogilefsd
 
  $ /usr/bin/mogilefsd
 +
 +
Or on a dreamhack, specify the local mogilefs directory:
  
The storage server can be configured with /etc/mogilefs/mogstored.conf as follows:
+
$ ~/mogilefs/mogilefsd --config=${HOME}/mogilefs/conf/mogilefsd.conf
 +
 +
=== mogstored ===
 +
 
 +
The storage server can be configured as follows, for non-dreamhacks:
  
 
  $ cat > /etc/mogilefs/mogstored.conf <<EOF
 
  $ cat > /etc/mogilefs/mogstored.conf <<EOF
Line 92: Line 117:
  
 
  $ /usr/bin/mogstored
 
  $ /usr/bin/mogstored
 +
 +
Users of dreamhacks should instead edit ~/mogilefs/conf/mogstored.conf, which specifies ~/.mogdata as the data store.  This directory will NOT be created automatically; you must do it yourself:
 +
 +
$ mkdir ~/.mogdata
 +
 +
Then start the storage server:
 +
 +
$ ~/mogilefs/mogstored --config=${HOME}/mogilefs/conf/mogstored.conf --daemon
 +
 +
=== mogadm and mogtool ===
  
 
Next you need to pass some configuration info to the tracker with mogadm, but mogadm needs a config file too:
 
Next you need to pass some configuration info to the tracker with mogadm, but mogadm needs a config file too:
Line 99: Line 134:
 
  EOF
 
  EOF
  
  $ /usr/bin/mogadm host add localhost --ip=127.0.0.1 --port=7500 --status=alive
+
Similarly for dreamhacks, except the correct file to use is ~/.mogilefs.conf and you will need to make sure the port is the same as in mogilefsd.conf.
  $ /usr/bin/mogadm device add localhost 1
+
 
 +
Once mogadm knows where to find the trackers, you will need to use it to add host and device information to your mogile storage. The port number to use on the first line below in this case is the same as is listed for "httplisten" in mogstored.conf.  You can use any name you like (the example below uses localhost) as long as you use the same name in both places.  The mogadm utility will be either in /usr/bin/mogadm or ~/mogilefs/mogadm depending on your installation.
 +
 
 +
$ mogadm host add localhost --ip=127.0.0.1 --port=7500 --status=alive
 +
  $ mogadm device add localhost 1
 
  $ mkdir /var/mogdata/dev1
 
  $ mkdir /var/mogdata/dev1
 +
 +
Don't forget the last step as, again, the system will not automatically create the directory if it doesn't exist.  On a dreamhack, this would be ~/.mogdata/dev1.
 +
 +
Once these steps are complete, you can use these mogadm commands to make sure things look healthy:
 +
 +
$ mogadm device list
 +
$ mogadm check
  
 
Create a MogileFS domain and storage classes to be used by Dreamwidth:
 
Create a MogileFS domain and storage classes to be used by Dreamwidth:
  
  $ /usr/bin/mogadm domain add dreamwidth
+
  $ mogadm domain add dreamwidth
  $ /usr/bin/mogadm class add dreamwidth temp --mindevcount=1
+
  $ mogadm class add dreamwidth temp --mindevcount=1
  $ /usr/bin/mogadm class add dreamwidth captcha --mindevcount=1
+
  $ mogadm class add dreamwidth userpics --mindevcount=1
  $ /usr/bin/mogadm class add dreamwidth userpics --mindevcount=1
+
  $ mogadm class add dreamwidth media --mindevcount=1
 +
$ mogadm class add dreamwidth vgifts --mindevcount=1
 +
 
 +
NOTE: if you have defined your classes in %LJ::MOGILEFS_CONFIG as shown below, you can run update-db.pl to create the classes automatically instead of doing it manually with mogadm.
  
Now you have a working MogileFS installation that you can test with mogtool:
+
Now you should have a working MogileFS installation that you can test with mogtool:
  
 
  $ cat > /etc/mogilefs/mogtool.conf <<EOF
 
  $ cat > /etc/mogilefs/mogtool.conf <<EOF
Line 116: Line 165:
 
  domain = dreamwidth
 
  domain = dreamwidth
 
  class = temp
 
  class = temp
 +
EOF
 +
 +
(On a dreamhack, you would use ~/.mogtool instead of /etc/mogilefs/mogtool.conf)
  
$ /usr/bin/mogtool inject /etc/hosts hosts
+
These sample commands test adding and deleting data:
$ /usr/bin/mogtool extract hosts -
+
$ /usr/bin/mogtool delete hosts
+
  
=== Set up Dreamwidth installation to work with MogileFS ===
+
$ mogtool inject /etc/hosts hosts
 +
$ mogtool extract hosts -
 +
$ mogtool delete hosts
  
Add to $LJHOME/etc/config-local.pl:
+
== Set up Dreamwidth installation to work with MogileFS ==
 +
 
 +
Add to $LJHOME/ext/local/etc/config-local.pl:
  
 
     %MOGILEFS_CONFIG = (
 
     %MOGILEFS_CONFIG = (
 
         hosts => [ '127.0.0.1:7001', ],
 
         hosts => [ '127.0.0.1:7001', ],
 
         domain => 'dreamwidth',
 
         domain => 'dreamwidth',
         classes => { 'temp' => 1, 'captcha' => 2, 'userpics' => 3, },
+
         classes => { 'userpics' => 1, 'media' => 1, 'vgifts' => 1, 'temp' => 1 },
 
     );
 
     );
 
     $USERPIC_MOGILEFS = 1;
 
     $USERPIC_MOGILEFS = 1;
    $CAPTCHA_MOGILEFS = 1;
 
  
Restart Apache, and try uploading an icon via editpics.bml
+
Restart Apache, and try uploading an icon!
 +
 
 +
Once your server has been configured to use MogileFS for storage, you will likely see errors (especially in userpics) if you run Apache without starting the MogileFS services.  Remember to always run BOTH mogstored and mogilefsd.
 +
 
 +
If you want to use MogileFS with the [[Dev Testing|automated test suite]], you will also need to add %MOGILEFS_CONFIG to your config-test.pl. If you want, you can repeat the "mogadm domain" and "mogadm class" steps above with a different domain name in order to create a different MogileFS domain for use with the test suite.
  
 
[[Category: Dreamwidth Installation]]
 
[[Category: Dreamwidth Installation]]
 +
[[Category:Dreamhack]]

Latest revision as of 02:19, 11 December 2018

Note: This page is deprecated. Dreamwidth now uses BlobStore for stuff. No new code that requires MogileFS will be accepted as of 2017.

When it wasn't deprecated, it was still a draft. Much of this information was pulled from the MogileFS Wiki combined with the INSTALL.txt instructions from the mogilefs source.

Configuring MogileFS for a Dreamwidth development installation

These are instructions for getting a simple MogileFS installation working on a single instance (likely development/testing) Dreamwidth installation. This is not a full setup document for a production, multi-server environment.

Overview

  • Install the MogileFS server and prepare the database
  • Configure and run MogileFS services
  • Set up Dreamwidth installation to work with MogileFS

Install the MogileFS server and prepare the database

Setup for a typical solo development environment (Debian)

Build and install MogileFS Debian packages

Follow the installation instructions on the MogileFS wiki.

Create MogileFS database

You should already have MySQL installed for your Dreamwidth installation. So log in as root there and create the mogilefs user:

$ mysql -u root -p
mysql> create database mogilefs;
mysql> grant all on mogilefs.* to 'mogile'@'%' identified by 'mogilepw';
mysql> flush privileges;
mysql> quit

Then go ahead and create the schema using mogdbsetup

$ /usr/bin/mogdbsetup --yes --dbname=mogilefs --dbuser=mogile --dbpassword=mogilepw

Setup for a shared development environment (Dreamhacks)

If you are using a dreamhack, the client libraries are already installed, so you only need to download and configure the MogileFS server on your account.

The canonical server code repositories are hosted on Github. There is a simplified and combined copy of the repository available from Kareila on Github.

The remainder of this document assumes that you have checked out that repository to your dreamhack home directory, like so:

$ git clone git@github.com:kareila/mogilefs-for-dreamhacks.git ~/mogilefs

Once it is checked out, edit the two configuration files in the mogilefs/conf directory according to the instructions in comments at the top of each file. You will need to contact Mark to get permission to use three additional ports on the dreamhack machine before proceeding with the configuration.

In order for the MogileFS scripts to be able to find their libraries, you will need to update your PERL5LIB environment variable. Here's what I have added to my .bash_profile:

$ export PERL5LIB=${HOME}/mogilefs/lib:${LJHOME}/cgi-bin

That tells Perl where to find both the MogileFS modules and the DW modules.

Once the scripts are functional, you will also need to run mogdbsetup as above, but without creating a separate database first. So the command you need to execute will look something like this:

$ ~/mogilefs/mogdbsetup --dbhost=localhost --dbport=xxxxx \
--dbname=dreamhack_username --dbuser=dh_username --dbpassword=xxxxx \
--type=MySQL

(The backslashes (\) indicate one long continued line, broken up for readability.)

Again, you need to use the proper username, password and ports for your particular installation. You will receive your port designations from Mark and your database password is in dw/ext/local/etc/config-private.pl.

Before we move on, one last thing I had to do was install the Net::Netmask module into mogilefs/lib, because it was missing from the preinstalled libraries.

$ cpanm -L ~/mogilefs Net::Netmask

Without this module, mogstored will not run.

Configure and run MogileFS services

Note: This section is not required for non-dreamhacks if you've run dpkg-reconfigure as in the link above.

You will need to set up at least one tracker (mogilefsd) and one storage server (mogstored). In a production environment, we would ideally want several of each on different machines for redundancy. Keep in mind that we are describing a simple development setup.

mogilefsd

For dreamhacks, your configuration file is in ~/mogilefs/conf/mogilefsd.conf and just needs updating for corrected information. For other installations, the tracker can be configured with /etc/mogilefs/mogilefsd.conf as follows:

$ cat > /etc/mogilefs/mogilefsd.conf <<EOF
daemonize = 1
db_dsn = DBI:mysql:mogilefs
db_user = mogile
db_pass = mogilepw
listen = 127.0.0.1:7001
conf_port = 7001
default_mindevcount = 1
EOF

To start the tracker on a custom installation:

$ /usr/bin/mogilefsd

Or on a dreamhack, specify the local mogilefs directory:

$ ~/mogilefs/mogilefsd --config=${HOME}/mogilefs/conf/mogilefsd.conf

mogstored

The storage server can be configured as follows, for non-dreamhacks:

$ cat > /etc/mogilefs/mogstored.conf <<EOF
daemonize = 1
httplisten = 127.0.0.1:7500
mgmtlisten = 127.0.0.1:7501
docroot = /var/mogdata
EOF
$ mkdir /var/mogdata

Then start the storage server:

$ /usr/bin/mogstored

Users of dreamhacks should instead edit ~/mogilefs/conf/mogstored.conf, which specifies ~/.mogdata as the data store. This directory will NOT be created automatically; you must do it yourself:

$ mkdir ~/.mogdata

Then start the storage server:

$ ~/mogilefs/mogstored --config=${HOME}/mogilefs/conf/mogstored.conf --daemon

mogadm and mogtool

Next you need to pass some configuration info to the tracker with mogadm, but mogadm needs a config file too:

$ cat > /etc/mogilefs/mogilefs.conf <<EOF
trackers = 127.0.0.1:7001
EOF

Similarly for dreamhacks, except the correct file to use is ~/.mogilefs.conf and you will need to make sure the port is the same as in mogilefsd.conf.

Once mogadm knows where to find the trackers, you will need to use it to add host and device information to your mogile storage. The port number to use on the first line below in this case is the same as is listed for "httplisten" in mogstored.conf. You can use any name you like (the example below uses localhost) as long as you use the same name in both places. The mogadm utility will be either in /usr/bin/mogadm or ~/mogilefs/mogadm depending on your installation.

$ mogadm host add localhost --ip=127.0.0.1 --port=7500 --status=alive
$ mogadm device add localhost 1
$ mkdir /var/mogdata/dev1

Don't forget the last step as, again, the system will not automatically create the directory if it doesn't exist. On a dreamhack, this would be ~/.mogdata/dev1.

Once these steps are complete, you can use these mogadm commands to make sure things look healthy:

$ mogadm device list
$ mogadm check

Create a MogileFS domain and storage classes to be used by Dreamwidth:

$ mogadm domain add dreamwidth
$ mogadm class add dreamwidth temp --mindevcount=1
$ mogadm class add dreamwidth userpics --mindevcount=1
$ mogadm class add dreamwidth media --mindevcount=1
$ mogadm class add dreamwidth vgifts --mindevcount=1

NOTE: if you have defined your classes in %LJ::MOGILEFS_CONFIG as shown below, you can run update-db.pl to create the classes automatically instead of doing it manually with mogadm.

Now you should have a working MogileFS installation that you can test with mogtool:

$ cat > /etc/mogilefs/mogtool.conf <<EOF
trackers = 127.0.0.1:7001
domain = dreamwidth
class = temp
EOF

(On a dreamhack, you would use ~/.mogtool instead of /etc/mogilefs/mogtool.conf)

These sample commands test adding and deleting data:

$ mogtool inject /etc/hosts hosts
$ mogtool extract hosts -
$ mogtool delete hosts

Set up Dreamwidth installation to work with MogileFS

Add to $LJHOME/ext/local/etc/config-local.pl:

   %MOGILEFS_CONFIG = (
       hosts => [ '127.0.0.1:7001', ],
       domain => 'dreamwidth',
       classes => { 'userpics' => 1, 'media' => 1, 'vgifts' => 1, 'temp' => 1 },
   );
   $USERPIC_MOGILEFS = 1;

Restart Apache, and try uploading an icon!

Once your server has been configured to use MogileFS for storage, you will likely see errors (especially in userpics) if you run Apache without starting the MogileFS services. Remember to always run BOTH mogstored and mogilefsd.

If you want to use MogileFS with the automated test suite, you will also need to add %MOGILEFS_CONFIG to your config-test.pl. If you want, you can repeat the "mogadm domain" and "mogadm class" steps above with a different domain name in order to create a different MogileFS domain for use with the test suite.