Difference between revisions of "Dev Maintenance"

From Dreamwidth Notes
Jump to: navigation, search
(Syncing code to live: Replacing instances of the "Deleting removed files" code with the snippet Dre made.)
(46 intermediate revisions by 11 users not shown)
Line 1: Line 1:
== Updating the Dreamwidth code ==
+
== Updating the Dreamwidth code on your Dreamhack ==
  
{{Warn|text='''VERY IMPORTANT:''' If you have custom changes that you have not committed or patched out, you want to do that first!  If you follow these steps without doing so, you will likely muck things up.}}
+
=== Shutting down apache ===
 
+
{{Warn|text=If you are using <tt>dw-nonfree</tt> code, see [[Keeping your site configs from wiping during updates]] to keep updates from wiping your site configs.}}
+
  
 
It is best to shut your Apache instance down before doing the update process, to make sure that everything, especially scripts in cgi-bin/ are reloaded properly:
 
It is best to shut your Apache instance down before doing the update process, to make sure that everything, especially scripts in cgi-bin/ are reloaded properly:
  
  stop-apache                     # for Dreamhacks
+
  stop-apache
sudo /etc/init.d/apache2 stop  # for people running their own
+
  
Okay.  Let's say you've been running your Dreamwidth install and you want to pull down the latest and greatest in fixes.  This is pretty easy.
+
=== Updating dw-free ===
  
{{Snippets/Updating_code|description=1}}
+
Okay.  Let's say you've been running your Dreamwidth install and you want to pull down the latest and greatest in fixes.  This is pretty easy.  First, if you are not somewhere in dw-free, get into that directory:
  
If you accidentally write over your config-local.pl file, and you're using a [[Dreamhacks|Dreamhack]], you will get a DB connect error when you try to run the above update-db.pl commands. In that case, you'll need to [[change config-private.pl to fix DBINFO|fix DBINFO in config-private.pl]] so that you can connect to the database.
+
cd $LJHOME
  
Now you can restart Apache:
+
Then grab updates from the development branch of Dreamwidth's repository:
  
  start-apache                    # for Dreamhacks
+
  git fetch dreamwidth
  sudo /etc/init.d/apache2 start  # for people running their own
+
  git checkout master && git pull --ff-only dreamwidth master:master
  
Of course, in a production environment, this whole process is not too recommended as you never know what kind of code you're going to get.  But for the most part, it's fairly straightforward.  (And if you're doing development, this is generally safe.)
+
You will also want to push the Dreamwidth repository changes to your fork on Github:
  
If your installation was before 2010-10-01, you will also want to [[Upgrading/Data Version 9|upgrade it to data version 9]].
+
git push origin master
  
=== Updating to the correct branch ===
+
If you have uncommitted changes, you'll get an error message and will need to use [[Git How To#How_to_stash_your_changes|git stash]].
  
Work is being done on the 'default' branch.  When you create a new copy of the repository, it automatically updates to the tip of the default branch.  This <i>should</i> be the correct tip.  You can also sync to the tip by using:
 
  
cd $LJHOME/cvs/dw-free
+
=== For those using DW non-free ===
hg update -C tip
+
  
{{Warn|text=The 'production' branch has been closed. All work is being done on the default branch. This should simplify things for everyone.}}
+
Repeat for dw-nonfree:
  
If for some reason the active branch is not pointing to 'default', then you can sync to 'default' manually by doing 'hg update -C default'. Be careful: the '-C' option will revert any modified files, and you may need to do another tip update after switching branches.
+
cd $LJHOME/ext/dw-nonfree
 +
 +
git fetch dreamwidth
 +
git checkout master && git pull --ff-only dreamwidth master:master
 +
git push origin master
  
=== Syncing code to live ===
+
=== Update your database ===
  
To be running production code, you also need to sync to your live code from the repository copy:
+
Now that your code has been updated, update the database:
  
  bin/cvsreport.pl -sync -cvsonly
+
  # order of commands is important
 +
$LJHOME/bin/upgrading/update-db.pl -r -p --innodb
 +
$LJHOME/bin/upgrading/update-db.pl -r --cluster=all --innodb
 +
$LJHOME/bin/upgrading/texttool.pl load
  
You will also want to delete files from your live code that have been deleted from the repository:
+
Remember that different branches might have different text strings you have added, and that you might need to update your database for different branches when you are working on them for those text strings to work.  The same applies for any database changes made by different branches you are working on.
  
{{Snippets/Deleting removed files}}
+
=== Compile the SCSS and other static files ===
  
== Updating your system ==
+
You'll also want to compile any changes to the static files. This will convert SCSS into CSS, as well as recompressing any changed javascript files, into the static build directory:
  
On Ubuntu, you can update the packages on your system with:
+
$LJHOME/bin/build-static.sh
 +
 
 +
=== Restart the server ===
 +
 
 +
Now you can restart Apache:
 +
 
 +
start-apache
 +
 
 +
Of course, in a production environment, this whole process is not too recommended as you never know what kind of code you're going to get.  But for the most part, it's fairly straightforward.  (And if you're doing development, this is generally safe.)
 +
 
 +
== Additional Information for Non-Dreamhack Users ==
 +
 
 +
This section is only for you if you're running your own installation.
 +
 
 +
Instead of using start-apache/stop-apache, which are Dreamhack-specific scripts, use these commands:
 +
 
 +
sudo /etc/init.d/apache2 start
 +
sudo /etc/init.d/apache2 stop
 +
 
 +
You'll also want to update packages on your system at some point. On Ubuntu, this would be done using:
  
 
  apt-get update
 
  apt-get update
Line 56: Line 77:
  
 
  sudo apt-get update && sudo apt-get upgrade
 
  sudo apt-get update && sudo apt-get upgrade
 
(Note, you may not need to do this step if you're on a Dreamhack and Sophie or Afuna has already done it for everyone.)
 
  
 
== Scripting ==
 
== Scripting ==
  
You can, of course, use scripts to make it easier for you to do most of this.
+
{{Warn|text="These scripts have not been thoroughly tested yet with the new system; they might not be updated entirely or right."}}
  
There is an "omnibus" script available at http://dw-dev.dreamwidth.org/94822.html which incorporates all the individual scripts listed here and some other functions, and includes help information.
+
You can, of course, use scripts to make it easier for you to do some of this. 
 +
 
 +
There is an epic "omnibus" script by <dwuser>jeshyr</dwuser> available at http://dw-dev.dreamwidth.org/94822.html which incorporates all the individual scripts listed here and some other useful functions, and includes help information. It'll save you a heap of angst as a beginner!
  
 
Some simpler example scripts are given below.
 
Some simpler example scripts are given below.
Line 72: Line 93:
  
 
<source lang="bash">#!/bin/bash
 
<source lang="bash">#!/bin/bash
hg -R $LJHOME/cvs/dw-free qpop -a && \
 
$LJHOME/bin/cvsreport.pl -u && \
 
hg -R $LJHOME/cvs/dw-free qpush -a</source>
 
  
Now, when you type 'dwu', this script will pop out any MQ patches you may have currently in dw-free, update the code to the latest available, and push your dw-free MQ patches back.
+
# make sure we are in the right directory;
 +
cd $LJHOME
  
If you need support for dw-nonfree too, just duplicate the lines mentioning dw-free and edit them to say dw-nonfree and make sure that all the lines apart from the first and last end with '&& \':
+
git fetch dreamwidth
  
<source lang="bash">#!/bin/bash
+
# pull changes from dreamwidth
hg -R $LJHOME/cvs/dw-free qpop -a && \
+
git checkout master
hg -R $LJHOME/cvs/dw-nonfree qpop -a && \
+
git pull --ff-only dreamwidth master
$LJHOME/bin/cvsreport.pl -u && \
+
hg -R $LJHOME/cvs/dw-free qpush -a && \
+
hg -R $LJHOME/cvs/dw-nonfree qpush -a</source>
+
  
=== dws - Syncing the live code ===
+
# push them to Github forks
 +
git push origin master
  
Put this code in a file called '''$LJHOME/bin/dws''' and make it executable with '''chmod ugo+x $LJHOME/bin/dws''':
+
# restore our old position
 +
git checkout -
  
<source lang="bash">#!/bin/bash
+
# check to see if we have a dw-nonfree directory; if we do, update that too
$LJHOME/bin/cvsreport.pl -s -c && \
+
if [ -d "$LJHOME/ext/dw-nonfree" ]; then
$LJHOME/bin/cvsreport.pl -s -c
+
  # change to dw-nonfree
cd $LJHOME; for i in `bin/cvsreport.pl -n -1`; do echo "Removing $i" && rm $i; done</source>
+
  cd $LJHOME/ext/dw-nonfree
 +
 
 +
  git fetch dreamwidth
 +
 
 +
  # pull changes from dw-nonfree
 +
  git checkout master
 +
  git pull --ff-only dreamwidth master
 +
 
 +
  # push them to Github forks
 +
  git push origin master
 +
 
 +
  # restore our old position
 +
  git checkout -
 +
fi
 +
</source>
  
This script will synchronise the live code twice when you type 'dws', and then remove any stale files that remain in the live code.
+
Now, when you type 'dwu', this script will update your dw-free and (if you have it) dw-nonfree repositories.
  
 
=== dwdb - Updating the database ===
 
=== dwdb - Updating the database ===
Line 103: Line 135:
  
 
<source lang="bash">#!/bin/bash
 
<source lang="bash">#!/bin/bash
$LJHOME/bin/upgrading/update-db.pl -r -p --innodb && \
+
$LJHOME/bin/upgrading/update-db.pl -r -p --innodb &&
$LJHOME/bin/upgrading/update-db.pl -r --cluster=all --innodb && \
+
$LJHOME/bin/upgrading/update-db.pl -r --cluster=all --innodb &&
 
$LJHOME/bin/upgrading/texttool.pl load</source>
 
$LJHOME/bin/upgrading/texttool.pl load</source>
  
 
This will update the database when you type 'dwdb'.
 
This will update the database when you type 'dwdb'.
  
=== tocvs - Switch between /cvs and live directories ===
+
=== comp - Recompiling the compass files ===
  
As of Jan 13, this is already in the default .bashrc in dreamhacks, so you shouldn't need to add it.
+
Put this code in a file called "'~/bin/comp'" and make it executable with "'chmod ugo+x ~/bin/comp'":
  
If you don't see the line "alias tocvs="source ~/bin/tocvs.real"" in your ~/.bashrc, you can add the following function yourself:
+
<source lang="bash">#!/bin/bash
 
+
cd "$LJHOME" &&
<source lang="bash">
+
compass compile &&
tocvs() {
+
cd ext/dw-nonfree &&
  DIR=$PWD
+
compass compile</source>
  REPO=${1-'dw-free'}
+
 
+
  if  [[ "$REPO" == "f" ]]; then REPO="dw-free";    fi
+
  if  [[ "$REPO" == "n" ]]; then REPO="dw-nonfree"; fi
+
 
+
  if [[ "$DIR" =~ ^$LJHOME ]]; then
+
    if [[ "$DIR" =~ ^$LJHOME/cvs/([^\/]+) ]]; then
+
      FROMREPO=${BASH_REMATCH[1]}
+
      if [[ "$1" == "" ]]; then
+
        DIR=${DIR/#$LJHOME\/cvs\/$FROMREPO/$LJHOME}
+
      else
+
        DIR=${DIR/#$LJHOME\/cvs\/$FROMREPO/$LJHOME\/cvs\/$REPO}
+
      fi
+
    else
+
      DIR=${DIR/#$LJHOME/$LJHOME\/cvs\/$REPO}
+
    fi
+
  fi
+
  cd $DIR
+
}
+
</source>
+
 
+
Then, start a new shell or log in again for the change to take effect. Now, whenever you're in a code directory, you can use 'tocvs' to switch back and forth from the live directory and the corresponding /cvs directory. It defaults to 'dw-free'; if you wish to go to another repository instead (for example, 'dw-nonfree'), give it as a parameter. (for example, 'tocvs dw-nonfree'). If you specify a repo name while you're already in a repository, it will switch to the named repository.
+
  
The repositories 'dw-free' and 'dw-nonfree' are aliased as 'f' and 'n' respectively.
+
This will recompile the CSS files.
  
 
==Cleaning up your directories==
 
==Cleaning up your directories==
Line 147: Line 157:
 
If you do any amount of work, you'll find that your directories get cluttered with .orig and .rej files everywhere. This script will clean those up:
 
If you do any amount of work, you'll find that your directories get cluttered with .orig and .rej files everywhere. This script will clean those up:
  
<source lang="bash">find -name *.orig -o -name *.rej | xargs rm</source>
+
<source lang="bash">git clean -f "*.rej" "*.orig"</source>
  
 
You can run it from the command line, or put it in a file called ~/bin/tidy and make it executable with "chmod +x ~/bin/tidy". If you do that, you'll be able to just type "tidy" to clean house.
 
You can run it from the command line, or put it in a file called ~/bin/tidy and make it executable with "chmod +x ~/bin/tidy". If you do that, you'll be able to just type "tidy" to clean house.
  
 
[[Category: Development]]
 
[[Category: Development]]
 +
[[Category: Dreamhack]]

Revision as of 17:03, 1 July 2019

Updating the Dreamwidth code on your Dreamhack

Shutting down apache

It is best to shut your Apache instance down before doing the update process, to make sure that everything, especially scripts in cgi-bin/ are reloaded properly:

stop-apache

Updating dw-free

Okay. Let's say you've been running your Dreamwidth install and you want to pull down the latest and greatest in fixes. This is pretty easy. First, if you are not somewhere in dw-free, get into that directory:

cd $LJHOME

Then grab updates from the development branch of Dreamwidth's repository:

git fetch dreamwidth
git checkout master && git pull --ff-only dreamwidth master:master

You will also want to push the Dreamwidth repository changes to your fork on Github:

git push origin master

If you have uncommitted changes, you'll get an error message and will need to use git stash.


For those using DW non-free

Repeat for dw-nonfree:

cd $LJHOME/ext/dw-nonfree

git fetch dreamwidth
git checkout master && git pull --ff-only dreamwidth master:master
git push origin master

Update your database

Now that your code has been updated, update the database:

# order of commands is important
$LJHOME/bin/upgrading/update-db.pl -r -p --innodb
$LJHOME/bin/upgrading/update-db.pl -r --cluster=all --innodb
$LJHOME/bin/upgrading/texttool.pl load

Remember that different branches might have different text strings you have added, and that you might need to update your database for different branches when you are working on them for those text strings to work. The same applies for any database changes made by different branches you are working on.

Compile the SCSS and other static files

You'll also want to compile any changes to the static files. This will convert SCSS into CSS, as well as recompressing any changed javascript files, into the static build directory:

$LJHOME/bin/build-static.sh

Restart the server

Now you can restart Apache:

start-apache

Of course, in a production environment, this whole process is not too recommended as you never know what kind of code you're going to get. But for the most part, it's fairly straightforward. (And if you're doing development, this is generally safe.)

Additional Information for Non-Dreamhack Users

This section is only for you if you're running your own installation.

Instead of using start-apache/stop-apache, which are Dreamhack-specific scripts, use these commands:

sudo /etc/init.d/apache2 start
sudo /etc/init.d/apache2 stop 

You'll also want to update packages on your system at some point. On Ubuntu, this would be done using:

apt-get update
apt-get upgrade

Or, if you want an easy command to run on your dw account in one swoop:

sudo apt-get update && sudo apt-get upgrade

Scripting

Warning: "These scripts have not been thoroughly tested yet with the new system; they might not be updated entirely or right."

You can, of course, use scripts to make it easier for you to do some of this.

There is an epic "omnibus" script by [info]jeshyr available at http://dw-dev.dreamwidth.org/94822.html which incorporates all the individual scripts listed here and some other useful functions, and includes help information. It'll save you a heap of angst as a beginner!

Some simpler example scripts are given below.

dwu - Updating the repos

Put this code in a file called ~/bin/dwu and make it executable with chmod ugo+x ~/bin/dwu:

#!/bin/bash
 
# make sure we are in the right directory;
cd $LJHOME
 
git fetch dreamwidth
 
# pull changes from dreamwidth
git checkout master
git pull --ff-only dreamwidth master
 
# push them to Github forks
git push origin master
 
# restore our old position
git checkout -
 
# check to see if we have a dw-nonfree directory; if we do, update that too
if [ -d "$LJHOME/ext/dw-nonfree" ]; then
  # change to dw-nonfree
  cd $LJHOME/ext/dw-nonfree
 
  git fetch dreamwidth
 
  # pull changes from dw-nonfree
  git checkout master
  git pull --ff-only dreamwidth master
 
  # push them to Github forks
  git push origin master
 
  # restore our old position
  git checkout -
fi

Now, when you type 'dwu', this script will update your dw-free and (if you have it) dw-nonfree repositories.

dwdb - Updating the database

Put this code in a file called ~/bin/dwdb and make it executable with chmod ugo+x ~/bin/dwdb:

#!/bin/bash
$LJHOME/bin/upgrading/update-db.pl -r -p --innodb &&
$LJHOME/bin/upgrading/update-db.pl -r --cluster=all --innodb &&
$LJHOME/bin/upgrading/texttool.pl load

This will update the database when you type 'dwdb'.

comp - Recompiling the compass files

Put this code in a file called "'~/bin/comp'" and make it executable with "'chmod ugo+x ~/bin/comp'":

#!/bin/bash
cd "$LJHOME" &&
compass compile &&
cd ext/dw-nonfree &&
compass compile

This will recompile the CSS files.

Cleaning up your directories

If you do any amount of work, you'll find that your directories get cluttered with .orig and .rej files everywhere. This script will clean those up:

git clean -f "*.rej" "*.orig"

You can run it from the command line, or put it in a file called ~/bin/tidy and make it executable with "chmod +x ~/bin/tidy". If you do that, you'll be able to just type "tidy" to clean house.