Difference between revisions of "Code push"

From Dreamwidth Notes
Jump to: navigation, search
(Created page with 'Category: Production Category:Dreamwidth.org The checklist for updating the code running on the live "dreamwidth.org" production site to the most recent version of the c…')
 
m (add note about updating angel configs)
 
(10 intermediate revisions by 2 users not shown)
Line 3: Line 3:
 
The checklist for updating the code running on the live "dreamwidth.org" production site to the most recent version of the code. Admins of other installations will most likely need to alter this checklist based on your site's unique needs.
 
The checklist for updating the code running on the live "dreamwidth.org" production site to the most recent version of the code. Admins of other installations will most likely need to alter this checklist based on your site's unique needs.
  
== Check for pushability==
+
== To do in advance of the push ==
  
Code pushes can be requested for business reasons or by members of the dev team, but the admin doing the push should take a few minutes to check over what will be pushed to make sure all patches are complete and in a releasable state. Before a push, check with Denise a few hours in advance to see if there are any patches in the review queue that should go into the release for business reasons.
+
=== Check for pushability===
  
== Update <dwcomm>dw_maintenance</dwcomm> in advance ==
+
Code pushes can be requested for business reasons or by members of the dev team, but the admin doing the push should take a few minutes to check over what will be pushed to make sure all patches are complete and in a releasable state. Before a push, check with Denise, Mark, or Kareila a few hours in advance to see if there are any pull requests in the review queue that should go into the release for business reasons.
  
Ideally, the comm should be updated a day or so in advance to let people know the timing of the push; at the very least it should be a few hours' notice. Give times in your local timezone and in GMT, and be sure to include the timezone information.
+
=== Announce in <dwcomm>dw_maintenance</dwcomm> that a code push is scheduled ===
  
== Update Twitter ==
+
Ideally, the comm should be updated a day or so in advance to let people know the timing of the push; at the very least it should be a few hours' notice. Give times in your local timezone and in GMT, and be sure to include the timezone information. Add a link to <a href="http://www.timeanddate.com/worldclock/fixedform.html">Time and Date's Event Time Announcer</a>.
 +
 
 +
== Preparing to push ==
 +
 
 +
=== Repository prep ===
 +
 
 +
For '''both''' dw-free and dw-nonfree:
 +
 
 +
* make sure previous release branch is merged into develop
 +
* make new release branches in both places, incrementing the previous release number
 +
* double check the new branches in Github, making sure each one looks identical to the develop branch for that repo
 +
 
 +
=== Production prep ===
 +
 
 +
Update to the new release branch on the admin box, being careful not to overwrite any live fixes in production that aren't checked in:
 +
 
 +
$ sudo su - dw
 +
$ cd $LJHOME # should put you in /home/dw/current
 +
$ git status # you'll see the live code
 +
$ git stash
 +
$ git fetch origin
 +
$ git checkout release-1.xx
 +
$ git stash pop # and handle any merge conflicts
 +
 
 +
Then repeat the same process in $LJHOME/ext/dw-nonfree (but there are almost never live code changes there).
 +
 
 +
Once this process is completed, git status in both $LJHOME and $LJHOME/ext/dw-nonfree should be clean (modulo the live changes) and should both be on the expected release branch - but not a bad idea to "git log" and verify the last commit is as expected.
 +
 
 +
=== Check for config changes ===
 +
 
 +
Do a diff against the previous release branch to see if any config changes were made, and reflect those changes in ext/local/etc/config.pl as needed.
 +
 
 +
If any workers were added or removed, update and run this script: <code>ext/local/etc/angel/regen-configs.pl</code> (we don’t use workers.conf in production).
 +
 
 +
=== Check for new module requirements ===
 +
 
 +
Run <code>bin/checkconfig.pl</code> and make sure we don’t require any additional modules.  If we do, make sure they get installed before proceeding.
 +
 
 +
=== Update Twitter ===
  
 
Update the @dreamwidth Twitter account when beginning the push to let people know the push is starting.
 
Update the @dreamwidth Twitter account when beginning the push to let people know the push is starting.
  
== Update dw_maintenance when beginning ==
+
=== Update <dwcomm>dw_maintenance</dwcomm> ===
 +
 
 +
When beginning the push, update <dwcomm>dw_maintenance</dwcomm> again to announce that it's starting. Include a downtime estimate, with an extra 20% or so added on top for unexpected problems.
 +
 
 +
=== Reload/recompile all the things ===
 +
 
 +
At this point you kind of have to begin to move quickly since this phase changes database stuff like styles, text, etc. - if you update those but don't push the code changes for a while, you can end up causing weird behavior on the site.
 +
 
 +
$ bin/build-static.sh  # this does compass compile, js minify
 +
$ bin/upgrading/texttool.pl load
 +
$ bin/upgrading/update-db.pl -r
 +
$ bin/upgrading/update-db.pl -r --cluster=all --drop
 +
$ bin/upgrading/update-db.pl -r -p
 +
 
 +
(The --drop is only needed if obsolete tables were removed in this update.)
 +
 
 +
If the proxy source code has changed, it will also need rebuilding:
 +
 
 +
$ cd src/proxy
 +
$ go build
 +
 
 +
=== Code Push + All The Restarts ===
 +
 
 +
If for some reason the site needs to be put into maintenance mode during the push, do that here:
 +
 
 +
$ sudo su -
 +
$ bin/maint-on
 +
 
 +
Then to do the actual code push:
 +
 
 +
$ sudo su -
 +
$ bin/sync-dw-home
 +
$ bin/restart-webs
 +
$ bin/restart-jobs  # to restart the workers
 +
 
 +
Once every thing is restarted, take things out of maintenance mode if needed:
 +
 
 +
$ bin/maint-off
  
When beginning the push, update dw_maintenance again to announce that it's starting. Include a downtime estimate, with an extra 20% or so added on top for unexpected problems.
+
It may also be necessary to restart the [[SSL Image Proxy]], but historically whenever that's changed it gets pushed separately.
  
 +
=== Spot-check the push ===
  
== Do technical magic ==
+
Inform the #dw-dev channel that the push has been completed and the site is back on. Ask them to spot-check a few functions by posting entries, reading their reading page, updating settings, etc. Watch the logs for any errors or problems (by running bin/tail-logs as root).
  
 +
==After the push is complete ==
  
== Spot-check the push ==
+
=== Update the dw_maintenance entry ===
  
Inform the #dw-work channel that the push has been completed and the site is back on. Ask them to spot-check a few functions by posting entries, reading their reading page, updating settings, etc. Watch the logs for any errors or problems.
+
Edit the <dwcomm>dw-maintenance</dwcomm> entry (don't post a new one) with the date/time (in both your local timezone, clearly marked, and in GMT) with a notice that the push has been completed. Let them know that if they spot any issues, they should contact Support or leave a comment.
  
== Increment repository tags ==
+
=== Update Twitter account ===
  
Increment the tags on any repository that has had commits to it since the last push. Use secondary versions (0.1.0 to 0.2.0, etc)
+
If needed, update the Twitter account again to let people know the push is complete, and that any problems should go to Support or <dwcomm>dw-maintenance</dwcomm>.
  
== Update the dw_maintenance entry ==
+
=== Highlight new journal themes ===
  
Edit the dw-maintenance entry (don't post a new one) with the date/time (in both your local timezone, clearly marked, and in GMT) with a notice that the push has been completed. Let them know that if they spot any issues, they should contact Support or leave a comment.
+
If there are any new journal themes in this push, use the /admin/themes interface to add them to the "New" category in the theme chooser.
  
== Monitor dw-maintenance entry for a few hours ==
+
=== Monitor <dwcomm>dw-maintenance</dwcomm> entry for a few hours ===
  
Stay available for a few hours as dw_maintenance comments come in, in case there's a major issue that needs to be resolved with another push. (You don't need to reply to the comments -- Denise will handle that.)
+
Make sure someone can stay available for a few hours as <dwcomm>dw_maintenance</dwcomm> comments come in, in case there's a major issue that needs to be resolved with another push. (You don't need to reply to the comments -- Denise or Kareila will handle that.)
  
== Increment repository tags again (if necessary) ==
+
=== Commit any needed fixes on the release branch ===
  
If there is a secondary push to resolve bugs, update the repo tags to a tertiary version (0.1.0 to 0.1.1) after a day or so, when all the issues are resolved. (The day after a code push should remain as a code freeze, for bugfixes with the previous push only.)
+
If there is a secondary push to resolve bugs, these commits should go directly on the release branch. The hotfixes will be merged back to the develop branch before the next push.

Latest revision as of 05:54, 26 February 2017


The checklist for updating the code running on the live "dreamwidth.org" production site to the most recent version of the code. Admins of other installations will most likely need to alter this checklist based on your site's unique needs.

To do in advance of the push

Check for pushability

Code pushes can be requested for business reasons or by members of the dev team, but the admin doing the push should take a few minutes to check over what will be pushed to make sure all patches are complete and in a releasable state. Before a push, check with Denise, Mark, or Kareila a few hours in advance to see if there are any pull requests in the review queue that should go into the release for business reasons.

Announce in [info]dw_maintenance that a code push is scheduled

Ideally, the comm should be updated a day or so in advance to let people know the timing of the push; at the very least it should be a few hours' notice. Give times in your local timezone and in GMT, and be sure to include the timezone information. Add a link to <a href="http://www.timeanddate.com/worldclock/fixedform.html">Time and Date's Event Time Announcer</a>.

Preparing to push

Repository prep

For both dw-free and dw-nonfree:

  • make sure previous release branch is merged into develop
  • make new release branches in both places, incrementing the previous release number
  • double check the new branches in Github, making sure each one looks identical to the develop branch for that repo

Production prep

Update to the new release branch on the admin box, being careful not to overwrite any live fixes in production that aren't checked in:

$ sudo su - dw
$ cd $LJHOME # should put you in /home/dw/current
$ git status # you'll see the live code
$ git stash
$ git fetch origin
$ git checkout release-1.xx
$ git stash pop # and handle any merge conflicts

Then repeat the same process in $LJHOME/ext/dw-nonfree (but there are almost never live code changes there).

Once this process is completed, git status in both $LJHOME and $LJHOME/ext/dw-nonfree should be clean (modulo the live changes) and should both be on the expected release branch - but not a bad idea to "git log" and verify the last commit is as expected.

Check for config changes

Do a diff against the previous release branch to see if any config changes were made, and reflect those changes in ext/local/etc/config.pl as needed.

If any workers were added or removed, update and run this script: ext/local/etc/angel/regen-configs.pl (we don’t use workers.conf in production).

Check for new module requirements

Run bin/checkconfig.pl and make sure we don’t require any additional modules. If we do, make sure they get installed before proceeding.

Update Twitter

Update the @dreamwidth Twitter account when beginning the push to let people know the push is starting.

Update [info]dw_maintenance

When beginning the push, update [info]dw_maintenance again to announce that it's starting. Include a downtime estimate, with an extra 20% or so added on top for unexpected problems.

Reload/recompile all the things

At this point you kind of have to begin to move quickly since this phase changes database stuff like styles, text, etc. - if you update those but don't push the code changes for a while, you can end up causing weird behavior on the site.

$ bin/build-static.sh  # this does compass compile, js minify
$ bin/upgrading/texttool.pl load
$ bin/upgrading/update-db.pl -r
$ bin/upgrading/update-db.pl -r --cluster=all --drop
$ bin/upgrading/update-db.pl -r -p

(The --drop is only needed if obsolete tables were removed in this update.)

If the proxy source code has changed, it will also need rebuilding:

$ cd src/proxy
$ go build

Code Push + All The Restarts

If for some reason the site needs to be put into maintenance mode during the push, do that here:

$ sudo su -
$ bin/maint-on

Then to do the actual code push:

$ sudo su -
$ bin/sync-dw-home
$ bin/restart-webs
$ bin/restart-jobs  # to restart the workers

Once every thing is restarted, take things out of maintenance mode if needed:

$ bin/maint-off

It may also be necessary to restart the SSL Image Proxy, but historically whenever that's changed it gets pushed separately.

Spot-check the push

Inform the #dw-dev channel that the push has been completed and the site is back on. Ask them to spot-check a few functions by posting entries, reading their reading page, updating settings, etc. Watch the logs for any errors or problems (by running bin/tail-logs as root).

After the push is complete

Update the dw_maintenance entry

Edit the [info]dw-maintenance entry (don't post a new one) with the date/time (in both your local timezone, clearly marked, and in GMT) with a notice that the push has been completed. Let them know that if they spot any issues, they should contact Support or leave a comment.

Update Twitter account

If needed, update the Twitter account again to let people know the push is complete, and that any problems should go to Support or [info]dw-maintenance.

Highlight new journal themes

If there are any new journal themes in this push, use the /admin/themes interface to add them to the "New" category in the theme chooser.

Monitor [info]dw-maintenance entry for a few hours

Make sure someone can stay available for a few hours as [info]dw_maintenance comments come in, in case there's a major issue that needs to be resolved with another push. (You don't need to reply to the comments -- Denise or Kareila will handle that.)

Commit any needed fixes on the release branch

If there is a secondary push to resolve bugs, these commits should go directly on the release branch. The hotfixes will be merged back to the develop branch before the next push.