Difference between revisions of "Workers"

From Dreamwidth Notes
Jump to: navigation, search
(Using worker-manager and workers.conf: unwanted workers)
(Cleaning the table up a bit.)
Line 32: Line 32:
  
 
== List of workers ==
 
== List of workers ==
 +
 +
Below is a list of all the workers in the <code>$LJHOME/bin/worker</code> directory.
  
 
Note: I probably got some of that wrong. Any corrections welcome, especially where I say "not sure".
 
Note: I probably got some of that wrong. Any corrections welcome, especially where I say "not sure".
Line 39: Line 41:
 
! Filename !! Type !! Category !! Function and notes
 
! Filename !! Type !! Category !! Function and notes
 
|-
 
|-
! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/birthday-notify worker/birthday-notify]
+
! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/birthday-notify birthday-notify]
 
| Manual || [[ESN]] || Fire LJ::Event::Birthday [[ESN#Events|ESN events]] for users whose birthdays are soon
 
| Manual || [[ESN]] || Fire LJ::Event::Birthday [[ESN#Events|ESN events]] for users whose birthdays are soon
 
|-
 
|-
! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/content-importer bin/worker/content-importer]
+
! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/content-importer content-importer]
 
| [[TheSchwartz]] || [[Importer]] || Import all content
 
| [[TheSchwartz]] || [[Importer]] || Import all content
 
|-
 
|-
! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/content-importer-lite bin/worker/content-importer-lite]
+
! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/content-importer-lite content-importer-lite]
 
| [[TheSchwartz]] || [[Importer]] || Import some content (everything except entries and comments)
 
| [[TheSchwartz]] || [[Importer]] || Import some content (everything except entries and comments)
 
|-
 
|-
! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/content-importer-verify bin/worker/content-importer-verify]
+
! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/content-importer-verify content-importer-verify]
 
| [[TheSchwartz]] || [[Importer]] || Verify import password
 
| [[TheSchwartz]] || [[Importer]] || Verify import password
 
|-
 
|-
! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/directory-meta bin/worker/directory-meta]
+
! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/directory-meta directory-meta]
 
| Manual || Directory/user search || Update data used in searching for users
 
| Manual || Directory/user search || Update data used in searching for users
 
|-
 
|-
! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/distribute-invites bin/worker/distribute-invites]
+
! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/distribute-invites distribute-invites]
 
| [[TheSchwartz]] || [[Invite code generation#Multiple_user_grants|Invite codes]] || Email distributed invite codes
 
| [[TheSchwartz]] || [[Invite code generation#Multiple_user_grants|Invite codes]] || Email distributed invite codes
 
|-
 
|-
Line 81: Line 83:
 
| [[TheSchwartz]] || [[ESN]] || [[ESN#Firing_events_and_delivering_notifications|Notification delivery]] stage 4 (mass)
 
| [[TheSchwartz]] || [[ESN]] || [[ESN#Firing_events_and_delivering_notifications|Notification delivery]] stage 4 (mass)
 
|-
 
|-
! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/expunge-users bin/worker/expunge-users]
+
! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/expunge-users expunge-users]
 
| Manual || Account deletion || Expunge (permanently) accounts deleted more than 60 days ago
 
| Manual || Account deletion || Expunge (permanently) accounts deleted more than 60 days ago
 
|-
 
|-
! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/import-scheduler bin/worker/import-scheduler]
+
! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/import-scheduler import-scheduler]
 
| Manual || [[Importer]] || Queue importing jobs in the right order - oneshot as of 20110925, but see [http://bugs.dwscoalition.org/show_bug.cgi?id=1491 bug 1491]
 
| Manual || [[Importer]] || Queue importing jobs in the right order - oneshot as of 20110925, but see [http://bugs.dwscoalition.org/show_bug.cgi?id=1491 bug 1491]
 
|-
 
|-
! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/incoming-email bin/worker/incoming-email]
+
! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/incoming-email incoming-email]
 
| [[TheSchwartz]] || Incoming email || Handles email posting, email support requests, email support follow-up
 
| [[TheSchwartz]] || Incoming email || Handles email posting, email support requests, email support follow-up
 
|-
 
|-
! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/latest-feed bin/worker/latest-feed]
+
! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/latest-feed latest-feed]
 
| [[TheSchwartz]] || Latest things || Heavy lifting for the Latest Things page
 
| [[TheSchwartz]] || Latest things || Heavy lifting for the Latest Things page
 
|-
 
|-
! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/lazy-cleanup bin/worker/lazy-cleanup]
+
! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/lazy-cleanup lazy-cleanup]
 
| [[TheSchwartz]] || Journal management || Postpone some of the work of entry deletion
 
| [[TheSchwartz]] || Journal management || Postpone some of the work of entry deletion
 
|-
 
|-
! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/load-friends-gm bin/worker/load-friends-gm]
+
! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/load-friends-gm load-friends-gm]
 
| [[Gearman]] || Old (LJ) friends system || Not used anymore, see [http://bugs.dwscoalition.org/show_bug.cgi?id=3971 bug 3971]
 
| [[Gearman]] || Old (LJ) friends system || Not used anymore, see [http://bugs.dwscoalition.org/show_bug.cgi?id=3971 bug 3971]
 
|-
 
|-
! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/paidstatus bin/worker/paidstatus]
+
! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/paidstatus paidstatus]
 
| Manual || Payments || Process paid carts, expired paid accounts, and paid accounts expiring soon
 
| Manual || Payments || Process paid carts, expired paid accounts, and paid accounts expiring soon
 
|-
 
|-
! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/ping-hubbub bin/worker/ping-hubbub]
+
! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/ping-hubbub ping-hubbub]
 
| [[TheSchwartz]] || Outbound syndication || Notify PubSubHubbub of DW journal updates
 
| [[TheSchwartz]] || Outbound syndication || Notify PubSubHubbub of DW journal updates
 
|-
 
|-
Line 111: Line 113:
 
| [[TheSchwartz]] || [[ESN]] || [[ESN#Firing_events_and_delivering_notifications|Notification delivery]] stage 1-4 combined (mass)
 
| [[TheSchwartz]] || [[ESN]] || [[ESN#Firing_events_and_delivering_notifications|Notification delivery]] stage 1-4 combined (mass)
 
|-
 
|-
! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/process-eventlogrecord bin/worker/process-eventlogrecord]
+
! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/process-eventlogrecord process-eventlogrecord]
 
| [[TheSchwartz]] || Disabled || Not used currently, see [http://bugs.dwscoalition.org/show_bug.cgi?id=3963 bug 3963]
 
| [[TheSchwartz]] || Disabled || Not used currently, see [http://bugs.dwscoalition.org/show_bug.cgi?id=3963 bug 3963]
 
|-
 
|-
! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/process-privacy bin/worker/process-privacy]
+
! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/process-privacy process-privacy]
 
| [[TheSchwartz]] || Entry privacy || Process mass entry privacy changes
 
| [[TheSchwartz]] || Entry privacy || Process mass entry privacy changes
 
|-
 
|-
! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/resolve-extacct bin/worker/resolve-extacct]
+
! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/resolve-extacct resolve-extacct]
 
| [[Gearman]] || External accounts || Determine the account type (personal/community/syndicated)
 
| [[Gearman]] || External accounts || Determine the account type (personal/community/syndicated)
 
|-
 
|-
! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/schedule-synsuck bin/worker/schedule-synsuck]
+
! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/schedule-synsuck schedule-synsuck]
 
| Manual || Syndication || Queue TheSchwartz jobs for updating syndicated accounts
 
| Manual || Syndication || Queue TheSchwartz jobs for updating syndicated accounts
 
|-
 
|-
! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/search-constraints bin/worker/search-constraints]
+
! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/search-constraints search-constraints]
 
| [[Gearman]] || Directory/user search || Something to do with partial search results? Not sure
 
| [[Gearman]] || Directory/user search || Something to do with partial search results? Not sure
 
|-
 
|-
! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/search-lookup]
+
! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/search-lookup search-lookup]
 
| [[Gearman]] || Directory/user search || Initiate search? Not sure
 
| [[Gearman]] || Directory/user search || Initiate search? Not sure
 
|-
 
|-
! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/search-updater bin/worker/search-updater]
+
! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/search-updater search-updater]
 
| Manual || Directory/user search || Not sure what it does, still used but has some bitrot. See [http://bugs.dwscoalition.org/show_bug.cgi?id=3968 bug 3968]
 
| Manual || Directory/user search || Not sure what it does, still used but has some bitrot. See [http://bugs.dwscoalition.org/show_bug.cgi?id=3968 bug 3968]
 
|-
 
|-
! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/send-email bin/worker/send-email]
+
! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/send-email send-email]
 
| [[TheSchwartz]] || [[TheSchwartz#Sending_non-ESN_email_to_a_user|Sending email]] || Send email to users
 
| [[TheSchwartz]] || [[TheSchwartz#Sending_non-ESN_email_to_a_user|Sending email]] || Send email to users
 
|-
 
|-
! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/send-email-mass bin/worker/send-email-mass]
+
! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/send-email-mass send-email-mass]
 
| [[TheSchwartz]] || [[TheSchwartz#Sending_non-ESN_email_to_a_user|Sending email]] || Send email to users (mass)
 
| [[TheSchwartz]] || [[TheSchwartz#Sending_non-ESN_email_to_a_user|Sending email]] || Send email to users (mass)
 
|-
 
|-
Line 141: Line 143:
 
| [[Gearman]] || Payments || Charge a user's credit card
 
| [[Gearman]] || Payments || Charge a user's credit card
 
|-
 
|-
! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/sphinx-copier bin/worker/sphinx-copier]
+
! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/sphinx-copier sphinx-copier]
 
| [[TheSchwartz]] || Full-text search || Keep the full-text search database in sync with actual entries (see [http://bugs.dwscoalition.org/show_bug.cgi?id=3966 bug 3966])
 
| [[TheSchwartz]] || Full-text search || Keep the full-text search database in sync with actual entries (see [http://bugs.dwscoalition.org/show_bug.cgi?id=3966 bug 3966])
 
|-
 
|-
! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/sphinx-search-gm bin/worker/sphinx-search-gm]
+
! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/sphinx-search-gm sphinx-search-gm]
 
| [[Gearman]] || Full-text search || Interface to the core sphinx search  
 
| [[Gearman]] || Full-text search || Interface to the core sphinx search  
 
|-
 
|-
! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/stats-collection bin/worker/stats-collection]
+
! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/stats-collection stats-collection]
 
| Manual || Site stats || Collect/compile statistics
 
| Manual || Site stats || Collect/compile statistics
 
|-
 
|-
! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/subscribe-hubbub bin/worker/subscribe-hubbub]
+
! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/subscribe-hubbub subscribe-hubbub]
 
| Manual || Syndication || Subscribe to hubbub for syndicated accounts so DW is notified of updates to feed sources
 
| Manual || Syndication || Subscribe to hubbub for syndicated accounts so DW is notified of updates to feed sources
 
|-
 
|-
! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/support-notify bin/worker/support-notify]
+
! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/support-notify support-notify]
 
| [[TheSchwartz]] || Support board || Email users about support answers they received or support actions they subscribed to
 
| [[TheSchwartz]] || Support board || Email users about support answers they received or support actions they subscribed to
 
|-
 
|-
! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/synsuck bin/worker/synsuck]
+
! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/synsuck synsuck]
 
| [[TheSchwartz]] || Syndication || Updates syndicated account from feed source, started by [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/schedule-synsuck bin/worker/schedule-synsuck] or through the hubbub system
 
| [[TheSchwartz]] || Syndication || Updates syndicated account from feed source, started by [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/schedule-synsuck bin/worker/schedule-synsuck] or through the hubbub system
 
|-
 
|-
! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/sysban-gm bin/worker/sysban-gm]
+
! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/sysban-gm sysban-gm]
 
| [[Gearman]] || Site administration || Retrieve sysban entries by type
 
| [[Gearman]] || Site administration || Retrieve sysban entries by type
 
|-
 
|-
! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/t-memlimit bin/worker/t-memlimit]
+
! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/t-memlimit t-memlimit]
 
| Manual || Testing || Memory stress test
 
| Manual || Testing || Memory stress test
 
|-
 
|-
! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/taglib-gm bin/worker/taglib-gm]
+
! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/taglib-gm taglib-gm]
 
| [[Gearman]] || Journal content || Retrieve user's tags
 
| [[Gearman]] || Journal content || Retrieve user's tags
 
|-
 
|-
! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/talklib-gm bin/worker/talklib-gm]
+
! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/talklib-gm talklib-gm]
 
| [[Gearman]] || Journal content || Update entry comment count, needed... not sure  
 
| [[Gearman]] || Journal content || Update entry comment count, needed... not sure  
 
|-
 
|-
! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/userpic-resize-gm bin/worker/userpic-resize-gm]
+
! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/userpic-resize-gm userpic-resize-gm]
 
| [[Gearman]] || Userpics || Resize userpics larger than maximum height or width
 
| [[Gearman]] || Userpics || Resize userpics larger than maximum height or width
 
|-
 
|-
! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/xpost bin/worker/xpost]
+
! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/xpost xpost]
 
| [[TheSchwartz]] || Crossposting || Crossposts new or edited entries
 
| [[TheSchwartz]] || Crossposting || Crossposts new or edited entries
 
|}
 
|}
  
 
[[Category: Installation]]
 
[[Category: Installation]]

Revision as of 19:48, 1 January 2012

Workers are programs that don't run under control of Apache, because they handle tasks that don't need to happen synchronously to a user's request (whether triggered by one or not), or because running them under Apache would consume excessive resources.

Some workers process TheSchwartz or Gearman requests. Those normally run under control of worker-manager (see below). Some workers are manual, and depending on their purpose, can run either under control of worker-manager or be started by cron, or even just once when {re)starting the server. (Manual workers are a mixed lot, and there's no "one size fits all" option for them.)

Starting and stopping workers

Using worker-manager and workers.conf

bin/worker-manager takes care of starting, stopping, and restarting workers to maintain the numbers of workers set in etc/workers.conf for each host it runs on. The etc/workers.conf syntax is pretty straightforward and described in the file itself. To start bin/worker-manager for normal use (daemonized), use:

$LJHOME/bin/worker-manager

Or, if you want it to stay in the foreground and display progress/debug messages, use:

$LJHOME/bin/worker-manager --debug

In both cases, killing it will also kill the workers it started.

If you find that you do not want to use some workers (for testing, temporarily, or long-term), you may turn them off in the configuration. In /etc/workers.conf, setting the count to 0 will not start any of that worker. Commenting out the line on which that worker appears (with a #) will not start any of that worker. These are best for temporary changes. For a permanent change, you may also entirely remove the line on which that worker appears .

Individually

You can also start individual workers from the shell prompt, by typing:

$LJHOME/bin/worker/(worker-name)

Most workers accept the -v or --verbose flag to request extra debugging output and stay in the foreground when started from the shell prompt, so ^C should work to kill them.

Using cron or at server boot time

Please refer to your OS documentation. The crontab(1) and init(8) manpages are probably most relevant.

List of workers

Below is a list of all the workers in the $LJHOME/bin/worker directory.

Note: I probably got some of that wrong. Any corrections welcome, especially where I say "not sure".

Workers in alphabetical order of filename
Filename Type Category Function and notes
birthday-notify Manual ESN Fire LJ::Event::Birthday ESN events for users whose birthdays are soon
content-importer TheSchwartz Importer Import all content
content-importer-lite TheSchwartz Importer Import some content (everything except entries and comments)
content-importer-verify TheSchwartz Importer Verify import password
directory-meta Manual Directory/user search Update data used in searching for users
distribute-invites TheSchwartz Invite codes Email distributed invite codes
esn-cluster-subs TheSchwartz ESN Notification delivery stage 2
esn-cluster-subs-mass TheSchwartz ESN Notification delivery stage 2 (mass)
esn-filter-subs TheSchwartz ESN Notification delivery stage 3
esn-filter-subs-mass TheSchwartz ESN Notification delivery stage 3 (mass)
esn-fired-event TheSchwartz ESN Notification delivery stage 1
esn-fired-event-mass TheSchwartz ESN Notification delivery stage 1 (mass)
esn-process-sub TheSchwartz ESN Notification delivery stage 4
esn-process-sub-mass TheSchwartz ESN Notification delivery stage 4 (mass)
expunge-users Manual Account deletion Expunge (permanently) accounts deleted more than 60 days ago
import-scheduler Manual Importer Queue importing jobs in the right order - oneshot as of 20110925, but see bug 1491
incoming-email TheSchwartz Incoming email Handles email posting, email support requests, email support follow-up
latest-feed TheSchwartz Latest things Heavy lifting for the Latest Things page
lazy-cleanup TheSchwartz Journal management Postpone some of the work of entry deletion
load-friends-gm Gearman Old (LJ) friends system Not used anymore, see bug 3971
paidstatus Manual Payments Process paid carts, expired paid accounts, and paid accounts expiring soon
ping-hubbub TheSchwartz Outbound syndication Notify PubSubHubbub of DW journal updates
process-esn TheSchwartz ESN Notification delivery stages 1-4 combined
process-esn-mass TheSchwartz ESN Notification delivery stage 1-4 combined (mass)
process-eventlogrecord TheSchwartz Disabled Not used currently, see bug 3963
process-privacy TheSchwartz Entry privacy Process mass entry privacy changes
resolve-extacct Gearman External accounts Determine the account type (personal/community/syndicated)
schedule-synsuck Manual Syndication Queue TheSchwartz jobs for updating syndicated accounts
search-constraints Gearman Directory/user search Something to do with partial search results? Not sure
search-lookup Gearman Directory/user search Initiate search? Not sure
search-updater Manual Directory/user search Not sure what it does, still used but has some bitrot. See bug 3968
send-email TheSchwartz Sending email Send email to users
send-email-mass TheSchwartz Sending email Send email to users (mass)
shop-creditcard-charge Gearman Payments Charge a user's credit card
sphinx-copier TheSchwartz Full-text search Keep the full-text search database in sync with actual entries (see bug 3966)
sphinx-search-gm Gearman Full-text search Interface to the core sphinx search
stats-collection Manual Site stats Collect/compile statistics
subscribe-hubbub Manual Syndication Subscribe to hubbub for syndicated accounts so DW is notified of updates to feed sources
support-notify TheSchwartz Support board Email users about support answers they received or support actions they subscribed to
synsuck TheSchwartz Syndication Updates syndicated account from feed source, started by bin/worker/schedule-synsuck or through the hubbub system
sysban-gm Gearman Site administration Retrieve sysban entries by type
t-memlimit Manual Testing Memory stress test
taglib-gm Gearman Journal content Retrieve user's tags
talklib-gm Gearman Journal content Update entry comment count, needed... not sure
userpic-resize-gm Gearman Userpics Resize userpics larger than maximum height or width
xpost TheSchwartz Crossposting Crossposts new or edited entries