Difference between revisions of "Workers"
(→Using worker-manager and workers.conf: unwanted workers) |
(Cleaning the table up a bit.) |
||
Line 32: | Line 32: | ||
== List of workers == | == List of workers == | ||
+ | |||
+ | Below is a list of all the workers in the <code>$LJHOME/bin/worker</code> directory. | ||
Note: I probably got some of that wrong. Any corrections welcome, especially where I say "not sure". | Note: I probably got some of that wrong. Any corrections welcome, especially where I say "not sure". | ||
Line 39: | Line 41: | ||
! Filename !! Type !! Category !! Function and notes | ! Filename !! Type !! Category !! Function and notes | ||
|- | |- | ||
− | ! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/birthday-notify | + | ! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/birthday-notify birthday-notify] |
| Manual || [[ESN]] || Fire LJ::Event::Birthday [[ESN#Events|ESN events]] for users whose birthdays are soon | | Manual || [[ESN]] || Fire LJ::Event::Birthday [[ESN#Events|ESN events]] for users whose birthdays are soon | ||
|- | |- | ||
− | ! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/content-importer | + | ! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/content-importer content-importer] |
| [[TheSchwartz]] || [[Importer]] || Import all content | | [[TheSchwartz]] || [[Importer]] || Import all content | ||
|- | |- | ||
− | ! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/content-importer-lite | + | ! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/content-importer-lite content-importer-lite] |
| [[TheSchwartz]] || [[Importer]] || Import some content (everything except entries and comments) | | [[TheSchwartz]] || [[Importer]] || Import some content (everything except entries and comments) | ||
|- | |- | ||
− | ! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/content-importer-verify | + | ! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/content-importer-verify content-importer-verify] |
| [[TheSchwartz]] || [[Importer]] || Verify import password | | [[TheSchwartz]] || [[Importer]] || Verify import password | ||
|- | |- | ||
− | ! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/directory-meta | + | ! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/directory-meta directory-meta] |
| Manual || Directory/user search || Update data used in searching for users | | Manual || Directory/user search || Update data used in searching for users | ||
|- | |- | ||
− | ! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/distribute-invites | + | ! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/distribute-invites distribute-invites] |
| [[TheSchwartz]] || [[Invite code generation#Multiple_user_grants|Invite codes]] || Email distributed invite codes | | [[TheSchwartz]] || [[Invite code generation#Multiple_user_grants|Invite codes]] || Email distributed invite codes | ||
|- | |- | ||
Line 81: | Line 83: | ||
| [[TheSchwartz]] || [[ESN]] || [[ESN#Firing_events_and_delivering_notifications|Notification delivery]] stage 4 (mass) | | [[TheSchwartz]] || [[ESN]] || [[ESN#Firing_events_and_delivering_notifications|Notification delivery]] stage 4 (mass) | ||
|- | |- | ||
− | ! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/expunge-users | + | ! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/expunge-users expunge-users] |
| Manual || Account deletion || Expunge (permanently) accounts deleted more than 60 days ago | | Manual || Account deletion || Expunge (permanently) accounts deleted more than 60 days ago | ||
|- | |- | ||
− | ! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/import-scheduler | + | ! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/import-scheduler import-scheduler] |
| Manual || [[Importer]] || Queue importing jobs in the right order - oneshot as of 20110925, but see [http://bugs.dwscoalition.org/show_bug.cgi?id=1491 bug 1491] | | Manual || [[Importer]] || Queue importing jobs in the right order - oneshot as of 20110925, but see [http://bugs.dwscoalition.org/show_bug.cgi?id=1491 bug 1491] | ||
|- | |- | ||
− | ! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/incoming-email | + | ! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/incoming-email incoming-email] |
| [[TheSchwartz]] || Incoming email || Handles email posting, email support requests, email support follow-up | | [[TheSchwartz]] || Incoming email || Handles email posting, email support requests, email support follow-up | ||
|- | |- | ||
− | ! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/latest-feed | + | ! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/latest-feed latest-feed] |
| [[TheSchwartz]] || Latest things || Heavy lifting for the Latest Things page | | [[TheSchwartz]] || Latest things || Heavy lifting for the Latest Things page | ||
|- | |- | ||
− | ! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/lazy-cleanup | + | ! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/lazy-cleanup lazy-cleanup] |
| [[TheSchwartz]] || Journal management || Postpone some of the work of entry deletion | | [[TheSchwartz]] || Journal management || Postpone some of the work of entry deletion | ||
|- | |- | ||
− | ! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/load-friends-gm | + | ! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/load-friends-gm load-friends-gm] |
| [[Gearman]] || Old (LJ) friends system || Not used anymore, see [http://bugs.dwscoalition.org/show_bug.cgi?id=3971 bug 3971] | | [[Gearman]] || Old (LJ) friends system || Not used anymore, see [http://bugs.dwscoalition.org/show_bug.cgi?id=3971 bug 3971] | ||
|- | |- | ||
− | ! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/paidstatus | + | ! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/paidstatus paidstatus] |
| Manual || Payments || Process paid carts, expired paid accounts, and paid accounts expiring soon | | Manual || Payments || Process paid carts, expired paid accounts, and paid accounts expiring soon | ||
|- | |- | ||
− | ! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/ping-hubbub | + | ! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/ping-hubbub ping-hubbub] |
| [[TheSchwartz]] || Outbound syndication || Notify PubSubHubbub of DW journal updates | | [[TheSchwartz]] || Outbound syndication || Notify PubSubHubbub of DW journal updates | ||
|- | |- | ||
Line 111: | Line 113: | ||
| [[TheSchwartz]] || [[ESN]] || [[ESN#Firing_events_and_delivering_notifications|Notification delivery]] stage 1-4 combined (mass) | | [[TheSchwartz]] || [[ESN]] || [[ESN#Firing_events_and_delivering_notifications|Notification delivery]] stage 1-4 combined (mass) | ||
|- | |- | ||
− | ! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/process-eventlogrecord | + | ! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/process-eventlogrecord process-eventlogrecord] |
| [[TheSchwartz]] || Disabled || Not used currently, see [http://bugs.dwscoalition.org/show_bug.cgi?id=3963 bug 3963] | | [[TheSchwartz]] || Disabled || Not used currently, see [http://bugs.dwscoalition.org/show_bug.cgi?id=3963 bug 3963] | ||
|- | |- | ||
− | ! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/process-privacy | + | ! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/process-privacy process-privacy] |
| [[TheSchwartz]] || Entry privacy || Process mass entry privacy changes | | [[TheSchwartz]] || Entry privacy || Process mass entry privacy changes | ||
|- | |- | ||
− | ! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/resolve-extacct | + | ! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/resolve-extacct resolve-extacct] |
| [[Gearman]] || External accounts || Determine the account type (personal/community/syndicated) | | [[Gearman]] || External accounts || Determine the account type (personal/community/syndicated) | ||
|- | |- | ||
− | ! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/schedule-synsuck | + | ! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/schedule-synsuck schedule-synsuck] |
| Manual || Syndication || Queue TheSchwartz jobs for updating syndicated accounts | | Manual || Syndication || Queue TheSchwartz jobs for updating syndicated accounts | ||
|- | |- | ||
− | ! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/search-constraints | + | ! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/search-constraints search-constraints] |
| [[Gearman]] || Directory/user search || Something to do with partial search results? Not sure | | [[Gearman]] || Directory/user search || Something to do with partial search results? Not sure | ||
|- | |- | ||
− | ! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/search-lookup] | + | ! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/search-lookup search-lookup] |
| [[Gearman]] || Directory/user search || Initiate search? Not sure | | [[Gearman]] || Directory/user search || Initiate search? Not sure | ||
|- | |- | ||
− | ! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/search-updater | + | ! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/search-updater search-updater] |
| Manual || Directory/user search || Not sure what it does, still used but has some bitrot. See [http://bugs.dwscoalition.org/show_bug.cgi?id=3968 bug 3968] | | Manual || Directory/user search || Not sure what it does, still used but has some bitrot. See [http://bugs.dwscoalition.org/show_bug.cgi?id=3968 bug 3968] | ||
|- | |- | ||
− | ! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/send-email | + | ! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/send-email send-email] |
| [[TheSchwartz]] || [[TheSchwartz#Sending_non-ESN_email_to_a_user|Sending email]] || Send email to users | | [[TheSchwartz]] || [[TheSchwartz#Sending_non-ESN_email_to_a_user|Sending email]] || Send email to users | ||
|- | |- | ||
− | ! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/send-email-mass | + | ! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/send-email-mass send-email-mass] |
| [[TheSchwartz]] || [[TheSchwartz#Sending_non-ESN_email_to_a_user|Sending email]] || Send email to users (mass) | | [[TheSchwartz]] || [[TheSchwartz#Sending_non-ESN_email_to_a_user|Sending email]] || Send email to users (mass) | ||
|- | |- | ||
Line 141: | Line 143: | ||
| [[Gearman]] || Payments || Charge a user's credit card | | [[Gearman]] || Payments || Charge a user's credit card | ||
|- | |- | ||
− | ! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/sphinx-copier | + | ! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/sphinx-copier sphinx-copier] |
| [[TheSchwartz]] || Full-text search || Keep the full-text search database in sync with actual entries (see [http://bugs.dwscoalition.org/show_bug.cgi?id=3966 bug 3966]) | | [[TheSchwartz]] || Full-text search || Keep the full-text search database in sync with actual entries (see [http://bugs.dwscoalition.org/show_bug.cgi?id=3966 bug 3966]) | ||
|- | |- | ||
− | ! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/sphinx-search-gm | + | ! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/sphinx-search-gm sphinx-search-gm] |
| [[Gearman]] || Full-text search || Interface to the core sphinx search | | [[Gearman]] || Full-text search || Interface to the core sphinx search | ||
|- | |- | ||
− | ! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/stats-collection | + | ! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/stats-collection stats-collection] |
| Manual || Site stats || Collect/compile statistics | | Manual || Site stats || Collect/compile statistics | ||
|- | |- | ||
− | ! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/subscribe-hubbub | + | ! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/subscribe-hubbub subscribe-hubbub] |
| Manual || Syndication || Subscribe to hubbub for syndicated accounts so DW is notified of updates to feed sources | | Manual || Syndication || Subscribe to hubbub for syndicated accounts so DW is notified of updates to feed sources | ||
|- | |- | ||
− | ! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/support-notify | + | ! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/support-notify support-notify] |
| [[TheSchwartz]] || Support board || Email users about support answers they received or support actions they subscribed to | | [[TheSchwartz]] || Support board || Email users about support answers they received or support actions they subscribed to | ||
|- | |- | ||
− | ! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/synsuck | + | ! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/synsuck synsuck] |
| [[TheSchwartz]] || Syndication || Updates syndicated account from feed source, started by [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/schedule-synsuck bin/worker/schedule-synsuck] or through the hubbub system | | [[TheSchwartz]] || Syndication || Updates syndicated account from feed source, started by [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/schedule-synsuck bin/worker/schedule-synsuck] or through the hubbub system | ||
|- | |- | ||
− | ! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/sysban-gm | + | ! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/sysban-gm sysban-gm] |
| [[Gearman]] || Site administration || Retrieve sysban entries by type | | [[Gearman]] || Site administration || Retrieve sysban entries by type | ||
|- | |- | ||
− | ! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/t-memlimit | + | ! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/t-memlimit t-memlimit] |
| Manual || Testing || Memory stress test | | Manual || Testing || Memory stress test | ||
|- | |- | ||
− | ! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/taglib-gm | + | ! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/taglib-gm taglib-gm] |
| [[Gearman]] || Journal content || Retrieve user's tags | | [[Gearman]] || Journal content || Retrieve user's tags | ||
|- | |- | ||
− | ! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/talklib-gm | + | ! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/talklib-gm talklib-gm] |
| [[Gearman]] || Journal content || Update entry comment count, needed... not sure | | [[Gearman]] || Journal content || Update entry comment count, needed... not sure | ||
|- | |- | ||
− | ! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/userpic-resize-gm | + | ! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/userpic-resize-gm userpic-resize-gm] |
| [[Gearman]] || Userpics || Resize userpics larger than maximum height or width | | [[Gearman]] || Userpics || Resize userpics larger than maximum height or width | ||
|- | |- | ||
− | ! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/xpost | + | ! [http://hg.dwscoalition.org/dw-free/file/tip/bin/worker/xpost xpost] |
| [[TheSchwartz]] || Crossposting || Crossposts new or edited entries | | [[TheSchwartz]] || Crossposting || Crossposts new or edited entries | ||
|} | |} | ||
[[Category: Installation]] | [[Category: Installation]] |
Revision as of 19:48, 1 January 2012
Workers are programs that don't run under control of Apache, because they handle tasks that don't need to happen synchronously to a user's request (whether triggered by one or not), or because running them under Apache would consume excessive resources.
Some workers process TheSchwartz or Gearman requests. Those normally run under control of worker-manager (see below). Some workers are manual, and depending on their purpose, can run either under control of worker-manager or be started by cron, or even just once when {re)starting the server. (Manual workers are a mixed lot, and there's no "one size fits all" option for them.)
Contents
Starting and stopping workers
Using worker-manager and workers.conf
bin/worker-manager takes care of starting, stopping, and restarting workers to maintain the numbers of workers set in etc/workers.conf for each host it runs on. The etc/workers.conf
syntax is pretty straightforward and described in the file itself. To start bin/worker-manager
for normal use (daemonized), use:
$LJHOME/bin/worker-manager
Or, if you want it to stay in the foreground and display progress/debug messages, use:
$LJHOME/bin/worker-manager --debug
In both cases, killing it will also kill the workers it started.
If you find that you do not want to use some workers (for testing, temporarily, or long-term), you may turn them off in the configuration. In /etc/workers.conf
, setting the count to 0 will not start any of that worker. Commenting out the line on which that worker appears (with a #) will not start any of that worker. These are best for temporary changes. For a permanent change, you may also entirely remove the line on which that worker appears .
Individually
You can also start individual workers from the shell prompt, by typing:
$LJHOME/bin/worker/(worker-name)
Most workers accept the -v
or --verbose
flag to request extra debugging output and stay in the foreground when started from the shell prompt, so ^C should work to kill them.
Using cron or at server boot time
Please refer to your OS documentation. The crontab(1) and init(8) manpages are probably most relevant.
List of workers
Below is a list of all the workers in the $LJHOME/bin/worker
directory.
Note: I probably got some of that wrong. Any corrections welcome, especially where I say "not sure".
Filename | Type | Category | Function and notes |
---|---|---|---|
birthday-notify | Manual | ESN | Fire LJ::Event::Birthday ESN events for users whose birthdays are soon |
content-importer | TheSchwartz | Importer | Import all content |
content-importer-lite | TheSchwartz | Importer | Import some content (everything except entries and comments) |
content-importer-verify | TheSchwartz | Importer | Verify import password |
directory-meta | Manual | Directory/user search | Update data used in searching for users |
distribute-invites | TheSchwartz | Invite codes | Email distributed invite codes |
esn-cluster-subs | TheSchwartz | ESN | Notification delivery stage 2 |
esn-cluster-subs-mass | TheSchwartz | ESN | Notification delivery stage 2 (mass) |
esn-filter-subs | TheSchwartz | ESN | Notification delivery stage 3 |
esn-filter-subs-mass | TheSchwartz | ESN | Notification delivery stage 3 (mass) |
esn-fired-event | TheSchwartz | ESN | Notification delivery stage 1 |
esn-fired-event-mass | TheSchwartz | ESN | Notification delivery stage 1 (mass) |
esn-process-sub | TheSchwartz | ESN | Notification delivery stage 4 |
esn-process-sub-mass | TheSchwartz | ESN | Notification delivery stage 4 (mass) |
expunge-users | Manual | Account deletion | Expunge (permanently) accounts deleted more than 60 days ago |
import-scheduler | Manual | Importer | Queue importing jobs in the right order - oneshot as of 20110925, but see bug 1491 |
incoming-email | TheSchwartz | Incoming email | Handles email posting, email support requests, email support follow-up |
latest-feed | TheSchwartz | Latest things | Heavy lifting for the Latest Things page |
lazy-cleanup | TheSchwartz | Journal management | Postpone some of the work of entry deletion |
load-friends-gm | Gearman | Old (LJ) friends system | Not used anymore, see bug 3971 |
paidstatus | Manual | Payments | Process paid carts, expired paid accounts, and paid accounts expiring soon |
ping-hubbub | TheSchwartz | Outbound syndication | Notify PubSubHubbub of DW journal updates |
process-esn | TheSchwartz | ESN | Notification delivery stages 1-4 combined |
process-esn-mass | TheSchwartz | ESN | Notification delivery stage 1-4 combined (mass) |
process-eventlogrecord | TheSchwartz | Disabled | Not used currently, see bug 3963 |
process-privacy | TheSchwartz | Entry privacy | Process mass entry privacy changes |
resolve-extacct | Gearman | External accounts | Determine the account type (personal/community/syndicated) |
schedule-synsuck | Manual | Syndication | Queue TheSchwartz jobs for updating syndicated accounts |
search-constraints | Gearman | Directory/user search | Something to do with partial search results? Not sure |
search-lookup | Gearman | Directory/user search | Initiate search? Not sure |
search-updater | Manual | Directory/user search | Not sure what it does, still used but has some bitrot. See bug 3968 |
send-email | TheSchwartz | Sending email | Send email to users |
send-email-mass | TheSchwartz | Sending email | Send email to users (mass) |
shop-creditcard-charge | Gearman | Payments | Charge a user's credit card |
sphinx-copier | TheSchwartz | Full-text search | Keep the full-text search database in sync with actual entries (see bug 3966) |
sphinx-search-gm | Gearman | Full-text search | Interface to the core sphinx search |
stats-collection | Manual | Site stats | Collect/compile statistics |
subscribe-hubbub | Manual | Syndication | Subscribe to hubbub for syndicated accounts so DW is notified of updates to feed sources |
support-notify | TheSchwartz | Support board | Email users about support answers they received or support actions they subscribed to |
synsuck | TheSchwartz | Syndication | Updates syndicated account from feed source, started by bin/worker/schedule-synsuck or through the hubbub system |
sysban-gm | Gearman | Site administration | Retrieve sysban entries by type |
t-memlimit | Manual | Testing | Memory stress test |
taglib-gm | Gearman | Journal content | Retrieve user's tags |
talklib-gm | Gearman | Journal content | Update entry comment count, needed... not sure |
userpic-resize-gm | Gearman | Userpics | Resize userpics larger than maximum height or width |
xpost | TheSchwartz | Crossposting | Crossposts new or edited entries |