Difference between revisions of "Escalation"

From Dreamwidth Notes
Jump to: navigation, search
(Downtime)
(User in peril phone tree: Update NZ information (same source as initial info))
 
(29 intermediate revisions by 2 users not shown)
Line 11: Line 11:
  
 
=User in peril phone tree=
 
=User in peril phone tree=
Who to contact first:  
+
'''Who to contact first: '''
  
 
Goes most direct to least direct
 
Goes most direct to least direct
Line 20: Line 20:
 
# File a Terms of Service report with "SUICIDE" or other similarly descriptive word in the subject
 
# File a Terms of Service report with "SUICIDE" or other similarly descriptive word in the subject
  
What ToS needs to know:  
+
'''What ToS needs to know: '''
 
# What is the username?  
 
# What is the username?  
 
# What sort of peril? Why are they in danger?  
 
# What sort of peril? Why are they in danger?  
## Links to entries, even locked entries, can save time for ToS.  
+
** Links to entries, even locked entries, can save time for ToS.  
 +
# Are there any factors that would put them in more danger if the wrong emergency service shows up? (Example: if police show up for a Black man with a medical or mental health emergency.)
  
What ToS needs to find:  
+
'''What ToS needs to find: '''
 
# Is there a reason to believe they may be in imminent danger of their life?
 
# Is there a reason to believe they may be in imminent danger of their life?
 
# (if so) What is their physical location?  
 
# (if so) What is their physical location?  
 +
# Would summoning emergency services help, or increase the danger?
 +
# (if help) What is the contact information for the appropriate local agency?
  
In case of a user with suicide plans, it can help if friends:
+
'''In case of a user with suicide plans, it can help if friends:'''
 
* [https://catespeaks.wordpress.com/2016/11/11/1479/ Reach out to them]
 
* [https://catespeaks.wordpress.com/2016/11/11/1479/ Reach out to them]
 
* Tell them they care
 
* Tell them they care
Line 36: Line 39:
 
* Refer them to help lines
 
* Refer them to help lines
 
** Lifeline - 13 11 14 (Australia)
 
** Lifeline - 13 11 14 (Australia)
 +
** OUTLine (LGBTIQ+) 08006885463 (New Zealand)
 +
** Depression Helpline free call or free text 1737 (New Zealand)
 +
** Shine (domestic abuse) 0508744 633 (New Zealand)
 +
** Youthline 0800376633 (New Zealand)
 +
** Lifeline 0800543354 or Suicide Crisis Helpline 0508 828 865 (New Zealand)
 +
** Parent Help (incl. child abuse) 0800568856 (New Zealand)
 
** 1-800-784-2433 (US)
 
** 1-800-784-2433 (US)
 
** Trevor project (LGBT* crisis) line - 866-488-7386 (US)
 
** Trevor project (LGBT* crisis) line - 866-488-7386 (US)
 
** Trans lifeline - 877-565-8860 (US)
 
** Trans lifeline - 877-565-8860 (US)
 
** [http://www.suicide.org/international-suicide-hotlines.html International suicide helplines by country]
 
** [http://www.suicide.org/international-suicide-hotlines.html International suicide helplines by country]
 +
* Make a commitment to follow up and help them navigate their options for non-crisis care
 +
* (privately, without making a fuss to them about it) make a commitment to check in regularly and just chat
  
Implementation:  
+
'''Implementation: '''
 
# Consult the User in Peril phone tree.  
 
# Consult the User in Peril phone tree.  
 
# Whoever is closest to the top of the phone tree, activate. Multiple people can activate if necessary.  
 
# Whoever is closest to the top of the phone tree, activate. Multiple people can activate if necessary.  
Line 53: Line 64:
 
# Ask how they know this.  
 
# Ask how they know this.  
 
# Ask for any links -- this can save time for ToS if these are already gathered. (This should be given in PM.)  
 
# Ask for any links -- this can save time for ToS if these are already gathered. (This should be given in PM.)  
 +
## If the user is having their crisis on another service, Dreamwidth may well not be able to read info and summon emergency crews -- in that case, refer to the other service's user-in-peril response crew if possible, and give the concerned friend whatever help we can as a public service.
 
# Ask for physical location. (This should be given in PM.)  
 
# Ask for physical location. (This should be given in PM.)  
 
# (if you have the username) Review the user's profile to see if they have set up a link for a post with contact information. If there is one, pass it to ToS privately.  
 
# (if you have the username) Review the user's profile to see if they have set up a link for a post with contact information. If there is one, pass it to ToS privately.  
Line 61: Line 73:
 
# Pass information to ToS as soon as they arrive, username and type of peril first, evidence for peril second, and everything else third.  
 
# Pass information to ToS as soon as they arrive, username and type of peril first, evidence for peril second, and everything else third.  
 
# As ToS arrives, they can take point on interviewing the person reporting.  
 
# As ToS arrives, they can take point on interviewing the person reporting.  
# Once the situation is handed off to ToS, de-escalate the channel. [https://twitter.com/rahaeli/media @rahaeli maintains a stash of kitten pictures] which can be helpful for this.  
+
# Once the situation is handed off to ToS, [[De-Escalation|de-escalate]] the channel. [https://twitter.com/rahaeli/media @rahaeli maintains a stash of kitten pictures] which can be helpful for this.  
 
# Some people in channel may be adversely affected. If you can, remain around to chat with anyone who seems to need someone.  
 
# Some people in channel may be adversely affected. If you can, remain around to chat with anyone who seems to need someone.  
 
# Adrenaline after-effects are fun! Check embodiment: food, drink, warmth, stretch/physical motion, medication, bathroom/washing, human contact, sleep.
 
# Adrenaline after-effects are fun! Check embodiment: food, drink, warmth, stretch/physical motion, medication, bathroom/washing, human contact, sleep.
Line 72: Line 84:
 
*Can you load other internet pages?  
 
*Can you load other internet pages?  
 
*What errors are you getting?  
 
*What errors are you getting?  
**How often?
+
*On refresh, do you get a working site?  
**Do they disappear upon refresh?
+
 
* Does [http://downforeveryoneorjustme.com/dreamwidth.org DownForEveryoneOrJustMe.com] think it's down?  
 
* Does [http://downforeveryoneorjustme.com/dreamwidth.org DownForEveryoneOrJustMe.com] think it's down?  
* Can other people in [[IRC]] load it?
+
* Can other people in [[IRC]] load it? Are there multiple reports?  
 
**What does Fig say? (The command "fig, is dw down?" will tell the bot to check stuff on the back end.)
 
**What does Fig say? (The command "fig, is dw down?" will tell the bot to check stuff on the back end.)
* How long has it been going on?
+
*How long has it been going on?
**Is it persistant or intermittent?
+
*Is it persistent or intermittent?
 +
 
 +
===What is "Down"?===
 +
The nature of the beast:
 +
* Complete downtime for everyone (in a way that isn't blatantly Someone Else's Fault, such as a Big Yellow Digger Error that takes out half a continent)
 +
* Complete downtime for a significant chunk of people
 +
* Significant site performance degradation for everyone or a significant chunk of people
 +
 
 +
Further troubleshooting steps:
 +
* Can you load the main page? https://www.dreamwidth.org / http://www.dreamwidth.org
 +
* Can you load the main page of your journal?
 +
* Can you load your reading page?
 +
* Can you load the main page of the other journal or community in question?
 +
* Site not responding or entirely fail to load site, with browser error message (what error message is it? investigate user and local connectivity issues)
 +
* Something responds, but whatever it is, isn't serving you Dreamwidth pages (what is it? Cloudflare, Amazon ELB, or something else?)
 +
** Does it look like a Dreamwidth-side caching layer, like Varnish, Perlbal, or nginx? (ESCALATE. As of 2016-ish, Cloudflare should be it.)
 +
* A page that looks like a Dreamwidth page is giving you an error message and not the content you expected (investigate further or escalate)
 +
* Dreamwidth pages load, with content, but in a strangely pantsless fashion (it's probably missing stylesheets, if this happens, and is most often caused by browser or internet problems on the user end)
 +
* Dreamwidth pages with content load, but a specific (and often uncommon) task fails. (File a ticket.)
 +
* Is it a Cloudflare outage?
 +
* Do subsequent refreshes also fail to result in a working site? (Stuff failing approximately 1/6 of the time can mean that one of the webservers is hosed in some hilarious way.)
 +
* Was there just a site update, and do the errors suggest that caches are refilling?
 +
* Was there just a news post, and do the errors suggest that there are delays in notifications being emailed?
  
 
===If others are able to reach the site:===
 
===If others are able to reach the site:===
Line 87: Line 120:
 
===Intermittent errors:===
 
===Intermittent errors:===
 
* Is it for everyone, or just some people?  
 
* Is it for everyone, or just some people?  
* Is there a pattern? If so, can you discern a pattern?  
+
* Can you discern a pattern to who gets it (if some), or when people get it (if everyone)?
  
 
===If it is down for everybody:===
 
===If it is down for everybody:===
Line 94: Line 127:
 
* Senior volunteers or non-Mark staff will confirm the error, and file/assist with filing support tickets and/or bug reports if warranted.  
 
* Senior volunteers or non-Mark staff will confirm the error, and file/assist with filing support tickets and/or bug reports if warranted.  
 
* After confirming that it is a hair-on-fire situation, [http://dw-lounge.dreamwidth.org/ Mark will be paged].
 
* After confirming that it is a hair-on-fire situation, [http://dw-lounge.dreamwidth.org/ Mark will be paged].
 +
** Mark's day job includes being paged at odd hours, so Mark is used to this.
 +
** This includes being paged at odd hours on holiday weekends and/or after major holidays.
  
 
=Other Site Weirdnesses=
 
=Other Site Weirdnesses=
Line 119: Line 154:
 
**This could be as simple as "when I post an entry, it explodes" or as complex as a step-by-step walk-through of everything you do, including URLs and possibly screenshots.
 
**This could be as simple as "when I post an entry, it explodes" or as complex as a step-by-step walk-through of everything you do, including URLs and possibly screenshots.
 
**The latter is preferred, but the former is ok too.
 
**The latter is preferred, but the former is ok too.
 +
 +
==Support vs. Github==
 +
If you're not familiar with Github, file a Support request.
 +
 +
Generally, Github issues are filed by
 +
* Site admins and system admins
 +
* DevCom
 +
* The <dwcomm>dw_suggestions</dwcomm> admin (Denise) when a suggestion is approved for future implementation
 +
** Generally, feature suggestions (new feature or changes to existing) should be sent to <dwcomm>dw_suggestions</dwcomm> for discussion.
 +
* Senior technical support volunteers
 +
* Technical support volunteers who have been instructed to do so by leadership
 +
* developers involved in the project
 +
* other developers who have taken a look through the existing issues and confirmed there's not a duplicate issue they can find to +1
  
 
=IRC problems=
 
=IRC problems=
Line 171: Line 219:
 
* DevCom: <dwstaff>karzilla</dwstaff>
 
* DevCom: <dwstaff>karzilla</dwstaff>
 
* Senior Volunteers:
 
* Senior Volunteers:
 +
** <dwuser>sporky_rat</dwuser>
 
** <dwuser>kaberett</dwuser>
 
** <dwuser>kaberett</dwuser>
  
 
==People who can page other people==  
 
==People who can page other people==  
 +
===Founders===
 
* <dwstaff>denise</dwstaff> can perhaps be reached via:  
 
* <dwstaff>denise</dwstaff> can perhaps be reached via:  
 
** <dwuser>sarah</dwuser>
 
** <dwuser>sarah</dwuser>
 
** <dwuser>azurelunatic</dwuser>
 
** <dwuser>azurelunatic</dwuser>
 +
** <dwuser>kaberett</dwuser>
 +
** <dwuser>kareila</dwuser>
 
* <dwstaff>mark</dwstaff> can perhaps be reached via:  
 
* <dwstaff>mark</dwstaff> can perhaps be reached via:  
 
** <dwuser>afuna</dwuser>
 
** <dwuser>afuna</dwuser>
 
** <dwuser>azurelunatic</dwuser>
 
** <dwuser>azurelunatic</dwuser>
 +
** <dwuser>kaberett</dwuser>
 +
** <dwuser>kareila</dwuser>
 +
 +
===Sysadmins, DevCom, ToS===
 +
* Robby can perhaps be reached via:
 +
** <dwuser>kareila</dwuser>
 
* <dwuser>jennifer</dwuser> can perhaps be reached via:  
 
* <dwuser>jennifer</dwuser> can perhaps be reached via:  
 
** <dwuser>azurelunatic</dwuser>
 
** <dwuser>azurelunatic</dwuser>
 +
** <dwuser>kareila</dwuser>
 +
* <dwuser>kareila</dwuser> can perhaps be reached via:
 +
** <dwuser>azurelunatic</dwuser>
 +
 +
===Senior Volunteers===
 
* <dwuser>azurelunatic</dwuser> can perhaps be reached via:  
 
* <dwuser>azurelunatic</dwuser> can perhaps be reached via:  
** fhocutt
+
** <dwuser>silveradept</dwuser>
** <dwuser>jd</dwuser> (DW private message)
+
** <dwuser>alexseanchai</dwuser>
** shadowspar
+
** <dwuser>norabombay</dwuser>
** Silver_Adept (et al)
+
** <dwuser>batrachian</dwuser>
** Woggy
+
** assorted other OSB attendees
 
* <dwuser>kaberett</dwuser> can perhaps be reached via:  
 
* <dwuser>kaberett</dwuser> can perhaps be reached via:  
** me_and  
+
** me_and
 +
 
 +
===Miscellaneous===
 
* Any member of <dwcomm>dw_lounge</dwcomm> may have contact information to page any other member.  
 
* Any member of <dwcomm>dw_lounge</dwcomm> may have contact information to page any other member.  
 
* Anyone with dreamwidth/delegate in their Freenode cloak is likely (but not guaranteed) to have extra ways to contact people.  
 
* Anyone with dreamwidth/delegate in their Freenode cloak is likely (but not guaranteed) to have extra ways to contact people.  

Latest revision as of 09:04, 17 August 2022

Technical support escalation

Goes lowest to highest.

  1. File a support request
  2. File a ticket for developers
  3. Support
  4. Senior volunteers/Non-sysadmin staff
  5. Senior staff (Rah)
  6. Sysadmin staff (Mark or Robby)

User in peril phone tree

Who to contact first:

Goes most direct to least direct

  1. Terms of Service/Senior staff (Rah, Mark)
  2. Anyone with contact info for Terms of Service/Senior staff (such as senior volunteers, non-sysadmin staff)
  3. Anyone with contact info for senior volunteers/non-sysadmin staff
  4. File a Terms of Service report with "SUICIDE" or other similarly descriptive word in the subject

What ToS needs to know:

  1. What is the username?
  2. What sort of peril? Why are they in danger?
    • Links to entries, even locked entries, can save time for ToS.
  1. Are there any factors that would put them in more danger if the wrong emergency service shows up? (Example: if police show up for a Black man with a medical or mental health emergency.)

What ToS needs to find:

  1. Is there a reason to believe they may be in imminent danger of their life?
  2. (if so) What is their physical location?
  3. Would summoning emergency services help, or increase the danger?
  4. (if help) What is the contact information for the appropriate local agency?

In case of a user with suicide plans, it can help if friends:

  • Reach out to them
  • Tell them they care
  • Tell them they would miss them
  • Tell them there is help
  • Refer them to help lines
    • Lifeline - 13 11 14 (Australia)
    • OUTLine (LGBTIQ+) 08006885463 (New Zealand)
    • Depression Helpline free call or free text 1737 (New Zealand)
    • Shine (domestic abuse) 0508744 633 (New Zealand)
    • Youthline 0800376633 (New Zealand)
    • Lifeline 0800543354 or Suicide Crisis Helpline 0508 828 865 (New Zealand)
    • Parent Help (incl. child abuse) 0800568856 (New Zealand)
    • 1-800-784-2433 (US)
    • Trevor project (LGBT* crisis) line - 866-488-7386 (US)
    • Trans lifeline - 877-565-8860 (US)
    • International suicide helplines by country
  • Make a commitment to follow up and help them navigate their options for non-crisis care
  • (privately, without making a fuss to them about it) make a commitment to check in regularly and just chat

Implementation:

  1. Consult the User in Peril phone tree.
  2. Whoever is closest to the top of the phone tree, activate. Multiple people can activate if necessary.
    1. Announce that you're present and whose contact info you have.
    2. Report who you've contacted and via which method, back to the coordinating channel.
    3. Use multiple methods to contact, such as Twitter DM *and* text message *and* phone call.
  3. Someone take responsibility for interviewing the person who is reporting the user in peril. Ideally this would be a senior volunteer or someone else with a strong commitment to confidentiality. This should perhaps not be someone who is actively ringing the phone tree.
  4. All information about the user in peril should be kept to PM and privileged channels, for their privacy.
  5. Ask for the username of the user in peril. (This should be given in PM.)
  6. Ask what type of peril (if it's not already apparent).
  7. Ask how they know this.
  8. Ask for any links -- this can save time for ToS if these are already gathered. (This should be given in PM.)
    1. If the user is having their crisis on another service, Dreamwidth may well not be able to read info and summon emergency crews -- in that case, refer to the other service's user-in-peril response crew if possible, and give the concerned friend whatever help we can as a public service.
  9. Ask for physical location. (This should be given in PM.)
  10. (if you have the username) Review the user's profile to see if they have set up a link for a post with contact information. If there is one, pass it to ToS privately.
  11. Ask the person if they are in communication with the user.
  12. If the person is, see helpful talking points.
  13. If the user is a member of any particular groups, there may be specialized resources for members of that group in crisis. The user's interests list can help here.
  14. If no one can contact anyone from Terms of Service in a timely manner, direct the person reporting to file a ToS ticket, with all the relevant information.
  15. Pass information to ToS as soon as they arrive, username and type of peril first, evidence for peril second, and everything else third.
  16. As ToS arrives, they can take point on interviewing the person reporting.
  17. Once the situation is handed off to ToS, de-escalate the channel. @rahaeli maintains a stash of kitten pictures which can be helpful for this.
  18. Some people in channel may be adversely affected. If you can, remain around to chat with anyone who seems to need someone.
  19. Adrenaline after-effects are fun! Check embodiment: food, drink, warmth, stretch/physical motion, medication, bathroom/washing, human contact, sleep.

Downtime

If the site is down, this wiki page may not be available. If that is the case, use your best judgment. Otherwise, try to follow these escalation procedures.

Before escalation, try to do some basic troubleshooting:

  • Is it down for you?
  • Can you load other internet pages?
  • What errors are you getting?
  • On refresh, do you get a working site?
  • Does DownForEveryoneOrJustMe.com think it's down?
  • Can other people in IRC load it? Are there multiple reports?
    • What does Fig say? (The command "fig, is dw down?" will tell the bot to check stuff on the back end.)
  • How long has it been going on?
  • Is it persistent or intermittent?

What is "Down"?

The nature of the beast:

  • Complete downtime for everyone (in a way that isn't blatantly Someone Else's Fault, such as a Big Yellow Digger Error that takes out half a continent)
  • Complete downtime for a significant chunk of people
  • Significant site performance degradation for everyone or a significant chunk of people

Further troubleshooting steps:

  • Can you load the main page? https://www.dreamwidth.org / http://www.dreamwidth.org
  • Can you load the main page of your journal?
  • Can you load your reading page?
  • Can you load the main page of the other journal or community in question?
  • Site not responding or entirely fail to load site, with browser error message (what error message is it? investigate user and local connectivity issues)
  • Something responds, but whatever it is, isn't serving you Dreamwidth pages (what is it? Cloudflare, Amazon ELB, or something else?)
    • Does it look like a Dreamwidth-side caching layer, like Varnish, Perlbal, or nginx? (ESCALATE. As of 2016-ish, Cloudflare should be it.)
  • A page that looks like a Dreamwidth page is giving you an error message and not the content you expected (investigate further or escalate)
  • Dreamwidth pages load, with content, but in a strangely pantsless fashion (it's probably missing stylesheets, if this happens, and is most often caused by browser or internet problems on the user end)
  • Dreamwidth pages with content load, but a specific (and often uncommon) task fails. (File a ticket.)
  • Is it a Cloudflare outage?
  • Do subsequent refreshes also fail to result in a working site? (Stuff failing approximately 1/6 of the time can mean that one of the webservers is hosed in some hilarious way.)
  • Was there just a site update, and do the errors suggest that caches are refilling?
  • Was there just a news post, and do the errors suggest that there are delays in notifications being emailed?

If others are able to reach the site:

  • Clear your browser cache, restart your browser, restart your computer (if possible). Also consider rebooting any network hardware, or if using WiFi, trying a different wireless network.
  • File a support request, including any error messages you may be getting. (See Filing support requests for instructions.)
    • Consider including a traceroute.

Intermittent errors:

  • Is it for everyone, or just some people?
  • Can you discern a pattern to who gets it (if some), or when people get it (if everyone)?

If it is down for everybody:

  • Locate a senior volunteer or staff (their IRC cloaks will contain "Delegate" or "Staff"), which can often be done by a simple inquiry of the form "Any staff or senior volunteers on duty? SOMETHING'S ON FIRE."
    • Try paging kareila, azurelunatic, kaberett.
  • Senior volunteers or non-Mark staff will confirm the error, and file/assist with filing support tickets and/or bug reports if warranted.
  • After confirming that it is a hair-on-fire situation, Mark will be paged.
    • Mark's day job includes being paged at odd hours, so Mark is used to this.
    • This includes being paged at odd hours on holiday weekends and/or after major holidays.

Other Site Weirdnesses

If something is not working as you expect it to, it is ok to poll IRC to see if it is not working for everyone else also.

If it is working for everyone else:

  • Check the DW FAQ to see if the issue is in there
  • File a Support request.
    • This helps staff track issues, and allows Support to do further troubleshooting

If it is not working for everyone else:

  • Ping kareila and let her know and/or mention it in #dreamwidth-support
  • then file the Support request

Filing support requests

Even if you report an error you are having in IRC, please also file a support request if you can. Support requests help staff remain accountable (so issues do not get forgotten) and keep track of how many people are experiencing a particular problem.

  • If you can reach Dreamwidth at all, file a support request from the web form.
  • If you cannot reach Dreamwidth, email support (at) dreamwidth (dot) org with a full description of what you are trying to do, what you expect will happen, what is happening instead, and the full text of any error messages you are getting.
  • Include steps for replication
    • This could be as simple as "when I post an entry, it explodes" or as complex as a step-by-step walk-through of everything you do, including URLs and possibly screenshots.
    • The latter is preferred, but the former is ok too.

Support vs. Github

If you're not familiar with Github, file a Support request.

Generally, Github issues are filed by

  • Site admins and system admins
  • DevCom
  • The [info]dw_suggestions admin (Denise) when a suggestion is approved for future implementation
    • Generally, feature suggestions (new feature or changes to existing) should be sent to [info]dw_suggestions for discussion.
  • Senior technical support volunteers
  • Technical support volunteers who have been instructed to do so by leadership
  • developers involved in the project
  • other developers who have taken a look through the existing issues and confirmed there's not a duplicate issue they can find to +1

IRC problems

Sometimes there are problems in IRC. Note that this applies to moderated channels.

  • Is it a situation where you (or someone without portfolio) saying something in that channel would help? For example, saying "I'm not comfortable with this topic" or "We don't use that kind of language in here" or deflecting the topic to something that is not currently on fire?
  • Does it require intervention from an IRC op? If so, (discreetly or otherwise) page someone who can attain that role. (It can help to holler both by role (can op up in #dreamwidth) and by name (almost anybody with /staff/ or /delegate/ in their cloak).)
    • PM people who can op up; if urgent, all of them in order of likely-to-be-there
    • holler in another channel which has people who can op up but doesn't have the individuals involved with the problem
  • Is it long-term & not time-sensitive? Emailing [info]denise is good, even if you were able to de-escalate it, to make sure that it's known in case it ever comes up again.
  • If there is an urgent type channel problem and none of #dreamwidth's own ops are around, seek help in #freenode

Who ya gonna call?

Classifications for people, and the sorts of emergency where these people might be called in.

Founders

An actual emergency concerning Dreamwidth-in-general (super rare).

Sysadmins

The server is on fire.

  • Mark
  • Robby

Terms of Service

A Dreamwidth user's life is in peril.

People with Terms of Service privs

Anti-Spam

Stuff is getting super spammed up.

People with power to work on Spam support requests

  • [info]azurelunatic (who may well page someone higher on the list)
  • Any Terms of Service person.

Members of the antispam community

Technical Support

There is something very wrong with the site, and people need information or instructions.

People who can page other people

Founders

Sysadmins, DevCom, ToS

Senior Volunteers

Miscellaneous

  • Any member of [info]dw_lounge may have contact information to page any other member.
  • Anyone with dreamwidth/delegate in their Freenode cloak is likely (but not guaranteed) to have extra ways to contact people.