Difference between revisions of "Bot Policy"
Foxfirefey (Talk | contribs) m (categorize) |
(Linkifying API (not in official version)) |
||
| (4 intermediate revisions by 3 users not shown) | |||
| Line 1: | Line 1: | ||
| − | This | + | This is a mirror of the Dreamwidth Studios "Bot Policy", the rules that third party applications and tools must abide by in order to prevent being blocked from accessing the Dreamwidth servers. |
| − | ''' | + | '''The official and most up to date version is always at [https://www.dreamwidth.org/site/bot Dreamwidth's bot policy] page.''' |
| − | == General | + | == General Guidelines == |
| − | + | The overall guidelines for bot/tool authors are pretty simple: | |
| − | * Good clients send a proper user-agent (or other info string) that includes a contact email address. | + | * Good clients send a proper user-agent (or other info string) that includes a contact email address. That will let us contact you if there are any problems. |
| − | * Be kind to the system | + | * Be kind to the system! Rate-limit your requests, and try to cache data whenever possible. |
| − | * | + | * Don't screen-scrape the HTML/BML output of the site. If you can't do something you want to do through an [[Dreamwidth_API|API]], let us know, and we'll try to add an API for it. |
| + | * Try to avoid asking for users' passwords. If there's something you can't do through the protocol or through our APIs, let us know, and we'll add it. If you have to ask for users' passwords, you should tell people what you'll do with them, and warn users to change their passwords before and after they access your client if they're worried. | ||
== Client Applications == | == Client Applications == | ||
| − | A client application is defined as a tool that runs under the control of an end user | + | A [[Compatible_clients|client application]] is defined as a tool that runs under the control of an end user (whether as a downloadable program or an in-browser object). |
| − | + | Generally, we don't restrict these tools. From time to time, we may need to restrict some of the API methods they use, if there's too much load on the site. We won't do this unless it's absolutely necessary to preserve the proper functioning of the site, and only on a temporary basis. | |
== Third Party Sites/Utilities == | == Third Party Sites/Utilities == | ||
| − | + | We place more restrictions on things that fall into this category, since they have the potential to seriously impact site load. We're here to serve our users (which includes client applications), but third-party sites that scrape data sources aren't our primary audience. | |
| − | Generally speaking, if your site is going to be | + | Generally speaking, if your site is going to be small and access Dreamwidth infrequently, you can go right ahead and do it. (Yes, these terms are purposefully vague -- use your best judgement, and contact us with any questions.) If your use becomes a problem, we'll contact you and let you know that your particular tool is turning out to be a load problem. This is one of the reasons why it's so critical to include contact information in your user agent. |
| − | + | If you intend to run a service (or your service gets popular), and you're making frequent requests, contact us and let us know. This will let us examine your tool and see what changes we can make on our end to better support your use, and what changes we can suggest in your tool to be gentler on the site. | |
| − | + | ||
| − | + | ||
| − | + | ||
| − | + | ||
== IP Rate Limiting == | == IP Rate Limiting == | ||
| − | From time to time you may find your IP address temporarily banned if we determine that your site is hitting | + | From time to time, you may find your IP address temporarily banned, if we determine that your site is hitting our servers too frequently or causing too much of a traffic or load spike. In this case, please contact us and let us know the IP that's been banned so we can work out a solution. |
[[Category: Dreamwidth.org]] | [[Category: Dreamwidth.org]] | ||
| + | [[Category: Data Sources]] | ||
Latest revision as of 21:02, 7 May 2017
This is a mirror of the Dreamwidth Studios "Bot Policy", the rules that third party applications and tools must abide by in order to prevent being blocked from accessing the Dreamwidth servers.
The official and most up to date version is always at Dreamwidth's bot policy page.
Contents
General Guidelines
The overall guidelines for bot/tool authors are pretty simple:
- Good clients send a proper user-agent (or other info string) that includes a contact email address. That will let us contact you if there are any problems.
- Be kind to the system! Rate-limit your requests, and try to cache data whenever possible.
- Don't screen-scrape the HTML/BML output of the site. If you can't do something you want to do through an API, let us know, and we'll try to add an API for it.
- Try to avoid asking for users' passwords. If there's something you can't do through the protocol or through our APIs, let us know, and we'll add it. If you have to ask for users' passwords, you should tell people what you'll do with them, and warn users to change their passwords before and after they access your client if they're worried.
Client Applications
A client application is defined as a tool that runs under the control of an end user (whether as a downloadable program or an in-browser object).
Generally, we don't restrict these tools. From time to time, we may need to restrict some of the API methods they use, if there's too much load on the site. We won't do this unless it's absolutely necessary to preserve the proper functioning of the site, and only on a temporary basis.
Third Party Sites/Utilities
We place more restrictions on things that fall into this category, since they have the potential to seriously impact site load. We're here to serve our users (which includes client applications), but third-party sites that scrape data sources aren't our primary audience.
Generally speaking, if your site is going to be small and access Dreamwidth infrequently, you can go right ahead and do it. (Yes, these terms are purposefully vague -- use your best judgement, and contact us with any questions.) If your use becomes a problem, we'll contact you and let you know that your particular tool is turning out to be a load problem. This is one of the reasons why it's so critical to include contact information in your user agent.
If you intend to run a service (or your service gets popular), and you're making frequent requests, contact us and let us know. This will let us examine your tool and see what changes we can make on our end to better support your use, and what changes we can suggest in your tool to be gentler on the site.
IP Rate Limiting
From time to time, you may find your IP address temporarily banned, if we determine that your site is hitting our servers too frequently or causing too much of a traffic or load spike. In this case, please contact us and let us know the IP that's been banned so we can work out a solution.
