Marc Mintel
5397309103
Some checks failed
Build & Deploy / 🔍 Prepare (push) Successful in 20s
Build & Deploy / 🧪 QA (push) Failing after 34s
Build & Deploy / 🏗️ Build (push) Has started running
Build & Deploy / 🚀 Deploy (push) Has been cancelled
Build & Deploy / 🧪 Smoke Test (push) Has been cancelled
Build & Deploy / ⚡ Lighthouse (push) Has been cancelled
Build & Deploy / 🔔 Notify (push) Has been cancelled
59 lines
3.1 KiB
Plaintext
59 lines
3.1 KiB
Plaintext
# Crawling
|
||
|
||
[RFC 9309](https://datatracker.ietf.org/doc/html/rfc9309) defines crawlers as automated clients.
|
||
|
||
Some web servers may reject requests that omit the `User-Agent` header or that use common defaults such as `'curl/7.79.1'`.
|
||
|
||
In **undici**, the default user agent is `'undici'`. Since undici is integrated into Node.js core as the implementation of `fetch()`, requests made via `fetch()` use `'node'` as the default user agent.
|
||
|
||
It is recommended to specify a **custom `User-Agent` header** when implementing crawlers. Providing a descriptive user agent allows servers to correctly identify the client and reduces the likelihood of requests being denied.
|
||
|
||
A user agent string should include sufficient detail to identify the crawler and provide contact information. For example:
|
||
|
||
```
|
||
AcmeCo Crawler - acme.co - contact@acme.co
|
||
```
|
||
|
||
When adding contact details, avoid using personal identifiers such as your own name or a private email address—especially in a professional or employment context. Instead, use a role-based or organizational contact (e.g., crawler-team@company.com) to protect individual privacy while still enabling communication.
|
||
|
||
If a crawler behaves unexpectedly—for example, due to misconfiguration or implementation errors—server administrators can use the information in the user agent to contact the operator and coordinate an appropriate resolution.
|
||
|
||
The `User-Agent` header can be set on individual requests or applied globally by configuring a custom dispatcher.
|
||
|
||
**Example: setting a `User-Agent` per request**
|
||
|
||
```js
|
||
import { fetch } from 'undici'
|
||
|
||
const headers = {
|
||
'User-Agent': 'AcmeCo Crawler - acme.co - contact@acme.co'
|
||
}
|
||
|
||
const res = await fetch('https://example.com', { headers })
|
||
```
|
||
|
||
## Best Practices for Crawlers
|
||
|
||
When developing a crawler, the following practices are recommended in addition to setting a descriptive `User-Agent` header:
|
||
|
||
* **Respect `robots.txt`**
|
||
Follow the directives defined in the target site’s `robots.txt` file, including disallowed paths and optional crawl-delay settings (see [W3C guidelines](https://www.w3.org/wiki/Write_Web_Crawler)).
|
||
|
||
* **Rate limiting**
|
||
Regulate request frequency to avoid imposing excessive load on servers. Introduce delays between requests or limit the number of concurrent requests. The W3C suggests at least one second between requests.
|
||
|
||
* **Error handling**
|
||
Implement retry logic with exponential backoff for transient failures, and stop requests when persistent errors occur (e.g., HTTP 403 or 429).
|
||
|
||
* **Monitoring and logging**
|
||
Track request volume, response codes, and error rates to detect misbehavior and address issues proactively.
|
||
|
||
* **Contact information**
|
||
Always include valid and current contact details in the `User-Agent` string so that administrators can reach the crawler operator if necessary.
|
||
|
||
## References and Further Reading
|
||
|
||
* [RFC 9309: The Robots Exclusion Protocol](https://datatracker.ietf.org/doc/html/rfc9309)
|
||
* [W3C Wiki: Write Web Crawler](https://www.w3.org/wiki/Write_Web_Crawler)
|
||
* [Ethical Web Crawling (WWW 2010 Conference Paper)](https://archives.iw3c2.org/www2010/proceedings/www/p1101.pdf)
|