# Crawl

An automated pass over many websites to measure them.

**Source:** “Throughout May 2024, the HTTP Archive data pipeline crawled several million websites, gathering the metadata to be used in the Web Almanac.” [Methodology (The Web Almanac by HTTP Archive, 2024)](https://almanac.httparchive.org/en/2024/methodology)

See also: [Crawler](/reference/glossary/crawler), [robots.txt](/reference/glossary/robots-txt).

---

Published 2026-10-29. Web page: https://www.mackenziebowes.com/reference/glossary/crawl