Crawl

An automated pass over many websites to measure them.

By Mackenzie Bowes · mackenzie@mackenziebowes.com · @_mackenziebowes on X

Source

“Throughout May 2024, the HTTP Archive data pipeline crawled several million websites, gathering the metadata to be used in the Web Almanac.” Methodology (The Web Almanac by HTTP Archive, 2024)

See also: Crawler, robots.txt.