# Crawler

Software that fetches web pages automatically.

**Source:** “OpenAI uses OAI-SearchBot and GPTBot robots.txt tags to enable webmasters to manage how their sites and content work with AI.” [Overview of OpenAI crawlers (OpenAI)](https://developers.openai.com/api/docs/bots)

See also: [Crawl](/reference/glossary/crawl), [robots.txt](/reference/glossary/robots-txt).

---

Published 2026-10-29. Web page: https://www.mackenziebowes.com/reference/glossary/crawler