# robots.txt

A file at the root of a site telling crawlers what they may fetch.

**Source:** “OpenAI uses OAI-SearchBot and GPTBot robots.txt tags to enable webmasters to manage how their sites and content work with AI.” [Overview of OpenAI crawlers (OpenAI)](https://developers.openai.com/api/docs/bots)

See also: [Crawler](/reference/glossary/crawler), [Crawl](/reference/glossary/crawl).

---

Published 2026-10-29. Web page: https://www.mackenziebowes.com/reference/glossary/robots-txt