robots.txt

A file at the root of a site telling crawlers what they may fetch.

By Mackenzie Bowes · mackenzie@mackenziebowes.com · @_mackenziebowes on X

Source

“OpenAI uses OAI-SearchBot and GPTBot robots.txt tags to enable webmasters to manage how their sites and content work with AI.” Overview of OpenAI crawlers (OpenAI)

See also: Crawler, Crawl.