Crawler

Software that fetches web pages automatically.

By Mackenzie Bowes · mackenzie@mackenziebowes.com · @_mackenziebowes on X

Source

“OpenAI uses OAI-SearchBot and GPTBot robots.txt tags to enable webmasters to manage how their sites and content work with AI.” Overview of OpenAI crawlers (OpenAI)

See also: Crawl, robots.txt.