robots.txt
A small file at the root of your site that tells bots which pages they may and may not crawl.
In plain English
Every major search and AI crawler checks robots.txt first. One wrong line can block an AI tool from your whole site, so it is worth checking which bots you allow.
In practice
A site’s robots.txt says “Disallow: /” for GPTBot, left over from an old setting, and the site never appears in ChatGPT citations.
robots.txtvsllms.txt
robots.txt sets which bots may crawl which pages. llms.txt is a proposed guide pointing AI tools to your best pages.
Related terms
How AI reads the webStarter AI crawler The bots AI companies send to read websites. If they are blocked, your pages cannot be read or quoted. In practice, and related terms → How AI reads the webPro llms.txt A proposed plain-text file that points AI tools to the most useful pages on a site. In practice, and related terms → How AI reads the webStarter Indexing When a search engine stores a page so it can show it in results. In practice, and related terms →