AI crawler
The bots AI companies send to read websites. If they are blocked, your pages cannot be read or quoted.
In plain English
Some crawlers collect pages for training, others fetch pages live to answer a question. Many sites block them by accident through security settings or old rules, which keeps them out of AI answers.
In practice
A firewall setting blocks AI bots on a contractor’s site, and ChatGPT says it “couldn’t access the page”.
Related terms
How AI reads the webPro robots.txt A small file at the root of your site that tells bots which pages they may and may not crawl. In practice, and related terms → How AI reads the webPro llms.txt A proposed plain-text file that points AI tools to the most useful pages on a site. In practice, and related terms → How AI reads the webStarter Indexing When a search engine stores a page so it can show it in results. In practice, and related terms →