The reading layer
Crawlers and answer-engine fetchers. They read a site so a model can be trained on it or so an assistant can quote it back to a person.
A training crawler re-indexing every page the day after a product launch. An assistant fetching one page because someone just asked it a question.
Declared user agents, the crawler category behind each one, and the share of all bot traffic that is AI rather than search or SEO.
- WEBREADINGMETA-EXTERNALAGENT · TRAINING CRAWLER
- WEBREADINGCHATGPT-USER · FETCH FOR ONE QUESTION
Web: the declared crawler or fetcher user agent, and its Web Bot Auth signature when it signs. Forms: a person who arrived from an AI answer. Phone and chat: not applicable — nothing reads a phone line.