GEO & AI Search Glossary
AI Crawler
An AI crawler is a bot that fetches web content for AI systems — either to train models (GPTBot, ClaudeBot) or to retrieve live answers for users (OAI-SearchBot, Perplexity-User).
Most AI crawlers do not execute JavaScript, so content rendered client-side is invisible to them. Static or server-rendered HTML, clean semantic markup and fast responses determine whether your pages can be read at all.
Policy matters too: robots.txt decides which bots you welcome. Brands pursuing AI visibility generally allow retrieval bots; training-bot policy is a business decision, but blocking everything removes you from AI answers.
Want this applied to your store?
Every term in this glossary maps to work we deliver. Tell us where you are and we will point to the highest-leverage next step.
Let's talk