robots.txt
The robots.txt is a simple text file in the root directory of a website that tells crawlers which areas they may visit and which not. It works like a notice at the entrance: search engine and AI crawlers read it first and (voluntarily) follow the rules. It controls access, not indexing itself.
Why the robots.txt matters for AI visibility
More and more AI systems like ChatGPT, Perplexity or Google AI Overviews send out their own crawlers to read web pages and later cite them in answers. These crawlers check your robots.txt first. If you lock them out there, your brand appears in AI answers less often or not at all. Conversely: whoever keeps important pages open increases the chance of being named as a source. The robots.txt is thus a silent gatekeeper for your AI visibility. Many operators don't even know that an old, restrictive file from SEO days blocks AI crawlers today. A deliberate approach therefore helps decide whether language models can even find and recommend you.
How the robots.txt works technically
The file always sits at your-domain.com/robots.txt and consists of simple rule blocks. Each block begins with User-agent (the name of the crawler), followed by Disallow (forbidden paths) or Allow (permitted paths). An asterisk as the User-agent applies to all crawlers. Important: the robots.txt is a request, not a technical lock. Reputable crawlers abide by it, disreputable ones ignore it. In addition, the path to the XML sitemap can be entered, so that crawlers find all important pages faster. The file prevents crawling but not necessarily indexing: if a blocked page is linked elsewhere, it can still end up in the index – without content. For real hiding you need a noindex meta tag.
Common mistakes
The most expensive mistake is a blanket Disallow: / that accidentally blocks the entire website – often a leftover from the development phase that was forgotten at launch. Equally risky: blocking AI crawlers like GPTBot even though you want to be visible in AI searches. Another classic is the belief that robots.txt protects sensitive data. The opposite is true, because the file is publicly viewable and even reveals blocked paths to the curious. Typos in paths or a wrong storage location (not in the root directory) also render the file ineffective. Check your robots.txt regularly, ideally with the testing tool in Google Search Console, and document every change cleanly.
Relation to AI recommendations and GEO
In Generative Engine Optimization (GEO) the aim is to be cited and recommended as a source by AI assistants. The robots.txt is the first switch point here. Only what an AI crawler may read can flow into a language model or be retrieved in real time. It's worth deciding deliberately: which AI crawlers do you allow, which not? Some operators deliberately open up for GPTBot and Perplexity but block training crawlers that bring no visibility benefit. The balance between reach and control is key. In addition, the llms.txt is gaining importance, a file that explains to AI systems in a structured way which content is especially relevant. Together, both files form the technical foundation of your AI visibility.
Example
An online shop for bicycle accessories notices that it never appears as a source in ChatGPT answers, even though it has detailed guides. A look into the robots.txt reveals: during the last relaunch, the rule User-agent: * with Disallow: / remained in place, dating from the test phase. The entire shop was blocked for all crawlers. After removing this line and adding the sitemap, AI and search engine crawlers are allowed to read the guide pages again. A few weeks later the shop is named as a source in AI answers about bicycle care.
Common questions
Can I use the robots.txt to block individual AI crawlers specifically?
Yes. You create a dedicated rule block, for example User-agent: GPTBot followed by Disallow: /, to lock out exactly this crawler. Other crawlers remain unaffected. Bear in mind, however, that you thereby also reduce your AI visibility for this system.
Does the robots.txt prevent my page from appearing in Google?
Not reliably. The file only stops crawling. A blocked but externally linked page can still end up in the index, then without a content preview. If you want to hide a page for sure, use a noindex meta tag instead of the robots.txt.