robots.txt for AI crawlers: steering GPTBot, ClaudeBot and others

What the robots.txt actually is

The robots.txt is a plain text file that sits in the root directory of your domain, that is, at yourdomain.de/robots.txt. Every bot that wants to visit your site fetches this file first and reads from it which areas it may enter and which not. The principle is ancient and dates from the 1990s, when it was mainly about classic search engines. What is new is that today AI crawlers like GPTBot or ClaudeBot evaluate the same file.

It is important to understand that the robots.txt is a request, not a door lock. It contains instructions that reputable bots abide by, because their operators have agreed to. A bot that ignores the file is not technically prevented. For the big AI providers, however, it holds that they document their crawlers publicly and respect the rules, because disregard would quickly become a reputation problem.

The structure is deliberately kept simple. With User-agent you name a particular bot and set beneath it, with Disallow or Allow, what applies. An asterisk as User-agent addresses all bots. It is exactly this simplicity that makes the file powerful: in a few lines you steer the behavior of dozens of crawlers, without writing a single line of program code.

Why AI crawlers are a topic of their own

Classic search engine bots read your page to take it into the index and show it in the results. In return they send you visitors. AI crawlers work differently: they collect content to train language models or to answer questions directly in a chat. A user then gets the answer without ever visiting your page. The classic deal of visibility for traffic thereby starts to wobble.

That is why different questions arise for AI crawlers than for Google. An online shop might want its product descriptions to appear in AI answers, but not its hard-earned guide content to simply wander into training. A law firm may not want client information to be processed by a model at all. A publisher, in turn, sells licenses and wants to prevent free siphoning. The interests are completely different depending on the industry.

A second point is the separation of purposes. Some providers use separate bots for training and for live answers. OpenAI, for instance, runs GPTBot for training and a separate user agent for answers in the ChatGPT context. This split allows you fine decisions: you can block training but allow your content to appear in real-time answers with source citation.

The most important AI bots and their names

For your rules to take effect, you have to know the correct user agent names. A typo leads to the rule running into the void and the bot crawling unhindered. The names change occasionally, and new bots are added, so a regular look into the providers' official documentation pays off. Do not rely on copied lists from forums, because outdated or wrong designations quickly creep in there.

Among the currently most relevant are the following crawlers. Note that a provider can run several bots, depending on whether it is about training, indexing or live answers.

If you are unsure which bots actually visit your site, a look into the server logs helps. There you see the user agent identifiers of the real accesses and can tune your robots.txt specifically to them, instead of entering dozens of names on suspicion, half of which never come by.

• GPTBot - crawler from OpenAI, collects content for model training
• OAI-SearchBot - OpenAI bot for search and answer functions in ChatGPT
• ClaudeBot - crawler from Anthropic for training the Claude models
• Claude-Web / Claude-User - Anthropic accesses in the context of user queries
• Google-Extended - steers usage by Google's AI products, separate from the normal Googlebot
• PerplexityBot - crawler of the answer engine Perplexity
• CCBot - crawler of Common Crawl, whose dataset many models use as a training source
• Bytespider - crawler from ByteDance, often with high retrieval volume
• Meta-ExternalAgent - crawler from Meta for AI training purposes

Concrete rules you can adopt directly

The simplest case: you want to lock out a particular bot completely. For that you write the name under User-agent and set Disallow to a single slash. The slash stands for the entire domain. Example: User-agent: GPTBot followed by Disallow: /. With that, GPTBot is barred from your complete site. You repeat this block for every further bot you want to block.

Often you want to proceed more finely. An education provider, for example, might want to release its public course descriptions but protect the internal members' area and downloads. Then you allow the bot in principle and block only individual paths: Disallow: /mitglieder/ and Disallow: /downloads/. Everything else stays accessible. This combination of targeted Disallow lines and optional Allow exceptions gives you fine control, without blocking the whole site.

Pay attention to the syntax details. Paths are to be given one per line, each bot block is separated by a blank line, and the order of the blocks does not matter. A common mistake is to cram several bots into a single User-agent line. That does not work reliably. Better write one clean block per bot, even if the file gets longer as a result.

• Block all AI bots: individual blocks per bot with Disallow: /
• Bar the whole site for everyone: User-agent: * plus Disallow: / - caution, that also hits Google
• Protect only one folder: Disallow: /path/ under the respective bot
• Allow an exception: Allow: /path/public/ after a broader Disallow

The honest limits of the robots.txt

The robots.txt does not protect, it signals. Bots that do not abide by the rules simply ignore your entries. Less well-known or aggressively operating crawlers in particular do this regularly. If content is really confidential, it belongs behind a login or a server-side access block, not in a public directory with a Disallow entry. The Disallow prevents crawling, but not the direct call by anyone who knows the URL.

A second fallacy concerns already trained models. If you lock out GPTBot today, that only affects future retrievals. Content that was already read before the block can no longer be removed from a trained model. The robots.txt thus works forward, not retroactively. Whoever reacts only late may have already distributed a large part of their content across various training datasets.

Third, a block changes nothing about content that others publish about you. If your company is described in press articles, directories or forums, these external texts still land in models, no matter what your own robots.txt says. You only steer access to your domain, not the whole picture that exists about you on the web.

Block or allow: the strategic trade-off

The reflexive answer of many operators is to block all AI bots. That is convenient, but rarely the best decision. More and more people look for information directly in AI assistants instead of classic search engines. Whoever does not show up there because they have blocked everything is simply not named in these answers. For a trade business or a local practice, that can mean staying invisible in a growing number of user queries.

It is therefore sensible to distinguish between training and answer. Many operators block pure training bots like GPTBot or CCBot, but allow the answer and search bots that retrieve content live and usually cite it with a source reference. This way you protect your content from the silent siphoning into training and still stay present in the AI answers that potentially bring you visibility and enquiries.

The right choice depends on the business model. A publisher that lives on licenses has good reasons for hard blocks. A service provider that wants to win customers benefits more from being found. Make the decision deliberately and document it, instead of adopting a copied standard configuration that fits a completely different case.

Implementation, testing and ongoing maintenance

You place the file as a pure text file named robots.txt in the root directory, so that it is reachable at yourdomain.de/robots.txt. A call in the browser shows you immediately whether it is delivered correctly. In many content management systems you edit it via an SEO extension or directly by file access. Make sure that no redirect and no error code is delivered, otherwise bots ignore the rules.

After every change you should check whether it takes effect. A simple way is to call up the file and compare the blocks line by line against your intention. The check only becomes really robust via the server logs, though: there you see which bots actually access and whether a blocked crawler still appears. If a blocked bot stays present, you either have the wrong name recorded or the bot ignores the rule.

Treat the robots.txt as a living document. Providers rename bots, new crawlers appear, and your own strategy changes. A fixed appointment, for example quarterly, to check the official bot lists and your own logs keeps the file current. This way you avoid a block set up two years ago being ineffective today, because the associated bot has long been operating under a different name.