Crawling
Crawling is the automated process in which a program – a so-called crawler or bot – systematically visits web pages, reads their content and follows links to discover further pages. Search engines and AI systems use crawling to capture what content exists on the web. Without crawling, a page stays invisible to machines and can later be neither indexed nor cited in answers.
Why crawling matters
Crawling is the first step in a long chain. Before a page appears in Google or is recommended by an AI assistant, a crawler must have found and read it at all. If your page is not crawled, it simply does not exist for machines – no matter how good the content is. For AI visibility this holds doubly true: language models draw their knowledge either from training data or from live fetches. In both cases, at the very beginning stands a crawler that collects your content. That is why the crawlability of your website helps decide whether you have any chance of visibility in search results and in AI answers.
How crawling works
A crawler starts with a list of known URLs, fetches each page and analyses the source code. Any contained links are added to the queue, so the crawler works its way from page to page. In doing so it observes rules from the robots.txt file, which defines which areas it may enter. An XML sitemap helps it find all important pages reliably. How often and how deeply crawling happens depends on the so-called crawl budget – the amount of resources a bot allocates to your site. Fast loading times, clean internal linking and clear URL structures ensure that the crawler works efficiently and overlooks no page.
Common mistakes
Many visibility problems arise as early as crawling. An accidental entry in the robots.txt can block entire directories. Content that is only loaded via JavaScript is not seen at all by some crawlers. Faulty redirects, endless parameter URLs or wrong status codes waste crawl budget and leave important pages undiscovered. A lack of internal linking is also tricky: if no other page points to a piece of content, the crawler often never finds it. So check regularly in Google Search Console which pages are being fetched. A clean technical foundation is the basis for good content to get a stage at all.
Relevance to AI recommendations
AI systems like ChatGPT, Perplexity or Google AI Overviews rely on specialised AI crawlers such as GPTBot to capture current web content. These bots work technically much like classic search-engine crawlers, but they pursue a different goal: they collect material from which answers and citations are later formed. Whoever wants to be mentioned in AI answers must allow these crawlers access and offer well-structured, machine-readable content. If you block AI crawlers in the robots.txt, you may be excluding yourself from recommendations. Conversely, a clear allowance combined with clean structure can raise your chance of appearing as a source in generative search results.
Example
Imagine a small bookshop with an online store. The owner publishes book tips every week but is puzzled that none of them appear in Google. A look into the robots.txt reveals the reason: the entire blog folder is accidentally locked to crawlers. The Googlebot was never allowed to read the articles. After removing the block and submitting a current XML sitemap, Google crawls and indexes the posts within a few days. Shortly afterwards the first book tips appear in the search results – and are even cited as a source in an AI answer.
Common questions
What is the difference between crawling and indexing?
Crawling is the reading of a page by a bot. Indexing is the next step: the captured content is added to the search engine's database so that it can appear in results. Only what has been crawled can be indexed – but not everything crawled automatically ends up in the index.
Can I control which crawlers visit my site?
Yes. Via the robots.txt file you define which bots may enter which areas. You can specifically allow or exclude individual crawlers such as GPTBot. But note: robots.txt is an instruction, not a technical enforcement. Reputable crawlers abide by it, but exclusion cannot be guaranteed this way.