# Crawling and indexing: basics

German title: Crawling und Indexierung: Grundlagen

> robots.txt controls crawling, not indexing; use noindex or password protection for that. noindex works only if the page is crawlable. Sitemaps are a hint, canonical a strong signal without binding effect. Crawl budget mainly concerns very large sites.

Manfred Sulzmann: “Technical SEO is unglamorous—but without working crawling there is no ranking and no AI visibility.”

## Facts

- **robots.txt:** controls crawling, not indexing
- **noindex:** works only on crawlable pages; meta tag or X-Robots-Tag
- **Sitemap limit:** 50 MB or 50,000 URLs per file
- **Canonical:** strong signal, not a directive
- **Crawl budget:** guide for very large sites

## What each tool does

Technical tools per Google documentationTool Does Does not
robots.txt Control crawler access to URLs, limit server load Keep a page out of Google
noindex Prevent indexing —
Sitemap Report URLs (XML, RSS/Atom, text) Guarantee crawling
rel=canonical Signal the preferred URL Act as a directive

## robots.txt and noindex

- Google writes that robots.txt is not a mechanism for keeping a web page out of Google. If other pages link to a blocked URL with descriptive text, Google may still index the URL without visiting it.
- To exclude a page from the index Google names noindex or password protection. noindex is set via  or the HTTP header X-Robots-Tag ; the header suits PDFs, videos and images.
- Prerequisite: the page must not be blocked by robots.txt, otherwise the crawler never sees the noindex rule and the page can still appear. Google does not support a noindex rule in robots.txt.
- Timing: Google has to recrawl the page to see the rule; depending on the page's importance this can take months. A recrawl can be requested via the URL Inspection tool.

## Sitemap and canonical

A sitemap may contain at most 50 MB (uncompressed) or 50,000 URLs; larger ones are split. Google uses lastmod if the value is “consistently and verifiably” accurate. For canonical, Google lists in order of strength: redirect, rel=canonical link, rel=canonical HTTP header, sitemap.

## Crawl budget

Google's guide targets sites with roughly one million pages and weekly changes, or roughly 10,000 pages with daily changes. According to Google, the numbers are rough estimates. Smaller sites generally do not need the guide.

## Relation to other reports

This report covers crawling and indexing. Load time and user experience are in the report on [Core Web Vitals](/reports/core-web-vitals-performance), titles and snippets in the report on [on-page SEO](/reports/onpage-seo-content-struktur).

## Not verified

The statement in the quote that without crawling there is no AI visibility is the author's assessment and is not verified in this report.

## Sources

- Google Search Central: Introduction to robots.txt (primary source)
- Google Search Central: Block Search indexing with noindex (primary source)
- Google Search Central: Build and submit a sitemap (primary source)
- Google Search Central: Specify a canonical URL (primary source)
- Google Search Central: Crawl budget for large sites (primary source)

Checked on: 24 October 2026.

## Questions

### noindex or robots.txt?

To keep a page out of the index, use noindex. robots.txt only blocks crawling; the URL can still be indexed.

### Do I need to worry about crawl budget?

Usually not. Google aimed the guide at very large or frequently updated sites; for small sites an up-to-date sitemap and checking the page indexing report is adequate.

### Is rel=canonical binding?

No, it is a strong signal but not a directive. Redirects are the stronger method.

### Why does my page still appear despite noindex?

Either robots.txt blocks the page so the rule is never seen, or Google has not recrawled the page since the rule was added. That can take months.

## Related

- [SEO basics for knowledge portals](https://munichmedia.com/reports/seo-basics-fuer-wissensportale.md)
- [On-page SEO & content structure](https://munichmedia.com/reports/onpage-seo-content-struktur.md)
- [Internal linking & topic silos](https://munichmedia.com/reports/interne-verlinkung-themen-silos.md)
- [Core Web Vitals & performance](https://munichmedia.com/reports/core-web-vitals-performance.md)

Canonical HTML: https://munichmedia.com/reports/technisches-seo-grundlagen