# Brand retrieval & RAG: what can be influenced

German title: Brand Retrieval & RAG: Was sich steuern lässt

> RAG combines a language model with a retrieval module and a searchable knowledge store. It follows that the retrieved documents shape the content of the answer; a brand cannot control this directly. The bots of OpenAI, Perplexity and Anthropic are documented. An effect of measures on being named is not verified.

## Facts

- **RAG:** language model + retrieval module + knowledge store
- **Lewis et al. (2020):** best results on three open-domain QA tasks
- **Bots:** 8 documented bots from OpenAI, Perplexity, Anthropic
- **Controllability:** not direct
- **Effect of measures:** not verified

## What is verified

Lewis et al. (2020) describe RAG as a combination of a pretrained sequence-to-sequence model (parametric memory) and a vector index over Wikipedia (non-parametric memory) searched by a neural retriever. The approach achieved best results on three open-domain question answering tasks. The paper does not describe how individual products such as ChatGPT or Perplexity choose their sources.

## Inference (not a statement of the paper)

If a system retrieves passages from a knowledge store and answers from them, the content of the answer depends on the retrieved documents. A brand owner therefore cannot directly control whether and how a brand appears. This is an inference from the method, not a statement by Lewis et al.

## Which bots fetch web pages?

Documented bots by providerProvider Bot Purpose per provider robots.txt
OpenAI OAI-SearchBot ChatGPT search features allow it to appear in search results
OpenAI GPTBot content that may be used for training disallowing signals: not for training; independent of OAI-SearchBot
OpenAI ChatGPT-User user-initiated actions robots.txt rules may not apply
Perplexity PerplexityBot showing and linking in Perplexity search results; no foundation model training allowing recommended
Perplexity Perplexity-User user-initiated fetch generally ignores robots.txt rules
Anthropic ClaudeBot content for training honors robots.txt
Anthropic Claude-User fetch on user requests disabling may reduce visibility in user-directed searches
Anthropic Claude-SearchBot improving search results disabling may reduce visibility in search results

According to OpenAI and Perplexity, robots.txt changes can take up to about 24 hours. Anthropic advises against IP blocking.

## Working model for brands (recommendation)

- Maintain a public page with verifiable product details (name, specification, prices, date).
- Keep the details identical across all own channels.
- Check whether the search bots are allowed in robots.txt.
- Document errors in third-party sources and report them to the operators.

These steps are recommendations. An effect on whether the brand is named is not verified.

## Not verified

This report describes a working model of the author. There is no primary source from Google, OpenAI, Perplexity or Anthropic for the individual steps; read them as recommendations, not as verified effects. The bot table only reproduces the providers' statements; how the bots influence the choice of sources in answers is not verified.

## Sources

- Lewis et al.: Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (arXiv 2005.11401v4) (primary source)
- OpenAI: Overview of OpenAI crawlers (primary source)
- Perplexity Docs: Crawlers (primary source)
- Anthropic Support: Does Anthropic crawl data from the web? (primary source)

Checked on: 24 October 2026.

## Questions

### Can I control RAG directly?

No. You can only influence what is on your own and third-party sources and whether the bots may fetch your pages.

### Which bots exist?

Documented are OAI-SearchBot, GPTBot and ChatGPT-User (OpenAI), PerplexityBot and Perplexity-User (Perplexity) and ClaudeBot, Claude-User and Claude-SearchBot (Anthropic).

### Does blocking GPTBot prevent ChatGPT search?

No. According to OpenAI the setting is independent of OAI-SearchBot.

### Is paid advertising enough?

There is no evidence for this.

### Are brands with a fact sheet named more often?

That is not verified.

## Related

- [Entities and information gain: concept](https://munichmedia.com/reports/aeo-entity-information-gain.md)
- [AEO tracking: metrics, tools and limits](https://munichmedia.com/reports/aeo-tracking-kpis.md)
- [Defensive measures against misinformation](https://munichmedia.com/reports/aeo-defensive-massnahmen.md)
- [Third-party platforms and sentiment: what is verified](https://munichmedia.com/reports/aeo-sentiment-drittplattformen.md)

Canonical HTML: https://munichmedia.com/reports/aeo-brand-retrieval-rag