# How to Allow AI Crawlers in robots.txt

> Source: Show Up On AI · https://showuponai.com/how-to/allow-ai-crawlers
> Updated: April 2026

The single highest-leverage 5-minute change you can make for AI search visibility. Here is the complete, current list of AI crawler user-agents — and exactly how to allow them.

## Step 1: Locate your robots.txt

It lives at https://yourdomain.com/robots.txt. Most CMSes either generate it dynamically or expose a file you can edit.

## Step 2: Paste this snippet

```
User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Perplexity-User
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: Claude-Web
Allow: /

User-agent: anthropic-ai
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: Applebot-Extended
Allow: /

User-agent: Bytespider
Allow: /

User-agent: CCBot
Allow: /

User-agent: cohere-ai
Allow: /

User-agent: Meta-ExternalAgent
Allow: /

Sitemap: https://yourdomain.com/sitemap.xml
```

## What each crawler does

- GPTBot — OpenAI's training-data crawler.
- OAI-SearchBot — OpenAI's live search crawler powering ChatGPT Search.
- ChatGPT-User — User-initiated browsing on behalf of a ChatGPT user.
- PerplexityBot — Perplexity's primary crawler.
- ClaudeBot / Claude-Web / anthropic-ai — Anthropic's family of crawlers.
- Google-Extended — Google's Gemini/Vertex training-data crawler (separate from Googlebot).
- Applebot-Extended — Apple Intelligence training crawler.
- Bytespider — ByteDance/TikTok AI crawler.
- CCBot — Common Crawl.
- cohere-ai — Cohere's training crawler.
- Meta-ExternalAgent — Meta AI training crawler.

## Related

- GEO Checklist: https://showuponai.com/geo-checklist
- How to Set Up llms.txt: https://showuponai.com/how-to/set-up-llms-txt