GEO Improvement: Add robots.txt to docs site
Source audit: GEO Audit Report — 2026-07-22
Audit date: 2026-07-22
Run: §29939896039
Finding
"Create robots.txt with Allow rules for AI bots (GPTBot, ClaudeBot, PerplexityBot)"
robots_txt check scored 0/18 on the docs site (github.github.com/gh-aw/) — no robots.txt file found, zero bots explicitly allowed, citation_bots_ok: false.
Why this matters
robots.txt is the primary crawl-permission signal that AI citation engines (ChatGPT, Claude, Perplexity, Google AI Overviews) check before indexing content. Without it, AI bots may skip crawling the docs site entirely, preventing gh-aw from appearing in AI-generated answers. This single file is worth +18 points — the largest individual score gap in the audit — and unblocks the llms.txt signal as well.
Suggested fix
- Create a
robots.txt file at the root of the docs site (github.github.com/gh-aw/robots.txt).
- Add explicit
Allow rules for the major AI bots. Minimal example:
User-agent: *
Allow: /
User-agent: GPTBot
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: anthropic-ai
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Google-Extended
Allow: /
User-agent: Bingbot
Allow: /
- If the site is generated (e.g., Docusaurus/Jekyll/Hugo), add this file to the
static/ or equivalent directory so it is included in the build output.
- Verify after deployment:
curl https://github.github.com/gh-aw/robots.txt
For a comprehensive bot list, refer to the robots.txt already present at the repository level (github.com/github/gh-aw), which already allows 27 AI bots and scored 15/18.
Generated by 🌍 GEO Optimizer Daily Audit · sonnet46 21.8 AIC · ⌖ 9.86 AIC · ⊞ 5K · ◷
GEO Improvement: Add
robots.txtto docs siteSource audit: GEO Audit Report — 2026-07-22
Audit date: 2026-07-22
Run: §29939896039
Finding
Why this matters
robots.txtis the primary crawl-permission signal that AI citation engines (ChatGPT, Claude, Perplexity, Google AI Overviews) check before indexing content. Without it, AI bots may skip crawling the docs site entirely, preventing gh-aw from appearing in AI-generated answers. This single file is worth +18 points — the largest individual score gap in the audit — and unblocks thellms.txtsignal as well.Suggested fix
robots.txtfile at the root of the docs site (github.github.com/gh-aw/robots.txt).Allowrules for the major AI bots. Minimal example:static/or equivalent directory so it is included in the build output.curl https://github.github.com/gh-aw/robots.txtFor a comprehensive bot list, refer to the
robots.txtalready present at the repository level (github.com/github/gh-aw), which already allows 27 AI bots and scored 15/18.