# ============================================================================== # GoKenyaHub Robots.txt - Search Engine & AI Crawler Optimization # ============================================================================== # Default Rules for Search Engines & General Crawlers User-agent: * Allow: / Allow: /blog Allow: /shop Allow: /academy Allow: /resources Allow: /search Disallow: /admin/ Disallow: /api/ Disallow: /unsubscribe # Specific Rules for AI Crawlers & LLM Indexers (ChatGPT, Gemini, Copilot, Perplexity, Claude) User-agent: GPTBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: Google-Extended Allow: / User-agent: PerplexityBot Allow: / User-agent: ClaudeBot Allow: / User-agent: Claude-Web Allow: / User-agent: Applebot-Extended Allow: / User-agent: Bytespider Allow: / User-agent: CCBot Allow: / User-agent: cohere-ai Allow: / # Sitemap & AI Discovery Documentation Directives Sitemap: https://gokenyahub.com/sitemap.xml Link: https://gokenyahub.com/llms.txt