# robots.txt for https://cobrain.jp/ # Crawl rules per RFC 9309 (https://www.rfc-editor.org/rfc/rfc9309) User-agent: * # Content Signals (https://contentsignals.org/) — content usage preferences, not access control. # search=yes : may index for search and link back # ai-input=yes : may use as input for AI answers / AI search at inference time # ai-train=no : may NOT use for AI model training Content-Signal: search=yes, ai-input=yes, ai-train=no Allow: / # --- AI crawlers --- # Policy: allow crawlers whose purpose is search-engine-equivalent indexing / # citation (surface our content and link back); disallow crawlers whose purpose # is AI model training or dataset collection for third-party AI services. # Per RFC 9309 a bot obeys only its most specific User-agent group, so each AI # bot needs explicit rules here (the "*" block above does not apply to them). # Search / AI-search crawlers (index & cite, link back) — allowed User-agent: OAI-SearchBot Allow: / User-agent: Claude-SearchBot Allow: / # Claude-User: real-time fetch to answer a specific user request # (inference-time retrieval, not training) — allowed. User-agent: Claude-User Allow: / User-agent: PerplexityBot Allow: / # AI training / dataset-collection crawlers — disallowed # Note: Google-Extended / Applebot-Extended are training opt-out tokens only; # disallowing them does not block Googlebot / Applebot search crawling (governed by "*"). User-agent: GPTBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: ClaudeBot Disallow: / User-agent: CCBot Disallow: / User-agent: Bytespider Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: Meta-ExternalAgent Disallow: / # Amazonbot: Amazon states crawled data may be used to train its generative AI # (third-party training under our policy) — disallowed. Amazon offers no separate # search-only token, so this also forgoes Alexa-style surfacing. User-agent: Amazonbot Disallow: / Sitemap: https://cobrain.jp/sitemap.xml