User-agent: * Disallow: / User-agent: AdsBot-Google Disallow: / # Block specific AI training/scraping crawlers User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: CCBot Disallow: /