SEO/robots.txt

robots.txtの書き方

画像検索ボットの排除

User-agent: Googlebot-Image
Disallow: /

User-agent: Yahoo-MMCrawler
Disallow: /

User-agent: *
Allow: /

サイトマップ登録

User-agent: *
Sitemap: http://example.com/sitemap.xml
Allow: /
Disallow: /js/
Disallow: /api/

ウェブ魚拓を拒否

User-agent: Megalodon
Disallow: /

不要bot排除

User-agent: MJ12bot
Disallow: /

User-agent: AhrefsBot
Disallow: /

User-agent: BLEXBot
Disallow: /

User-agent: Yandex
Disallow: /

User-agent: baiduspider
Disallow: /

User-agent: SemrushBot
Disallow: /

参考：https://oxynotes.com/?p=9056

画像を登録しない

nginx

location ~* \.pdf$ {
  add_header X-Robots-Tag "noindex, nofollow";
}

apache

<Files ~ "\.pdf$">
  Header set X-Robots-Tag "noindex, nofollow"
</Files>

参考：https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag?hl=ja#apache

絶対パス指定の場合(/assets/img/test-min.pngなど)

nginx

location ~ \/assets\/img\/*min\.png$ {
  add_header X-Robots-Tag "noindex, nofollow";
}