SurfyBot

SurfyBot is the crawler behind Surfy open search. It reads public web pages so people can find them on Surfy. It crawls for search, not for AI training.

User agentSurfyBot/1.0 (+https://surfy.one/bot)

How it works

Opt out

  1. Robots.txtBlock SurfyBot to stop its visits:
    User-agent: SurfyBot
    Disallow: /
    Pages already in results stay there. To remove them, add noindex first and let SurfyBot see it.
  2. Meta tagsAdd these to a robots meta tag or an X-Robots-Tag header. SurfyBot applies them on its next visit, so keep the page open in robots.txt.
    • noindex: the page is removed from results.
    • nosnippet: none of its text is used.
    • max-snippet:N: no more than N characters are used.
    • noarchive: short passages can be used, never the full text.
    • unavailable_after: the page leaves results after the date you set.
    • nofollow: its links aren't followed.