SurfyBot is the crawler behind Surfy open search. It reads public web pages so people can find them on Surfy. It crawls for search, not for AI training.
Crawl-delay asks for longer, SurfyBot waits longer.Retry-After says, up to an hour. Without Retry-After, it pauses for a minute.User-agent: SurfyBot
Disallow: /Pages already in results stay there. To remove them, add noindex first and let SurfyBot see it.X-Robots-Tag header. SurfyBot applies them on its next visit, so keep the page open in robots.txt.noindex: the page is removed from results.nosnippet: none of its text is used.max-snippet:N: no more than N characters are used.noarchive: short passages can be used, never the full text.unavailable_after: the page leaves results after the date you set.nofollow: its links aren't followed.