AsiBot

AsiBot is the crawler of ASI (Agent Search Index), a small independent search engine operated by xynatec. It visits English-language web pages so that they can be found in ASI's search results, which show a title, the link and a short snippet. AsiBot does not collect content for training AI models.

How AsiBot behaves

• It identifies itself with the user agent AsiBot/0.1; its robots token is AsiBot.
• It obeys robots.txt, including Crawl-delay, and re-reads it at least once a day.
• It waits at least two seconds between requests to the same host and backs off on errors, 429 and 503.
• It honours noindex, nofollow, nosnippet and max-snippet in <meta name="robots"> and <meta name="asibot">, and noindex in the X-Robots-Tag header.
• It never stores raw HTML; only the text needed for search is kept.

Opting out

To keep AsiBot away from your whole site, add this to your robots.txt:

User-agent: AsiBot
Disallow: /

Pages that are already in the index are removed after AsiBot's next visit.

Contact

Questions or problems: [email protected]

ASI search