LargerioBot

What it reads

On each site, LargerioBot reads the home page and at most a few linked pages, such as contact, about and legal pages. It records the technologies it detects, the social accounts and company details the site declares, and the contact e-mail addresses published there.

It does not submit forms, does not log in, and does not download images, fonts or media. Rendered scans do load the scripts, stylesheets and data requests the page needs: your logs show those too.

How often

A site among the top million is typically revisited every one to two weeks, other sites less often. It never requests two pages of the same site at once, and pauses between pages.

It honours Crawl-delay up to 10 seconds between pages; if a site asks for more, it reads the home page only. When a site answers 429, it slows down and honours Retry-After.

How to recognise it

Quick scans and lookups send this user agent:

Mozilla/5.0 (compatible; LargerioBot/1.0; +https://larger.io/bot)

Rendered scans use headless Chrome and send this one, where <version> is the version of the Chrome in use:

Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/<version> Safari/537.36 (compatible; LargerioBot/1.0; +https://larger.io/bot)

Anyone can send these user agents. To check that a visitor really is LargerioBot, look up the name of its address: it must end in .larger.io. Then look that name up: it must resolve back to the same address.

host <ip address>
host <name returned>

How to limit or block it

LargerioBot reads robots.txt before any other page. It follows the group addressed to LargerioBot, or the * group when there is none. To keep it away from your whole site:

User-agent: LargerioBot
Disallow: /

To make it wait 10 seconds between two pages:

User-agent: LargerioBot
Crawl-delay: 10

Contact

Email crawler@larger.io to have your domain removed from the public index and from exports, or with any question about the bot.