CubDomainBot

What it does

When someone asks for a report on your site, or checks it with one of our tools, CubDomainBot loads your homepage and a few public files — robots.txt, security.txt, ads.txt and app-ads.txt, and your MTA-STS policy and gpc.json where you publish them — to check SSL, headers, technologies and SEO basics. It doesn't log in, submit forms or crawl the whole site.

Three things go further, and only when a person asks for them:

• the sitemap generator follows the links on your site to list its pages;
• the browser check opens your homepage once in a headless Chrome, which loads the page's images, scripts and styles like any browser;
• refreshing a company's blog on CubDomain reads your site's RSS or Atom feed.

How to recognise it

It identifies itself with this user agent:

Mozilla/5.0 (compatible; CubDomainBot/2.0; +https://www.cubdomain.com/bot)

It makes at most 120 requests a minute to any one site — your www and bare domain count as one — and respects robots.txt. The images, scripts and styles a browser check loads with your homepage come on top, as they would for any visitor.

Limit or block it

Add rules to your robots.txt. CubDomainBot follows a group for CubDomainBot if there is one, and the * group if not. To block it completely:

User-agent: CubDomainBot
Disallow: /

To keep it out of part of your site:

User-agent: CubDomainBot
Disallow: /private/

Blocking it means your report will only show data we can get without visiting your site, such as DNS and WHOIS, and our tools won't check it. We read robots.txt again at least once a day, so a change takes effect within 24 hours.

One exception: when you verify that you own your domain on CubDomain, we fetch the verification file or page you asked us to check, even if robots.txt blocks us. Crawl-delay isn't read; the limit above always applies.