SeoGuard

Our crawler

About SeoGuardBot

SeoGuardBot is the scanner behind SeoGuard. If it is in your server's logs, one of two things happened. Somebody pasted one of your pages into our checker to see how it looks in search results and link previews: that is one page, once. Or somebody who looks after your site has asked us to watch it, and we read its pages on a schedule to tell them what changed.

How to recognise it

Every request it makes carries this user agent:

SeoGuardBot/1.0

What it fetches

One scan is one page, fetched once, plus at most five small requests about it:

  • the page itself (we stop reading at 2 MB and after 10 seconds);
  • /robots.txt, to see whether search engines may read the page;
  • the page's share image: a HEAD, then only its first 64 KB, which is where its dimensions are;
  • /favicon.ico, if the page links no icon;
  • the http:// version of the address, to see that it redirects to https://.

It doesn't run JavaScript, submit forms, follow links or keep cookies. It never runs more than two scans of one site at a time, and never more than six a minute, whoever is asking. A repeat scan of a page within ten minutes is answered from our copy, unless the person asks for the page to be fetched again.

When it is watching a whole site

A monitored site is read from its sitemap.xml, or by following links from the home page when there is none, once a month, a week or a day depending on the plan. While it does:

  • it obeys robots.txt for SeoGuardBot, and never requests a page it is disallowed;
  • it asks for one page a second at most, two at once at most, and waits longer if your robots.txt sets a Crawl-delay (we honour up to 30 seconds);
  • a 429 or 503 makes it back off for as long as Retry-After says, and a run of refusals ends the scan;
  • it reads at most 25 pages of a site until its owner has proved the site is theirs, and stays on the one site.

If it is being blocked

A firewall or bot filter that turns SeoGuardBot away makes our report say "we were blocked" rather than anything about your page. To let it through, allow the user agent above:

  • Cloudflare: Security → WAF → Custom rules → a rule with User Agent contains SeoGuardBot and the action Skip.
  • Wordfence: Firewall → Blocking → Advanced → add the user agent to the allowlist.
  • Anything else: allow requests whose user agent contains SeoGuardBot.

If you would rather it didn't visit

Block the same user agent in your firewall and every scan of your site will stop at the first request. A one-off scan is made on a person's say-so, like a link preview, so it isn't governed by robots.txt; our site monitoring, which crawls, is, under this same name: 

User-agent: SeoGuardBot
Disallow: /

Something wrong?

If SeoGuardBot is misbehaving, or someone is using it against your site, write to hello@getseoguard.com with the time and the address it requested. We keep a keyed hash of each requester's address, not the address itself, which is enough to stop them.