Back to Home

RankifyBot

The crawler behind Rankify's SEO and AI-visibility tools.

What it is

Rankify is an SEO platform for small businesses. RankifyBot reads web pages when a Rankify user asks us to check their website — an SEO audit, an AI-visibility scan, or a map of their site's internal links — and, for some tools, other public pages, such as competitors' pages or articles that mention the user's business. Rankify does not train AI models on the pages it fetches.

User agent

  • RankifyBot/1.0 (+https://gorankify.com) — all tools.

What it fetches

  • Single pages a user asks about. One page per request, the way a person opening it would — so, like ChatGPT-User and Perplexity-User, these checks don't consult robots.txt.
  • Site maps. When a user maps their own site, RankifyBot reads up to 100 pages, follows the robots.txt rules for RankifyBot (or *), fetches three pages at a time with pauses in between, and slows down when a site answers 429 (too many requests).
  • robots.txt and sitemap files, to find pages and to report which crawlers — including AI search crawlers — a site allows.
  • It never logs in or submits forms.

If a site's firewall refuses requests from cloud servers, Rankify may retry a page a signed-in user asked for through a scraping provider. When a page builds its text with JavaScript, Rankify may also load it in that provider's browser to show the user what Google sees. These requests still identify themselves as RankifyBot, so a rule that blocks RankifyBot by name still applies, but they don't carry the signature described below.

Verifying a request

RankifyBot signs its requests with Web Bot Auth (HTTP Message Signatures). Signed requests carry a Signature-Agent header pointing to https://www.gorankify.com, and our public keys are published at https://www.gorankify.com/.well-known/http-message-signatures-directory. Apart from the scraping-provider requests above, a request that only claims to be RankifyBot without a valid signature may not be us.

Allowing RankifyBot

If you use Rankify and your security settings block it, allow the user agent RankifyBot in your firewall or security plugin. On Cloudflare, a WAF custom rule that matches the user agent and uses the Skip action works on paid plans; the free plan's Bot Fight Mode can't make exceptions, so it would need to be turned off while you scan.

Blocking RankifyBot

To stop RankifyBot from mapping your site, add this to your robots.txt:

User-agent: RankifyBot
Disallow: /

To stop all requests, block the user agent RankifyBot in your firewall, or email us and we'll help.

Contact

Questions or problems: support@gorankify.com