SerpexBot

SerpexBot exists for one job: building the Serpex index, the retrieval layer behind our /search and /contents API and MCP tools. We crawl to answer an agent's question with clean, citeable text — not to train a model.

That distinction is worth being explicit about:

• Not a training crawler. Crawled pages feed a search index, the same category as Googlebot or Bingbot — not a foundation-model training crawler like GPTBot. We don't train generative models on the content we index.
• Not a data reseller. The product is ranked search results and cleaned page text served through our API — not your content sold wholesale as a raw scrape or dataset.
• Quality-gated, not bulk. We only keep pages that pass our quality filters; spam, junk, and adult material are dropped before they ever reach the index.

Every request SerpexBot makes sends this exact User-Agent, with no rotation and no variants:

SerpexBot/1.0 (+https://serpex.ai/bot)

We never disguise our crawler as a browser or another company's bot, and we never rotate identities to get past a block. If a request claims to be SerpexBot, you can confirm it (below) — anything that fails verification is an impostor, not us.

SerpexBot currently crawls from a single, fixed IP address: 46.4.31.29. Three independent signals confirm a request is genuinely us:

# 1. The User-Agent is an exact, unspoofed match — always
SerpexBot/1.0 (+https://serpex.ai/bot)

# 2. Cross-check the source IP against our published list
$ curl -s https://serpex.ai/serpexbot.json
{ "ipv4": ["46.4.31.29/32"], "ipv6": [] }

# 3. That IP already resolves forward from crawler.serpex.ai
$ host crawler.serpex.ai
crawler.serpex.ai has address 46.4.31.29

SerpexBot fully honors the Robots Exclusion Protocol. We fetch and cache each domain's robots.txt for up to 24 hours, so changes take effect the same day. Four common cases:

Allow everything

# Allow SerpexBot to crawl your whole site
User-agent: SerpexBot
Allow: /

Block us entirely

# Block SerpexBot entirely — nothing gets indexed
User-agent: SerpexBot
Disallow: /

Slow us down without blocking

# Keep indexing, just slow us down (seconds between requests)
User-agent: SerpexBot
Crawl-delay: 10
Allow: /

We accept a Crawl-delay of 1–60 seconds; values outside that range are clamped to the nearest bound rather than ignored.

Keep specific paths out

# Allow the site, but keep specific paths out of the index
User-agent: SerpexBot
Disallow: /admin/
Disallow: /cart/
Allow: /

We also respect page-level noindex and drop a URL from the index on our next refresh pass after finding it. There's no partial mode within a Disallow: it always means we don't fetch the page at all.

Beyond whatever you set in robots.txt, SerpexBot polices itself per domain:

• At least 1 second between requests to the same host, even if robots.txt says nothing about rate — and a default cap of 10,000 pages per domain per day.
• Automatic back-off on 429 responses, rather than retrying immediately.
• Conditional GET (If-Modified-Since / ETag) whenever your server supports it, so an unchanged page costs you a 304, not a full re-download, on every refresh.

Need something more specific than robots.txt allows — a lower rate, a crawl window, a one-off pause? Email us and we'll hand-configure it.

• Honest. One truthful User-Agent, no browser or identity spoofing, no anti-bot evasion.
• Selective. We index quality content only — no spam, no adult material.
• Respectful. We obey robots.txt, noindex, and remove content on request.
• Transparent. This page reflects our current setup as-is, including what we're still rolling out — no claims we can't back up.

Questions, crawl-rate requests, or content removal: bot@serpex.ai. We respond within two business days.