RoasanBot

RoasanBot is the crawler of Roasan, a service that builds store intelligence from the public data of Shopify stores. This page explains what it collects, how it behaves and how to have a store removed.

How to recognise it

RoasanBot sends this User-Agent with every request to a store:

RoasanBot/1.0 (+https://roasan.com/bot; [email protected])

Its robots.txt product token is RoasanBot.

Why we crawl

To tell Shopify stores apart from other sites and to describe them — theme, detected apps, catalogue size and public products — for people researching the e-commerce market. Stores are found through public sources: Certificate Transparency logs, public web archives and datasets such as Common Crawl.

What we collect

Whether a site runs on Shopify, its myshopify address and the domains it uses.
The theme and the apps visible in the public storefront page.
The product count and the public catalogue: product titles, prices and image addresses. Images are not copied.
Currency, language and a country hint.
The business e-mail address and phone number a store publishes on its own storefront — on its contact page, its homepage or its policy page.
When each field was observed and on which page.

What we never collect

Anything a store does not publish on its own storefront; personal names; the addresses of embedded apps or of Shopify itself. Stores run by an individual get no public page.
Checkout, cart and customer account pages: they are never requested.
Anything behind a password or a login: RoasanBot never signs in and skips password-protected stores.
Product descriptions and product images.

How it behaves

It follows robots.txt: the rules for RoasanBot first, otherwise those for all agents. A Crawl-delay of up to 10 seconds is honoured. robots.txt is re-read at least every 24 hours.
It sends one request at a time to a store, at least 250 ms apart, and limits its overall pace across all Shopify stores.
On 429 or 503 answers it backs off for 30 seconds, then 2 minutes, 10 minutes and 1 hour; after 5 in a row it leaves the site alone for 24 hours.
It never connects to private or internal network addresses.

Blocking RoasanBot

Add these lines to your store's robots.txt:

User-agent: RoasanBot
Disallow: /

The change takes effect the next time RoasanBot reads robots.txt, within 24 hours. Blocking stops new visits; to have data already collected deleted, use the opt-out below.

Opt out and removal

To have a store's data deleted and its public page removed, e-mail [email protected] with the store's domain.

We check that the request comes from the store — through a DNS TXT record or a message to a role address of the domain such as info@ — so that nobody can remove a competitor's store.

Then all data about the store is deleted, the store is not crawled again and its public page answers "410 Gone". This happens within 7 days of verification, usually within 24 hours.

A self-service form will be added to this page; until then, e-mail is the way to reach us.

Contact

Questions about RoasanBot: [email protected]. We answer within 2 business days.

Why we crawl

What we collect

Whether a site runs on Shopify, its myshopify address and the domains it uses.

The theme and the apps visible in the public storefront page.

The product count and the public catalogue: product titles, prices and image addresses. Images are not copied.

Currency, language and a country hint.

The business e-mail address and phone number a store publishes on its own storefront — on its contact page, its homepage or its policy page.

When each field was observed and on which page.

What we never collect

Anything a store does not publish on its own storefront; personal names; the addresses of embedded apps or of Shopify itself. Stores run by an individual get no public page.

Checkout, cart and customer account pages: they are never requested.

Anything behind a password or a login: RoasanBot never signs in and skips password-protected stores.

Product descriptions and product images.

How it behaves

It follows robots.txt: the rules for RoasanBot first, otherwise those for all agents. A Crawl-delay of up to 10 seconds is honoured. robots.txt is re-read at least every 24 hours.

It sends one request at a time to a store, at least 250 ms apart, and limits its overall pace across all Shopify stores.

On 429 or 503 answers it backs off for 30 seconds, then 2 minutes, 10 minutes and 1 hour; after 5 in a row it leaves the site alone for 24 hours.

It never connects to private or internal network addresses.

Opt out and removal

To have a store's data deleted and its public page removed, e-mail [email protected] with the store's domain.

We check that the request comes from the store — through a DNS TXT record or a message to a role address of the domain such as info@ — so that nobody can remove a competitor's store.

Then all data about the store is deleted, the store is not crawled again and its public page answers "410 Gone". This happens within 7 days of verification, usually within 24 hours.

A self-service form will be added to this page; until then, e-mail is the way to reach us.