apifystats
Log in Start free
Actor/sbio-apps

Robots.txt Auditor | Crawl Rules & Sitemap Discovery

Monitoring

Read robots.txt for supplied websites and export user-agent groups, allow/disallow rules, sitemap declarations, extension directives and parsing warnings. Identify missing files and compare content hashes in your crawl configuration workflows.

Publisher description, verbatim
Last 7 dayson this actor

No changes seen in the last 7 days.

The morning one happens, followers hear first.Follow this actor
Key figures17 readings
Price / 1k$5.00per 1,000 robots-checked
Users / 30d1→ 0 over 17 readings
Users, all time2→ 0 over 17 readings
Runs / 30d16↑ 15 over 17 readings
Runs / user / mo16.0runs per monthly user
Recorded changes
  • 2026-09-09changeListed—on record
showing 1 of 1 in the last 30 days · followers get the next one by email

History

17 readings since 2026-09-09
Users / 30d17 readings
09-09→ 009-25
Users, all time17 readings
09-09→ 009-25
Runs / 30d17 readings
09-09↑ 1509-25

Follow this actor, compare it against its rivals, and get every overnight change in one morning report.

Publish it yourself? Claim your publisher page and its full history is yours free, forever.

Open in the app Claim this page

Numbers on this page are measured only by an independent census of the Apify Store — not self-reported. Not indexed: too little has happened to this actor for us to have anything worth saying yet.