Discoverability
Store classification
Categories
AUTOMATION
OTHER
OPEN_SOURCE
Store labels
No public values available.
Catalog ID GpfpXrCoIzZ1hXMb9
Public Apify Actor
scrapeworks/robots-txt
Fetch and parse robots.txt for any site: user-agent rules, crawl-delay, and declared sitemaps. Optionally test whether specific URLs are allowed for a given user-agent, using correct longest-match rules.
Active users / 30d
1
Total users
2
Runs / 30d
29
Total runs
108
Rating
—
No reviews
Bookmarks
0
Usage history
Latest captured Active users / 30d
1
0 since 17 Aug
30 of 30 UTC days captured
Latest captured Runs / 30d
29
0 since 17 Aug
30 of 30 UTC days captured
1
Users / 7d
Discoverability
Categories
Store labels
No public values available.
Monetization
Pay per event
2 charge events configured
Actor Start
apify-actor-start
Charged when the Actor starts running. Number of events charged depends on Actor memory (one event per GB, minimum one event).
$0.00005
1
Users / 30d
1
Users / 90d
result
apify-default-dataset-item
Single result in the default dataset.
$0.001
Loading public Actor documentation
Reading the current public Store definition and reviews.
More from this builder
Otta Jobs Scraper (Welcome to the Jungle)
9 active users · 122 runs / 30d
Website Performance Analyzer — Speed Test & Core Web Vitals
7 active users · 29 runs / 30d
GitHub Repository Search & Scraper
6 active users · 42 runs / 30d
eBay Search Scraper - Listings, Prices & Sellers
4 active users · 898 runs / 30d