Stive Hassan

Turn RSS 2.0, Atom and JSON Feed sources into one uniform JSON schema: id, title, url, published, author, summary, categories. Up to 50 feeds per run, format auto-detected. Built for pipelines and AI agents that want one schema instead of three. Missing data is null, never fabricated.

Check up to 100 sites for llms.txt and get the parsed structure back as JSON: title, summary, sections and every link with its notes. Sites without the file return not-found so you can measure adoption across a list. Built for AI agents and pipelines that want machine readable site maps.

Parse robots.txt for up to 100 sites into one JSON record each: rule groups, sitemaps and crawl delays, plus an AI crawler policy audit across 17 bots like GPTBot, ClaudeBot and PerplexityBot. Each bot rated allowed, partial or blocked, with an open, selective or closed summary per site.

Extract every URL from XML sitemaps and sitemap indexes: url, lastmod, changefreq, priority and source sitemap. Handles nested indexes, gzip and plain text formats. Up to 20 roots and 50,000 URLs per run. Clean JSON for SEO audits, migrations and crawl planning. Missing fields are null.