Website Content Crawler for LLMs & RAG

page-crawled

The primary event, charged once per page returned. It covers the fetch and the whole extracted record: the page as Markdown with headings, lists, tables and fenced code intact, the title, description, author, published and modified dates, language, canonical URL and Open Graph type, every link and image found, the heading outline, and character, word and token counts. It also covers the boilerplate pass — the navigation, footer and menus stripped out by comparing blocks across the whole crawl, which is work done on the corpus rather than on any single page. Nothing is charged for a page below your minimum word count once boilerplate is gone: a redirect stub or an empty tag page is furniture, not content. Nothing is charged for a page robots.txt refused, for a URL that answered with an error, or for a response that was not a readable page. A page that came back client-rendered IS charged, because reporting that it needs a browser is the finding you came for and the fetch happened either way.

chunk-produced

Charged per chunk, and only when chunking is switched on. Each chunk arrives ready to embed: a stable id derived from the URL and the chunk index so a re-crawl updates rows instead of duplicating them, its own token count, the heading it sits under so it stays self-describing out of context, and an explicit count of the tokens overlapped from the previous chunk. Chunks break on block boundaries, so one never starts mid-sentence and a code fence is never cut in half. A run that leaves chunking off never incurs this event, and the page record already carries the full text.