Shinobu Otani

Breaking changes between two versions of a Python package: execution-verified migration cases (LibPulse corpus) plus raw GitHub release notes.

For every pinned dependency in a requirements.txt: how far behind it is and which execution-verified breaking changes sit in the upgrade path (LibPulse corpus).

Turn Markdown documents into structured JSON: nested heading tree with section text, fenced code blocks, links, parsed tables, and size statistics. Pure parsing, no LLM cost.

Convert scraped HTML into clean Markdown and plain text: headings, nested lists, links, images, code blocks, blockquotes, and tables. Drops scripts, styles, and structural boilerplate (nav/footer/aside) so only content remains. Pure parsing, no LLM cost.

Normalize Japanese text for data pipelines: Unicode NFKC (full/half-width unification), wave-dash unification, whitespace cleanup, hiragana/katakana conversion, Japanese-aware sentence splitting, and per-script character stats.

Split Markdown or plain text into retrieval-ready chunks for RAG pipelines: cuts at headings, packs whole sentences up to a size limit with optional overlap, and tags every chunk with its heading breadcrumb. Handles Japanese sentence boundaries. No LLM cost.

Detect exact and near-duplicate texts in scraped or collected datasets: normalized hashing for exact matches plus character-shingle Jaccard similarity for near matches. Works on Japanese (no word tokenization needed). Returns per-text verdicts and duplicate clusters. No LLM cost.

Extract structured entities from free text: email addresses, URLs, phone numbers (incl. Japanese formats and full-width digits), dates (ISO, slash, Japanese 年月日) and IP addresses. Deterministic regex extraction with per-kind counts — fast, cheap, no LLM.