Ireland Open Data Scraper - data.gov.ie Portal

search-page

One page of up to 200 datasets read from the data.gov.ie CKAN index. This is the fixed cost of paging through the portal, spread across every row that page yields.

resource-row

One downloadable file or service: its direct download URL, normalised and raw format, MIME type, byte size, checksum, whether it is machine readable, whether it is loaded into the portal DataStore, its locator protocol, and the identity, publisher and theme of the dataset it belongs to.

organization-row

One of the 174 publishing bodies on the portal, with its acronym, full description, logo, how many datasets it publishes, its follower count and the date it joined.

harvest-source-row

One of the 31 upstream feeds data.gov.ie harvests, with its URL, harvester type, collection name, default licence and language, schedule, next scheduled run, last harvest request and job count.

showcase-row

One of the portal showcases: an application, map or story built on Irish open data, with its description, external URL, author and tags.

data-request-row

One public data request, what a citizen or company asked a public body to publish, with the body it targets, the intended use, the vote count, whether it was closed and which dataset was accepted in answer.

activity-row

One recent change to a dataset on the portal, with its timestamp, the change type and the dataset it touched. This is the feed to poll for what moved since your last run.

theme-row

One of the 14 portal themes with its plain-English label and its live dataset count under whatever filters the run set.

hvd-category-row

One of the six high-value dataset categories from Implementing Regulation (EU) 2023/138, with its EU URI and how many Irish datasets fall under it.

licence-row

One licence id the portal uses, with a plain-English label, an Open Definition check and its live dataset count. The portal marks 732 CC-BY-4.0 datasets as not open, and this row is where that shows up.

frequency-row

One declared update frequency with a plain-English label and its live dataset count, so you can see how much of the portal is actually maintained.

collection-row

One of the 27 harvest collections with its live dataset count, showing how much of the national portal each contributing feed accounts for.

format-row

One file-format spelling with its canonical name, whether it is machine readable, and its dataset count. The portal uses 132 spellings for about 30 real formats and the index is case sensitive, so the raw spellings matter.

tag-row

One keyword from the portal vocabulary of 22,742 tags with the number of datasets carrying it, ranked, honouring whatever filters the run set.

openness-score

Optional. The portal's own 0 to 5 star openness rating for a dataset plus the reason it was given, fetched from the record view that carries it. Not charged when the portal returned no rating.

datastore-probe

Optional. One file that is loaded into the portal DataStore queried for its real column names, per-column SQL types, true row count and a sample of rows. Nothing in the harvested metadata carries this. Not charged when the table could not be read.

link-check

Optional. One file URL fetched from the publisher's own server to see whether it is actually alive: HTTP status, redirect target, content type and byte size. The portal harvests 31 external feeds and never revalidates the links.

organization-profile

Optional. The publishing body's acronym, description, logo, dataset count and follower count added to a dataset row. Each publisher is fetched once per run and reused across every dataset it published.