Uruguay Open Data Scraper - CKAN Datos Abiertos

dataset-row

One dataset with its full catalogue record: title, description, publishing organization with its own profile, author and maintainer with emails, licence, themes, tags, declared update cadence with a computed overdue flag, harvest origin, and every file nested with its format, size, checksum, dates and whether it is queryable through the portal DataStore.

resource-row

One downloadable file or service: its direct download URL, format, MIME type, byte size, checksum and dates, plus whether it is loaded into the DataStore, carrying the identity, publisher, licence, themes and tags of the dataset it belongs to.

datastore-table-row

One resource the portal will answer live queries about, with the real column names and PostgreSQL types read from the DataStore, its exact row count, its table and index sizes, and ready-made query and SQL endpoints. Includes the DataStore read, so it is never also billed as a schema block.

contact-row

One deduplicated person or unit named on the datasets scanned: name, email, email domain, the roles they hold, how many datasets they are responsible for, which organizations and themes those cover, and a sample of the datasets themselves.

organization-row

One of the 73 publishing bodies on the portal, with its description, logo, creation date, its page on the catalogue, its total dataset count and how many of its datasets match the filters on this run.

category-row

One of the 22 themes the portal groups datasets under, with its Spanish and English titles, its description and image, and dataset counts both catalogue-wide and within the current query.

tag-row

One keyword from the 1,487-entry catalogue vocabulary with its rank, the number of datasets carrying it catalogue-wide and under the current query, and a link to the portal search for it.

format-row

One file format with its true dataset count, measured by querying the index for the union of every raw spelling rather than summing facets, plus each of those raw spellings with its own entry count and whether the format is tabular.

license-row

One licence from the portal registry with its title, URL, Open Definition and Open Source conformance, and how many datasets declare it catalogue-wide and under the current query.

harvest-source-row

One upstream portal the catalogue copies datasets from: its URL, harvester type, refresh frequency, owning organization, how many datasets it contributed, and its last harvest job with the added, updated, unchanged, errored and deleted counts.

link-check

Optional. One file URL fetched to see whether it is actually retrievable: HTTP status, content type, byte size, redirect target and latency. The catalogue harvests external portals and never re-verifies what it copied, and 8 of 90 sampled URLs did not answer.

datastore-schema

Optional. The real column names, PostgreSQL types, exact row count and table size of one DataStore-backed file, read from the portal without downloading a byte. Not charged for files with no DataStore table, which are skipped without a request.

datastore-preview

Optional. The first rows of one DataStore table, returned verbatim from the portal query API including its own NULLs, so you see the data itself rather than its description. Up to 100 rows for the same single request.

view-stats

Optional. The portal own page-view counters for one dataset, total and recent, which is the only public signal of what people actually use. Costs a separate request because the counters cannot be read from the search index.

resource-views

Optional. The previews the portal has configured for one file: data explorer, map, chart, PDF or web page viewer, each with its type and title.

harvest-provenance

Optional. Where one copied dataset really came from: the upstream portal URL, the harvester type, the refresh frequency and whether that source is still active. Only fires on datasets that were actually harvested, and the lookup is cached per source.

organization-profile

Optional. One extra query per publisher returning what it actually publishes: its top themes, file formats, tags and licences, each with counts.