External Data Fetchers¶

External data fetchers periodically pull data from a third-party HTTP API and store the response on a content object, so the frontend can consume external data through the normal REST API without calling the foreign service itself. Two content types are provided: the generic APIDataFetcher and the specialised RBSDataFetcher for room-reservation data.

Overview¶

An APIDataFetcher is placed inside a parent content object. On a configurable interval a background job calls the configured endpoint, optionally transforms the response, and stores it on the fetcher. Selected values from the response can be copied up into fields on the parent object, so the parent serializes a clean, ready-to-use result.

Key configuration on the generic fetcher:

endpoint – a static URL to fetch.
dynamic_endpoint – a TALES expression producing the URL at fetch time (used instead of the static endpoint when set).
interval – how often (in seconds) the background job re-fetches.
request_headers – additional request headers (defaults to Accept: application/json).
response_timeout – per-request timeout in seconds.
transform_script_name – the name of a traversable script that post-processes the raw response before it is stored.
parent_field_config – a mapping of response values (dot-notation paths) to fields on the parent object, copied up after each fetch.
OpenAPI authentication – optional openapi_login_endpoint, openapi_username and openapi_password; the fetcher logs in, stores the auth response, and injects a Bearer token into the request headers, refreshing it when expired.

Stored response fields¶

After a successful fetch the fetcher carries:

response – the fetched (and optionally transformed) payload.
response_time – when the response was received.
response_duration – how long the request took.
response_error – an error message if the last fetch failed, otherwise empty.

REST API¶

Reading fetched data¶

The fetcher serializes like any content object; the fetched payload is available under response, along with the timing and error fields.

const response = await fetch('/Plone/my-page/weather-fetcher', {
    headers: { 'Accept': 'application/json' }
});
const fetcher = await response.json();
console.log(fetcher.response);          // the stored external payload
console.log(fetcher.response_time);     // when it was last fetched
console.log(fetcher.response_error);    // null when the last fetch succeeded

When a dynamic endpoint is configured, the serialized endpoint reflects the resolved URL.

In most cases the frontend does not read the fetcher directly: selected values are copied into the parent object via parent_field_config, so the parent’s own serialization already contains the data.

RBSDataFetcher¶

The RBSDataFetcher is a specialised fetcher for room-reservation data from the RBS (egovcenter) API. It targets a fixed upstream endpoint and derives the rooms to query from object_ids configured on its parent object, so editors only manage the list of rooms.

Instead of storing the response on a single field for direct consumption, the RBS fetcher indexes each reservation as a synthetic Reservation document in Elasticsearch, scoped to the fetcher’s path. Updates are differential – only added and removed reservations are written – and editors can define additional_reservations on the parent to inject extra closure entries (for example “facility closed”). Removing the fetcher clears its reservations from the index.

Reservations are consumed through the raw search endpoint only:

const response = await fetch('/Plone/@raw-search', {
    method: 'POST',
    headers: {
        'Accept': 'application/json',
        'Content-Type': 'application/json',
    },
    body: JSON.stringify({ query: { term: { portal_type: 'Reservation' } } }),
});
const data = await response.json();
data.hits.hits.forEach(hit => {
    const reservation = hit._source;
    console.log(reservation.Title, reservation.start, reservation.end);
});

Each indexed reservation provides a start and end date-time, a title combining organiser and event, and a description, making the data directly usable for calendar or listing views.

Keeping reservations out of content search¶

Reservations reach content search unless a site excludes them, and they do so twice over:

A reservation document is indexed under the fetcher’s path, so a hit in @es-search renders as a duplicate of the fetcher rather than as the reservation.
The fetcher stores its response as a blob, and collective.elasticsearch feeds every blob field through the attachment pipeline. The reservation payload therefore ends up in the fetcher’s own SearchableText, making the fetcher match arbitrary upstream terms.

Whether that matters is a per-site decision, so it is configuration rather than package policy. A site running RBS excludes both portal types through the existing wcs.backend.search.filter registry record (search control panel), which appends raw Elasticsearch clauses to every query the assembler builds:

[
  {"bool": {"must_not": {"terms": {"portal_type": ["Reservation", "RBSDataFetcher"]}}}}
]

@raw-search does not use the query assembler, so this filter never affects how the reservations themselves are read.

EduDocDataFetcher¶

The EduDocDataFetcher harvests the edudoc.ch OAI-PMH repository – the Swiss education document server – and indexes both the bibliographic metadata and the full text of the linked PDFs into the same Elasticsearch index Plone content uses. The records are therefore found by ordinary site search, without any separate index or endpoint.

Configuration:

oai_base_url – the OAI-PMH endpoint, https://edudoc.ch/oai2d by default.
oai_set – the set to harvest, edk by default (roughly 3300 records).
interval – how often the harvest job runs, daily by default.
last_datestamp – the harvest watermark. It advances automatically to the newest record seen; clearing it forces a full re-harvest.
pdf_request_interval – seconds between PDF downloads, 2.0 by default. Raise it if edudoc.ch keeps answering 429.
languages – only index records written in the selected languages; empty means all of them.

OAI-PMH can only select by set and datestamp – there is no language parameter, and none of edudoc.ch’s 26 sets is language-based – so the language filter is applied after fetching, on MARC 041. A record that declares no language at all (about 4% of the edk set) is not known to be in a wanted language and is therefore skipped as well. Narrowing the filter only drops the records a later harvest happens to return; to re-align a populated index, clear it and load again.

How harvesting works¶

The job walks the whole set through the OAI resumptionToken, parsing MARCXML. Once a watermark exists, subsequent runs only request records changed since then. Because edudoc.ch declares a persistent deleted-record policy, those selective runs also report removals, so deletions are picked up without ever re-harvesting the whole set.

Only records whose OAI datestamp actually changed are re-indexed, and each of those gets its PDF downloaded and pushed through the Elasticsearch ingest-attachment pipeline, which extracts the text and appends it to SearchableText. PDFs larger than 20 MB are skipped. Removing the fetcher clears its records from the index.

The downloads are deliberately paced. A file URL redirects to a registerDownload endpoint, so edudoc.ch counts every harvest fetch and starts answering 429 Too Many Requests when a batch arrives too quickly. Each download is therefore scheduled pdf_request_interval seconds after its predecessor, which spreads a full 3300-record harvest across roughly two hours at the default of 2 seconds. A download that is rate-limited anyway is retried with an escalating backoff of 5, 15 and 60 minutes rather than failing the record.

Recovering from failed downloads¶

Because only changed records are re-indexed, a PDF whose download failed for good – it exhausted its retries, or the harvest predates the pacing – is never fetched again on its own: the record is already in the index with an unchanged datestamp, so the next harvest skips it. Its metadata is searchable, its full text is not.

The Clear indexed records button on the fetcher removes every document belonging to it and resets the watermark. The next data load is then a full harvest that re-indexes all records and queues all PDFs again.

Field mapping¶

The records reuse the existing Plone catalog fields – no EduDoc-specific index is added:

MARC field	Plone field
001	`id`, `getId`
245$a	`Title`, `sortable_title`
520$a	`Description`
700$a, falling back to 710$a	`Creator`
269$a	`effective`, `Date`, `created`, `EffectiveDate`, `CreationDate`
OAI header datestamp	`modified`, `ModificationDate`
690$a, 955$a, 041$a, 490$a, 336$a/980$a	`Subject`
derived from 001	`getRemoteUrl`
242$a, 710$a, 260, 300$a, 954$a, 957$a and the PDF text	`SearchableText`

Each date is indexed twice because Plone keeps two flavours of it: the ISO index fields (effective, modified) that queries sort and filter on, and the DateTime metadata columns (EffectiveDate, ModificationDate) that the search templates format for display. A real brain always carries both.

The bilingual MARC 690 keywords (Berufsbildung = enseignement professionnel) are split so both languages are searchable on their own. Language, series and document type land in Subject as prefixed entries (Sprache: Deutsch, Reihe: EDK-Dossier 38, Typ: Monograph) so they can be grouped out of the otherwise flat keyword facet. There is no dedicated language field: this catalog has no Language index unless plone.app.multilingual is installed, and the mapping deliberately reuses existing Plone fields only. Narrowing a search by language therefore means &Subject=Sprache: Deutsch.

Consuming the records¶

The records carry the portal type EduDocRecord and are searched like any other content – but every query must include a text term. collective.elasticsearch only routes a catalog query to Elasticsearch when it mentions one of the ES-only indexes (SearchableText, Title, Description, topics_filter_uids); anything else is answered straight from the ZODB catalog, where these documents do not exist. A pure filter query such as ?portal_type=EduDocRecord therefore returns nothing, while ?portal_type=EduDocRecord&SearchableText=… works. Use @raw-search to browse or facet without a search term.

const response = await fetch(
    '/Plone/@es-search?portal_type=EduDocRecord&SearchableText=Lehrerbildung' +
    '&metadata_fields=getRemoteUrl',
    { headers: { 'Accept': 'application/json' } }
);
const data = await response.json();
data.items.forEach(record => {
    console.log(record.title, record.getRemoteUrl);
});

The records exist only in the index, so their @id points at a synthetic path below the fetcher. Opening it answers 302 Found with the publication’s page on edudoc.ch in the Location header – for browser requests as well as for API ones – which means search results are clickable without any special handling in the frontend. getRemoteUrl carries the same target for clients that prefer to link out directly, and is the better choice for a client that would otherwise follow the redirect and try to parse the edudoc.ch page as JSON.

The fetcher has the same blob problem described for RBS above: its harvest is stored as a blob, so the whole payload lands in the fetcher’s own SearchableText and it matches arbitrary EduDoc terms. The records are meant to be searchable, the fetcher is not, so exclude only the fetcher:

[
  {"bool": {"must_not": {"terms": {"portal_type": ["EduDocDataFetcher"]}}}}
]