External Data Fetchers

External data fetchers periodically pull data from a third-party HTTP API and store the response on a content object, so the frontend can consume external data through the normal REST API without calling the foreign service itself. Two content types are provided: the generic APIDataFetcher and the specialised RBSDataFetcher for room-reservation data.

Overview

An APIDataFetcher is placed inside a parent content object. On a configurable interval a background job calls the configured endpoint, optionally transforms the response, and stores it on the fetcher. Selected values from the response can be copied up into fields on the parent object, so the parent serializes a clean, ready-to-use result.

Key configuration on the generic fetcher:

• endpoint – a static URL to fetch.

• dynamic_endpoint – a TALES expression producing the URL at fetch time (used instead of the static endpoint when set).

• interval – how often (in seconds) the background job re-fetches.

• request_headers – additional request headers (defaults to Accept: application/json).

• response_timeout – per-request timeout in seconds.

• transform_script_name – the name of a traversable script that post-processes the raw response before it is stored.

• parent_field_config – a mapping of response values (dot-notation paths) to fields on the parent object, copied up after each fetch.

• OpenAPI authentication – optional openapi_login_endpoint, openapi_username and openapi_password; the fetcher logs in, stores the auth response, and injects a Bearer token into the request headers, refreshing it when expired.

Stored response fields

After a successful fetch the fetcher carries:

• response – the fetched (and optionally transformed) payload.

• response_time – when the response was received.

• response_duration – how long the request took.

• response_error – an error message if the last fetch failed, otherwise empty.

REST API

Reading fetched data

The fetcher serializes like any content object; the fetched payload is available under response, along with the timing and error fields.

const response = await fetch('/Plone/my-page/weather-fetcher', {
    headers: { 'Accept': 'application/json' }
});
const fetcher = await response.json();
console.log(fetcher.response);          // the stored external payload
console.log(fetcher.response_time);     // when it was last fetched
console.log(fetcher.response_error);    // null when the last fetch succeeded

When a dynamic endpoint is configured, the serialized endpoint reflects the resolved URL.

In most cases the frontend does not read the fetcher directly: selected values are copied into the parent object via parent_field_config, so the parent’s own serialization already contains the data.

RBSDataFetcher

The RBSDataFetcher is a specialised fetcher for room-reservation data from the RBS (egovcenter) API. It targets a fixed upstream endpoint and derives the rooms to query from object_ids configured on its parent object, so editors only manage the list of rooms.

Instead of storing the response on a single field for direct consumption, the RBS fetcher indexes each reservation as a synthetic Reservation document in Elasticsearch, scoped to the fetcher’s path. Updates are differential – only added and removed reservations are written – and editors can define additional_reservations on the parent to inject extra closure entries (for example “facility closed”). Removing the fetcher clears its reservations from the index.

Reservations are therefore consumed like any other indexed content, through the search endpoints:

const response = await fetch(
    '/Plone/@es-search?portal_type=Reservation',
    { headers: { 'Accept': 'application/json' } }
);
const data = await response.json();
data.items.forEach(reservation => {
    console.log(reservation.title, reservation.start, reservation.end);
});

Each indexed reservation provides a start and end date-time, a title combining organiser and event, and a description, making the data directly usable for calendar or listing views.