Fairspec Catalog
Updated Nov 12, 2026
| Authors | Evgeny Karev |
|---|---|
| Profile | https://fairspec.org/profiles/latest/catalog.json |
Fairspec Catalog is a simple JSON Lines-based replication format that allows to sync a catalog of Fairspec Datasets. Dynamic search, sorting and similar capabilities are no goals of this specification.
Language
The key words MUST, MUST NOT, REQUIRED, SHALL, SHALL NOT, SHOULD, SHOULD NOT, RECOMMENDED, MAY, and OPTIONAL in this document are to be interpreted as described in RFC 2119.
Descriptor
A Fairspec Catalog is a JSON Lines resource with each line represents a Fairspec Dataset and its last updated time. The dataset locations MUST be unique within a catalog, except for the tombstone and entry that share an upd as del sets out, and the datasets MUST be sorted by the last updated time in descending order.
A line with no characters at all is skipped and is not an entry, so an empty file is a valid catalog with no datasets.
Catalog
Each line of the catalog MUST be a Dataset object. Read in order, the lines form the array that the profile validates.
For example:
{"loc": "https://example.com/dataset1.json", "upd": "2023-10-01T00:00:00Z"}
{"loc": "https://example.com/dataset3.json", "upd": "2023-09-01T00:00:00Z"}
{"loc": "https://example.com/dataset2.json", "upd": "2023-08-01T00:00:00Z"}Dataset
A catalog entry pointing to a Fairspec Dataset. It MUST have the following properties (all required unless otherwise stated):
loc [required] [#loc]
URL of the Fairspec Dataset descriptor. The property MUST be an absolute HTTP or HTTPS URL: a JSON Schema URI whose scheme is http or https, in any case as in RFC 3986.
For example:
{
"loc": "https://example.com/dataset.json"
}upd [required] [#upd]
The last updated time of the dataset. The property MUST be JSON Schema date-time (this format requires a timezone component).
For example:
{
"upd": "2023-10-01T00:00:00Z"
}del
An optional boolean marking the dataset as removed from the catalog: a tombstone. It is sorted by upd, the time of the removal, like any other entry. A client that reads a tombstone MUST drop the dataset at loc. A publisher SHOULD keep a tombstone for as long as a client may still hold the dataset.
A dataset removed and added again within one timestamp has a tombstone and an entry with the same loc and upd. The later of the two lines wins: a client MUST keep the dataset when the entry comes after the tombstone, and drop it when the tombstone comes after the entry.
For example:
{
"loc": "https://example.com/dataset.json",
"upd": "2023-11-01T00:00:00Z",
"del": true
}Extension
Properties not defined by this specification are allowed, so that a descriptor written for a later version stays readable: an implementation MUST ignore them and MAY warn about them. Fairspec Catalog has no extension mechanism: such properties carry no meaning.
Streaming
Fairspec Catalog is designed to be used in a streaming manner. Because datasets are sorted by their last updated time in descending order, a client SHOULD synchronize with a high-water mark: the newest upd it has processed. To synchronize, it reads the catalog line-by-line while upd is greater than or equal to the mark, and stops at the first line older than the mark; the new mark is the upd of the first line. Datasets tied at the mark are read again rather than skipped, since several datasets can share a timestamp and their order is not defined, so processing a dataset a client has already seen MUST be harmless. This allows efficient synchronization without processing the entire catalog file.
Comparison
Created with ❤ and Livemark