# Fairspec Catalog

<table>
  <tr>
    <th>Authors</th>
    <td>Evgeny Karev</td>
  </tr>
  <tr>
    <th>Profile</th>
    <td>
      <a href="https://fairspec.org/profiles/latest/catalog.json">https://fairspec.org/profiles/latest/catalog.json</a>
    </td>
  </tr>
</table>

Fairspec Catalog is a simple [JSON Lines](https://jsonlines.org/)-based replication format that allows to sync a catalog of Fairspec Datasets. Dynamic search, sorting and similar capabilities are no goals of this specification.

## Language

The key words `MUST`, `MUST NOT`, `REQUIRED`, `SHALL`, `SHALL NOT`, `SHOULD`, `SHOULD NOT`, `RECOMMENDED`, `MAY`, and `OPTIONAL` in this document are to be interpreted as described in [RFC 2119](https://www.ietf.org/rfc/rfc2119.txt).

## Descriptor

A Fairspec Catalog is a [JSON Lines](https://jsonlines.org/) resource with each line represents a [Fairspec Dataset](../dataset) and its last updated time. The dataset locations `MUST` be unique within a catalog, except for the tombstone and entry that share an `upd` as [`del`](#del) sets out, and the datasets `MUST` be sorted by the last updated time in descending order.

A line with no characters at all is skipped and is not an entry, so an empty file is a valid catalog with no datasets.

## Catalog

Each line of the catalog `MUST` be a [Dataset](#dataset) object. Read in order, the lines form the array that the [profile](https://fairspec.org/profiles/latest/catalog.json) validates.

For example:

```jsonl
{"loc": "https://example.com/dataset1.json", "upd": "2023-10-01T00:00:00Z"}
{"loc": "https://example.com/dataset3.json", "upd": "2023-09-01T00:00:00Z"}
{"loc": "https://example.com/dataset2.json", "upd": "2023-08-01T00:00:00Z"}
```

## Dataset

A catalog entry pointing to a [Fairspec Dataset](../dataset). It `MUST` have the following properties (all required unless otherwise stated):

### `loc` [required] [#loc]

URL of the [Fairspec Dataset](../dataset) descriptor. The property `MUST` be an absolute HTTP or HTTPS URL: a [JSON Schema URI](https://json-schema.org/understanding-json-schema/reference/type#resource-identifiers) whose scheme is `http` or `https`, in any case as in [RFC 3986](https://datatracker.ietf.org/doc/html/rfc3986#section-3.1).

For example:

```json
{
  "loc": "https://example.com/dataset.json"
}
```

### `upd` [required] [#upd]

The last updated time of the dataset. The property `MUST` be [JSON Schema date-time](https://json-schema.org/understanding-json-schema/reference/type#dates-and-times) (this format requires a timezone component).

For example:

```json
{
  "upd": "2023-10-01T00:00:00Z"
}
```

### `del`

An optional boolean marking the dataset as removed from the catalog: a tombstone. It is sorted by `upd`, the time of the removal, like any other entry. A client that reads a tombstone `MUST` drop the dataset at `loc`. A publisher `SHOULD` keep a tombstone for as long as a client may still hold the dataset.

A dataset removed and added again within one timestamp has a tombstone and an entry with the same `loc` and `upd`. The later of the two lines wins: a client `MUST` keep the dataset when the entry comes after the tombstone, and drop it when the tombstone comes after the entry.

For example:

```json
{
  "loc": "https://example.com/dataset.json",
  "upd": "2023-11-01T00:00:00Z",
  "del": true
}
```

## Extension

> [!TIP]
> Additional properties are allowed.

Properties not defined by this specification are allowed, so that a descriptor written for a later version stays readable: an implementation `MUST` ignore them and `MAY` warn about them. Fairspec Catalog has no extension mechanism: such properties carry no meaning.

## Streaming

Fairspec Catalog is designed to be used in a streaming manner. Because datasets are sorted by their last updated time in descending order, a client `SHOULD` synchronize with a high-water mark: the newest `upd` it has processed. To synchronize, it reads the catalog line-by-line while `upd` is greater than or equal to the mark, and stops at the first line older than the mark; the new mark is the `upd` of the first line. Datasets tied at the mark are read again rather than skipped, since several datasets can share a timestamp and their order is not defined, so processing a dataset a client has already seen `MUST` be harmless. This allows efficient synchronization without processing the entire catalog file.

## Comparison

> [!NOTE]
> This section is under development.
