Skip to content

Observables

What are observables?

Observables are something that can be "observed" from a dataset. The definition of an observable is deliberately vague in order to allow for multiple use-cases. Typically, observables refer to dataset aggregates such as what values are represented, data health, etc. More advanced use-cases are also supported such as timeliness.

What value does observables bring?

Observables are primarily created to aid with search. Basically providing up-to-date metrics so that users may find datasets relevant to geography, values, etc.

Supported observables

AttributeObservable Abstract Base Class for observables relating to a specific attribute

StaticObservable Abstract Base Class for observables that are manually maintained

StaticCoverage 1D coverage of a specific attribute, i.e. min/max-values of an attribute (column, etc)

StaticGeometricCoverage Geometric coverage, i.e. bounding box or convex hull covering all datapoints in a dataset

How to work with observables

Interacting with observables are demonstrated in our SDK examples.

Examples of adding observables

Observable can be added through catalog's UI as time or geography:

img_32.png
Observables.

If we select time observable, we can select catalog's timeframe:

img_33.png
Time observable's prompt.

If we select geography observable, we can select catalog's location:

img_34.png
Geography observable's prompt.

To add observable to the dataset, you need to POST https://api.hubocean.earth/catalog with request body containing observable information. Here is an example with bounding box observable:

{
  "kind": "catalog.hubocean.io/observable",
  "version": "v1alpha2",
  "metadata": {
    "name": "test-bbox-observable"
  },
  "spec": {
    "ref": "catalog.hubocean.io/dataset/test",
    "observable_class": "catalog.hubocean.io/observableClass/static-geometric-coverage",
    "details": {
      "value": {
        "coordinates": [
          [
            [
              -3.5155975174915,
              58.66634174511648
            ],
            [
              4.188130631827647,
              61.29673427997298
            ],
            [
              8.885525844826816,
              53.98531912117838
            ],
            [
              2.684964163667672,
              52.524167960060595
            ],
            [
              -3.5155975174915,
              58.66634174511648
            ]
          ]
        ],
        "type": "Polygon"
      },
      "attribute": "coverage"
    }
  }
}

Note: spec.ref has to be catalog.hubocean.io/dataset/{dataset-name}

Location can also be added to the file. When we update file's metadata in PATCH https://api.hubocean.earth/data/{dataset_uuid}/{file_name}/metadata, we can update geo_location field with such example:

{
  name: "test.csv",
  mime_type: "text/csv",
  metadata: {
    "region": "Setubal",
    "date": "10/09/2024",
  },
  geo_location: {
    "type": "Feature",
    "geometry": {
      "type": "Point",
      "coordinates": [125.6, 10.1]
    },
    "properties": {
      "name": "Dinagat Islands"
    }
  }
}

Another complex location example is here.

Note: do pay attention to brackets when adding complex locations, as this part is error prone.

Here is an example how to add observable with SDK:

from odp.client import OdpClient
from odp.dto import Metadata
from odp.dto.catalog import ObservableDto, ObservableSpec

# Instantiate the client without specifying a token provider.
# The token provider will be set based on the environment.
client = OdpClient()

# Declare a new observable to be added to the data catalog
manifest = ObservableDto(
    metadata=Metadata(
        name=client.personalize_name("sdk-observable-example"),
        display_name="Test Observable for time",
        description="A test observable for time",
        labels={"hubocean.io/test": True},
    ),
    spec=ObservableSpec(
        # Update test-dataset to the dataset's name for whom observable will be added
        ref="catalog.hubocean.io/dataset/test-dataset", 
        observable_class="catalog.hubocean.io/observableClass/static-coverage",
        details={"value": [0, 1684147082], "attribute": "test"},
    ),
)

# The observable is created in the catalog.
# The return value is the full manifest of the created observable.
manifest = client.catalog.create(manifest)

We can also add location observable to the File. When creating FileMetadataDto, need to update geo_location parameter:

FileMetadataDto(
  name="test.csv",
  mime_type="text/csv",
  metadata={
    "region": "Setubal",
    "date": "10/09/2024",
  },
  geo_location={
    "type": "Feature",
    "geometry": {
      "type": "Point",
      "coordinates": [125.6, 10.1]
    },
    "properties": {
      "name": "Dinagat Islands"
    }
  }
)