API: openeo.extra.artifacts

Added in version 0.45.0.

Warning

This is a new experimental API, subject to change.

Important

The artifacts functionality relies on extra Python packages. They can be installed using:

pip install "openeo[artifacts]" --upgrade

When running openEO jobs it is not uncommon to require artifacts that should be accessible during job execution. This requires the artifacts to be accessible from within the openEO processing environment. openeo.extra.artifacts tries to perform the heavy lifting for this use case by allowing staging artifacts to a secure but temporary location using 3 simple steps:

  1. Connect to your openEO backend

  2. Create an artifact helper from your openEO connection

  3. Upload your file using the artifact helper and optionally get a presigned URI

So in code this looks like:

import openeo
from openeo.extra.artifacts import build_artifact_helper

connection = openeo.connect("my-openeo.prod.example").authenticate_oidc()

artifact_helper = build_artifact_helper(connection)
storage_uri = artifact_helper.upload_file(path, object_name)
presigned_uri = artifact_helper.get_presigned_url(storage_uri)

Note:

  • The presigned_uri should be used for accessing the objects. It has authentication details embedded so if your data is sensitive you must make sure to keep this URL secret. You can lower expires_in_seconds in openeo.extra.artifacts._artifact_helper_abc.ArtifactHelperABC.get_presigned_url() to limit the time window in which the URI can be used.

  • The openEO backend must expose additional metadata in its capabilities doc to make this possible. Implementers of a backend can check the extra documentation For backend providers.

User facing API

openeo.extra.artifacts.build_artifact_helper(connection, config=None)[source]
Parameters:

  • connection (Connection) – openeo.Connection connection to an openEOBackend

  • config (Optional[ArtifactsStorageConfigABC]) – Optional This parameter should only be used when instructed by the maintainer of the OpenEO backend. object to specify configuration for Artifact storage. If omitted the helper will try to get the preferred config as advertised by the OpenEO backend.

Return type:

ArtifactHelperABC

Returns:

An Artifact helper instance that can be used to manage artifacts

class openeo.extra.artifacts._artifact_helper_abc.ArtifactHelperABC(config)[source]

This class defines the interface that an artifact helper should implement and support. This is used by OpenEO users willing to manage artifacts.

Instances that implement it get created by the openeo.extra.artifacts.build_artifact_helper factory

abstractmethod get_presigned_url(storage_uri, expires_in_seconds=604800)[source]

A method to get a signed https URL for a given StorageURI which can be accessed via normal http libraries.

These URIs should be kept secret as they provide access to the data.

Parameters:

  • storage_uri (StorageURI) – URI to the artifact that is stored by a previous upload_file call

  • expires_in_seconds (int) – Optional how long expressed in seconds before the returned signed URL becomes invalid

Return type:

str

Returns:

The signed https URI.

abstractmethod upload_file(path, object_name='')[source]

A method to store an artifact remotely and get a StorageURI which points to the stored data.

Parameters:

  • path (str | Path) – Location of the file to be uploaded absolute path or relative to current working directory.

  • object_name (str) – Optional name you want to give to the object. If not specified the filename will be used.

Return type:

StorageURI

Returns:

If you want to use the StorageURI in a processgraph convert it using Python’s built-in str() function which is understood by the OpenEO processor.

How does it work ?

  1. openeo.extra.artifacts.build_artifact_helper() is a factory method that will create an artifact helper where the type is defined by the config type. The openEO connection object is used to see if the openEO backend advertises a preferred config.

  2. openeo.extra.artifacts._artifact_helper_abc.ArtifactHelperABC.upload_file() and openeo.extra.artifacts._artifact_helper_abc.ArtifactHelperABC.get_presigned_url() do the heavy lifting to store your artifact in provider managed storage and to return references that can be used. In case the backend uses an Object storage that has an S3 API it will:

    1. Get temporary S3 credentials based on config advertised by the backend and the session from your connection

    2. Upload the file into object storage and return an S3 URI which the backend can resolve

    3. Optional the openeo.extra.artifacts._artifact_helper_abc.ArtifactHelperABC.get_presigned_url() makes a URI signed with the temporary credentials such that it works standalone (Some tools and execution steps do not support handling of internal references. presigned URLs should work in any tool).

For backend providers

Warning

Investigation is ongoing whether this would be a good fit for the workspace extension https://github.com/Open-EO/openeo-api/issues/566 which would mean a big overhaul for backend implementations. If you are a backend provider and interested in this feature please create an issue to allow collaboration.

Artifacts exceptions

When using artifacts your interactions can result in the following exceptions.

exception openeo.extra.artifacts.exceptions.ArtifactsException[source]

Family of exceptions related to artifacts

exception openeo.extra.artifacts.exceptions.NoAdvertisedProviders[source]

The providers list is empty. This OpenEO backend does not seem to support artifacts or at least not advertise its capabilities.

exception openeo.extra.artifacts.exceptions.UnsupportedArtifactsType(type_id)[source]

The artifacts type is not supported. This should only occur if you specify config manually and the artifact type is not supported by the backend.

Parameters:

type_id (str) – The type_id that is not supported but which was attempted to be used

exception openeo.extra.artifacts.exceptions.NoDefaultConfig(key)[source]

This OpenEO backend does not seem to advertise a default value for a config parameter. This is likely a bug in the capabilities advertised by the backend. If you are not specify config manually you should contact support of the backend provider.

Parameters:

key (str) – The key for which no config value was found

exception openeo.extra.artifacts.exceptions.InvalidProviderConfig[source]

The backend has an invalid provider config. This must be fixed by the provider of the backend. If you are not specify config manually you should contact support of the backend provider.

exception openeo.extra.artifacts.exceptions.ProviderSpecificException[source]

This is an exception specific to the type of artifact that is used. The exception itself will contain additional information.