Custom Metadata in Active

Active supports filtering, creating Collections, and by extension, creating Datasets and Projects based on the custom metadata on your data.

Prereqs

Before you can filter your data or create a Collection based on your data's custom metadata, the custom metadata must exist in your Annotate Project.

ℹ️
Note
This content applies to custom metadata (clientMetadata), which is the metadata associated with individual data units. This is distinct from videoMetadata that is used to specify video parameters when using Strict client-only access. It is also distinct from patient metadata in DICOM files.

Custom metadata (clientMetadata) is accessed by specifying the dataset using the <dataset_hash>. All Projects that have the specified Dataset attached contain custom metadata.

READ THIS FIRST

While not required, we strongly recommend importing a metadata schema before importing custom metadata into Encord. The process we recommend:

Import a metadata schema.
Import your custom metadata.

If a metadata schema already exists, you can import metadata. You can run a small piece of code to verify that a metadata schema exists.

ℹ️
Note
Performing multiple schema imports overwrites the current schema with the new schema.

Import metadata schema

Before importing your custom metadata to Encord, we recommend that you import a metadata schema. Encord uses metadata schemas to validate custom metadata uploaded to Encord and to instruct Active how to display your metadata.

Metadata schemas support the following data types for your metadata:

NUMBER = "number"
STRING = "string"
BOOLEAN = "boolean"
DATETIME = "datetime"

ℹ️
Note

Encord supports ONE metadata schema for each organisation. Performing multiple imports overwrites the current schema with the new schema.

NUMBER supports float values

Boolean supports "true" and "false"

DATETIME format is ISO 8601 date time (for example: yyyymmdd, yyyy-mm-dd, yyyy-mm-ddThh:mm:ss.msusps)

ℹ️
Note
Incorrectly specifying a data type in the schema can cause errors when filtering your data in Active. If you encounter errors while filtering, verify your schema is correct. If your schema has errors, correct the errors, re-import the schema, and then re-sync your Active Project.

Import your metadata schema to Encord


# Import dependencies
from encord import EncordUserClient

SSH_PATH = "<file-path-to-ssh-private-key>"

# Authenticate with Encord using the path to your private key
user_client: EncordUserClient = EncordUserClient.create_with_ssh_private_key(
    ssh_private_key_path=SSH_PATH
)

user_client.set_client_metadata_schema_from_dict({'metadata_1': 'data type', 'metadata_2': 'data type', 'metadata_3': 'data type'})


# Import dependencies
from encord import EncordUserClient

SSH_PATH = "/Users/chris-encord/ssh-private-key.txt"

# Authenticate with Encord using the path to your private key
user_client: EncordUserClient = EncordUserClient.create_with_ssh_private_key(
    ssh_private_key_path=SSH_PATH
)

user_client.set_client_metadata_schema_from_dict({'captured_at': 'datetime', 'city': 'datetime', 'dark': 'boolean'})

Verify your schema

After importing your schema to Encord we recommend that you verify that the import is successful. Run the following code to verify your metadata schema imported and that the schema is correct.


# Import dependencies
from encord import EncordUserClient

SSH_PATH = "<file-path-to-ssh-private-key>"

# Authenticate with Encord using the path to your private key
user_client: EncordUserClient = EncordUserClient.create_with_ssh_private_key(
    ssh_private_key_path=SSH_PATH
)

schema = client.get_client_metadata_schema


# Import dependencies
from encord import EncordUserClient

SSH_PATH = "/Users/chris-encord/ssh-private-key.txt"

# Authenticate with Encord using the path to your private key
user_client: EncordUserClient = EncordUserClient.create_with_ssh_private_key(
    ssh_private_key_path=SSH_PATH
)

schema = client.get_client_metadata_schema

List custom metadata

The following code lists the custom metadata of all data units in the specified Dataset. The code prints the custom metadata along with the data unit's index within the dataset.


# Import dependencies
from encord import EncordUserClient
from encord.client import DatasetAccessSettings

# Authenticate with Encord using the path to your private key
client = EncordUserClient.create_with_ssh_private_key(
    ssh_private_key_path="<private_key_path>"
)

# Specify a dataset to read or write metadata to
dataset = client.get_dataset("<dataset_hash>")

# Fetch the dataset's metadata
dataset.set_access_settings(DatasetAccessSettings(fetch_client_metadata=True))

# Read the metadata of all data units in the dataset.
for data_unit, data_row in enumerate(dataset.data_rows):
    print(f"{data_row.client_metadata} - Data Unit: {data_unit}")

Import custom metadata

❗️
CRITICAL INFORMATION
Before importing custom metadata to Encord, first import a metadata schema.

Import custom metadata to a specific data unit in your Annotate Project

You can import custom metadata (clientMetadata) to specific data units in the dataset.

👍
Tip
You can find the <data unit number> by reading all metadata in the dataset. The sample code provided prints all <data unit number>s.


# Import dependencies
from encord import EncordUserClient
from encord.client import DatasetAccessSettings

# Authenticate with Encord using the path to your private key
client = EncordUserClient.create_with_ssh_private_key(
    ssh_private_key_path="<private_key_path>"
)

# Specify a dataset to read or write metadata to
dataset = client.get_dataset("<dataset_hash>")

# Fetch the dataset's metadata
dataset.set_access_settings(DatasetAccessSettings(fetch_client_metadata=True))

# Add metadata to a specific data unit by replacing <data unit number> with the number of the data unit
data_row = dataset.data_rows[<data unit number>]

# Replace {"my": "metadata"} with the metadata you want to add
data_row.client_metadata= {"my": "metadata"}
data_row.save()
print(data_row.client_metadata)

Import custom metadata (`clientMetadata`) to all data units in a dataset

The following code adds the same custom metadata (clientMetadata) to each data unit in the specified dataset. The code prints the custom metadata along with the data units index within the dataset, so that you can verify that the custom metadata was set correctly.


# Import dependencies
from encord import EncordUserClient
from encord.client import DatasetAccessSettings

# Authenticate with Encord using the path to your private key
client = EncordUserClient.create_with_ssh_private_key(
    ssh_private_key_path="<private_key_path>"
)

# Specify a dataset to read or write metadata to
dataset = client.get_dataset("<dataset_hash>")

# Fetch the dataset's metadata
dataset.set_access_settings(DatasetAccessSettings(fetch_client_metadata=True))

# Add metadata to all data units in the dataset.
# Replace {"my": "metadata"} with the metadata you want to add
for data_unit, data_row in enumerate(dataset.data_rows):
    data_row.client_metadata = {"my": "metadata"}
    data_row.save()
    print(f"{data_row.client_metadata} - Data Unit: {data_unit}")