Import and verify custom metadata

ℹ️

Note

This content applies to custom metadata (clientMetadata), which is the metadata associated with individual data units. This is distinct from videoMetadata that is used to specify video parameters when using Strict client-only access. It is also distinct from patient metadata in DICOM files.

Custom metadata (clientMetadata) is accessed by specifying the dataset using the <dataset_hash>. All Projects that have the specified Dataset attached contain custom metadata.

READ THIS FIRST

While not required, we strongly recommend importing a metadata schema before importing custom metadata into Encord. The process we recommend:

  1. Import a metadata schema.
  2. Import your custom metadata.

If a metadata schema already exists, you can import metadata. You can run a small piece of code to verify that a metadata schema exists.

ℹ️

Note

Performing multiple schema imports overwrites the current schema with the new schema.

Import metadata schema

Before importing your custom metadata to Encord, we recommend that you import a metadata schema. Encord uses metadata schemas to validate custom metadata uploaded to Encord and to instruct Active how to display your metadata.

Metadata schemas support the following data types for your metadata:

  • NUMBER = "number"
  • STRING = "string"
  • BOOLEAN = "boolean"
  • DATETIME = "datetime"
  • GEOSPATIAL = "geospatial"

ℹ️

Note

  • Encord supports ONE metadata schema for each organisation. Performing multiple imports overwrites the current schema with the new schema.
  • NUMBER supports float values
  • Boolean supports "true" and "false"
  • DATETIME format is ISO date time (for example: yyyymmdd, yyyy-mm-dd, yyyy-mm-ddThh:mm:ss.msusps)

ℹ️

Note

Incorrectly specifying a data type in the schema can cause errors when filtering your data in Active. If you encounter errors while filtering, verify your schema is correct. If your schema has errors, correct the errors, re-import the schema, and then re-sync your Active Project.

Get your Org ID

Before you can add your metadata schema to Encord, you need to get your Organization's unique ID (org_id). The org_id is available from the address bar in Encord.

To get your org_id:

  1. Log on to the Encord platform with an account that has admin access.
    The Encord homepage appears.

  2. Click the Profile icon.
    A drop-down menu appears.

  3. Click Encord.
    The Encord platform page appears.

  4. Record your organization's ID from the browser address bar.

Import your metadata schema to Encord

Make sure you have your org_id before running this code.


# Import dependencies
from encord import EncordUserClient

SSH_PATH = "<file-path-to-ssh-private-key>"

# Authenticate with Encord using the path to your private key
user_client: EncordUserClient = EncordUserClient.create_with_ssh_private_key(
    ssh_private_key_path=SSH_PATH
)

user_client.set_client_metadata_schema_from_dict(org_id, {'metadata_1': 'data type', 'metadata_2': 'data type', 'metadata_3': 'data type'})



# Import dependencies
from encord import EncordUserClient

SSH_PATH = "/Users/chris-encord/ssh-private-key.txt"

# Authenticate with Encord using the path to your private key
user_client: EncordUserClient = EncordUserClient.create_with_ssh_private_key(
    ssh_private_key_path=SSH_PATH
)

user_client.set_client_metadata_schema_from_dict(42, {'captured_at': 'datetime', 'city': 'datetime', 'dark': 'boolean'})

Verify your schema

After importing your schema to Encord we recommend that you verify that the import is successful. Run the following code to verify your metadata schema imported and that the schema is correct.


# Import dependencies
from encord import EncordUserClient

SSH_PATH = "<file-path-to-ssh-private-key>"

# Authenticate with Encord using the path to your private key
user_client: EncordUserClient = EncordUserClient.create_with_ssh_private_key(
    ssh_private_key_path=SSH_PATH
)

schema = client.get_client_metadata_schema(org-id)



# Import dependencies
from encord import EncordUserClient

SSH_PATH = "/Users/chris-encord/ssh-private-key.txt"

# Authenticate with Encord using the path to your private key
user_client: EncordUserClient = EncordUserClient.create_with_ssh_private_key(
    ssh_private_key_path=SSH_PATH
)

schema = client.get_client_metadata_schema(42)

List custom metadata

The following code lists the custom metadata of all data units in the specified Dataset. The code prints the custom metadata along with the data unit's index within the dataset. 


# Import dependencies
from encord import EncordUserClient
from encord.client import DatasetAccessSettings

# Authenticate with Encord using the path to your private key
client = EncordUserClient.create_with_ssh_private_key(
    ssh_private_key_path="<private_key_path>"
)

# Specify a dataset to read or write metadata to
dataset = client.get_dataset("<dataset_hash>")

# Fetch the dataset's metadata
dataset.set_access_settings(DatasetAccessSettings(fetch_client_metadata=True))

# Read the metadata of all data units in the dataset.
for data_unit, data_row in enumerate(dataset.data_rows):
    print(f"{data_row.client_metadata} - Data Unit: {data_unit}")

Import custom metadata

❗️

CRITICAL INFORMATION

Before importing custom metadata to Encord, first import a metadata schema.

Import custom metadata to a specific data unit in your Annotate Project

You can import custom metadata (clientMetadata) to specific data units in the dataset.

👍

Tip

You can find the <data unit number> by reading all metadata in the dataset. The sample code provided prints all <data unit number>s.


# Import dependencies
from encord import EncordUserClient
from encord.client import DatasetAccessSettings

# Authenticate with Encord using the path to your private key
client = EncordUserClient.create_with_ssh_private_key(
    ssh_private_key_path="<private_key_path>"
)

# Specify a dataset to read or write metadata to
dataset = client.get_dataset("<dataset_hash>")

# Fetch the dataset's metadata
dataset.set_access_settings(DatasetAccessSettings(fetch_client_metadata=True))

# Add metadata to a specific data unit by replacing <data unit number> with the number of the data unit
data_row = dataset.data_rows[<data unit number>]

# Replace {"my": "metadata"} with the metadata you want to add
data_row.client_metadata= {"my": "metadata"}
data_row.save()
print(data_row.client_metadata)

Import custom metadata (clientMetadata) to all data units in a dataset

The following code adds the same custom metadata (clientMetadata) to each data unit in the specified dataset. The code prints the custom metadata along with the data units index within the dataset, so that you can verify that the custom metadata was set correctly. 


# Import dependencies
from encord import EncordUserClient
from encord.client import DatasetAccessSettings

# Authenticate with Encord using the path to your private key
client = EncordUserClient.create_with_ssh_private_key(
    ssh_private_key_path="<private_key_path>"
)

# Specify a dataset to read or write metadata to
dataset = client.get_dataset("<dataset_hash>")

# Fetch the dataset's metadata
dataset.set_access_settings(DatasetAccessSettings(fetch_client_metadata=True))

# Add metadata to all data units in the dataset.
# Replace {"my": "metadata"} with the metadata you want to add
for data_unit, data_row in enumerate(dataset.data_rows):
    data_row.client_metadata = {"my": "metadata"}
    data_row.save()
    print(f"{data_row.client_metadata} - Data Unit: {data_unit}")