GCP storage

Suggest Edits

There are two parts to setting up a GCP integration in Encord:

  1. Setting up a GCP bucket on the Google Cloud Platform so that it can be integrated with Encord. This includes setting up a CORS configuration for your bucket.

ℹ️

Note

Ensure all objects in the bucket have the Standard Storage storage class.

  1. Integrating your GCP bucket with Encord's platform. Strict client-only access can also be added in this step if necessary.

1. Setting up a GCP bucket

To get started, please log into your Google Cloud account and navigate to the GCP bucket you wish to integrate with Encord.

Creating a service account for Encord

In the bucket's PERMISSION tab, click Grant Access to add a service account for encord.

Add 'cord-cloud-storage-integrator@cord-ai-platform.iam.gserviceaccount.com' as a New Principal and assign the following roles:

  • Storage Object Viewer

  • Storage Object Creator

Creating a CORS configuration

A CORS configuration needs to be applied to the bucket you wish to integrate with Encord. This will enable your user's browsers to request resources from the specified bucket.

Example gsutil command

Below is an example gsutil command to be used in your command line or terminal to create a CORS configuration.

ℹ️

Note

The only way to create a CORS configuration is using the command line, or terminal tool.

gsutil cors set example_cors_file.json gs://example_bucket
  • Replace 'example_bucket' with the name of the bucket you wish to integrate.
  • Replace 'example_cors_file' with the name of your CORS file. A sample JSON file is provided.

JSON configuration file

Below is a simple JSON file that should work with your GCP bucket. 

[
    {
      "origin": ["https://app.encord.com", "https://api.encord.com", "https://dicom.encord.com"],
      "method": ["GET"],
      "responseHeader": ["Content-Type"],
      "maxAgeSeconds": 3600
    }
]

ℹ️

Note

  • Preflight requests are valid for 1 hour. Use the maxAgeSeconds variable to adjust the number of seconds the browser is allowed to make requests before it must repeat the preflight request.
  • Successful browser requests return the Content-Type of the resource in the response, as set by the responseHeader.

To customize the file further please consult the GCP documentation.

2. Creating the integration on Encord

Now that your GCP storage has been set up and configured, you are ready to create the integration in the Encord platform.

In the Integrations section of the Encord platform click Add integration to create a new integration.

Click + Add integration.

Create the integration by selecting GCP at the top of the chooser. Enter a name for the integration, and enter the name of the bucket you wish to make available in the second dropdown of the GCP integration window.

Optionally check the box to enable Strict client-only access, server-side media features will not be available if you would like Encord to sign URLs, but refrain from downloading any media files onto Encord servers. Read more about this feature here.

Click Create to create the GCP integration.

3. Testing your integration

Click the Run a URL test button to test the integration.

Paste the URL of any object in the bucket and click Check Encord can access this URL. If the test is successful a green tick appears next to Encord infrastructure and This machine.

Creating a Dataset with GCP data

We recommend setting the expiration time for presigned URLs, in your cloud storage settings, to be greater than the time it takes to complete an annotation task. More information can be found in the GCP documentation.

Please navigate to the private cloud integration page for guidance on how to create a dataset with data stored in GCP.

Updated 3 months ago