Consensus Workflows (BETA)

Suggest Edits
Aligned Image with Page Break

Consensus, in Encord, provides the ability for one or more reviewers to directly compare the labels on a data unit, generated by 2 or more annotators, for agreement/correctness on their labels. Annotators do not see what other annotators have labeled.

If consensus is not reached, labels can be directed to a second Review block, where a reviewer can adjust any labels on a data unit from Consensus block or add additional labels to the data unit.

❗️

CRITICAL INFORMATION

Consensus in Encord is based on Consensus for ALL labels on a data unit. A data unit being an image, an image group, an image sequence, a video, or a DICOM series. Encord Consensus is not granular to the label itself (yet). This means reviewers must ensure ALL labels on a data unit are in consensus before deciding consensus is reached.

Consensus supports Objects and Classifications on images, image groups, image sequences, videos.

Consensus basics

Workflows direct who and how annotations, and reviews of those annotations, are performed on data units. The addition of the Consensus and Archive blocks brings the ability to get consensus on your annotations.

Consensus blocks include an ANNOTATE and REVIEW stage in a single block. The following settings must be configured for a Consensus block:

  • # of annotators required for task: Specify the number of annotators required to label a task (data unit) for the task to move to the REVIEW stage in the Consensus block.

  • # of annotators required to agree: Specify the number of annotators that a reviewer decides have labels that are correct and in agreement on a task.

  • Specify annotators and reviewers in the ANNOTATE and REVIEW sections of Consensus blocks. On a Consensus Project, Admins/Team Managers must specify the pool of annotators and reviewers for annotating and reviewing. This must be done for each Consensus block in a Consensus Project.

Archive blocks provide a location for any task that reviewers deem "not in consensus".

Review blocks added to the No consensus output, of a Consensus block, allow reviewers to approve and refine/edit all labels, from all annotators from the Consensus block.

Tasks move through Consensus Projects in the following way:

  1. When the number of annotators specified in # of annotators required for task label the task, the task moves to REVIEW.

  2. Reviewers compare the labels on the task and decide on the correctness and annotator agreement specified in # of annotators required to agree. The task moves to the next stage based on label correctness and whether consensus is reached.

  3. Adding multiple Consensus blocks in sequence, means all labels from all previous Consensus blocks display for the reviewers in each subsequent Consensus block.

Encord provides the following default Consensus Workflow, when creating a Project:

The default Consensus Workflow Consensus uses 2 Consensus blocks with the second block acting as a "tie-breaker".

ℹ️

Note

When using any Consensus Workflow in a Project, do not forget to add your annotators and reviewers to the ANNOTATE and REVIEW sections. Also, if needed, edit the values for # of annotators required for task and # of annotators required to agree, for each Consensus block.

Create a Consensus Project

  1. Log in to Encord.
    The Encord homepage appears.

  2. Go to Annotate > Projects > Annotation projects.
    The Annotation projects page appears.

  3. Click the New annotation project button.
    The Create new project page appears.

  4. Click the Load workflow template button.
    The Load from workflow template modal appears.

  5. Select a Workflow template that uses Consensus blocks from the workflow template list.
    Consensus, the default Consensus Workflow, appears at the top of the template list.

  6. Click Load template.
    The Workflow template appears in the Project workspace.

  7. Add your annotators and reviewers to the ANNOTATE and REVIEW sections of the Consensus block.

  8. Specify the required value for # of annotators required for task.

  9. Specify the required value for # of annotators required to agree.

  10. Complete configuration of the Project.

  11. Click Create project.
    The Project appears in the Annotation project list.

Consensus Advanced

When the default Consensus Workflow template does not suit your requirements, build your own Consensus Workflow template. You can add multiple Consensus, Annotate, Review, and Router blocks to a Workflow.

Here are a couple of examples.

One Consensus Block Workflow

In this Consensus Workflow:

  • Total # of annotators: 6
  • Total # of reviewers: 2
  • # of annotators required for task: 4
  • # of annotators required to agree: 3

The following diagrams follow a task through the Consensus Workflow until the labels on the task are correct and in consensus or incorrect and not in consensus.

No Consensus

Consensus Reached

Two Consensus Blocks Workflow

In this Consensus Workflow:

Consensus Block 1:

  • Total # of annotators: 6
  • Total # of reviewers: 2
  • # of annotators required for task: 4
  • # of annotators required to agree: 3
  • Reviewers see labels from all 4 annotators.

Consensus Block 2:

  • Total # of expert annotators: 3
  • Total # of expert reviewers: 2
  • # of annotators required for task: 1
  • # of annotators required to agree: 3
  • Expert reviewers see labels from all 5 annotators (4 from Consensus Block 1 and 1 from Consensus Block 2).

The following diagrams follow a task through the Consensus Workflow until the labels on the task are correct and in consensus or incorrect and not in consensus. If consensus happens in the first Consensus block, the task goes directly to Complete.

Two Consensus Blocks - No Consensus

Two Consensus Blocks - Immediate Consensus

Two Consensus Blocks - Consensus

Review and Refine Consensus labels

Adding a REVIEW block to the No consensus output of a CONSENSUS block changes the REVIEW block into a REVIEW and REFINE block. CONSENSUS blocks allow reviewers to approve the best labels from one annotator on a data unit, but reviewers cannot select the best specific labels or edit specific labels on a data unit. The REVIEW and REFINE block allows reviewers to select and edit the best labels from any annotator and to add new labels to a data unit.

Consensus with Review and Refine Block Workflow

In this Consensus Workflow:

Consensus Block 1:

  • Total # of annotators: 6
  • Total # of reviewers: 2
  • # of annotators required for task: 4
  • # of annotators required to agree: 3
  • Reviewers see labels from all 4 annotators.

Consensus Block 2:

  • Total # of expert annotators: 3
  • Total # of expert reviewers: 2
  • # of annotators required for task: 1
  • # of annotators required to agree: 3
  • Expert reviewers see labels from all 5 annotators (4 from Consensus Block 1 and 1 from Consensus Block 2).

Review Block

  • Total number of reviewers: 2
  • Reviewers can edit the "best representative" set of labels on a data unit.

Review and Refine Block:

  • Total number of reviewers: 2
  • Reviewers can select and edit the best specific labels on any object or classification label from any of the 5 annotators (4 from Consensus Block 1 and 1 from Consensus Block 2). Reviewers can also create new labels for the data unit.

Create a Consensus Workflow Template

To create a consensus Workflow template, your template must include one (or more) Consensus block and one Archive block. Consensus blocks can be used with Annotate Percentage Router blocks and Review blocks to build out your Workflow template.

👍

Tip

For more information on creating Workflow templates see our documentation here.

Example Consensus Workflow Template

In the following Workflow we have 2 Consensus blocks, an Archive block, a Percentage Router, and a Reviewer block.

The first Consensus block requires each data unit to be labelled by 3 different annotators and consensus is only reached when the reviewer decides that all three reviewers agree/have correctly annotated the data unit.

If consensus is reached, the task goes to the Percentage Router. 25% of the approved tasks go to a Review Audit. Any task rejected during the audit go directly to the second Consensus block for annotation and review.

The second Consensus block requires each data unit to be labelled by 1 expert annotator. The expert annotator from the second Consensus block is added to the annotators from the first Consensus block and consensus is only reached when the reviewer decides that three of the reviewers agree/have correctly annotated the data unit. Rejected tasks and tasks where consensus is not reached go to the second Consenus block for annotation and review. Any tasks that reach consensus go to the Complete stage. Any tasks that do not reach consensus go to the Archive block. The fate of any tasks in the Archive block is then decided by a Team Manager or Admin.

To create a Consensus Workflow template:

  1. Log in to Encord.
    The Encord homepage appears.

  2. Go to Annotate > Workflow templates.
    The Workflow templates page appears.

  3. Click the New workflow template button.
    The Create Workflow template page appears.

  4. Click the Invite collaborators icon.
    The Workflow Collaborators dialog appears.

  5. Add the required collaborators for the template.

    ℹ️

    Note

    Remember that consensus requires at least 2 annotators for the first Consensus block, and we recommend at least 2 reviewers per Consensus block.

  6. Click the Add stage icon.
    The Add stage dailog appears.

  7. Drag and drop at least 1 Consensus block and 1 Archive block into the template creation workspace.

  8. Connect the No Consensus output of the Consensus block (the final Consensus block if you have more than one Consensus block) to the input of the Archive block.

  9. Add other blocks to the template as you require.

  10. In each block, specify specific collaborators to perform tasks.

  11. Specify a value for # of annotators required for task for the Annotate portion of the Consensus blocks.

ℹ️

Note

# of annotators required for task is the number of annotators required to annotate a task (data unit) before the task (data unit) moves to the Review stage in the Consensus block.
If you use 2 or more Consensus blocks in your workflow, we recommend specifying only 1 annotator for your second and subsequent Consensus blocks. This is because all annotators, and their annotations, from previous blocks appear in the review of subsequent reviews.

  1. Specify a value for # of annotators required to agree for the Review stage in the Consensus blocks.

ℹ️

Note

# of annotators required to agree is the number of annotators required to agree on ALL labels on a data unit before consensus is reached. Reviewers decide if annotators have correctly labeled a data unit and if the annotators are in agreement.

  1. Complete configuration of the template.

  2. Save the template.

What do different user roles see?

Annotators

When working on consensus Projects, annotators have no changes to their workflow or within the Encord platform. Annotators simply need to annotate the tasks (data units) in their queue.

Annotators do not see the work of other annotators as they annotate their tasks in Consensus Projects.

Reviewers

When reviewing labels from a REVIEW block on Consensus Projects, reviewers have a new UI when they initiate a task for review.

ℹ️

Note

The only annotator's work/labels that display in audit Review blocks or the Complete block, is the annotator you specify as the "best annotator".

When reviewing and refining labels from a REVIEW and REFINE block, reviewers see all the labels from all annotators.

Admins and Team Managers

Admins and Team Managers can do the following:

  • Act as a reviewer on Review tasks
  • View tasks
  • Monitor the entire queue (Annotate and Review tasks). This includes which tasks are in progress in terms of the number of annotators working on a task with the number of annotators required to finish a task.
  • Reassign annotation tasks

Admins and Team Managers CANNOT annotate tasks in a Consensus Workflow.

Export Labels from Consensus Projects

❗️

CRITICAL INFORMATION

Consensus Projects export ONLY the labels that come from the annotator's that are marked as "Best" on each data unit by reviewers. This means that only the best labels are exported for your ML training.

To export labels from Consensus Projects

  1. Log in to Encord.
    The Encord homepage appears.

  2. Go to Annotate > Projects > Annotate projects.
    The Annotate projects page appears.

  3. Click a Consensus Project from the Project list.
    The Summary page for the Project appears.

  4. Click Labels & Export.
    The Data tab appears on the Labels & Export page.

  5. Filter the data to only display data from the Complete stage.
    Only data from the Complete stage appears in the list.

  6. Select the data units you want to export the labels from.

  7. Click the Select all XX rows with labels button.

  8. Click Export & Save.
    The Export labels and save a version dialog appears.

  9. Provide a meaningful name for the version of the export.

  10. Specify the format, label types for export, whether to generate signed URLs, and specify the Label approval status.

  11. Click Export and save.
    A message appears showing the export progress.

  12. Click the Download when the collection of labels completes or go to the Versions page and download the labels from Versions page.

Reopen/Reassign Consensus Annotate tasks

As annotators work through their queue of annotation tasks, each task is "locked" to the annotator that first opened that specific task. The task is no longer visible to other annotators in the Project. In a Project that uses a Consensus Workflow, tasks get locked once multiple annotators (as specified by the number of annotators required to annotate a task in a Consensus block) open a specific task. All annotators must complete the annotation task before the task can move to the Review stage. Admins and Team Managers can reassign/reopen a task. Reopening the task makes the task visible to all annotators in the Project, and any annotator in the Project can then work on the task. Reassigning the task allows Admins or Team Managers to specify the annotator that works on the task.

To reopen/reassign an Annotate task:

  1. Log in to Encord.
    The Encord homepage appears.

  2. Go to Annotate > Projects > Annotate projects.
    The Annotate projects page appears.

  3. Click a Consensus Project from the Project list.
    The Summary page for the Project appears.

  4. Click the Annotate block with the task you want reassigned.

    Reopen/Reassign Task

  5. Expand the task you want to reassign.
    The row expands to display the users currently assigned to the task.

  6. To reopen the task to any annotator on the Project:

    a. Click the 3 dots icon next to the annotator you want the task reopened from.
    The Reopen button appears.

    b. Click Reopen.
    The task becomes visible to all annoatators in the Annotate queue. Any annotator can then "lock" the task (including the annotator the task was reopened from).

  7. To assign a specific annotator to the task:

    a. Click the down arrow next to the annotator you want the task reassigned from.
    A drop down menu appears with all annotators available on the Project

    b. Click the down arrow next to the user you want changed.
    A drop down menu appears with other annotators available to perform the task.

    c. Select the annotator you want the task assigned to.

Updated 1 day ago