Review Consensus Projects (BETA)

Suggest Edits
Aligned Image with Page Break

Consensus Projects have the following types of reviewers:

  • REVIEW blocks nested inside CONSENSUS blocks: Reviewers in these blocks are responsible for ensuring that labels made by one or more annotators are correct across an entire data unit.
  • REVIEW and REFINE blocks connected to No consensus output from CONSENSUS blocks: These reviewers can select and edit the best specific label on an object or frame, from one or more annotators.

Review Labels for Consensus

Reveiwers on Consensus Projects have a different goal from Reviewers on other types of Encord Projects. Reviewers on Consensus Projects review ALL the labels on a data unit, made by a specified number of annotators. The reviewers then make a decision on whether there is agreement/consensus on ALL the labels on a task (data unit).

If the reviewer decides that ALL the labels on a data unit are correct/in agreement, the reviewer selects the annotators that are in agreement and clicks Consensus reached. If the reviewer decides that there is no agreement on the labels or that the labels are incorrect, the reviewer clicks No Consensus.

Currently, Object labels and Classifications are supported for consensus review on all data unit types. Data units are images, image groups, image sequences, videos, and DICOM series.

To review labels on a Consensus Project:

  1. Log in to Encord.
    Your review task queue appears for the latest Project you are assigned for review.

  2. Go to Annotate > Projects > Annotate projects.
    The Annotate projects page appears displaying a list of Projects you are assigned to review.

  3. Click a Consensus Project.
    The Project opens with the Review populated with review tasks.

  4. Click Initiate on a task.
    The Reviewing page appears.

  5. Review the labels from each annotator.

❗️

CRITICAL INFORMATION

Consensus in Encord is based on Consensus for ALL labels on a data unit. A data unit being an image, an image group, an image sequence, a video, or a DICOM series. Encord Consensus is not granular to the label itself (yet). This means reviewers must ensure ALL labels on a data unit are in consensus before deciding consensus is reached.

The Reviewing page is different for single images and all other data units. When reviewing single images, expand each user to display their labels and decide on consensus. When reviewing videos, image groups, image sequences, and DICOM series a timeline appears under the data unit. Use the timeline to speed up reviewing all the labels on the data unit.

Single Image, Image Group

Video, Image Sequence, DICOM Series

  1. If the labels are correct:

a. Select the annotators that have done the best job of labeling the data unit.
The Consensus reached button becomes clickable once the required number of annotators required are selected.

b. Click the Consensus reached button.

c. Select the annotator that did the best job. That annotator's work on the data unit is used if an Admin or Team Manager export the labels for the data unit.

d. Click Submit.

Single Image, Image Group

Video, Image Sequence, DICOM Series

  1. Click No consensus if the labels are not correct OR if not enough of the annotators have labeled the data unit correctly.

Review and Refine Consensus Labels

Reviewers working in the Review and Refine block can see all labels from all annotators from a Consensus block. The reviewer is able to select the best labels from each object on a data unit and edit those labels or they can create new labels for the data unit.

The Review and Refine page is different for single images and all other data units. When reviewing and refining single images, expand each user to display their labels and decide on the best labels. When reviewing and refining videos, image groups, image sequences, and DICOM series, a timeline appears under the data unit. Use the timeline to speed up reviewing and refining all the labels on the data unit.

To review and refine labels from a Consensus Project:

Only data units for which consensus is not reached can be reviewed and refined.

  1. Log in to Encord.
    Your review and refine task queue appears for the latest Project you are assigned for review and refinement.

  2. Go to Annotate > Projects > Annotate projects.
    The Annotate projects page appears displaying a list of Projects you are assigned to review and refine.

  3. Click a Consensus Project.
    The Project opens with Review and Refine populated with tasks.

  4. Click Initiate on a task.
    The Review and refine page appears.

  5. Review and refine the labels on each data unit.

Single Image, Image Group

Video, Image Sequence, DICOM Series

  1. Click Edit approved labels to add new labels to the data unit.

  2. Select the best labels for each object on the data unit.

  3. Click Approve labels.
    The labels appear in the Approved labels list.

  4. Select a label from the Approved labels list.

  5. Click Edit approved labels if the labels need to be adjusted.

  6. Click Approve task after selecting, refining, and creating the labels needed for the data unit.

  7. Click Reject task if the labels on the data unit cannot be added or corrected so the labels on the data unit meet your needs.

Updated 2 days ago