AIMABEC

Annotators and Training

We provide trainings and upskilling opportunities on the subject of AI and also provide services related to the development of AI models through our social enterprise. We are conscious that each person comes with their own sets of values, beliefs, and personal biases, and we raise awareness among our annotators about the potential biases they might be exhibiting and transmitting to the data while performing the annotation. The training module on “Bias in AI” on the Humans in the Loop Training Center should be completed by all annotators in order to be eligible for annotation projects by earning the badge of “Ethical AI Annotator”.

• However, we also promote a view of annotation as interpretative and collaborative work where a shared understanding must be built between annotators and the client.

• We promote the use of clear guidelines, open communication, edge case examples, as well as straightforward annotation interfaces and processes, as ways to mitigate potential biases.

Humans in the Loop believes that informed annotators are empowered to make better decisions when annotating the dataset. Therefore we always share the background information of the AI project and its purpose in the Guidelines which are shared with annotators.

Best Practice

Special attention is paid to the following projects, with the following best practices for each one:

• Projects which require the annotation of subjective human self-held characteristics and protected attributes, such as gender, race, disability, and others.

• Instead of annotating race or ethnicity, skin color should be annotated according to a widely-accepted dermatological convention, in addition to other objective characteristics such as the presence of an eye crease.

• Instead of annotating gender (especially with binary variables), context-aware labels should be used for objective annotation, such as beard, makeup, shirt

• Projects which require the annotation of image-level tags for classification purposes should be discouraged given the challenges in interpretability or model predictions.

• We promote the use of clear guidelines, open communication, edge case examples, as well as straightforward annotation interfaces and processes, as ways to mitigate potential biases.