The project started with an efficient selection of subject matter experts from DataForce’s community of one million people. We built the team gradually from screened candidates, handpicking the reviewers who had a background in pharmacovigilance. A combination of expert supervision, a blind annotation process, and the implementation of specific quality metrics such as Kohen’s Kappa coefficient resulted in quick quality improvements. As the annotation process continued, we scaled up the annotation team to shorten the deliverycycle. Our team reviewed thousands of social media posts in a few days.
To guarantee accuracy in annotation and categorization, we performed double-blind annotation on each task, followed by thorough quality controls. Any disagreements were reconciled by a reviewer who came up with the final categorization for each entry.
We implemented feedback promptly with the team as more data sets were coming in, and updated guidelines to reflect edge cases and unexpected experiences within social media posts. Our internal annotation platform, DataForce, updated seamlessly to adapt to all project changes and allowed us to complete the dataset with a scalable model of constant improvement.
The client reported that this human-in-the-loop model proved to be a much more scalable and efficient option than traditional manual categorization.