Crowd-Kit: Computational Quality Control for Crowdsourcing
Crowd-Kit is a powerful Python library that implements commonly-used aggregation methods for crowdsourced annotation and offers the relevant metrics and datasets. We strive to implement functionality that simplifies working with crowdsourced data.
Currently, Crowd-Kit contains:
- implementations of commonly-used aggregation methods for categorical, pairwise, textual, and segmentation responses
- metrics of uncertainty, consistency, and agreement with aggregate
- loaders for popular crowdsourced datasets
The library is currently in a heavy development state, and interfaces are subject to change.
Installing
Installing Crowd-Kit is as easy as pip install crowd-kit
Getting Started
This example shows how to use Crowd-Kit for categorical aggregation using the classical Dawid-Skene algorithm.
First, let us do all the necessary imports.
from crowdkit.aggregation import DawidSkene
from crowdkit.datasets import load_dataset
import pandas as pd
Then, you need to read your annotations into Pandas DataFrame with columns task, performer, label. Alternatively, you can download an example dataset.
df = pd.read_csv('results.csv') # should contain columns: task, performer, label
# df, ground_truth = load_dataset('relevance-2') # or download an example dataset
Then you can aggregate the performer responses as easily as in scikit-learn:
aggregated_labels = DawidSkene(n_iter=100).fit_predict(df)
Implemented Aggregation Methods
Below is the list of currently implemented methods, including the already available (
Categorical Responses
| Method | Status |
|---|---|
| Majority Vote | |
| Dawid-Skene | |
| Gold Majority Vote | |
| M-MSR | |
| Wawa | |
| Zero-Based Skill | |
| GLAD | |
| BCC | |
Textual Responses
| Method | Status |
|---|---|
| RASA | |
| HRRASA | |
| ROVER | |
Image Segmentation
| Method | Status |
|---|---|
| Segmentation MV | |
| Segmentation RASA | |
| Segmentation EM | |
Pairwise Comparisons
| Method | Status |
|---|---|
| Bradley-Terry | |
| Noisy Bradley-Terry | |
Citation
- Ustalov D., Pavlichenko N., Losev V., Giliazev I., and Tulin E. A General-Purpose Crowdsourcing Computational Quality Control Toolkit for Python. The Ninth AAAI Conference on Human Computation and Crowdsourcing: Works-in-Progress and Demonstration Track. HCOMP 2021. 2021. arXiv: 2109.08584 [cs.HC].
@inproceedings{HCOMP2021/CrowdKit,
author = {Ustalov, Dmitry and Pavlichenko, Nikita and Losev, Vladimir and Giliazev, Iulian and Tulin, Evgeny},
title = {{A General-Purpose Crowdsourcing Computational Quality Control Toolkit for Python}},
year = {2021},
booktitle = {The Ninth AAAI Conference on Human Computation and Crowdsourcing: Works-in-Progress and Demonstration Track},
series = {HCOMP~2021},
eprint = {2109.08584},
eprinttype = {arxiv},
eprintclass = {cs.HC},
url = {https://www.humancomputation.com/assets/wips_demos/HCOMP_2021_paper_85.pdf},
language = {english},
}
Questions and Bug Reports
- For reporting bugs please use the Toloka/bugreport page.
- Join our English-speaking slack community for both tech and abstract questions.
License
© YANDEX LLC, 2020-2021. Licensed under the Apache License, Version 2.0. See LICENSE file for more details.
16 Feb 06, 2022
1 Jan 13, 2022
4 Aug 07, 2022
3 May 17, 2022
3 Jan 10, 2022
3.3k Jan 07, 2023
5 Nov 22, 2022
41 Sep 17, 2022
18 Jul 19, 2022
6 Jul 07, 2022
33 Nov 30, 2022
1.5k Jan 05, 2023
224 Dec 08, 2022
295 Dec 26, 2022
15 Dec 22, 2022
651 Dec 12, 2022
100 Dec 31, 2022
2 Apr 13, 2022
7 Nov 24, 2022
8 Apr 11, 2022