General tricks that may help you find bad, or noisy, labels in your dataset

Related tags

Miscellaneousdoubtlab
Overview

doubtlab

A lab for bad labels.

Warning still in progress.

This repository contains general tricks that may help you find bad, or noisy, labels in your dataset. The hope is that this repository makes it easier for folks to quickly check their own datasets before they invest too much time and compute on gridsearch.

Install

You can install the tool via pip.

python -m pip install doubtlab

Quickstart

Doubtlab allows you to define "reasons" for a row of data to deserve another look. These reasons can form a pipeline which can be used to retreive a sorted list of examples worth checking again.

from doubtlab import DoubtLab
from doubtlab.reasons import ProbaReason, WrongPredictionReason

# Let's say we have some model already
model.fit(X, y)

# Next we can the reasons for doubt. In this case we're saying
# that examples deserve another look if the associated proba values
# are low or if the model output doesn't match the associated label.
reasons = {
    'proba': ProbaReason(model=model),
    'wrong_pred': WrongPredictionReason(model=model)
}

# Pass these reasons to a doubtlab instance.
doubt = DoubtLab(**reasons)

# Get the predicates, or reasoning, behind the order
predicates = doubt.get_predicates(X, y)
# Get the ordered indices of examples worth checking again
indices = doubt.get_indices(X, y)
# Get the (X, y) candidates worth checking again
X_check, y_check = doubt.get_candidates(X, y)

Features

The library implemented many "reaons" for doubt.

  • ProbaReason: assign doubt when a models' confidence-values are low
  • RandomReason: assign doubt randomly, just for sure
  • LongConfidenceReason: assign doubt when a wrong class gains too much confidence
  • ShortConfidenceReason: assign doubt when the correct class gains too little confidence
  • DisagreeReason: assign doubt when two models disagree on a prediction
  • CleanLabReason: assign doubt according to cleanlab

Related Projects

  • The cleanlab project was an inspiration for this one. They have a great heuristic for bad label detection but I wanted to have a library that implements many. Be sure to check out their work on the labelerrors.com project.
  • My employer, Rasa, has always had a focus on data quality. Some of that attitude is bound to have seeped in here. Be sure to check out Rasa X if you're working on virtual assistants.
Comments
  • `QuantileDifferenceReason` and `StandardDeviationReason`

    `QuantileDifferenceReason` and `StandardDeviationReason`

    Hey! I was thinking if it would make sense to add two more reasons for regressions tasks, namely something like HighLeveragePointReason and HighStudentizedResidualReason.

    Citing Wikipedia:

    • Leverage is a measure of how far away the independent variable values of an observation are from those of the other observations. High-leverage points, if any, are outliers with respect to the independent variables (link)
    • A studentized residual is the quotient resulting from the division of a residual by an estimate of its standard deviation. [...] This is an important technique in the detection of outliers. (link)
    opened by FBruzzesi 31
  • Doubt Reason Based on Entropy

    Doubt Reason Based on Entropy

    If a machine learning model is very "confident" then the proba scores will have low entropy. The most uncertain outcome is a uniform distribution which would contain high entropy. Therefore, it could be sensible to add entropy as a reason for doubt.

    opened by koaning 10
  • Add staticmethods to reasons to prevent re-compute.

    Add staticmethods to reasons to prevent re-compute.

    I really like the current design with reasons just being function calls.

    However, when working with large datasets or in use cases where you already have the predictions of a model, I wonder if you have thought about letting users to pass either a sklearn model or the pre-computed probas (for those Reasons where it make sense). For threshold-based reasons and large datasets this could save some time and compute, allow for faster iteration, and it would open up the possibility of using other models beyond sklearn.

    I understand that the design wouldn't be as clean as it is right now, might cause miss-alignments if users don't send the correct shapes/positions, but I wonder if you have considered this (or any other way to pass pre-computed predictions).

    Just to illustrate what I mean (sorry about the dirty-pseudo code):

    class ProbaReason:
    
        def __init__(self, model=None, probas=None, max_proba=0.55):
            if not model or probas:
                 print("You should at least pass a model or probas")
            self.model = model
            self.probas = probas
            self.max_proba = max_proba
    
        def __call__(self, X, y=None):
            probas = probas if self.probas else self.model.predict_proba(X)
            result = probas.max(axis=1) <= self.max_proba
            return result.astype(np.float16)
    
    opened by dvsrepo 9
  • "Fair" Sorting

    Suppose there are 5 reasons for doubt, 4 of which overlap a lot. Then we may end up in a situation where we ignore a reason. That could be bad ... maybe it's worth exploring voting systems a bit to figure out alternative sorting methods.

    opened by koaning 7
  • Add example to docs that shows lambda X, y: y.isna()

    Add example to docs that shows lambda X, y: y.isna()

    Hey! First of all: this is a very cool project ;) I have been thinking about potential new "reasons" to doubt and I personally often look into predictions generated by a model whenever the data instance had missing values (and part of the model-pipeline imputes them)... So I wonder if it would be useful to have a FillNaNReason (or something similar) based, for example in the MissingIndicator transformer.

    opened by juanitorduz 4
  • added conda-install-option and badges to readme

    added conda-install-option and badges to readme

    This closes #14: doubtlab can now be installed with conda from conda-forge channel.

    • [x] Created conda-forge/doubtlab-feedstock to make doubtlab available on conda-forge channel.
    • [x] Added conda install option to readme.
    • [x] Added the following badges to readme.

    GitHub - License PyPI - Python Version PyPI - Package Version PyPI - Downloads Conda - Platform Conda (channel only) Docs - GitHub.io

    opened by sugatoray 4
  • Added a LICENSE

    Added a LICENSE

    Hi @koaning,

    I am assuming MIT License is okay for this repository. If you think otherwise, please feel free to make changes in the PR accordingly.

    • [x] Added an MIT License
    • [x] ~~Added a Citation file~~ Removed the citation file and updated the name of the PR. - ~~If you have an orcid, please consider adding it to the citation.cff file.~~
    opened by sugatoray 4
  • Add a conda installation option using conda-forge channel

    Add a conda installation option using conda-forge channel

    I have already started this one. Will push a PR once the conda installation option is available.

    See: Adding doubtlab from PyPI to conda-forge channel.

    @koaning As the primary maintainer of this repo, would you like to be listed as one of the maintainers of doubtlab on conda-forge channel? Please let me know, I will add your name as another maintainer of conda-forge/doubtlab-feedstock, once it is accepted.

    opened by sugatoray 3
  • Doubt about MarginConfidenceReason :-)

    Doubt about MarginConfidenceReason :-)

    Hi Vincent,

    Nice library! As mentioned a while ago on Twitter I'm doing a review to understand and compare different approaches to find label errors.

    I'm playing with the AG News dataset, which we know it contains a lot of errors from our own previous experiments with Rubrix (using the training loss and using cleanlab).

    While playing with the different reasons, I'm having difficulties to understand the reasoning behind the MarginConfidenceReason. As far as I can tell, if the model is doubting the margin between the top two predicted labels should be small, and that could point to an ambiguous example and/or a label error. If I read the code and description correctly, MarginConfidenceReason is doing the opposite, so I'd love to know the reasoning behind this to make sure I'm not missing something.

    For context, using the MarginConfidenceReason with the AG News training set yields almost the entire dataset (117788 examples for the default threshold of 0.2, and 112995 for threshold=0.5). I guess this could become useful when there's overlap with other reasons, but I want to make sure about the reasoning :-).

    opened by dvsrepo 2
  • updated docs: installation and badges

    updated docs: installation and badges

    Updated docs:

    • [x] updated installation (with conda)
    • [x] ~~added badges from readme~~

    @koaning I am not sure if you would prefer to include the badges in the docs (website). If you don't, please feel free to remove them.

    UPDATE: removed badges from the docs (docs/index.md).

    opened by sugatoray 2
  • Issue with cleanlab upgrading to v2

    Issue with cleanlab upgrading to v2

    Issue

    image

    Environment details

    image

    Temporary fix

    pip install "doubtlab==1.0.0"

    More permanent fix

    Pin doubtlab dependency to "doubtlab<2.0.0"

    More more permanent fix

    They've made some changes to their API

    Let me know if you'd like me to make a PR

    Thanks for a great package @koaning 😄

    opened by duarteocarmo 1
  • Consider a fairlearn demo.

    Consider a fairlearn demo.

    When two models disagree something interesting might be happening. But that'll only happen if you have two models that are actually different.

    What if you have one model that's better at accuracy and another one that's better at fairness.

    Maybe these labels deserve more attention too.

    opened by koaning 0
  • Assign Doubt for Dissimilarity from Labelled Set

    Assign Doubt for Dissimilarity from Labelled Set

    Suppose that y can contain NaN values if they aren't labeled. In that case, we may want to favor a subset of these NaN values. In particular: if they differ substantially from the already labeled datapoints.

    The idea here is that we may be able to sample more diverse datapoints.

    opened by koaning 10
  • Does it make sense to add an ensemble for spaCy?

    Does it make sense to add an ensemble for spaCy?

    This seems to be a like-able method of dealing with text outside the realm of scikit-learn. But I prefer to delay this feature until I really understand the use-case. For anything related to entities we cannot use sklearn, but tags/classes should work fine as-is.

    opened by koaning 1
Releases(0.2.4)
Owner
vincent d warmerdam
Solving problems involving data. Mostly NLP these days. AskMeAnything[tm].
vincent d warmerdam
Demo of using DataLoader to prevent out of memory

Demo of using DataLoader to prevent out of memory

3 Jun 25, 2022
Python Commodore BBS multi-client

python-cbm-bbs-petscii Python Commodore BBS multi-client This is intended for commodore 64, c128 and most commodore compatible machines (as the new Co

7 Sep 16, 2022
El_Binario - A converter for Binary, Decimal, Hexadecimal and Octal numbers

El_Binario El_Binario es un conversor de números Binarios, Decimales, Hexadecima

2 Jan 28, 2022
Modern robots.txt Parser for Python

Robots Exclusion Protocol Parser for Python Robots.txt parsing in Python. Goals Fetching -- helper utilities for fetching and parsing robots.txts, inc

Moz 176 Dec 16, 2022
Scripts for BGC analysis in large MAGs and results of their application to soil metagenomes within Chernevaya Taiga RSF-funded project

Scripts for BGC analysis in large MAGs and results of their application to soil metagenomes within Chernevaya Taiga RSF-funded project

1 Dec 06, 2021
Adversarial Robustness with Non-uniform Perturbations

Adversarial Robustness with Non-uniform Perturbations This repository hosts the code to replicate experiments of the paper Adversarial Robustness with

5 May 20, 2022
Find out where all films you want to watch are streaming

Just Watch Letterboxd Find out where all films you want to watch are streaming Ever wonder what films you want to watch are already on the streaming p

Jordan Oslislo 2 Feb 04, 2022
Convert Roman numerals to modern numerals and vice-versa

Roman Numeral Conversion Utilities This is a utility module for converting from and to Roman numerals. It supports numbers upto 3,999,999, using the v

Fictive Kin 1 Dec 17, 2021
Its a simple and fun to use application. You can make your own quizes and send the lik of the quiz to your friends.

Quiz Application Its a simple and fun to use application. You can make your own quizes and send the lik of the quiz to your friends. When they would a

Atharva Parkhe 1 Feb 23, 2022
NASH 2021 project... this may or may not end up working 🤷‍♂️

wavespace synthesiser this is my NASH 2021 project, which may or may not end up working 🤷‍♂️ what is going on? imagine you have a big folder of audio

Ben Hayes 12 May 17, 2022
A quick experiment to demonstrate Metamath formula parsing, where the grammar is embedded in a few additional 'syntax axioms'.

Warning: Hacked-up code ahead. (But it seems to work...) What it does This demonstrates an idea which I posted about several times on the Metamath mai

Marnix Klooster 1 Oct 21, 2021
Tools for collecting social media data around focal events

Social Media Focal Events The focalevents codebase provides tools for organizing data collected around focal events on social media. It is often diffi

Ryan Gallagher 80 Nov 28, 2022
From "fixed RAnDom CRashes" to "[FIX] Fixed random crashes."

Clean Commit From fixed RAnDom CRashes to [FIX] Fixed random crashes. Clean commit helps you by auto-formating your commits to make your repos better

Mathias 3 Dec 26, 2021
Zotero references script (and app)

A little script (and PyInstaller build) for a very specific, somewhat hack-ish purpose: managing and exporting project references with Zotero and its API.

Marius Rödder 0 Dec 05, 2021
The code for 2021 MGTV AI Challenge Anti Stealing Link, and the online result ranks 10th.

赛题介绍 芒果TV-第二届“马栏山杯”国际音视频算法大赛-防盗链 随着业务的发展,芒果的视频内容也深受网友的喜欢,不少视频网站和应用开始盗播芒果的视频内容,盗链网站不经过芒果TV的前端系统,跳过广告播放,且消耗大量的服务器、带宽资源,直接给公司带来了巨大的经济损失,因此防盗链在日常运营中显得尤为重要

tongji40 16 Jun 17, 2022
The dynamic code loading framework used in LocalStack

localstack-plugin-loader localstack-plugin-loader is the dynamic code loading framework used in LocalStack. Install pip install localstack-plugin-load

LocalStack 5 Oct 09, 2022
A play store search module

A play store search module

Fayas Noushad 5 Dec 01, 2021
Pipenv-local-deps-repro - Reproduction of a local transitive dependency on pipenv

Reproduction of the pipenv bug with transitive local dependencies. Clone this re

Lucas Duailibe 2 Jan 11, 2022
Análise do Aplicativo Prévias PSDB 2021

Análise do Aplicativo Prévias PSDB 2021 Com a recente polêmica sobre o aplicativo usado nas Prévias do PSDB de 2021, fiquei curioso para saber como er

Paulo Matias 18 Jul 31, 2022
A collection of Python library code for building Python applications.

Abseil Python Common Libraries This repository is a collection of Python library code for building Python applications. The code is collected from Goo

Abseil 2k Jan 07, 2023