General tricks that may help you find bad, or noisy, labels in your dataset

Related tags

Miscellaneousdoubtlab
Overview

doubtlab

A lab for bad labels.

Warning still in progress.

This repository contains general tricks that may help you find bad, or noisy, labels in your dataset. The hope is that this repository makes it easier for folks to quickly check their own datasets before they invest too much time and compute on gridsearch.

Install

You can install the tool via pip.

python -m pip install doubtlab

Quickstart

Doubtlab allows you to define "reasons" for a row of data to deserve another look. These reasons can form a pipeline which can be used to retreive a sorted list of examples worth checking again.

from doubtlab import DoubtLab
from doubtlab.reasons import ProbaReason, WrongPredictionReason

# Let's say we have some model already
model.fit(X, y)

# Next we can the reasons for doubt. In this case we're saying
# that examples deserve another look if the associated proba values
# are low or if the model output doesn't match the associated label.
reasons = {
    'proba': ProbaReason(model=model),
    'wrong_pred': WrongPredictionReason(model=model)
}

# Pass these reasons to a doubtlab instance.
doubt = DoubtLab(**reasons)

# Get the predicates, or reasoning, behind the order
predicates = doubt.get_predicates(X, y)
# Get the ordered indices of examples worth checking again
indices = doubt.get_indices(X, y)
# Get the (X, y) candidates worth checking again
X_check, y_check = doubt.get_candidates(X, y)

Features

The library implemented many "reaons" for doubt.

  • ProbaReason: assign doubt when a models' confidence-values are low
  • RandomReason: assign doubt randomly, just for sure
  • LongConfidenceReason: assign doubt when a wrong class gains too much confidence
  • ShortConfidenceReason: assign doubt when the correct class gains too little confidence
  • DisagreeReason: assign doubt when two models disagree on a prediction
  • CleanLabReason: assign doubt according to cleanlab

Related Projects

  • The cleanlab project was an inspiration for this one. They have a great heuristic for bad label detection but I wanted to have a library that implements many. Be sure to check out their work on the labelerrors.com project.
  • My employer, Rasa, has always had a focus on data quality. Some of that attitude is bound to have seeped in here. Be sure to check out Rasa X if you're working on virtual assistants.
Comments
  • `QuantileDifferenceReason` and `StandardDeviationReason`

    `QuantileDifferenceReason` and `StandardDeviationReason`

    Hey! I was thinking if it would make sense to add two more reasons for regressions tasks, namely something like HighLeveragePointReason and HighStudentizedResidualReason.

    Citing Wikipedia:

    • Leverage is a measure of how far away the independent variable values of an observation are from those of the other observations. High-leverage points, if any, are outliers with respect to the independent variables (link)
    • A studentized residual is the quotient resulting from the division of a residual by an estimate of its standard deviation. [...] This is an important technique in the detection of outliers. (link)
    opened by FBruzzesi 31
  • Doubt Reason Based on Entropy

    Doubt Reason Based on Entropy

    If a machine learning model is very "confident" then the proba scores will have low entropy. The most uncertain outcome is a uniform distribution which would contain high entropy. Therefore, it could be sensible to add entropy as a reason for doubt.

    opened by koaning 10
  • Add staticmethods to reasons to prevent re-compute.

    Add staticmethods to reasons to prevent re-compute.

    I really like the current design with reasons just being function calls.

    However, when working with large datasets or in use cases where you already have the predictions of a model, I wonder if you have thought about letting users to pass either a sklearn model or the pre-computed probas (for those Reasons where it make sense). For threshold-based reasons and large datasets this could save some time and compute, allow for faster iteration, and it would open up the possibility of using other models beyond sklearn.

    I understand that the design wouldn't be as clean as it is right now, might cause miss-alignments if users don't send the correct shapes/positions, but I wonder if you have considered this (or any other way to pass pre-computed predictions).

    Just to illustrate what I mean (sorry about the dirty-pseudo code):

    class ProbaReason:
    
        def __init__(self, model=None, probas=None, max_proba=0.55):
            if not model or probas:
                 print("You should at least pass a model or probas")
            self.model = model
            self.probas = probas
            self.max_proba = max_proba
    
        def __call__(self, X, y=None):
            probas = probas if self.probas else self.model.predict_proba(X)
            result = probas.max(axis=1) <= self.max_proba
            return result.astype(np.float16)
    
    opened by dvsrepo 9
  • "Fair" Sorting

    Suppose there are 5 reasons for doubt, 4 of which overlap a lot. Then we may end up in a situation where we ignore a reason. That could be bad ... maybe it's worth exploring voting systems a bit to figure out alternative sorting methods.

    opened by koaning 7
  • Add example to docs that shows lambda X, y: y.isna()

    Add example to docs that shows lambda X, y: y.isna()

    Hey! First of all: this is a very cool project ;) I have been thinking about potential new "reasons" to doubt and I personally often look into predictions generated by a model whenever the data instance had missing values (and part of the model-pipeline imputes them)... So I wonder if it would be useful to have a FillNaNReason (or something similar) based, for example in the MissingIndicator transformer.

    opened by juanitorduz 4
  • added conda-install-option and badges to readme

    added conda-install-option and badges to readme

    This closes #14: doubtlab can now be installed with conda from conda-forge channel.

    • [x] Created conda-forge/doubtlab-feedstock to make doubtlab available on conda-forge channel.
    • [x] Added conda install option to readme.
    • [x] Added the following badges to readme.

    GitHub - License PyPI - Python Version PyPI - Package Version PyPI - Downloads Conda - Platform Conda (channel only) Docs - GitHub.io

    opened by sugatoray 4
  • Added a LICENSE

    Added a LICENSE

    Hi @koaning,

    I am assuming MIT License is okay for this repository. If you think otherwise, please feel free to make changes in the PR accordingly.

    • [x] Added an MIT License
    • [x] ~~Added a Citation file~~ Removed the citation file and updated the name of the PR. - ~~If you have an orcid, please consider adding it to the citation.cff file.~~
    opened by sugatoray 4
  • Add a conda installation option using conda-forge channel

    Add a conda installation option using conda-forge channel

    I have already started this one. Will push a PR once the conda installation option is available.

    See: Adding doubtlab from PyPI to conda-forge channel.

    @koaning As the primary maintainer of this repo, would you like to be listed as one of the maintainers of doubtlab on conda-forge channel? Please let me know, I will add your name as another maintainer of conda-forge/doubtlab-feedstock, once it is accepted.

    opened by sugatoray 3
  • Doubt about MarginConfidenceReason :-)

    Doubt about MarginConfidenceReason :-)

    Hi Vincent,

    Nice library! As mentioned a while ago on Twitter I'm doing a review to understand and compare different approaches to find label errors.

    I'm playing with the AG News dataset, which we know it contains a lot of errors from our own previous experiments with Rubrix (using the training loss and using cleanlab).

    While playing with the different reasons, I'm having difficulties to understand the reasoning behind the MarginConfidenceReason. As far as I can tell, if the model is doubting the margin between the top two predicted labels should be small, and that could point to an ambiguous example and/or a label error. If I read the code and description correctly, MarginConfidenceReason is doing the opposite, so I'd love to know the reasoning behind this to make sure I'm not missing something.

    For context, using the MarginConfidenceReason with the AG News training set yields almost the entire dataset (117788 examples for the default threshold of 0.2, and 112995 for threshold=0.5). I guess this could become useful when there's overlap with other reasons, but I want to make sure about the reasoning :-).

    opened by dvsrepo 2
  • updated docs: installation and badges

    updated docs: installation and badges

    Updated docs:

    • [x] updated installation (with conda)
    • [x] ~~added badges from readme~~

    @koaning I am not sure if you would prefer to include the badges in the docs (website). If you don't, please feel free to remove them.

    UPDATE: removed badges from the docs (docs/index.md).

    opened by sugatoray 2
  • Issue with cleanlab upgrading to v2

    Issue with cleanlab upgrading to v2

    Issue

    image

    Environment details

    image

    Temporary fix

    pip install "doubtlab==1.0.0"

    More permanent fix

    Pin doubtlab dependency to "doubtlab<2.0.0"

    More more permanent fix

    They've made some changes to their API

    Let me know if you'd like me to make a PR

    Thanks for a great package @koaning 😄

    opened by duarteocarmo 1
  • Consider a fairlearn demo.

    Consider a fairlearn demo.

    When two models disagree something interesting might be happening. But that'll only happen if you have two models that are actually different.

    What if you have one model that's better at accuracy and another one that's better at fairness.

    Maybe these labels deserve more attention too.

    opened by koaning 0
  • Assign Doubt for Dissimilarity from Labelled Set

    Assign Doubt for Dissimilarity from Labelled Set

    Suppose that y can contain NaN values if they aren't labeled. In that case, we may want to favor a subset of these NaN values. In particular: if they differ substantially from the already labeled datapoints.

    The idea here is that we may be able to sample more diverse datapoints.

    opened by koaning 10
  • Does it make sense to add an ensemble for spaCy?

    Does it make sense to add an ensemble for spaCy?

    This seems to be a like-able method of dealing with text outside the realm of scikit-learn. But I prefer to delay this feature until I really understand the use-case. For anything related to entities we cannot use sklearn, but tags/classes should work fine as-is.

    opened by koaning 1
Releases(0.2.4)
Owner
vincent d warmerdam
Solving problems involving data. Mostly NLP these days. AskMeAnything[tm].
vincent d warmerdam
Adansons Base is a data management tool that organizes metadata of unstructured data and creates and organizes datasets.

Adansons Base is a data management tool that organizes metadata of unstructured data and creates and organizes datasets. It makes dataset creation more effective and helps find essential insights fro

Adansons Inc 27 Oct 22, 2022
MeepoBenchmark - This project aims at providing the scripts, logs, and analytic results for Meepo Blockchain

MeepoBenchmark - This project aims at providing the scripts, logs, and analytic results for Meepo Blockchain

Peilin Zheng 3 Aug 16, 2022
A Python library that helps data scientists to infer causation rather than observing correlation.

A Python library that helps data scientists to infer causation rather than observing correlation.

QuantumBlack Labs 1.7k Jan 04, 2023
Exploring basic lambda calculus in Python

Lambda Exploring basic lambda calculus in Python. In this repo I have used the lambda function built into python to get a more intiutive feel of lambd

Bhardwaj Bhaskar 2 Nov 12, 2021
Create rangebased on lists or values of the range itself. Range any type. Can you imagine?

funcao-allrange-for-python3 Create rangebased on lists or values of the range itself. Range any type. Can you imagine? WARNING!!! THIS MODULE DID NOT

farioso-fernando 1 Feb 09, 2022
A wide AOI generator tool.

Dark Generator A wide AOI generator tool. Information Installation To Install you have to have python 3.x and pip installed on your system. If you hav

Darkest Surface 12 Dec 26, 2022
My Dotfiles of Arco Linux

Arco-DotFiles My Dotfiles of Arco Linux Apps Used Htop LightDM lightdm-webkit2-greeter Alacritty Qtile Cava Spotify nitrogen neofetch Spicetify Thunar

$BlueDev5 6 Dec 11, 2022
python scripts and other files to generate induction encoder PCBs in Kicad

induction_encoder python scripts and other files to generate induction encoder PCBs in Kicad Targeting the Renesas IPS2200 encoder chips.

Taylor Alexander 8 Feb 16, 2022
Would upload anything I do with/related to brainfuck

My Brainfu*k Repo Basically wanted to create something with Brainfu*k but realized that with the smol brain I have, I need to see the cell values real

Rafeed 1 Mar 22, 2022
A discord group chat creator just made it because i saw people selling this stuff for like up to 40 bucks

gccreator some discord group chat tools just made it because i saw people selling this stuff for like up to 40 bucks (im currently working on a faster

baum1810 6 Oct 03, 2022
A collection of convenient parsers for Advent of Code problems.

Advent of Code Parsers A collection of convenient Python parsers for Advent of Code problems. Installation pip install aocp Quickstart You can import

Miguel Blanco Marcos 3 Dec 13, 2021
The presented desktop application was made to solve 1d schrodinger eqation

schrodinger_equation_1d_solver The presented desktop application was made to solve 1d schrodinger eqation. It implements Numerov's algorithm (step by

Artem Kashapov 2 Dec 29, 2021
This is a simple web interface for SimplyTranslate

SimplyTranslate Web This is a simple web interface for SimplyTranslate List of Instances You can find a list of instances here: SimplyTranslate Projec

4 Dec 14, 2022
Parser for the GeoSuite[tm] PRV export format

Parser for the GeoSuite[tm] PRV export format This library provides functionality to parse geotechnical investigation data in .prv files generated by

EMerald Geomodelling 1 Dec 17, 2021
A collection of common regular expressions bundled with an easy to use interface.

CommonRegex Find all times, dates, links, phone numbers, emails, ip addresses, prices, hex colors, and credit card numbers in a string. We did the har

Madison May 1.5k Dec 31, 2022
Create standalone, installable R Shiny apps using Electron

Create standalone, installable R Shiny apps using Electron

Chase Clark 5 Dec 24, 2021
Utility functions for working with data from Nix in Python

Pynixutil - Utility functions for working with data from Nix in Python Examples Base32 encoding/decoding import pynixutil input = "v5sv61sszx301i0x6x

Tweag 11 Dec 16, 2022
A Python program that generates a maze that solves itself using DFS

Maze Generator And Solver Program Purpose: Generates a maze that then solves itself Language: Python and Pygame Algorithm: Randomized DFS / Floodfill

Joshua Liu 1 Jul 25, 2022
Construção de um jogo Dominó na linguagem python com base em algoritmos personalizados.

Domino (projecto-python) Construção de um jogo Dominó na linguaguem python com base em algoritmos personalizados e na: Monografia apresentada ao curso

Nuninha-GC 1 Jan 12, 2022
Paxos in Python, tested with Jepsen

Python implementation of Multi-Paxos with a stable leader and reconfiguration, roughly following "Paxos Made Moderately Complex". Run python3 paxos/st

A. Jesse Jiryu Davis 25 Dec 15, 2022