Dataset for the Research2Clinics @ NeurIPS 2021 Paper: What Do You See in this Patient? Behavioral Testing of Clinical NLP Models

Last update: Sep 20, 2022

Overview

Behavioral Testing of Clinical NLP Models

This repository contains code for testing the behavior of clinical prediction models based on patient letters. For a detailed description of the testing framework see our paper What Do You See in this Patient? Behavioral Testing of Clinical NLP Models.

Usage

Install requirements: pip install -r requirements.txt

Run main.py, e.g. for diagnosis prediction test on gender, age and ethnicity:

python main.py 
    --test_set_path ./path_to_test_set
    --model_path bvanaken/CORe-clinical-diagnosis-prediction
    --task diagnosis
    --shift_keys gender,age,ethnicity
    --save_dir ./results
    --gpu False

Parameter	Description
test_set_path	Path to original test set file
model_path	Path to model or Huggingface model hub checkpoint
task	Current options: diagnosis, mortality
shift_keys	Which patient characteristics to test. Current options: age, gender, ethnicity, weight, intersectional (gender + ethnicity)
save_dir	Directory to save results, default: "./results"
gpu	Whether to use a gpu during inference or not, default: False

Using Non-Transformer models

The framework currently focuses on testing Transformer-based models. However, it is easy to extend it to any other prediction model. To do so, simply create a new class implementing the Predictor interface and add it to the TASK_MAP in main.py.

Cite

@inproceedings{vanAken2021,
  author    = {Betty van Aken and
               Sebastian Herrmann and
               Alexander Löser},
  title     = {What Do You See in this Patient? Behavioral Testing of Clinical NLP Models},
  booktitle = {Bridging the Gap: From Machine Learning Research to Clinical Practice, 
               Research2Clinics Workshop @ NeurIPS 2021},
  year      = {2021}
}

Dataset for the Research2Clinics @ NeurIPS 2021 Paper: What Do You See in this Patient? Behavioral Testing of Clinical NLP Models

Related tags

Overview

Behavioral Testing of Clinical NLP Models

Usage

Using Non-Transformer models

Cite

Owner

Betty van Aken

Plugin adapted from Ultralytics to bring YOLOv5 into Napari

In the case of your data having only 1 channel while want to use timm models

Efficient Sharpness-aware Minimization for Improved Training of Neural Networks

LLVM-based compiler for LightGBM gradient-boosted trees. Speeds up prediction by ≥10x.

Use tensorflow to implement a Deep Neural Network for real time lane detection

Statsmodels: statistical modeling and econometrics in Python

Cartoon-StyleGan2 🙃 : Fine-tuning StyleGAN2 for Cartoon Face Generation

Code for Learning Manifold Patch-Based Representations of Man-Made Shapes, in ICLR 2021.

Denoising Diffusion Probabilistic Models

Python Environment for Bayesian Learning

Implementation of "Deep Implicit Templates for 3D Shape Representation"

PyTorch code for training MM-DistillNet for multimodal knowledge distillation

DetCo: Unsupervised Contrastive Learning for Object Detection

PyTorch Implementation of Sparse DETR

Moment-DETR code and QVHighlights dataset

Reduce end to end training time from days to hours (or hours to minutes), and energy requirements/costs by an order of magnitude using coresets and data selection.

Official PyTorch Implementation of SSMix (Findings of ACL 2021)

This is the official repository of the paper Stocastic bandits with groups of similar arms (NeurIPS 2021). It contains the code that was used to compute the figures and experiments of the paper.

K-Means Clustering and Hierarchical Clustering Unsupervised Learning Solution in Python3.

PyTorch implementation of deep GRAph Contrastive rEpresentation learning (GRACE).