Sparse Progressive Distillation: Resolving Overfitting under Pretrain-and-Finetune Paradigm

Last update: Dec 05, 2022

Overview

Sparse Progressive Distillation: Resolving Overfitting under Pretrain-and-Finetune Paradigm

This is the Pytorch implementation for sparse progressive distillation (SPD). For more details about the motivation, techniques and experimental results, refer to our paper here.

Running

Environment Preparation (using python3)
```
pip install -r requirements.txt
```
Dataset Preparation

The original GLUE dataset could be downloaded here.

BERT_base fine-tuning on GLUE

We use finetuned BERT_base as the teacher. For each task of GLUE benchmark, we obtain the finetuned model using the original huggingface transformers code with the following script.

python run_glue.py \
          --model_name_or_path $INT_DIR \
          --task_name $TASK_NAME \
          --do_train \
          --do_eval \
          --data_dir $GLUE_DIR/$TASK_NAME/ \
          --max_seq_length 128 \
          --per_gpu_train_batch_size 32 \
          --per_gpu_eval_batch_size 32 \
          --learning_rate 3e-5 \
          --num_train_epochs 4.0 \
          --output_dir $OUT_DIR \
          --evaluate_during_training \
          --overwrite_output_dir \
          --logging_steps 400 \
          --logging_dir $OUT_DIR \
          --save_steps 10000

Sparse Progressive Distillation

We use run_glue.py to run the sparse progressive distillation. --num_prune_epochs is the epochs for pruning. --num_train_epochs is the total number of epochs (pruning, progressive distillation, finetuning).

python run_glue.py \
  --model_name_or_path PATH_TO_FINETUNED_MODEL \
  --task_name $TASK_NAME \
  --do_train \
  --do_eval \
  --do_lower_case \
  --data_dir $GLUE_DIR/$TASK_NAME/ \
  --max_seq_length 128 \
  --per_gpu_train_batch_size 32 \
  --per_gpu_eval_batch_size 32 \
  --learning_rate 6.4e-4 \
  --save_steps 50 \
  --num_prune_epochs 30 \
  --num_train_epochs 60 \
  --sparsity 0.9 \
  --output_dir $OUT_DIR \
  --evaluate_during_training \
  --replacing_rate 0.8 \
  --overwrite_output_dir \
  --steps_for_replacing 0 \
  --scheduler_type linear

To Dos

Provide our teacher model for each task.
Provide best performed model checkpoint for each task.

Sparse Progressive Distillation: Resolving Overfitting under Pretrain-and-Finetune Paradigm

Related tags

Overview

Sparse Progressive Distillation: Resolving Overfitting under Pretrain-and-Finetune Paradigm

Running

BERT_base fine-tuning on GLUE

Sparse Progressive Distillation

To Dos

Owner

Official implementation of the paper Do pedestrians pay attention? Eye contact detection for autonomous driving

Free-duolingo-plus - Duolingo account creator that uses your invite code to get you free duolingo plus

Traductor de lengua de señas al español basado en Python con Opencv y MedaiPipe

A demo of how to use JAX to create a simple gravity simulation

Educational API for 3D Vision using pose to control carton.

Code for Contrastive-Geometry Networks for Generalized 3D Pose Transfer

Implementation of the Point Transformer layer, in Pytorch

Code base for NeurIPS 2021 publication titled Kernel Functional Optimisation (KFO)

Source Code for our paper: Understand me, if you refer to Aspect Knowledge: Knowledge-aware Gated Recurrent Memory Network

PyTorch Implementation of ByteDance's Cross-speaker Emotion Transfer Based on Speaker Condition Layer Normalization and Semi-Supervised Training in Text-To-Speech

Algebraic effect handlers in Python

Pytorch implementation of MalConv

an implementation of Revisiting Adaptive Convolutions for Video Frame Interpolation using PyTorch

Attention-based Transformation from Latent Features to Point Clouds (AAAI 2022)

“英特尔创新大师杯”深度学习挑战赛赛道3：CCKS2021中文NLP地址相关性任务

Classify bird species based on their songs using SIamese Networks and 1D dilated convolutions.

Hcaptcha-challenger - Gracefully face hCaptcha challenge with Yolov5(ONNX) embedded solution

code for our ECCV 2020 paper "A Balanced and Uncertainty-aware Approach for Partial Domain Adaptation"

Supporting code for the paper "Dangers of Bayesian Model Averaging under Covariate Shift"

Google Recaptcha solver.

Sparse Progressive Distillation: Resolving Overfitting under Pretrain-and-Finetune Paradigm

Related tags

Overview

Sparse Progressive Distillation: Resolving Overfitting under Pretrain-and-Finetune Paradigm

Running

BERT_base fine-tuning on GLUE

Sparse Progressive Distillation

To Dos

Owner

Official implementation of the paper Do pedestrians pay attention? Eye contact detection for autonomous driving

Free-duolingo-plus - Duolingo account creator that uses your invite code to get you free duolingo plus

Traductor de lengua de señas al español basado en Python con Opencv y MedaiPipe

A demo of how to use JAX to create a simple gravity simulation

Educational API for 3D Vision using pose to control carton.

Code for Contrastive-Geometry Networks for Generalized 3D Pose Transfer

Implementation of the Point Transformer layer, in Pytorch

Code base for NeurIPS 2021 publication titled Kernel Functional Optimisation (KFO)

Source Code for our paper: Understand me, if you refer to Aspect Knowledge: Knowledge-aware Gated Recurrent Memory Network

PyTorch Implementation of ByteDance's Cross-speaker Emotion Transfer Based on Speaker Condition Layer Normalization and Semi-Supervised Training in Text-To-Speech

Algebraic effect handlers in Python

Pytorch implementation of MalConv

an implementation of Revisiting Adaptive Convolutions for Video Frame Interpolation using PyTorch

Attention-based Transformation from Latent Features to Point Clouds (AAAI 2022)

“英特尔创新大师杯”深度学习挑战赛 赛道3：CCKS2021中文NLP地址相关性任务

Classify bird species based on their songs using SIamese Networks and 1D dilated convolutions.

Hcaptcha-challenger - Gracefully face hCaptcha challenge with Yolov5(ONNX) embedded solution

code for our ECCV 2020 paper "A Balanced and Uncertainty-aware Approach for Partial Domain Adaptation"

Supporting code for the paper "Dangers of Bayesian Model Averaging under Covariate Shift"

Google Recaptcha solver.

“英特尔创新大师杯”深度学习挑战赛赛道3：CCKS2021中文NLP地址相关性任务