PyCIL: A Python Toolbox for Class-Incremental Learning
Introduction • Methods Reproduced • Reproduced Results • How To Use • License • Acknowledgements • Contact
The code repository for "PyCIL: A Python Toolbox for Class-Incremental Learning" [paper] in PyTorch. If you use any content of this repo for your work, please cite the following bib entry:
@misc{zhou2021pycil,
title={PyCIL: A Python Toolbox for Class-Incremental Learning},
author={Da-Wei Zhou and Fu-Yun Wang and Han-Jia Ye and De-Chuan Zhan},
year={2021},
eprint={2112.12533},
archivePrefix={arXiv},
primaryClass={cs.LG}
}
Introduction
Traditional machine learning systems are deployed under the closed-world setting, which requires the entire training data before the offline training process. However, real-world applications often face the incoming new classes, and a model should incorporate them continually. The learning paradigm is called Class-Incremental Learning (CIL). We propose a Python toolbox that implements several key algorithms for class-incremental learning to ease the burden of researchers in the machine learning community. The toolbox contains implementations of a number of founding works of CIL such as EWC and iCaRL, but also provides current state-of-the-art algorithms that can be used for conducting novel fundamental research. This toolbox, named PyCIL for Python Class-Incremental Learning, is open source with an MIT license.
Methods Reproduced
-
FineTune: Baseline method which simply updates parameters on new task, suffering from Catastrophic Forgetting. By default, weights corresponding to the outputs of previous classes are not updated. -
EWC: Gradient Episodic Memory for Continual Learning. [paper] -
LwF: Learning without Forgetting. [paper] -
Replay: Baseline method with exemplars. -
GEM: Gradient Episodic Memory for Continual Learning. [paper] -
iCaRL: Incremental Classifier and Representation Learning. [paper] -
BiC: Large Scale Incremental Learning. [paper] -
WA: Maintaining Discrimination and Fairness in Class Incremental Learning. [paper] -
PODNet: PODNet: Pooled Outputs Distillation for Small-Tasks Incremental Learning. [paper] -
DER: DER: Dynamically Expandable Representation for Class Incremental Learning. [paper] -
Coil: Co-Transport for Class-Incremental Learning. [paper]
Reproduced Results
CIFAR100
Imagenet100
More experimental details and results are shown in our paper.
How To Use
Clone
Clone this github repository:
git clone https://github.com/G-U-N/PyCIL.git
cd PyCIL
Dependencies
Run experiment
- Edit the
[MODEL NAME].jsonfile for global settings. - Edit the hyperparameters in the corresponding
[MODEL NAME].pyfile (e.g.,models/icarl.py). - Run:
python main.py --config=./exps/[MODEL NAME].json
where [MODEL NAME] should be chosen from: finetune, ewc, lwf, replay, gem, icarl, bic, wa, podnet, der.
hyper-parameters
When using PyCIL, you can edit the global parameters and algorithm-specific hyper-parameter in the corresponding json file.
These parameters include:
- memory-size: The total exemplar number in the incremental learning process. Assuming there are $K$ classes at current stage, the model will preserve $\left[\frac{memory-size}{K}\right]$ exemplar per class.
- init-cls: The number of classes in the first incremental stage. Since there are different settings in CIL with a different number of classes in the first stage, our framework enables different choices to define the initial stage.
- increment: The number of classes in each incremental stage $i$, $i$ > 1. By default, the number of classes per incremental stage is equivalent per stage.
- convnet-type: The backbone network for the incremental model. According to the benchmark setting,
ResNet32is utilized forCIFAR100, andResNet18is utilized forImageNet. - seed: The random seed adopted for shuffling the class order. According to the benchmark setting, it is set to 1993 by default.
Other parameters in terms of model optimization, e.g., batch size, optimization epoch, learning rate, learning rate decay, weight decay, milestone, temperature, can be modified in the corresponding Python file.
Datasets
We have implemented the pre-processing of CIFAR100, imagenet100 and imagenet1000. When training on CIFAR100, this framework will automatically download it. When training on imagenet100/1000, you should specify the folder of your dataset in utils/data.py.
def download_data(self):
assert 0,"You should specify the folder of your dataset"
train_dir = '[DATA-PATH]/train/'
test_dir = '[DATA-PATH]/val/'
License
Please check the MIT license that is listed in this repository.
Acknowledgements
We thank the following repos providing helpful components/functions in our work.
Contact
If there are any questions, please feel free to propose new features by opening an issue or contact with the author: Da-Wei Zhou([email protected]) and Fu-Yun Wang([email protected]). Enjoy the code.
![why loss_clf = F.cross_entropy(logits[:, self._known_classes:], fake_targets)?](https://avatars.githubusercontent.com/u/51309340?v=4)
170.1k Jan 04, 2023
64 Jan 07, 2023
55 Sep 13, 2022
115 Jan 06, 2023
4 Nov 27, 2022
9 Jul 19, 2022
1.2k Jan 05, 2023
3 Aug 03, 2022
66 Dec 13, 2022
5 Aug 02, 2022
31 Jul 16, 2022
1 Jan 05, 2022
2 Dec 08, 2022
26 Dec 22, 2022
733 Dec 29, 2022
190 Dec 30, 2022
10 Sep 27, 2022
371 Jan 03, 2023
201 Dec 29, 2022
15 Dec 16, 2022