RBERT for Relation Extraction task for KLUE
Project Description
- Relation Extraction task is one of the task of Korean Language Understanding Evaluation(KLUE) Benchmark.
- Relation extraction can be defined as multiclass classification task for relationship between subject entity and object entity.
- Classes are such as
no_relation,per:employee_of,org:founded_by... totaling 30 labels. - This repo contains custom fine-tuning method utilizing monologg's R-BERT Implementation.
- Custom punctuations with Pororo NER has been added to the dataset prior to the model's training.
- If you want to refer to the experimentation note such as punctuation method of the entity, please refer to the blog post
Arguments Usage
| Argument | type | Default | Explanation |
|---|---|---|---|
| batch_size | int | 40 | batch size for training and inferece |
| num_folds | int | 5 | number of fold for Stratified KFold |
| num_train_epochs | int | 5 | number of epochs for training |
| loss | str | focalloss | loss function |
| gamma | float | 1.0 | focalloss's gamma value |
| optimizer | str | adamp | optimizer for training |
| scheduler | str | get_cosine_schedule_with_warmup | learning rate scheduler |
| learning_rate | float | 0.00005 | initial learning rate |
| weight_decay | float | 0.01 | Loss function's weight decay, preventing overfit |
| warmup_step | int | 500 | |
| debug | bool | false | debug with CPU device for better error representation |
| dropout_rate | float | 0.1 | |
| save_steps | int | 100 | number of steps for saving the model |
| evaluation_steps | int | 100 | number of step until the evaluation |
| metric_for_best_model | str | eval/loss | the metric for determining which is the best model |
| load_best_model_at_end | bool | True |
References
- monologg's R-BERT Implementation in Pytorch
- Enriching Pre-trained Language Model with Entity Information for Relation Classification
- KLUE: Korean Language Understanding Evaluation
- KLUE RE (Relation Extraction) Dataset / CC BY-SA
- Pororo's NER Documentation
Authorship
Hardware
GPU : Tesla V100 32GB
29 Oct 23, 2022
9 Dec 08, 2022
1 Feb 11, 2022
34 Nov 24, 2022
11 Aug 26, 2022
289 Jan 06, 2023
169 Dec 21, 2022
2 Feb 17, 2022
4.8k Jan 03, 2023
5 Jan 03, 2022
1.4k Dec 30, 2022
213 Dec 22, 2022
102 Dec 19, 2022
8 Dec 22, 2022
611 Dec 16, 2022
1.2k Dec 23, 2022
2 Oct 02, 2022
409 Jan 06, 2023
1.1k Dec 27, 2022
3.9k Jan 03, 2023