“英特尔创新大师杯”深度学习挑战赛 赛道3：CCKS2021中文NLP地址相关性任务

Related tags

Deep Learningccks2021-track3-top1
Overview

ccks2021-track3

CCKS2021中文NLP地址相关性任务-赛道三-冠军方案

团队：我的加菲鱼- wodejiafeiyu

初赛第二/复赛第一/决赛第一

前言

19年开始，陆陆续续参加了一些比赛，拿到过一些top，比较懒一直都没分享过，这次比较幸运又拿了top1，打算分享下

分类的任务用这个框架基本都能top10，之前个人参加的《全球人工智能大赛-赛道一》和这个也差不多，只是有些具体任务的trick不同

比赛主页

https://tianchi.aliyun.com/competition/entrance/531901/introduction

环境

  • torch==1.6.0
  • transformers=3.0.2

预训练模型

- nezha-base 
- nezha-wwm
- macbert
  • 下载预训练模型，放到文件夹user_data/model_param/pretrain_model_param/下，文件夹和模型名字一一对应

全流程脚本

sh run.sh

核心思路

  • 预训练mlm中的mask策略使用ngram-mask，相比原始的动态mask，提升了预训练难度
  • 标签也包含语义信息，预训练部分融入标签信息，提升预训练效果
  • 混合精度预训练，损失一部分精度，提升整体的训练速度，实际测试结果精度损失不大，速度提升明显
  • 对抗训练，已经是一个比较常用的trick了
  • 后12层加上embedding的cls动态加权平均
  • multi-sample dropout
  • 推理的时候，阈值搜索
  • 三折交叉验证，每折使用不同的随机数种子使用dynamic pad
  • 加权平均，模型融合

总结

模块 提分点分析 提升
ngram-mask 相比于单个字的遮蔽，ngram-mask加大了预训练任务的难度，从而提升效果 2.6个千分点
融入标签信息预训练 标签也含有语义信息，模型学习的更多 1个千分点
后12层加上embedding的cls动态加权平均+multi-sample dropout 加权平均能增强向量的语义表征能力从而提升效果。multi-sample dropout能加速训练，增强泛化能力 3.3个千分点
对抗训练fgm 生成对抗样本，对抗样本的训练，增加模型泛化能力 2.2个千分点
多分类阈值搜索 缩放系数使得f1最优 0.7个千分点
模型融合 提升模型的鲁棒性和泛化能力 1.5个千分点
动态pad和预训练混合精度训练和每折使用不同的随机数种子 提升模型训练速度，不同的随机数种子加大了模型的差异，提升融合的效果
Owner
shaochenjie
shaochenjie
GitHub Repository
2021:"Bridging Global Context Interactions for High-Fidelity Image Completion"

TFill arXiv | Project This repository implements the training, testing and editing tools for "Bridging Global Context Interactions for High-Fidelity I

Chuanxia Zheng 111 Jan 08, 2023
Official code for MPG2: Multi-attribute Pizza Generator: Cross-domain Attribute Control with Conditional StyleGAN

This is the official code for Multi-attribute Pizza Generator (MPG2): Cross-domain Attribute Control with Conditional StyleGAN. Paper Demo Setup Envir

Fangda Han 5 Sep 01, 2022
Exadel CompreFace is a free and open-source face recognition GitHub project

Exadel CompreFace is a leading free and open-source face recognition system Exadel CompreFace is a free and open-source face recognition service that

Exadel 2.6k Jan 04, 2023
A modular framework for vision & language multimodal research from Facebook AI Research (FAIR)

MMF is a modular framework for vision and language multimodal research from Facebook AI Research. MMF contains reference implementations of state-of-t

Facebook Research 5.1k Jan 04, 2023
Graph Convolutional Networks in PyTorch

Graph Convolutional Networks in PyTorch PyTorch implementation of Graph Convolutional Networks (GCNs) for semi-supervised classification [1]. For a hi

Thomas Kipf 4.5k Dec 31, 2022
A Python package for causal inference using Synthetic Controls

Synthetic Control Methods A Python package for causal inference using synthetic controls This Python package implements a class of approaches to estim

Oscar Engelbrektson 107 Dec 28, 2022
Code for "Learning Structural Edits via Incremental Tree Transformations" (ICLR'21)

Learning Structural Edits via Incremental Tree Transformations Code for "Learning Structural Edits via Incremental Tree Transformations" (ICLR'21) 1.

NeuLab 40 Dec 23, 2022
Histology images query (unsupervised)

110-1-NTU-DBME5028-Histology-images-query Final Project: Histology images query (unsupervised) Kaggle: https://www.kaggle.com/c/histology-images-query

1 Jan 05, 2022
Alpha-IoU: A Family of Power Intersection over Union Losses for Bounding Box Regression

Alpha-IoU: A Family of Power Intersection over Union Losses for Bounding Box Regression YOLOv5 with alpha-IoU losses implemented in PyTorch. Example r

Jacobi(Jiabo He) 147 Dec 05, 2022
Breast Cancer Classification Model is applied on a different dataset

Breast Cancer Classification Model is applied on a different dataset

1 Feb 04, 2022
Stochastic Tensor Optimization for Robot Motion - A GPU Robot Motion Toolkit

STORM Stochastic Tensor Optimization for Robot Motion - A GPU Robot Motion Toolkit [Install Instructions] [Paper] [Website] This package contains code

NVIDIA Research Projects 101 Dec 12, 2022
Improving adversarial robustness by a coupling rejection strategy

Adversarial Training with Rectified Rejection The code for the paper Adversarial Training with Rectified Rejection. Environment settings and libraries

Tianyu Pang 29 Jan 06, 2023
A pre-trained language model for social media text in Spanish

RoBERTuito A pre-trained language model for social media text in Spanish READ THE FULL PAPER Github Repository RoBERTuito is a pre-trained language mo

25 Dec 29, 2022
Repository relating to the CVPR21 paper TimeLens: Event-based Video Frame Interpolation

TimeLens: Event-based Video Frame Interpolation This repository is about the High Speed Event and RGB (HS-ERGB) dataset, used in the 2021 CVPR paper T

Robotics and Perception Group 544 Dec 19, 2022
Machine learning algorithms for many-body quantum systems

NetKet NetKet is an open-source project delivering cutting-edge methods for the study of many-body quantum systems with artificial neural networks and

NetKet 413 Dec 31, 2022
Python scripts form performing stereo depth estimation using the CoEx model in ONNX.

ONNX-CoEx-Stereo-Depth-estimation Python scripts form performing stereo depth estimation using the CoEx model in ONNX. Stereo depth estimation on the

Ibai Gorordo 8 Dec 29, 2022
Convolutional neural network web app trained to track our infant’s sleep schedule using our Google Nest camera.

Machine Learning Sleep Schedule Tracker What is it? Convolutional neural network web app trained to track our infant’s sleep schedule using our Google

g-parki 7 Jul 15, 2022
ImageBART: Bidirectional Context with Multinomial Diffusion for Autoregressive Image Synthesis

ImageBART NeurIPS 2021 Patrick Esser*, Robin Rombach*, Andreas Blattmann*, Björn Ommer * equal contribution arXiv | BibTeX | Poster Requirements A sui

CompVis Heidelberg 110 Jan 01, 2023
Core ML tools contain supporting tools for Core ML model conversion, editing, and validation.

Core ML Tools Use coremltools to convert machine learning models from third-party libraries to the Core ML format. The Python package contains the sup

Apple 3k Jan 08, 2023
Weakly Supervised Posture Mining with Reverse Cross-entropy for Fine-grained Classification

Fine-grainedImageClassification Weakly Supervised Posture Mining with Reverse Cross-entropy for Fine-grained Classification We trained model here: lin

ZhenchaoTang 14 Oct 21, 2022