ALBERT-pytorch-implementation - ALBERT pytorch implementation

Last update: Oct 06, 2022

Overview

ALBERT-pytorch-implementation

developing...

모델의 개념이해를 돕기 위한 구현물로 현재 변수명을 상세히 적었고 변수를 Configuration화 하지 않고 중복하여 적어 구현하였습니다.

2022-01-05

MultiHeadAttention.py 구현 완료

2022-01-06

Transformer.py 구현 완료
- Position-wise FeedForward Network 구현
- Sublayer Connection 구현
- GELU이용
ALBERT.py 구현 중 ... (Embedding 미구현)

2022-01-07

Embeddings.py 구현 완료
- Position Embedding 구현
- Token Embedding 구현
- Segment Embedding 구현 (SOP 적용할 시 필요하여 일단 구현)
ALBERT.py 구현 완료
- Hidden Projection 구현
- Staked Layer iteration 가능

TODO Lists.

~~Embedding 구현~~
Masked Language Model + SOP 구현
변수 Configuration 화
코드 정리 및 Package 화

IDEA (Further Implementations)

Compression
- Pruning
- Quantization (clustering method)
- Distillation

Owner

BG Kim

GitHub Repository

Application of K-means algorithm on a music dataset after a dimensionality reduction with PCA

PCA for dimensionality reduction combined with Kmeans Goal The Goal of this notebook is to apply a dimensionality reduction on a big dataset in order

0 Sep 17, 2022

Code of the paper "Part Detector Discovery in Deep Convolutional Neural Networks" by Marcel Simon, Erik Rodner and Joachim Denzler

Part Detector Discovery This is the code used in our paper "Part Detector Discovery in Deep Convolutional Neural Networks" by Marcel Simon, Erik Rodne

17 Feb 22, 2022

Learning from Synthetic Data with Fine-grained Attributes for Person Re-Identification

Less is More: Learning from Synthetic Data with Fine-grained Attributes for Person Re-Identification Suncheng Xiang Shanghai Jiao Tong University Over

68 Dec 13, 2022

Small utility to demangle Nim symbols in callgrind files

nim_callgrind A small utility to demangle Nim symbols from callgrind files. Usage Run your (Nim) program with something like this: valgrind --tool=cal

3 Feb 15, 2022

Meta graph convolutional neural network-assisted resilient swarm communications

Resilient UAV Swarm Communications with Graph Convolutional Neural Network This repository contains the source codes of Resilient UAV Swarm Communicat

62 Dec 06, 2022

Replication Package for AequeVox:Automated Fariness Testing for Speech Recognition Systems

AequeVox Replication Package for AequeVox:Automated Fariness Testing for Speech Recognition Systems README under development. Python Packages Required

2 Aug 28, 2022

Code release for paper: The Boombox: Visual Reconstruction from Acoustic Vibrations

The Boombox: Visual Reconstruction from Acoustic Vibrations Boyuan Chen, Mia Chiquier, Hod Lipson, Carl Vondrick Columbia University Project Website |

12 Nov 30, 2022

Official Tensorflow implementation of U-GAT-IT: Unsupervised Generative Attentional Networks with Adaptive Layer-Instance Normalization for Image-to-Image Translation (ICLR 2020)

U-GAT-IT — Official TensorFlow Implementation (ICLR 2020) : Unsupervised Generative Attentional Networks with Adaptive Layer-Instance Normalization fo

6.2k Jan 04, 2023

Active learning for Mask R-CNN in Detectron2

MaskAL - Active learning for Mask R-CNN in Detectron2 Summary MaskAL is an active learning framework that automatically selects the most-informative i

49 Dec 20, 2022

MonoRec: Semi-Supervised Dense Reconstruction in Dynamic Environments from a Single Moving Camera

494 Jan 06, 2023

Ensemble Learning Priors Driven Deep Unfolding for Scalable Snapshot Compressive Imaging [PyTorch]

Ensemble Learning Priors Driven Deep Unfolding for Scalable Snapshot Compressive Imaging [PyTorch] Abstract Snapshot compressive imaging (SCI) can rec

6 Nov 01, 2022

Semantic Edge Detection with Diverse Deep Supervision

Semantic Edge Detection with Diverse Deep Supervision This repository contains the code for our IJCV paper: "Semantic Edge Detection with Diverse Deep

12 Dec 31, 2022

Riemannian Convex Potential Maps

Modeling distributions on Riemannian manifolds is a crucial component in understanding non-Euclidean data that arises, e.g., in physics and geology. The budding approaches in this space are limited b

61 Nov 28, 2022

This is the repository for the AAAI 21 paper [Contrastive and Generative Graph Convolutional Networks for Graph-based Semi-Supervised Learning].

CG3 This is the repository for the AAAI 21 paper [Contrastive and Generative Graph Convolutional Networks for Graph-based Semi-Supervised Learning]. R

12 Oct 28, 2022

Learning Synthetic Environments and Reward Networks for Reinforcement Learning

Learning Synthetic Environments and Reward Networks for Reinforcement Learning We explore meta-learning agent-agnostic neural Synthetic Environments (

16 Sep 02, 2022

Implementation of FSGNN

FSGNN Implementation of FSGNN. For more details, please refer to our paper Experiments were conducted with following setup: Pytorch: 1.6.0 Python: 3.8

19 Dec 05, 2022

Optimizing DR with hard negatives and achieving SOTA first-stage retrieval performance on TREC DL Track (SIGIR 2021 Full Paper).

Optimizing Dense Retrieval Model Training with Hard Negatives Jingtao Zhan, Jiaxin Mao, Yiqun Liu, Jiafeng Guo, Min Zhang, Shaoping Ma 🔥 News 2021-10

99 Dec 27, 2022

A module that used for encrypt code which includes RSA and AES

软件加密模块 requirement： Crypto,pycryptodome,pyqt5 本地加密信息为随机字符串使用说明命令行参数 -h 帮助 -checkWorking 检查是否能正常工作，后接1确认指令 -checkEndDate 检查截至日期，后接1确认指令 -activateCode

2 Sep 27, 2022

MolRep: A Deep Representation Learning Library for Molecular Property Prediction

MolRep: A Deep Representation Learning Library for Molecular Property Prediction Summary MolRep is a Python package for fairly measuring algorithmic p

83 Dec 24, 2022

ANN model for prediction a spatio-temporal distribution of supercooled liquid in mixed-phase clouds using Doppler cloud radar spectra.

VOODOO Revealing supercooled liquid beyond lidar attenuation Explore the docs » Report Bug · Request Feature Table of Contents About The Project Built

2 Apr 28, 2022