Implementation of Fast Transformer in Pytorch

Last update: Dec 27, 2022

Overview

Fast Transformer - Pytorch

Implementation of Fast Transformer in Pytorch. This only work as an encoder.

Install

$ pip install fast-transformer-pytorch

Usage

import torch
from fast_transformer_pytorch import FastTransformer

model = FastTransformer(
    num_tokens = 20000,
    dim = 512,
    depth = 2,
    max_seq_len = 4096,
    absolute_pos_emb = True   # default uses relative positional encoding, but if that isn't working, then turn on absolute positional embedding by setting this to True
)

x = torch.randint(0, 20000, (1, 4096))
mask = torch.ones(1, 4096).bool()

logits = model(x, mask = mask) # (1, 4096, 20000)

Citations

@misc{wu2021fastformer,
    title   = {Fastformer: Additive Attention is All You Need}, 
    author  = {Chuhan Wu and Fangzhao Wu and Tao Qi and Yongfeng Huang},
    year    = {2021},
    eprint  = {2108.09084},
    archivePrefix = {arXiv},
    primaryClass = {cs.CL}
}

A Transformer Implementation that is easy to understand and customizable.

Simple Transformer I've written a series of articles on the transformer architecture and language models on Medium. This repository contains an implem

4 Jan 20, 2022

Fast topic modeling platform

The state-of-the-art platform for topic modeling. Full Documentation User Mailing List Download Releases User survey What is BigARTM? BigARTM is a pow

633 Dec 21, 2022

Easy, fast, effective, and automatic g-code compression!

Getting to the meat of g-code. Easy, fast, effective, and automatic g-code compression! MeatPack nearly doubles the effective data rate of a standard

97 Nov 21, 2022

Library for fast text representation and classification.

fastText fastText is a library for efficient learning of word representations and sentence classification. Table of contents Resources Models Suppleme

24.1k Jan 5, 2023

💥 Fast State-of-the-Art Tokenizers optimized for Research and Production

Provides an implementation of today's most used tokenizers, with a focus on performance and versatility. Main features: Train new vocabularies and tok

6.2k Dec 31, 2022

✨Fast Coreference Resolution in spaCy with Neural Networks

✨ NeuralCoref 4.0: Coreference Resolution in spaCy with Neural Networks. NeuralCoref is a pipeline extension for spaCy 2.1+ which annotates and resolv

2.6k Jan 4, 2023

:house_with_garden: Fast & easy transfer learning for NLP. Harvesting language models for the industry. Focus on Question Answering.

(Framework for Adapting Representation Models) What is it? FARM makes Transfer Learning with BERT & Co simple, fast and enterprise-ready. It's built u

1.6k Dec 27, 2022

Library for fast text representation and classification.

fastText fastText is a library for efficient learning of word representations and sentence classification. Table of contents Resources Models Suppleme

22.2k Feb 18, 2021

💥 Fast State-of-the-Art Tokenizers optimized for Research and Production

Provides an implementation of today's most used tokenizers, with a focus on performance and versatility. Main features: Train new vocabularies and tok

4.3k Feb 18, 2021

Releases(0.0.4)

0.0.4(Aug 25, 2021)

Source code(tar.gz)
Source code(zip)
0.0.3(Aug 24, 2021)

Source code(tar.gz)
Source code(zip)
0.0.2(Aug 23, 2021)

Source code(tar.gz)
Source code(zip)
0.0.1(Aug 23, 2021)

Source code(tar.gz)
Source code(zip)

Owner

Phil Wang

Working with Attention. It's all we need

GitHub Repository

Implementation of Token Shift GPT - An autoregressive model that solely relies on shifting the sequence space for mixing

Token Shift GPT Implementation of Token Shift GPT - An autoregressive model that relies solely on shifting along the sequence dimension and feedforwar

32 Oct 14, 2022

OpenChat: Opensource chatting framework for generative models

OpenChat is opensource chatting framework for generative models.

427 Jan 06, 2023

Making text a first-class citizen in TensorFlow.

TensorFlow Text - Text processing in Tensorflow IMPORTANT: When installing TF Text with pip install, please note the version of TensorFlow you are run

1k Dec 26, 2022

SpikeX - SpaCy Pipes for Knowledge Extraction

SpikeX is a collection of pipes ready to be plugged in a spaCy pipeline. It aims to help in building knowledge extraction tools with almost-zero effort.

384 Dec 12, 2022

A framework for training and evaluating AI models on a variety of openly available dialogue datasets.

ParlAI (pronounced “par-lay”) is a python framework for sharing, training and testing dialogue models, from open-domain chitchat, to task-oriented dia

9.7k Jan 09, 2023

SpeechBrain is an open-source and all-in-one speech toolkit based on PyTorch.

The goal is to create a single, flexible, and user-friendly toolkit that can be used to easily develop state-of-the-art speech technologies, including systems for speech recognition, speaker recognit

5.1k Jan 09, 2023

This repository contains the official release of the model "BanglaBERT" and associated downstream finetuning code and datasets introduced in the paper titled "BanglaBERT: Combating Embedding Barrier in Multilingual Models for Low-Resource Language Understanding".

BanglaBERT This repository contains the official release of the model "BanglaBERT" and associated downstream finetuning code and datasets introduced i

197 Dec 25, 2022

Implementation of Fast Transformer in Pytorch

Related tags

Overview

Fast Transformer - Pytorch

Install

Usage

Citations

You might also like...

A Transformer Implementation that is easy to understand and customizable.

Fast topic modeling platform

Easy, fast, effective, and automatic g-code compression!

Library for fast text representation and classification.

💥 Fast State-of-the-Art Tokenizers optimized for Research and Production

✨Fast Coreference Resolution in spaCy with Neural Networks

:house_with_garden: Fast & easy transfer learning for NLP. Harvesting language models for the industry. Focus on Question Answering.

Library for fast text representation and classification.

💥 Fast State-of-the-Art Tokenizers optimized for Research and Production

Releases(0.0.4)

0.0.4(Aug 25, 2021)

0.0.3(Aug 24, 2021)

0.0.2(Aug 23, 2021)

0.0.1(Aug 23, 2021)

Owner

Phil Wang

Implementation of Token Shift GPT - An autoregressive model that solely relies on shifting the sequence space for mixing

OpenChat: Opensource chatting framework for generative models

Making text a first-class citizen in TensorFlow.

SpikeX - SpaCy Pipes for Knowledge Extraction

A framework for training and evaluating AI models on a variety of openly available dialogue datasets.

SpeechBrain is an open-source and all-in-one speech toolkit based on PyTorch.

This repository contains the official release of the model "BanglaBERT" and associated downstream finetuning code and datasets introduced in the paper titled "BanglaBERT: Combating Embedding Barrier in Multilingual Models for Low-Resource Language Understanding".

用Resnet101+GPT搭建一个玩王者荣耀的AI

ACL'22: Structured Pruning Learns Compact and Accurate Models

A Japanese tokenizer based on recurrent neural networks

The code for two papers: Feedback Transformer and Expire-Span.

pytorch implementation of Attention is all you need

CodeBERT: A Pre-Trained Model for Programming and Natural Languages.

Sentiment Analysis Project using Count Vectorizer and TF-IDF Vectorizer

Gathers machine learning and Tensorflow deep learning models for NLP problems, 1.13 < Tensorflow < 2.0

Statistics and Mathematics for Machine Learning, Deep Learning , Deep NLP

Natural Language Processing Specialization

voice2json is a collection of command-line tools for offline speech/intent recognition on Linux

Implementation of paper Does syntax matter? A strong baseline for Aspect-based Sentiment Analysis with RoBERTa.

Prompt tuning toolkit for GPT-2 and GPT-Neo