GPT - gMLP
This repository will attempt to crack long context autoregressive language modeling (GPT) using variations of gMLPs. Specifically, it will contain a variant that does gMLP for local sliding windows. The hope is to be able to stretch a single GPU to be able to train context lengths of 4096 and above efficiently and well.
GPT is technically a misnomer now, since there will be no attention (transformer) at all contained in the architecture.
Install
$ pip install g-mlp-gpt
Usage
import torch
from g_mlp_gpt import gMLPGPT
model = gMLPGPT(
num_tokens = 20000,
dim = 512,
depth = 4,
seq_len = 1024,
window_size = (128, 256, 512, 1024) # window sizes for each depth
)
x = torch.randint(0, 20000, (1, 1000))
logits = model(x) # (1, 1000, 20000)
16k context length
import torch
from g_mlp_gpt import gMLPGPT
model = gMLPGPT(
num_tokens = 20000,
dim = 512,
seq_len = 16384,
depth = 8,
reversible = True,
window = (128, 128, 256, 512, 1024, 1024, 2048, 2048, 4096, 4096, 8192, 8192),
axial = (1, 1, 1, 1, 1, 1, 2, 2, 4, 4, 8, 8)
).cuda()
x = torch.randint(0, 20000, (1, 16384)).cuda()
logits = model(x) # (1, 16384, 20000)
Citations
@misc{liu2021pay,
title = {Pay Attention to MLPs},
author = {Hanxiao Liu and Zihang Dai and David R. So and Quoc V. Le},
year = {2021},
eprint = {2105.08050},
archivePrefix = {arXiv},
primaryClass = {cs.LG}
}
9 Nov 16, 2022
24 Oct 26, 2022
331 Jan 3, 2023
74 Dec 5, 2022
28 Feb 20, 2022
81 Nov 26, 2022
39 Nov 30, 2022
49 Dec 22, 2022
120 Dec 12, 2022
1.3k Jan 08, 2023
1 Dec 14, 2021
34 Nov 07, 2022
8 Jan 02, 2023
1 Jan 16, 2022
114 Dec 30, 2022
57 Nov 17, 2022
213 Dec 17, 2022
54 Dec 07, 2022
17 Dec 10, 2022
153 Jan 04, 2023
50 Nov 18, 2022
2.2k Jan 05, 2023
18 Dec 23, 2022
23 Dec 26, 2022
246 Dec 13, 2022
19 Dec 12, 2022
236 Dec 21, 2022
9 Dec 07, 2022