Language Modelling On Wikitext 103

평가 지표

Number of params

Test perplexity

평가 결과

이 벤치마크에서 각 모델의 성능 결과

비교 표

모델 이름	Number of params	Test perplexity
improving-neural-language-models-by	257M	17.4
alleviating-sequence-information-loss-with	-	32.85
fast-parametric-learning-with-activation	-	34.3
reformer-the-efficient-transformer-1	-	26.0
how-much-complexity-does-an-rnn-architecture	-	-
language-modeling-with-gated-convolutional	-	44.9
transformer-xl-attentive-language-models	257M	18.3
how-much-complexity-does-an-rnn-architecture	-	-
subformer-a-parameter-reduced-transformer	96M	20.39
efficient-content-based-sparse-attention-with-1	-	15.8
when-attention-meets-fast-recurrence-training	148M	18.3
when-attention-meets-fast-recurrence-training	234M	17.1
differentiable-model-compression-via-pseudo	-	18.0
the-information-pathways-hypothesis	-	17.18
language-models-are-unsupervised-multitask	774M	22.05
primal-attention-self-attention-through	-	31.0
shortformer-better-language-modeling-using	247M	17.56
hungry-hungry-hippos-towards-language	355M	16.9
dynamic-evaluation-of-transformer-language	257M	16.4
how-much-complexity-does-an-rnn-architecture	-	-
hungry-hungry-hippos-towards-language	-	18.5
generalization-through-memorization-nearest	247M	15.79
fast-parametric-learning-with-activation	-	29.7
190409408	395M	20.4
finetuning-pretrained-transformers-into-rnns	-	19.6
improving-neural-language-modeling-via	-	28.0
an-analysis-of-neural-language-modeling-at	151M	33.0
on-the-adequacy-of-untuned-warmup-for	-	-
deep-equilibrium-models	180M	29.0
transformer-xl-attentive-language-models	151M	24.0
rethinking-attention-with-performers	-	26.8
adaptive-input-representations-for-neural	247M	18.70
accessing-higher-level-representations-in	139M	18.2
augmenting-self-attention-with-persistent	133M	20.6
all-nlp-tasks-are-generation-tasks-a-general	10000M	12.22
improving-neural-language-models-with-a	-	44.8
infty-former-infinite-memory-transformer	-	16.64
gateloop-fully-data-controlled-linear	125M	13.4
improving-neural-language-models-with-a	-	40.8
transformers-are-rnns-fast-autoregressive	-	25.6
general-purpose-long-context-autoregressive	-	18.4
hungry-hungry-hippos-towards-language	1300M	12.5
efficiently-modeling-long-sequences-with-1	249M	21.28
infty-former-infinite-memory-transformer	-	24.22
language-models-are-unsupervised-multitask	124M	37.50
language-models-are-unsupervised-multitask	1542M	17.48
dynamic-evaluation-of-transformer-language	257M	17.0
you-can-t-pick-your-neighbors-or-can-you-when	247M	15.5
random-feature-attention-1	-	30.5
infty-former-infinite-memory-transformer	-	16.61
the-information-pathways-hypothesis	-	17.60
language-models-are-unsupervised-multitask	355M	26.37
hyena-hierarchy-towards-larger-convolutional	-	18.6
improving-transformer-models-by-reordering	247M	17.96
an-empirical-evaluation-of-generic	-	45.19
language-modeling-with-gated-convolutional	-	37.2
advancing-state-of-the-art-in-language	-	13.29
time-aware-large-kernel-convolutions	240M	23.3
segabert-pre-training-of-segment-aware-bert	257M	17.1
convolutional-sequence-modeling-revisited	-	45.2
megatron-lm-training-multi-billion-parameter	8300M	10.81
improving-neural-language-models-with-a	-	48.7
pay-attention-when-required	-	22.7
compressive-transformers-for-long-range-1	-	17.1
deep-equilibrium-models	110M	23.2
mega-moving-average-equipped-gated-attention	252M	18.07
fnetar-mixing-tokens-with-autoregressive	144.4M	25.81
pay-attention-when-required	-	18.4
generalization-through-memorization-nearest	247M	16.12
random-feature-attention-1	-	23.5
fast-parametric-learning-with-activation	-	36.4
accessing-higher-level-representations-in	44M	22.4
delight-very-deep-and-light-weight	99M	24.14
hungry-hungry-hippos-towards-language	2700M	10.6
memory-efficient-stochastic-methods-for	122M	22.91
relational-recurrent-neural-networks	-	31.6
all-nlp-tasks-are-generation-tasks-a-general	10000M	11.33
improving-language-models-by-retrieving-from	7532M	2.4
deep-equilibrium-models	138M	32.4
hyena-hierarchy-towards-larger-convolutional	-	18.5
fast-parametric-learning-with-activation	-	29.2
infty-former-infinite-memory-transformer	-	24.22
infty-former-infinite-memory-transformer	-	24.22
shortformer-better-language-modeling-using	247M	18.15
trellis-networks-for-sequence-modeling	-	29.19
revisiting-simple-neural-probabilistic	148M	25.2
infty-former-infinite-memory-transformer	-	16.61
infty-former-infinite-memory-transformer	-	16.61
hungry-hungry-hippos-towards-language	125M	23.7