Language Modelling On One Billion Word

评估指标

Number of params

PPL

评测结果

各个模型在此基准测试上的表现结果

模型名称	Number of params	PPL	Paper Title	Repository
OmniNetT (Large)	100M	21.5	OmniNet: Omnidirectional Representations from Transformers	-
LSTM-8192-1024 + CNN Input	1.04B	30.0	Exploring the Limits of Language Modeling	-
Cohere Large	-	25.06	-	-
Transformer-XL Large	0.8B	21.8	Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context	-
Transformer-XL Base	0.46B	23.5	Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context	-
Adaptive Input Large	0.46B	23.91	Adaptive Input Representations for Neural Language Modeling	-
DynamicConv	0.34B	26.67	Pay Less Attention with Lightweight and Dynamic Convolutions	-
Adaptive Input Very Large	1.0B	23.02	Adaptive Input Representations for Neural Language Modeling	-
RNN-1024 + 9 Gram	20B	51.3	One Billion Word Benchmark for Measuring Progress in Statistical Language Modeling	-
OmniNetB (Large)	-	22	OmniNet: Omnidirectional Representations from Transformers	-
GPT-2	1.54B	42.16	Language Models are Unsupervised Multitask Learners	-
Evolved Transformer Big	-	28.6	The Evolved Transformer	-
OmniNetP (Large)	100M	21.6	OmniNet: Omnidirectional Representations from Transformers	-
Mesh Tensorflow	4.9B	24.0	Mesh-TensorFlow: Deep Learning for Supercomputers	-
SRU++ Large	465M	23.5	When Attention Meets Fast Recurrence: Training Language Models with Reduced Compute	-
Low-Budget MoE	5B	34.1	Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer	-
BIG G-LSTM-2	-	36.0	Factorization tricks for LSTM networks	-
LSTM-8192-1024	1.8B	30.6	Exploring the Limits of Language Modeling	-
GCNN-14 bottleneck	-	31.9	Language Modeling with Gated Convolutional Networks	-
Sparse Non-Negative	33B	52.9	Skip-gram Language Modeling Using Sparse Non-negative Matrix Probability Estimation	-

0 of 27 row(s) selected.