Language Modelling On Enwiki8

评估指标

Bit per Character (BPC)

Number of params

评测结果

各个模型在此基准测试上的表现结果

模型名称	Bit per Character (BPC)	Number of params	Paper Title	Repository
64-layer Character Transformer Model	1.11	44M	Character-Level Language Modeling with Deeper Self-Attention
Transformer-XL (12 layers)	1.06	41M	Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context
Longformer (12 layers, h=512)	1.00	41M	Longformer: The Long-Document Transformer
SHA-LSTM (4 layers, h=1024, no attention head)	1.33	51M	Single Headed Attention RNN: Stop Thinking With Your Head
Transformer-LS (large)	0.97	110M	Long-Short Transformer: Efficient Transformers for Language and Vision
Large mLSTM	1.24	46M	Multiplicative LSTM for sequence modelling	-
Hypernetworks	1.34	27M	HyperNetworks
Feedback Transformer	0.96	77M	Addressing Some Limitations of Transformers with Feedback Memory
Transformer (12 layers, 8k adaptive span)	1.02	39M	Adaptive Attention Span in Transformers
GPT-2 (48 layers, h=1600)	0.93	1542M	Language Models are Unsupervised Multitask Learners	-
Transformer-XL (24 layers)	0.99	277M	Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context
Cluster-Former (#C=512)	1.22	-	Cluster-Former: Clustering-based Sparse Transformer for Long-Range Dependency Encoding	-
Transformer-XL (18 layers)	1.03	88M	Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context
Compressive Transformer (24 layers)	0.97	277M	Compressive Transformers for Long-Range Sequence Modelling
LSTM (7 layers)	1.67	-	Generating Sequences With Recurrent Neural Networks
Transformer (24 layers, 8k adaptive span)	0.98	209M	Adaptive Attention Span in Transformers
Focus	0.940	22M	Focus Your Attention (with Adaptive IIR Filters)	-
Transformer-XL (24 layers, RMS dynamic eval, decay)	0.940	277M	Dynamic Evaluation of Transformer Language Models
LN HM-LSTM	1.32	35M	Hierarchical Multiscale Recurrent Neural Networks
Expire-Span (24 layers)	0.95	208M	Not All Memories are Created Equal: Learning to Forget by Expiring	-

0 of 42 row(s) selected.