Language Modelling On Enwiki8

Metrics

Bit per Character (BPC)

Number of params

Results

Performance results of various models on this benchmark

Model Name	Bit per Character (BPC)	Number of params	Paper Title	Repository
64-layer Character Transformer Model	1.11	44M	Character-Level Language Modeling with Deeper Self-Attention	-
Transformer-XL (12 layers)	1.06	41M	Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context	-
Longformer (12 layers, h=512)	1.00	41M	Longformer: The Long-Document Transformer	-
SHA-LSTM (4 layers, h=1024, no attention head)	1.33	51M	Single Headed Attention RNN: Stop Thinking With Your Head	-
Transformer-LS (large)	0.97	110M	Long-Short Transformer: Efficient Transformers for Language and Vision	-
Large mLSTM	1.24	46M	Multiplicative LSTM for sequence modelling	-
Hypernetworks	1.34	27M	HyperNetworks	-
Feedback Transformer	0.96	77M	Addressing Some Limitations of Transformers with Feedback Memory	-
Transformer (12 layers, 8k adaptive span)	1.02	39M	Adaptive Attention Span in Transformers	-
GPT-2 (48 layers, h=1600)	0.93	1542M	Language Models are Unsupervised Multitask Learners	-
Transformer-XL (24 layers)	0.99	277M	Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context	-
Cluster-Former (#C=512)	1.22	-	Cluster-Former: Clustering-based Sparse Transformer for Long-Range Dependency Encoding	-
Transformer-XL (18 layers)	1.03	88M	Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context	-
Compressive Transformer (24 layers)	0.97	277M	Compressive Transformers for Long-Range Sequence Modelling	-
LSTM (7 layers)	1.67	-	Generating Sequences With Recurrent Neural Networks	-
Transformer (24 layers, 8k adaptive span)	0.98	209M	Adaptive Attention Span in Transformers	-
Focus	0.940	22M	Focus Your Attention (with Adaptive IIR Filters)	-
Transformer-XL (24 layers, RMS dynamic eval, decay)	0.940	277M	Dynamic Evaluation of Transformer Language Models	-
LN HM-LSTM	1.32	35M	Hierarchical Multiscale Recurrent Neural Networks	-
Expire-Span (24 layers)	0.95	208M	Not All Memories are Created Equal: Learning to Forget by Expiring	-

0 of 42 row(s) selected.