Language Modelling On Enwiki8

المقاييس

Bit per Character (BPC)

Number of params

النتائج

نتائج أداء النماذج المختلفة على هذا المعيار القياسي

اسم النموذج	Bit per Character (BPC)	Number of params	Paper Title	Repository
64-layer Character Transformer Model	1.11	44M	Character-Level Language Modeling with Deeper Self-Attention
Transformer-XL (12 layers)	1.06	41M	Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context
Longformer (12 layers, h=512)	1.00	41M	Longformer: The Long-Document Transformer
SHA-LSTM (4 layers, h=1024, no attention head)	1.33	51M	Single Headed Attention RNN: Stop Thinking With Your Head
Transformer-LS (large)	0.97	110M	Long-Short Transformer: Efficient Transformers for Language and Vision
Large mLSTM	1.24	46M	Multiplicative LSTM for sequence modelling	-
Hypernetworks	1.34	27M	HyperNetworks
Feedback Transformer	0.96	77M	Addressing Some Limitations of Transformers with Feedback Memory
Transformer (12 layers, 8k adaptive span)	1.02	39M	Adaptive Attention Span in Transformers
GPT-2 (48 layers, h=1600)	0.93	1542M	Language Models are Unsupervised Multitask Learners	-
Transformer-XL (24 layers)	0.99	277M	Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context
Cluster-Former (#C=512)	1.22	-	Cluster-Former: Clustering-based Sparse Transformer for Long-Range Dependency Encoding	-
Transformer-XL (18 layers)	1.03	88M	Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context
Compressive Transformer (24 layers)	0.97	277M	Compressive Transformers for Long-Range Sequence Modelling
LSTM (7 layers)	1.67	-	Generating Sequences With Recurrent Neural Networks
Transformer (24 layers, 8k adaptive span)	0.98	209M	Adaptive Attention Span in Transformers
Focus	0.940	22M	Focus Your Attention (with Adaptive IIR Filters)	-
Transformer-XL (24 layers, RMS dynamic eval, decay)	0.940	277M	Dynamic Evaluation of Transformer Language Models
LN HM-LSTM	1.32	35M	Hierarchical Multiscale Recurrent Neural Networks
Expire-Span (24 layers)	0.95	208M	Not All Memories are Created Equal: Learning to Forget by Expiring	-

0 of 42 row(s) selected.