Language Modelling On Text8

评估指标

Bit per Character (BPC)

评测结果

各个模型在此基准测试上的表现结果

模型名称	Bit per Character (BPC)	Paper Title	Repository
Transformer-LS (small)	1.09	Long-Short Transformer: Efficient Transformers for Language and Vision
Focus	0.98	Focus Your Attention (with Adaptive IIR Filters)	-
GPT-2	0.98	Language Models are Unsupervised Multitask Learners	-
All-attention network - 36 layers	1.08	Augmenting Self-attention with Persistent Memory
12L Transformer + 8K adaptive span	1.11	Adaptive Attention Span in Transformers
BP-Transformer - 12 Layers	1.11	BP-Transformer: Modelling Long-Range Context via Binary Partitioning	-
12-layer Character Transformer Model	1.18	Character-Level Language Modeling with Deeper Self-Attention
All-attention network - 18 layers	1.11	Augmenting Self-attention with Persistent Memory
td-LSTM (Zhang et al., 2016)	1.63	Architectural Complexity Measures of Recurrent Neural Networks	-
mLSTM + dynamic eval	1.19	Dynamic Evaluation of Neural Sequence Models
Large mLSTM +emb +WN +VD	1.27	Multiplicative LSTM for sequence modelling	-
LayerNorm HM-LSTM	1.29	Hierarchical Multiscale Recurrent Neural Networks
td-LSTM-large	1.49	Architectural Complexity Measures of Recurrent Neural Networks	-
GAM-RHN-10	1.157	Recurrent Highway Networks with Grouped Auxiliary Memory
Unregularised mLSTM	1.40	Multiplicative LSTM for sequence modelling	-
Transformer-XL + RMS dynamic eval + decay	1.038	Dynamic Evaluation of Transformer Language Models
Bipartite flows (8 flows)	1.23	Discrete Flows: Invertible Generative Models of Discrete Data
BFN	1.41	Bayesian Flow Networks
24L Transformer + 8K adaptive span	1.07	Adaptive Attention Span in Transformers
64-layer Character Transformer Model	1.13	Character-Level Language Modeling with Deeper Self-Attention

0 of 24 row(s) selected.