Language Modelling On Text8

評価指標

Bit per Character (BPC)

評価結果

このベンチマークにおける各モデルのパフォーマンス結果

モデル名	Bit per Character (BPC)	Paper Title	Repository
Transformer-LS (small)	1.09	Long-Short Transformer: Efficient Transformers for Language and Vision	-
Focus	0.98	Focus Your Attention (with Adaptive IIR Filters)	-
GPT-2	0.98	Language Models are Unsupervised Multitask Learners	-
All-attention network - 36 layers	1.08	Augmenting Self-attention with Persistent Memory	-
12L Transformer + 8K adaptive span	1.11	Adaptive Attention Span in Transformers	-
BP-Transformer - 12 Layers	1.11	BP-Transformer: Modelling Long-Range Context via Binary Partitioning	-
12-layer Character Transformer Model	1.18	Character-Level Language Modeling with Deeper Self-Attention	-
All-attention network - 18 layers	1.11	Augmenting Self-attention with Persistent Memory	-
td-LSTM (Zhang et al., 2016)	1.63	Architectural Complexity Measures of Recurrent Neural Networks	-
mLSTM + dynamic eval	1.19	Dynamic Evaluation of Neural Sequence Models	-
Large mLSTM +emb +WN +VD	1.27	Multiplicative LSTM for sequence modelling	-
LayerNorm HM-LSTM	1.29	Hierarchical Multiscale Recurrent Neural Networks	-
td-LSTM-large	1.49	Architectural Complexity Measures of Recurrent Neural Networks	-
GAM-RHN-10	1.157	Recurrent Highway Networks with Grouped Auxiliary Memory
Unregularised mLSTM	1.40	Multiplicative LSTM for sequence modelling	-
Transformer-XL + RMS dynamic eval + decay	1.038	Dynamic Evaluation of Transformer Language Models	-
Bipartite flows (8 flows)	1.23	Discrete Flows: Invertible Generative Models of Discrete Data	-
BFN	1.41	Bayesian Flow Networks	-
24L Transformer + 8K adaptive span	1.07	Adaptive Attention Span in Transformers	-
64-layer Character Transformer Model	1.13	Character-Level Language Modeling with Deeper Self-Attention	-

0 of 24 row(s) selected.