Language Modelling On Wikitext 103

评估指标

Number of params

Test perplexity

评测结果

各个模型在此基准测试上的表现结果

模型名称	Number of params	Test perplexity	Paper Title	Repository
Transformer-XL Large + Phrase Induction	257M	17.4	Improving Neural Language Models by Segmenting, Attending, and Predicting the Future	-
AWD-LSTM-MoS + ATOI	-	32.85	Alleviating Sequence Information Loss with Data Overlapping and Prime Batch Sizes	-
LSTM (Hebbian)	-	34.3	Fast Parametric Learning with Activation Memorization	-
Reformer 125M	-	26.0	Reformer: The Efficient Transformer	-
LSTM	-	-	How much complexity does an RNN architecture need to learn syntax-sensitive dependencies?	-
GCNN-8	-	44.9	Language Modeling with Gated Convolutional Networks	-
Transformer-XL Large	257M	18.3	Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context	-
GRU	-	-	How much complexity does an RNN architecture need to learn syntax-sensitive dependencies?	-
Subformer	96M	20.39	Subformer: A Parameter Reduced Transformer	-
Routing Transformer	-	15.8	Efficient Content-Based Sparse Attention with Routing Transformers	-
SRU++ Base	148M	18.3	When Attention Meets Fast Recurrence: Training Language Models with Reduced Compute	-
SRU++ Large	234M	17.1	When Attention Meets Fast Recurrence: Training Language Models with Reduced Compute	-
DIFFQ (λ=1, g=16)	-	18.0	Differentiable Model Compression via Pseudo Quantization Noise	-
Transformer+SSA+Self-ensemble	-	17.18	The Information Pathways Hypothesis: Transformers are Dynamic Self-Ensembles	-
GPT-2 Large	774M	22.05	Language Models are Unsupervised Multitask Learners	-
Primal.+Trans.	-	31.0	Primal-Attention: Self-attention through Asymmetric Kernel SVD in Primal Representation	-
Staged Training	247M	17.56	Shortformer: Better Language Modeling using Shorter Inputs	-
Hybrid H3 (355M)	355M	16.9	Hungry Hungry Hippos: Towards Language Modeling with State Space Models	-
Transformer-XL (RMS dynamic eval)	257M	16.4	Dynamic Evaluation of Transformer Language Models	-
Decay RNN	-	-	How much complexity does an RNN architecture need to learn syntax-sensitive dependencies?	-

0 of 89 row(s) selected.