Language Modelling On Wikitext 103

Metrics

Number of params

Test perplexity

Results

Performance results of various models on this benchmark

Model Name	Number of params	Test perplexity	Paper Title	Repository
Transformer-XL Large + Phrase Induction	257M	17.4	Improving Neural Language Models by Segmenting, Attending, and Predicting the Future
AWD-LSTM-MoS + ATOI	-	32.85	Alleviating Sequence Information Loss with Data Overlapping and Prime Batch Sizes
LSTM (Hebbian)	-	34.3	Fast Parametric Learning with Activation Memorization	-
Reformer 125M	-	26.0	Reformer: The Efficient Transformer
LSTM	-	-	How much complexity does an RNN architecture need to learn syntax-sensitive dependencies?
GCNN-8	-	44.9	Language Modeling with Gated Convolutional Networks
Transformer-XL Large	257M	18.3	Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context
GRU	-	-	How much complexity does an RNN architecture need to learn syntax-sensitive dependencies?
Subformer	96M	20.39	Subformer: A Parameter Reduced Transformer	-
Routing Transformer	-	15.8	Efficient Content-Based Sparse Attention with Routing Transformers
SRU++ Base	148M	18.3	When Attention Meets Fast Recurrence: Training Language Models with Reduced Compute
SRU++ Large	234M	17.1	When Attention Meets Fast Recurrence: Training Language Models with Reduced Compute
DIFFQ (λ=1, g=16)	-	18.0	Differentiable Model Compression via Pseudo Quantization Noise
Transformer+SSA+Self-ensemble	-	17.18	The Information Pathways Hypothesis: Transformers are Dynamic Self-Ensembles
GPT-2 Large	774M	22.05	Language Models are Unsupervised Multitask Learners	-
Primal.+Trans.	-	31.0	Primal-Attention: Self-attention through Asymmetric Kernel SVD in Primal Representation
Staged Training	247M	17.56	Shortformer: Better Language Modeling using Shorter Inputs
Hybrid H3 (355M)	355M	16.9	Hungry Hungry Hippos: Towards Language Modeling with State Space Models
Transformer-XL (RMS dynamic eval)	257M	16.4	Dynamic Evaluation of Transformer Language Models
Decay RNN	-	-	How much complexity does an RNN architecture need to learn syntax-sensitive dependencies?

0 of 89 row(s) selected.