Language Modelling On Wikitext 2

Métriques

Number of params

Test perplexity

Validation perplexity

Résultats

Résultats de performance de divers modèles sur ce benchmark

Nom du modèle	Number of params	Test perplexity	Validation perplexity	Paper Title	Repository
adversarial + AWD-LSTM-MoS + dynamic eval	35M	38.65	40.27	Improving Neural Language Modeling via Adversarial Training	-
AWD-LSTM-DOC	37M	58.03	60.29	Direct Output Connection for a High-Rank Language Model	-
Mogrifier LSTM	35M	55.1	57.3	Mogrifier LSTM	-
GPT-2 (fine-tuned)	1542M	15.17	15.69	Hydra: A System for Large Multi-Model Deep Learning	-
AWD-LSTM + dynamic eval	33M	44.3	46.4	Dynamic Evaluation of Neural Sequence Models	-
AWD-LSTM + ATOI	33M	64.73	67.47	Alleviating Sequence Information Loss with Data Overlapping and Prime Batch Sizes	-
Grave et al. (2016) - LSTM	-	99.3	-	Improving Neural Language Models with a Continuous Cache	-
OPT-175B (50% Sparsity)	-	234.77	-	SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot	-
GPT-2 (medium)	345M	22.76	-	Language Models are Unsupervised Multitask Learners	-
SparseGPT (175B, 50% Sparsity)	-	8.21	-	SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot	-
FRAGE + AWD-LSTM-MoS + dynamic eval	35M	39.14	40.85	FRAGE: Frequency-Agnostic Word Representation	-
GL-LWGC + AWD-MoS-LSTM + dynamic eval	38M	40.46	42.19	Gradual Learning of Recurrent Neural Networks	-
AWD-FWM Schlag et al. (2020)	37M	61.65	54.48	Learning Associative Inference Using Fast Weight Memory	-
GPT-2 (large)	762M	19.93	-	Language Models are Unsupervised Multitask Learners	-
AWD-LSTM-DRILL	34M	61.9	64.9	Deep Residual Output Layers for Neural Language Generation	-
GPT-2	1542M	18.34	-	Language Models are Unsupervised Multitask Learners	-
AWD-LSTM + continuous cache pointer	33M	52.0	53.8	Regularizing and Optimizing LSTM Language Models	-
Melis et al. (2017) - 1-layer LSTM (tied)	24M	65.9	69.3	On the State of the Art of Evaluation in Neural Language Models	-
Inan et al. (2016) - Variational LSTM (tied) (h=650)	-	87.7	92.3	Tying Word Vectors and Word Classifiers: A Loss Framework for Language Modeling	-
AWD-LSTM-MoS + dynamic eval	35M	40.68	42.41	Breaking the Softmax Bottleneck: A High-Rank RNN Language Model	-

0 of 38 row(s) selected.