Modelpublic

neulab/gpt2-finetuned-wikitext103

sourceHugging Faceupdated 4y agoView on Hugging Face

3likes453downloads

Model Card

This is a gpt2 model, finetuned on the Wikitext-103 dataset.

It achieves a perplexity of 14.84 using a "sliding window" context, using the run_clm.py script at https://github.com/neulab/knn-transformers.

Base LM:	`distilgpt2`	`gpt2`
base perplexity	18.25	14.84
+kNN-LM	15.03	12.57
+RetoMaton	14.70	12.46

This model was released as part of the paper "Neuro-Symbolic Language Modeling with Automaton-augmented Retrieval" (ICML'2022).

For more information, see: https://github.com/neulab/knn-transformers

If you use this model, please cite:

@inproceedings{alon2022neuro,
  title={Neuro-Symbolic Language Modeling with Automaton-augmented Retrieval},
  author={Alon, Uri and Xu, Frank and He, Junxian and Sengupta, Sudipta and Roth, Dan and Neubig, Graham},
  booktitle={International Conference on Machine Learning},
  pages={468--485},
  year={2022},
  organization={PMLR}
}