funnel-transformer/medium-base

1---2language: en3license: apache-2.04datasets:5- bookcorpus6- wikipedia7- gigaword8---9 10# Funnel Transformer medium model (B6-3x2-3x2 without decoder)11 12Pretrained model on English language using a similar objective objective as [ELECTRA](https://huggingface.co/transformers/model_doc/electra.html). It was introduced in13[this paper](https://arxiv.org/pdf/2006.03236.pdf) and first released in14[this repository](https://github.com/laiguokun/Funnel-Transformer). This model is uncased: it does not make a difference15between english and English.16 17Disclaimer: The team releasing Funnel Transformer did not write a model card for this model so this model card has been18written by the Hugging Face team.19 20## Model description21 22Funnel Transformer is a transformers model pretrained on a large corpus of English data in a self-supervised fashion. This means it23was pretrained on the raw texts only, with no humans labelling them in any way (which is why it can use lots of24publicly available data) with an automatic process to generate inputs and labels from those texts. 25 26More precisely, a small language model corrupts the input texts and serves as a generator of inputs for this model, and27the pretraining objective is to predict which token is an original and which one has been replaced, a bit like a GAN training.28 29This way, the model learns an inner representation of the English language that can then be used to extract features30useful for downstream tasks: if you have a dataset of labeled sentences for instance, you can train a standard31classifier using the features produced by the BERT model as inputs.32 33**Note:** This model does not contain the decoder, so it ouputs hidden states that have a sequence length of one fourth34of the inputs. It's good to use for tasks requiring a summary of the sentence (like sentence classification) but not if35you need one input per initial token. You should use the `medium` model in that case.36 37## Intended uses & limitations38 39You can use the raw model to extract a vector representation of a given text, but it's mostly intended to40be fine-tuned on a downstream task. See the [model hub](https://huggingface.co/models?filter=funnel-transformer) to look for41fine-tuned versions on a task that interests you.42 43Note that this model is primarily aimed at being fine-tuned on tasks that use the whole sentence (potentially masked)44to make decisions, such as sequence classification, token classification or question answering. For tasks such as text45generation you should look at model like GPT2.46 47### How to use48 49 50Here is how to use this model to get the features of a given text in PyTorch:51 52```python53from transformers import FunnelTokenizer, FunnelBaseModel54tokenizer = FunnelTokenizer.from_pretrained("funnel-transformer/medium-base")55model = FunnelBaseModel.from_pretrained("funnel-transformer/medium-base")56text = "Replace me by any text you'd like."57encoded_input = tokenizer(text, return_tensors='pt')58output = model(**encoded_input)59```60 61and in TensorFlow:62 63```python64from transformers import FunnelTokenizer, TFFunnelBaseModel65tokenizer = FunnelTokenizer.from_pretrained("funnel-transformer/medium-base")66model = TFFunnelBaseModel.from_pretrained("funnel-transformer/medium-base")67text = "Replace me by any text you'd like."68encoded_input = tokenizer(text, return_tensors='tf')69output = model(encoded_input)70```71 72## Training data73 74The BERT model was pretrained on:75- [BookCorpus](https://yknzhu.wixsite.com/mbweb), a dataset consisting of 11,038 unpublished books,76- [English Wikipedia](https://en.wikipedia.org/wiki/English_Wikipedia) (excluding lists, tables and headers),77- [Clue Web](https://lemurproject.org/clueweb12/), a dataset of 733,019,372 English web pages,78- [GigaWord](https://catalog.ldc.upenn.edu/LDC2011T07), an archive of newswire text data,79- [Common Crawl](https://commoncrawl.org/), a dataset of raw web pages.80 81 82### BibTeX entry and citation info83 84```bibtex85@misc{dai2020funneltransformer,86    title={Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processing},87    author={Zihang Dai and Guokun Lai and Yiming Yang and Quoc V. Le},88    year={2020},89    eprint={2006.03236},90    archivePrefix={arXiv},91    primaryClass={cs.LG}92}93```94 95