Master-AI-Lab/EnergyBERT

1---2license: mit3language:4- en5pipeline_tag: fill-mask6---7 8# Model Details9 10Domain Specific BERT model for Text Mining in Energy & Material Field11## Model Description12 13<!-- Provide a longer summary of what this model is. -->14 15- **Developed by:** Tong Xie, Yuwei Wan, Juntao Fang, Prof. Bram Hoex16- **Supported by:** University of New South Wales, National Computational Infrastructure Australia17- **Model type:** Transformer18- **Language(s) (NLP):** EN19- **License:** MIT20 21## Model Sources22 23<!-- Provide the basic links for the model. -->24 25- **Repository:** [Github](https://github.com/MasterAI-EAM/EnergyBERT)26- **Paper:** [Under Prepreation]27 28# Uses29 30<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->31 32## Direct Use33 34<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->35 36Text Mining in Energy & Material Fields37 38## Downstream Use 39 40<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->41 42The EnergyBERT model can be expanded way beyond just text classification. It can be fine-tuned to perform various other downstream NLP tasks in the domain of Energy & Material43 44## How to Get Started with the Model45 46Use the code below to get started with the model.47 48```python49from transformers import pipeline50unmasker = pipeline('fill-mask', model='EnergyBERT')51unmasker("Hello I'm a <mask> model.")52```53 54 Training Details55 56## Training Data57 58<!-- This should link to a Data Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->59 601.2M Published full-text literature corpus from 2000 to 2021.61 62## Training Procedure 63 64<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->65 66BERT is trained on two unsupervised tasks during its pre-training period: masked language modeling and next sentence prediction. A masked language model involves masking some of the input tokens at random and training the model to predict the masked tokens based on the context surrounding the input tokens. Next sentence prediction involves training the model to predict whether two sentences follow each other logically.67 68### Training Hyperparameters69 70- **Training regime:**71