cglez/bert-dapt-wiki_toxic-uncased

Model Card: BERT-DAPT-WikiToxic

A domain-adapted BERT-base model, further pre-trained on the WikiToxic dataset text.

Model Details

Description

This model is based on the BERT base (uncased) architecture and was further pre-trained (domain-adapted) using the text in WikiToxic dataset, excluding its test split. Only the masked language modeling (MLM) objective was used during domain adaptation.

• —Developed by: Cesar Gonzalez-Gutierrez
• —Funded by: ERC
• —Architecture: BERT-base
• —Language: English
• —License: Apache 2.0
• —Base model: BERT base model (uncased)

Checkpoints

Intermediate checkpoints from the pre-training process are available and can be accessed using specific tags, which correspond to training epochs and steps:

To load a model from a specific intermediate checkpoint, use the revision parameter with the corresponding tag:

from transformers import AutoModelForMaskedLM

model = AutoModelForMaskedLM.from_pretrained("<model-name>", revision="<checkpoint-tag>")

Sources

• —Paper: [Information pending]

Training Details

For more details on the training procedure, please refer to the base model's documentation: Training procedure.

Training Data

All texts from WikiToxic dataset, excluding the test partition.

Preprocessing

All markup and symbols were removed from the texts, including punctuation.

Training Hyperparameters

• —Precision: fp16
• —Batch size: 32
• —Gradient accumulation steps: 3

Uses

For typical use cases and limitations, please refer to the base model's guidance: Inteded uses & limitations.

Bias, Risks, and Limitations

This model inherits potential risks and limitations from the base model. Refer to: Limitations and bias.

Environmental Impact

• —Hardware Type: NVIDIA Tesla V100 PCIE 32GB
• —Cluster Provider: Artemisa
• —Compute Region: EU

Citation

BibTeX:

[More Information Needed]