CoolFace
Modelpublic

lmvasque/readability-es-benchmark-bertin-es-sentences-2class

sourceHugging Facecc-by-4.0updated 4y agoView on Hugging Face
0likes8downloads
Model Card

Readability benchmark (ES): bertin-es-sentences-2class

This project is part of a series of models from the paper "A Benchmark for Neural Readability Assessment of Texts in Spanish". You can find more details about the project in our GitHub.

Models

Our models were fine-tuned in multiple settings, including readability assessment in 2-class (simple/complex) and 3-class (basic/intermediate/advanced) for sentences and paragraph datasets. You can find more details in our paper. These are the available models you can use (current model page in bold):

ModelGranularity# classes
BERTIN (ES)paragraphs2
BERTIN (ES)paragraphs3
mBERT (ES)paragraphs2
mBERT (ES)paragraphs3
mBERT (EN+ES)paragraphs3
[BERTIN (ES)](https://huggingface.co/lmvasque/readability-es-benchmark-bertin-es-sentences-2class)sentences2
BERTIN (ES)sentences3
mBERT (ES)sentences2
mBERT (ES)sentences3
mBERT (EN+ES)sentences3

For the zero-shot setting, we used the original models BERTIN and mBERT with no further training.

Results

These are our results for all the readability models in different settings. Please select your model based on the desired performance:

GranularityModelF1 Score (2-class)Precision (2-class)Recall (2-class)F1 Score (3-class)Precision (3-class)Recall (3-class)
ParagraphBaseline (TF-IDF+LR)0.8290.8320.8270.5560.5630.550
ParagraphBERTIN (Zero)0.3080.2220.5000.2270.2840.338
ParagraphBERTIN (ES)0.9240.9230.9250.7720.7760.768
ParagraphmBERT (Zero)0.3080.2220.5000.2530.3120.368
ParagraphmBERT (EN)---0.5050.5600.552
ParagraphmBERT (ES)0.9330.9320.9360.7760.7770.778
ParagraphmBERT (EN+ES)---0.7790.7830.779
SentenceBaseline (TF-IDF+LR)0.8110.8140.8080.5250.5310.521
SentenceBERTIN (Zero)0.3670.2900.5000.1880.2320.335
SentenceBERTIN (ES)0.9000.9000.9000.6990.7010.698
SentencemBERT (Zero)0.3670.2900.5000.2780.3290.351
SentencemBERT (EN)---0.5210.5650.539
SentencemBERT (ES)0.8930.8910.8960.6880.6860.691
SentencemBERT (EN+ES)---0.6790.6760.682

Citation

If you use our results and scripts in your research, please cite our work: "A Benchmark for Neural Readability Assessment of Texts in Spanish" (to be published)

@inproceedings{vasquez-rodriguez-etal-2022-benchmarking,
    title = "A Benchmark for Neural Readability Assessment of Texts in Spanish",
    author = "V{\'a}squez-Rodr{\'\i}guez, Laura  and
      Cuenca-Jim{\'\e}nez, Pedro-Manuel and
      Morales-Esquivel, Sergio Esteban and
      Alva-Manchego, Fernando",
    booktitle = "Workshop on Text Simplification, Accessibility, and Readability (TSAR-2022), EMNLP 2022",
    month = dec,
    year = "2022",
}