vulturuldemare/Estonian-Text-Simplification

Estonian Text Simplification

This repository contains resources and models for Estonian text simplification, including datasets and pre-trained models.

Files

Dataset Files

• —`simplification_training_set.json`: A dataset used to fine-tune LLaMA 3.1 for text simplification.
• —`src`: The source of the data.
• —`original`: The original sentence to be simplified.
• —`simpl_lex`: A lexical simplification (may be empty).
• —`simpl_final`: The final simplified sentence.

• —`manually_checked_simplifications.json`: Simplifications verified by human raters.
• —`original`: The original sentence to be simplified.
• —`simplified`: The automatically simplified sentence.
• —`corrected`: The sentence corrected by the annotator.

• —`wiki_large.json`: Automatically translated WikiLarge corpus to Estonian.
• —`eng`: The English sentence from WikiLarge.
• —`machine_translation`: The Estonian sentence produced by machine translation.

Models

• —`llama31-model.zip`: A fine-tuned version of LLaMA 3.1 for Estonian text simplification, trained on simplification_training_set.json.
• —`openNMT-SimplificationModel.pt`: An OpenNMT model trained for Estonian text simplification that combines wiki_large.json with simplification_training_set.json.

How to Use

1. 1.Download the dataset files for analysis or further fine-tuning.
2. 2.Use the models in your text simplification pipelines.

Licensing

This repository is licensed under CC BY.

Citation

These resources were created in the EKTB55 project "Teksti lihtsustamine eesti keeles".

Credits

Coordinator: Eduard Barbu Data Custodian: Meeri-Ly Muru Annotators: Piret Kivi and Mihkel Rünkla