datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
fineweb-2-dutchDutchMedicalTextV3Translated mix of PMC/Pubmed, existing data sets from Apollo/Meditron, various Dutch sources and now with
the medically filtered texts from fineweb and finepdf.
Be aware of spurious repetitions due to the use of neural machine translation. You can mitigate it partially by iteratively applying a suitable regex-replace
operation. Example (bleeding eyes) code is
RE_SPURIOUS_CHARS = re.compile(r'([^\w])\1{3,}')
RE_SPURIOUS_WORDS = re.compile(r'(\b[\w\-\s\;\:\,\.]+\b)\1{4,}')
RE_MULTISPACE =… See the full description on the dataset page: https://huggingface.co/datasets/UMCU/DutchMedicalTextV3.DutchMedicalTextV2Translated mix of PMC/Pubmed, existing data sets from Apollo/Meditron and various Dutch sources.
Be aware of spurious repetitions due to the use of neural machine translation. You can mitigate it partially by iteratively applying a suitable regex-replace
operation. Example (bleeding eyes) code is
RE_SPURIOUS_CHARS = re.compile(r'([^\w])\1{3,}')
RE_SPURIOUS_WORDS = re.compile(r'(\b[\w\-\s\;\:\,\.]+\b)\1{4,}')
RE_MULTISPACE = re.compile(r'\s{2,}')
def apply_until_stable(pattern, repl, text… See the full description on the dataset page: https://huggingface.co/datasets/UMCU/DutchMedicalTextV2.PubmedAbstracts_Dutch_v1
License:
All Rights Reserved with Limited ML Training License.
This data may only be used for training machine-learning models and may not be redistributed in its original or substantially similar form.
This refers explicitly to sharing the data as-is, or as-derived. It does not prohibit you to use this data for training machine learning models as this would constitute FAIR use depending on the application.
Articles available from PubMed Central (PMC) are provided by the respective… See the full description on the dataset page: https://huggingface.co/datasets/UMCU/PubmedAbstracts_Dutch_v1.ultra_feedback_dutch_cleaned
Ultra Feedback Dutch Cleaned
This is a cleaned version of BramVanroy/ultra_feedback_dutch, based on the cleaning done by Argilla on the original Ultra Feedback dataset. Another difference is that we only include GEITje 7B Ultra and GPT-4-Turbo. GEITje chat, which was used in the original dataset, is not used.
After cleaning I also generated replies for other models (like TowerInstruct, Mistral), but the results were too poor (in Dutch) to include so we only kept the GEITje Ultra and… See the full description on the dataset page: https://huggingface.co/datasets/BramVanroy/ultra_feedback_dutch_cleaned.orca_dpo_pairs_dutch_cleaned
Dataset Card for Orca DPO Pairs Dutch Cleaned
Citation
If you use this dataset, GEITje 7B Ultra (SFT) or any of its derivatives or quantizations, place cite the following paper:
@misc{vanroy2024geitje7bultraconversational,
title={GEITje 7B Ultra: A Conversational Model for Dutch},
author={Bram Vanroy},
year={2024},
eprint={2412.04092},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2412.04092},
}… See the full description on the dataset page: https://huggingface.co/datasets/BramVanroy/orca_dpo_pairs_dutch_cleaned.common_corpus_dutch_pd
Common Corpus v2 - Dutch Public Domain collection
This is a version of Common Corpus v2 filtered to keep only the rows where collection is "Dutch-PD".
Looking for all Dutch-language documents in Common Corpus, regardless of the collection they are in?
Then you might want to look at Rijgersberg/common_corpus_nl.
Common Corpus is a very large open and permissible licensed text dataset created by Pleias.
Please be sure to acknowledge the creators of the original dataset when using this… See the full description on the dataset page: https://huggingface.co/datasets/Rijgersberg/common_corpus_dutch_pd.apollo_english_guidelines_translated_to_dutch_with_nllb200
Data description
Translation of the English medical guidelines that are part of the Apollo corpus, using the NLLB200-600M NTM.
Acknowledgement
The work received funding from the European Union's Horizon Europe research
and innovation programme under Grant Agreement No. 101057849 (DataTools4Heart project).
For more information on the background, see Datatools4Heart Huggingface/Website/Git
biolord_dutch_marianmt
Dataset Card for Biolord Dutch Marianmt
MariaNMT translation of BioLORD definitions.
The source language: English
The original data source: Original Data Source
Data description
Dutch translation of BioLord definitions using MariaMT
Acknowledgement
This is part of the DT4H project with attribution [Cite the paper].
Doi and reference
[DOI and reference to the source paper/s]
apollo_english_guidelines_translated_to_dutch_with_marianmt
Data description
Apollo corpus, English guidelines translated to Dutch using MariaNMT.
Acknowledgement
The work received funding from the European Union's Horizon Europe research
and innovation programme under Grant Agreement No. 101057849 (DataTools4Heart project).
For more information on the background, see Datatools4Heart Huggingface/Website/Git
epfl_guidelines_dutch_marianmt
Dataset Card for Epfl English Guidelines Translated To Dutch With MariaNMT
This dataset was created by the EPFL, and can found in it original form here
The source language: English
The original data source: Original Data Source
The MariaNMT model used can be found: here
Data description
Translation of the English medical guidelines that are part of the Meditron corpus, using the LLM GPT 4o mini
Acknowledgement
This is part of the DT4H project with… See the full description on the dataset page: https://huggingface.co/datasets/UMCU/epfl_guidelines_dutch_marianmt.