huggingface/text-data-filtering

Update requirements.txt

Update requirements.txt (#3)

Update requirements.txt

add pdf

Delete explanation_filtering_pipeline.pdf

Update app.py

Upload explanation_filtering_pipeline.pdf

Delete explanation_filtering_pipeline.pdf

remove arabic and viet models

Add Portuguese

delete unused models

Merge branch 'main' of https://huggingface.co/spaces/huggingface/text-data-filtering

back to before portuguese

Update app.py

Update app.py

update visu for Portuguese

7 languages supported

new kenlm models

add register information

new filter on word repetition ratio

visualization: small step for the slider on flagged words ratio

visualization: choose between several languages

fix bug

distributions for the filters on words and discarded words by filter

visualization: upload our own stop words and flagged words list

quick fix

everything in expanders

display distributions in sidebar and filtering parameters in expanders

rename badwords to flagged words + new flagged words list of 68 words

button to download parameters

add warning message

better visualization

fix division by 0 in compute_special_characters_ratio

new tool to analyse our own doc

fix requirements

fix packages

test

correction of bug

delete app_2

merge

filter on repetition removal

Update app.py

chinese visu

Delete en_examples_with_stats_no_small_docs.json

Delete en_examples_with_stats_ldnoob.json

Delete en_examples_with_stats.json

remove zipf's law and update of the doc

visu with discarded documents by filter

new visu

faster visu (less documents)