sawalni-ai/fw-darija

Gherbal’ing Multilingual Fineweb 2 🍵 Following up on their previous release, the fineweb team has been hard at work on their upcoming multilingual fineweb dataset which contains a massive collection of 50M+ sentences across 100+ languages. The data, sourced from the Common Crawl corpus, has been classified into these languages using GlotLID, a model able to recognize more than 2000 languages. The performance of GlotLID is quite impressive, considering the complexity of the… See the full description on the dataset page: https://huggingface.co/datasets/sawalni-ai/fw-darija.