Datasetpublic

oscar-corpus/mOSCAR

More info can be found here: https://oscar-project.github.io/documentation/versions/mOSCAR/ Paper link: https://arxiv.org/abs/2406.08707 New features: Additional filtering steps were applied to remove toxic content (more details in the next version of the paper, coming soon). Spanish split is now complete. Face detection in images to blur them once downloaded (coordinates are reported on images of size 256 respecting aspect ratio). Additional language identification of the documents to… See the full description on the dataset page: https://huggingface.co/datasets/oscar-corpus/mOSCAR.

sourceHugging Facecc-by-4.0updated 2y agoView on Hugging Face

18likes9.5kdownloads

Dataset Card

More info can be found here: https://oscar-project.github.io/documentation/versions/mOSCAR/

Paper link: https://arxiv.org/abs/2406.08707

New features:

—Additional filtering steps were applied to remove toxic content (more details in the next version of the paper, coming soon).
—Spanish split is now complete.
—Face detection in images to blur them once downloaded (coordinates are reported on images of size 256 respecting aspect ratio).
—Additional language identification of the documents to improve document-language matching.
—Replace most of Personal Identifiable Information by generic strings.

Previous version remains available, to continue using it:

dataset = load_dataset("oscar-corpus/mOSCAR", revision="v1")

Layout

{
    'images': [{'img_idx': '#000002',
                'sha512': '65c1e5605d48f8753256f758bd442cbdd43e6987691227b1ea6b81430ff36609f46d448c8171546232fe0c258d9e44ce4378f32e8ada5c43c314df5a5e230de2',
                'url': 'https://actuconsommation.fr/wp-content/uploads/2020/05/Disneylands-Japon-1068x712.jpg',
                'faces_loc': [x0, y0, x1, y1]}],
    'metadata': [{'node_order': 'img_#000002|txt_#000000|txt_#000001|txt_#000002|txt_#000003|txt_#000004|txt_#000005|txt_#000006|txt_#000009',
                  'url': 'https://actuconsommation.fr/2020/05/11/disneyland-une-reouverture-sous-haute-securite-a-shanghai-ce-lundi/'}],
    'text': [{'text': 'Disneyland : une réouverture sous haute sécurité à Shanghai ce lundi', 'text_idx': '#000000'},
             {'text': 'Des milliers de visiteurs ont pu pénétrer lundi dans le Disneyland de Shanghai, le premier des six parcs de [...]', text_idx': '#000001'},
             [...] ]
}

Citation

@article{futeral2024moscar,
  title={mOSCAR: A Large-scale Multilingual and Multimodal Document-level Corpus},
  author={Futeral, Matthieu and Zebaze, Armel and Suarez, Pedro Ortiz and Abadji, Julien and Lacroix, R{\'e}mi and Schmid, Cordelia and Bawden, Rachel and Sagot, Beno{\^\i}t},
  journal={arXiv preprint arXiv:2406.08707},
  year={2024}
}