Datasetpublic

FrancophonIA/MIC21

[!NOTE] Dataset origin: https://live.european-language-grid.eu/catalogue/corpus/18029/ MIC21 Original description One of the processing tasks for large multimodal data streams is automatic image description (image classification, object segmentation and classification). Although the number and the diversity of image datasets is constantly expanding, still there is a huge demand for more datasets in terms of variety of domains and object classes covered. The goal… See the full description on the dataset page: https://huggingface.co/datasets/FrancophonIA/MIC21.

sourceHugging Faceupdated 5mo agoView on Hugging Face

0likes14kdownloads

Dataset Card

[!NOTE] Dataset origin: https://live.european-language-grid.eu/catalogue/corpus/18029/

MIC21

Original description

One of the processing tasks for large multimodal data streams is automatic image description (image classification, object segmentation and classification). Although the number and the diversity of image datasets is constantly expanding, still there is a huge demand for more datasets in terms of variety of domains and object classes covered. The goal of the project Multilingual Image Corpus (MIC 21) is to provide a large image dataset with annotated objects and object descriptions in 24 languages. The Multilingual Image Corpus consists of an Ontology of visual objects (based on WordNet) and a collection of thematically related images whose objects are annotated with segmentation masks and labels describing the ontology classes. The dataset is designed both for image classification and object detection and for semantic segmentation. The main contributions of our work are: a) the provision of large collection of high quality copyright-free images; b) the formulation of the Ontology of visual objects based on WordNet noun hierarchies; c) the precise manual correction of automatic object segmentation within the images and the annotation of object classes; and d) the association of objects and images with extended multilingual descriptions based on WordNet inner- and interlingual relations. The dataset can be used also for multilingual image caption generation, image-to-text alignment and automatic question answering for images and videos.

This Hugging Face dataset

The original data can be found via the link above. The present dataset converts the original data into a Hugging Face dataset. We have in particular:

—convert everything in parquet,
—convert image as PIL,
—create one column per language.

Note: The paper indicates 21,316 images. In practice, we propose only 21,140 due to corrupted annotations that could not be correctly associated with 176 images.

Usage

Since the global dataset is just about 70GB, we have chosen to present it as 4 splits so that users can use the one of their choice if they don’t need all of it.

python

from datasets import load_dataset

ds = load_dataset("FrancophonIA/MIC21", "arts")
ds = load_dataset("FrancophonIA/MIC21", "sport")
ds = load_dataset("FrancophonIA/MIC21", "security")
ds = load_dataset("FrancophonIA/MIC21", "transport")
ds = load_dataset("FrancophonIA/MIC21") # load the 4 splits in one shot

Configs

Config	Content
`arts`	Artists, musicians, dancers, sculptors...
`security`	Security personnel, equipment...
`sport`	Athletes and sporting activities
`transport`	Vehicles and means of transport

Column Schema

Column	Type	Description
`image`	`Image`	Embedded PIL image (RGB JPEG)
`image_id`	`int64`	Unique image identifier (COCO)
`file_name`	`string`	Source file name
`width`	`int32`	Image width in pixels
`height`	`int32`	Image height in pixels
`domain`	`string`	Domain (`ARTS`, `SECURITY`, `SPORT`, `TRANSPORT`)
`class_name`	`string`	Main class of the image (e.g. `accordionist`)
`label_en`	`string`	English translation of the class
`label_fr`	`string`	French translation of the class
`label_XX`	`string`	Same for all 25 languages (see table below)
`objects`	`list`	List of annotated objects (COCO format, see below)

`objects` field

Each entry in the list corresponds to one COCO annotation:

Field	Type	Description
`annotation_id`	`int64`	Annotation identifier
`category_id`	`int64`	Category identifier
`category_name`	`string`	Category name
`bbox`	`list[float32]`	Bounding box `[x, y, width, height]` (COCO format)
`area`	`float32`	Object area in pixels²
`iscrowd`	`int8`	`1` if dense crowd, `0` otherwise
`segmentation`	`list[list[float32]]`	COCO segmentation polygons
`metadata`	`string`	Additional metadata (serialized JSON)

Available languages (`label_XX`)

Code	Language	Code	Language	Code	Language
`en`	English	`fr`	French	`de`	German
`es`	Spanish	`it`	Italian	`pt`	Portuguese
`nl`	Dutch	`pl`	Polish	`ru`	Russian
`bg`	Bulgarian	`sr`	Serbian	`hr`	Croatian
`sk`	Slovak	`sl`	Slovenian	`ro`	Romanian
`el`	Greek	`ca`	Catalan	`gl`	Galician
`eu`	Basque	`sq`	Albanian	`fi`	Finnish
`sv`	Swedish	`da`	Danish	`lt`	Lithuanian
`is`	Icelandic

Citation

@inproceedings{koeva-etal-2022-multilingual,
    title = "Multilingual Image Corpus {--} Towards a Multimodal and Multilingual Dataset",
    author = "Koeva, Svetla  and
      Stoyanova, Ivelina  and
      Kralev, Jordan",
    editor = "Calzolari, Nicoletta  and
      B{\'e}chet, Fr{\'e}d{\'e}ric  and
      Blache, Philippe  and
      Choukri, Khalid  and
      Cieri, Christopher  and
      Declerck, Thierry  and
      Goggi, Sara  and
      Isahara, Hitoshi  and
      Maegaard, Bente  and
      Mariani, Joseph  and
      Mazo, H{\'e}l{\`e}ne  and
      Odijk, Jan  and
      Piperidis, Stelios",
    booktitle = "Proceedings of the Thirteenth Language Resources and Evaluation Conference",
    month = jun,
    year = "2022",
    address = "Marseille, France",
    publisher = "European Language Resources Association",
    url = "https://aclanthology.org/2022.lrec-1.162",
    pages = "1509--1518",
    abstract = "One of the processing tasks for large multimodal data streams is automatic image description (image classification, object segmentation and classification). Although the number and the diversity of image datasets is constantly expanding, still there is a huge demand for more datasets in terms of variety of domains and object classes covered. The goal of the project Multilingual Image Corpus (MIC 21) is to provide a large image dataset with annotated objects and object descriptions in 24 languages. The Multilingual Image Corpus consists of an Ontology of visual objects (based on WordNet) and a collection of thematically related images whose objects are annotated with segmentation masks and labels describing the ontology classes. The dataset is designed both for image classification and object detection and for semantic segmentation. The main contributions of our work are: a) the provision of large collection of high quality copyright-free images; b) the formulation of the Ontology of visual objects based on WordNet noun hierarchies; c) the precise manual correction of automatic object segmentation within the images and the annotation of object classes; and d) the association of objects and images with extended multilingual descriptions based on WordNet inner- and interlingual relations. The dataset can be used also for multilingual image caption generation, image-to-text alignment and automatic question answering for images and videos.",
}