fschieber/wit-captions
Dataset derived from the original WIT dataset, with the following changes: Removed all columns except for 'image_url', 'caption_reference_description', 'caption_attribution_description', 'mime_type', 'original_height', 'original_width' All rows without either a 'caption_reference_description' or 'caption_attribution_description' have been removed The image references were deduplicated on the image_url preserving the entry with the longest caption_reference_description A new 'text' column was… See the full description on the dataset page: https://huggingface.co/datasets/fschieber/wit-captions.
Dataset derived from the original WIT dataset, with the following changes:
- Removed all columns except for 'imageurl', 'captionreferencedescription', 'captionattributiondescription', 'mimetype', 'originalheight', 'originalwidth'
- All rows without either a 'captionreferencedescription' or 'captionattributiondescription' have been removed
- The image references were deduplicated on the imageurl preserving the entry with the longest captionreference_description
- A new 'text' column was added containing the longest caption between 'captionreferencedescription' and 'captionattributiondescription'
- Note: captionattributiondescription originally may contain many languages. A simple regex was used to parse the available languages and select the longest one. This may not capture all languages correctly or may leave some captions with "artifacts" in case the regex fails to capture the language correctly.\
- The dataset was finally split into 8 files
All 8 files combined contains 11,419,525 rows.
The goal of this dataset is to be used to generate a new dataset with the embedding vectors for the 'text' column.