CoolFace
Datasetpublic

ServiceNow/BigDocs-7.5M

BigDocs-7.5M Training data for the paper: BigDocs: An Open and Permissively-Licensed Dataset for Training Multimodal Models on Document and Code Tasks 🌐 Homepage | 📖 arXiv Guide on Data Loading Some parts of BigDocs-7.5M are distributed without their "image" column, and instead have an "img_id" column. The file get_bigdocs_75m.py, part of this repository, provides tooling to substitutes such images back in. from get_bigdocs_75m import… See the full description on the dataset page: https://huggingface.co/datasets/ServiceNow/BigDocs-7.5M.

sourceHugging Facecc-by-4.0updated 1y agoView on Hugging Face
8likes2.3kdownloads
Dataset Card

BigDocs-7.5M

Training data for the paper: BigDocs: An Open and Permissively-Licensed Dataset for Training Multimodal Models on Document and Code Tasks

🌐 Homepage | 📖 arXiv

Guide on Data Loading

Some parts of BigDocs-7.5M are distributed without their "image" column, and instead have an "imgid" column. The file `getbigdocs_75m.py`, part of this repository, provides tooling to substitutes such images back in.

python
from get_bigdocs_75m import get_bigdocs_75m

arxivocr = get_bigdocs_75m("ArxivOCR")
arxivtablecap = get_bigdocs_75m("ArxivTableCap")
cocotext = get_bigdocs_75m("COCOtext", user_local_path=".../train2014")
pubtables1m = get_bigdocs_75m("pubtables-1m", user_local_path=".../PubTables-1M-Detection/images")
textocr = get_bigdocs_75m("TextOCR", user_local_path=".../train")
tabfact = get_bigdocs_75m("TabFact", user_local_path=".../Table-Fact-Checking")
open4business = get_bigdocs_75m("Open4Business", user_local_path=".../Open4Business")
wikitq = get_bigdocs_75m("WikiTQ", user_local_path=".../WikiTableQuestions")

When specified, user_local_path must point to one of the third-party datasets listed below.

  • —COCOtext: http://images.cocodataset.org/zips/train2014.zip
  • —pubtables-1m: https://www.microsoft.com/en-us/research/publication/pubtables-1m
  • —TextOCR: https://dl.fbaipublicfiles.com/textvqa/images/trainvalimages.zip
  • —TabFact: https://github.com/wenhuchen/Table-Fact-Checking
  • —Open4Business: https://github.com/amanpreet692/Open4Business
  • —WikiTQ: https://github.com/ppasupat/WikiTableQuestions

You may specify num_proc as you would for datasets.map. See the docstring in get_bigdocs_75m.py for more details.

Licensing

The part of this repository generated by us is Copyright ServiceNow 2024 and licensed under the CC-BY-4.0 license.

Multiple datasets, documents, and tools were involved in the generation of BigDocs-Bench. We document these dependencies on a per-sample basis through the query_info, annotation_info and image_info fields, respectively documenting the query, annotations and image fields of our datasets.