Datasetpublic
knowurknottty/MathNet

Quick Start · Overview · Tasks · Comparison · Dataset Stats · Data Sources · Pipeline · Schema · License · Citation This is the official MathNet v0. A larger version v1 will be uploaded soon (more countires, problems and richer metadata). Schema is stable but field values may be revised in v1. Quick start from datasets import load_dataset # Default: all problems ds = load_dataset("ShadenA/MathNet", split="train") # Or a specific country / competition-body config… See the full description on the dataset page: https://huggingface.co/datasets/knowurknottty/MathNet.
sourceHugging Facecc-by-4.0updated 5mo agoView on Hugging Face
1likes1.6kdownloads
root__pycache__build_dataset.cpython-310.pyc
build_dataset.cpython-310.pyc128 linesDownload Raw Back to __pycache__
1o

2���i�6�
@sTdZddlZddlZddlZddlZddlmZddlm	Z	m3Z4ddlmZm
Z
mZmZmZed�Zed�Zed�Zed	Zed5Zejddd�ejddd�ed
ZedZejdddZe�d�Ze�d�ZdZdHdedede fdd�Z!dd�Z"dd�Z#dd�Z$d e defd!d"�Z%d e dedBfd#d$�Z&d%e d&e'dedBfd'd(�Z(d)e d&e'd*e'd+e)de f6d,d-�Z*d.e'd/e'dBd&e'd0e de'f7d1d2�Z+e
ed3�ed3�ed3�ed3�eed3��eed3��ed3�ed3�ee��ed3�ed3�d4��Z,d5e de fd6d7�Z-d8e'defd9d:�Z.dId=e)d>ed?ede)fd@dA�Z/d=e)dBe fdCdD�Z0dEdF�Z1e2dGk�r(e1�dSdS)Ju�Build the full MathNet HuggingFace dataset.8 9Schema (exactly these fields, in this order):10    id, country, competition, problem_markdown, solutions_markdown,11    topics_flat, language, booklet_source, images, problem_type, final_answer12 13Images: every `![](ref)` in problem/solution markdown is rewritten to14`![](attached_image_N.png)` and the corresponding bytes are attached to the15row in the same order. Three upstream ref kinds are resolved:16  1. `images/<name>`                        — read from the local images dir17  2. `attached_image_*.png`                 — mapped via the raw jsonl's18                                              `images_map` to a local file19  3. `https://cdn.mathpix.com/...`          — downloaded once to a cache dir20 21Unrecoverable refs (5 total across the corpus, e.g. `path/to/image.png`) have22their `![](...)` tag stripped from the markdown.23 24IDs: 4-char base36, assigned in a stable sort order (country, year,25problem_number, unique_id) so rebuilds are deterministic.26�N)�Path)�defaultdict�Counter)�Dataset�Features�Value�Sequence�Imagez8/afs/csail.mit.edu/group/vision/torralba/mathnet_websitezG/afs/csail.mit.edu/group/vision/torralba/mathnet_website_archive/sourcez3/afs/csail.mit.edu/group/vision/torralba/mathnet_hf�data�_mathpix_cacheT)�parents�exist_ok�imageszproblems.jsonl�mathnet_website_archive�source�metadata.jsonlz!\[[^\]]*\]\(([^)]+)\)�$0123456789abcdefghijklmnopqrstuvwxyz��n�width�returncCsF|dkrtd��d}|rt|d�\}}t||}|s|pd�|d�S)Nrzn must be non-negative��$�0)�27ValueError�divmod�BASE36�rjust)rr�s�r�r �D/afs/csail.mit.edu/group/vision/torralba/mathnet_hf/build_dataset.py�	to_base360s�r"c	Csli}ttdd�d��D]&}t|��
}t�|�}Wd�n1s#wY|D]}|||d<q*q
|S)Nr28�fullz*.json�	unique_id)�sorted�ROOT�glob�open�json�load)�by_id�jf�f�arr�pr r r!�load_full_problems:s29��r0cCsji}tt��%}|D]}t�|�}|�d�}|�d�}|r"|r"|||<q	Wd�|S1s.wY|S)zLunique_id -> images_map dict from the raw problems.jsonl (keyed by hash_id).�hash_id�30images_mapN)r(�	RAW_JSONLr)�loads�get)�resultr-�line�d�h�imr r r!�load_raw_images_mapDs31323334��35��r;cCsxi}t��std}|��r|}n|St}t|��}|D]
}t�|�}|||d<qWd�|S1s5wY|S)Nrr$)�METADATA_JSONL�existsr&r(r)r4)r+�alt�pathr-r7r8r r r!�
load_metadataQs 3637�38��r@�urlcCsBt�|�����dd�}t|�d�d�jpd}t|�|��S)N��?rz.jpg)�hashlib�md5�encode�	hexdigestr�split�suffix�
MATHPIX_CACHE)rAr9�extr r r!�mathpix_cache_pathcsrLc39Cs�t|�}|��r|��jdkr|Sz.tjj|ddid�}tjj|dd��}|�|�	��Wd�|WS1s8wY|WSt40y`}ztd|dd��d	|���WYd}~dSd}~ww)41Nrz42User-AgentzMozilla/5.0)�headers�)�timeoutz  [WARN] download failed �Fz... )rLr=�stat�st_size�urllib�request�Request�urlopen�write_bytes�read�	Exception�print)rA�dest�req�resp�er r r!�download_mathpixis 43����r_�refr2cCs|��}|�d�rt|td�d�}|��r|SdSd|vrM|�d�sM|�|�}t|t�rK|�d�p4d}|rK|�ddd��	d�}t|}|��rK|SdS|�d	�sW|�d44�r�|�45�D]-}t|t�r�|�d�|kr�|�d�pod}|r�|�ddd��	d�}t|}|��r�|Sq[t|�SdS)z:Resolve a markdown image ref to a local file path on disk.zimages/N�attached_image�httpr?r��/zhttp://zhttps://�original_url)�strip�46startswith�IMG_DIR�lenr=r5�47isinstance�dict�replace�lstrip�valuesr_)r`r2r/�entry�ipath�namer r r!�resolve_refws4484950�rr�text�assignments�ordercs(dtjdtf���fdd�}t�||�S)a:Rewrite every ![](ref) in `text` to ![](attached_image_N.png).51 52    `assignments` maps original-ref -> attached index (1-based), shared across53    problem/solution texts of one problem.54    `order` is the parallel list of resolved Paths in the assignment order.55    Unrecoverable refs have their tag stripped.56    �mrcsj|�d���}|�vr�|}d|�d�St|��}|dur dSt��d}|�|<��|�d|�d�S)Nrcz![](attached_image_z.png)r)�grouprfrrri�append)rvr`�idx�resolved�rtr2rur r!�repl�s5758zrewrite_markdown.<locals>.repl)�re�Match�str�FULL_IMG_TAG_RE�sub)rsr2rtrur|r r{r!�rewrite_markdown�s
r��problem�meta�short_idc59
s�i�g�t|�d�pd����}���fdd�|�d�pgD�}g}�D]
}|�|��|jd��q$|�d�p8g}dd�|D�}	||�d	�|�d60�|||	|�d�|�d�||pXi�d
�|p^i�d�d�S)N�problem_markdownrcsg|]	}t|�����qSr )r�)�.0rr{r r!�61<listcomp>�szbuild_row.<locals>.<listcomp>�solutions_markdown)�bytesr?�topicscSs4g|]}t|�dkrd�|dd��nd�|��qS)rcz > N)ri�join)r��tr r r!r��s4�country�competition�booklet_language�booklet_source�problem_type�final_answer��idr�r�r�r��topics_flat�languager�rr�r�)r�r5rx�62read_bytesrq)63r�r�r2r��p_md�solsrr/r�r�r r{r!�	build_row�s* �r��stringr��ccCs|�dd��dd��dd�S)N� �_�(r�))rl�r�r r r!�slug�sr��rowcCs�d}|dr|dtt|d�d�d7}t|d�dkr"|d7}nt|d�d	kr.|d647}|dr6|d7}|dr>|d
7}t|d�}d|krNdkrUn|S|d7}|S)zEHigher = more visually interesting for the HF dataset viewer preview.rr�dr�65r���(rc�r�r��r���i�)�minri)r�r�plenr r r!�showcase_score�s 66�r��x��rows�	head_size�seedcsddl}tt�}|D]
}||dpd�|�q67|D]}||jtdd�qdd�|��D�}t|���}g}	t	��t68|	�|krrd	}69|D]$}t70|	�|krNn||}|sUqD|�d�}|	�|���|d71�d}72qD|73slnt74|	�|ks@|�
|�}
�fdd�|D�}|
�|�|	|S)
u�Put a curated, country-diverse head at the top, then shuffle the tail.75 76    HF's dataset viewer shows the first ~100 rows — this makes that preview77    look appealing without changing ids (already assigned).78    rNr��UnknownT)�key�reversecSs i|]\}}|dd�|D��qS)cSsg|]}|dr|�qS)rr �r�rr r r!r�79sz2reorder_for_preview.<locals>.<dictcomp>.<listcomp>r )r�r��br r r!�80<dictcomp>81s z'reorder_for_preview.<locals>.<dictcomp>Fr�csg|]82}|d�vr|�qS)r�r r���used_idsr r!r� �z'reorder_for_preview.<locals>.<listcomp>)�randomr�listrx�sortr��itemsr%�keys�setri�pop�add�Random�shuffle)r�r�r�r��bucketsrr��queues�
country_order�head�83progressed�q�pick�rng�tailr r�r!�reorder_for_preview�s:8485�8687r��cfg_namec8889Cs�t|}|jdd�|�d�D]}|��qtj|td�}t|�}d}td||d|�}|dkr=|�	t90|d��dSt|�D]}|j||dd�}	|	�	t91|d	|d92�d|d93�d���qAdS)
NT)r
�*)�features��rcztrain-00000-of-00001.parquet)�94num_shards�index�95contiguousztrain-�05dz-of-z.parquet)
�DATA_OUT�mkdirr'�unlinkr�	from_list�FEATURESri�max�96to_parquetr�range�shard)97r�r�r8r-�dsr�98shard_size�n_shards�ir�r r r!�write_config%s99&�r�c	sntd�t�}tdt|��d��td�t�}tdt|��d��td�t�}tdt|��d��dd	�}t|��|d100�}td�i}t|�D]\}}t|dd
�}t|�dkr`t	d|����|||d<qItdtt|�d����td�g}	d}101d}d}t|�D]Z\}}|�102|di�}
|�103d�p�dg|�104d�p�g}t||�105|d�|
||d�}|	�|�|dr�|d7}|106t|d�7}107|dddkr�td|d�dt|��d|108�d��q�tdt|	��d|�d|109���td �t
|	d!d"d#�}td$d%�|dd&�D��}td'd%�|dd&�D��}td(t|��d)|�d*��td+�t|d,�td-d.�|	D��}td/t|��d0��|D]"��fd1d2�|	D�}t|t���tdt���d3t|��d4���q>t|	�t|�|||110td5d%�|	D����td6d.�|	D��d7�}ttd8d9��}tj||d:td;�Wd�n	1�s�wYtd<�ttjd=d>�|��D�d:td;��dS)?NzLoading full problems...z  z	 problemszLoading raw images_map...z problems have images_mapzLoading enriched metadata...z metadata recordscSs2|�d�pd|�d�pdt|�d�pd�|dfS)Nr�r�yearr�problem_numberr$)r5r)r/r r r!�sort_keyEs111�zmain.<locals>.sort_key)r�z)Assigning sequential 4-char base36 IDs...r)rz(ran out of 4-char base36 space at index r$z  last id: rcz#Building rows (resolving images)...rr�rr�rr�z  built rdz rows, z images so farz  total rows: z, with images: z, total images: z*112Reordering 'all' for appealing preview...r�r�)r�r�css�|]}|dVqdS)r�Nr r�r r r!�	<genexpr>os�zmain.<locals>.<genexpr>r�css�|]	}|drdVqdS)rrcNr r�r r r!r�ps�z  top-100 covers z countries, z have imageszWriting 'all' config...�allcS�h|]113}|dr|d�qS�r�r r�r r r!�	<setcomp>ur�zmain.<locals>.<setcomp>zWriting z per-country configs...csg|]114}|d�kr|�qSr�r r�r�r r!r�xr�zmain.<locals>.<listcomp>z: z rowscss �|]}|dr|dVqdS)r�Nr r�r r r!r��s�cSr�)r�r r�r r r!r��r�)�115n_problems�n_countries�	countries�
n_with_images�n_total_images�	languages�competitionsz116stats.json�wr�)�indent�defaultz
117Done. Stats:cSsi|]\}}|dvr||�qS))r�r�r )r��k�vr r r!r��szmain.<locals>.<dictcomp>)rZr0rir;r@r%rn�	enumerater"�RuntimeErrorr5r�rxr�r�sumr�r��most_commonr(�OUTr)�dumpr�dumpsr�)�problems�raw_mapsr�r��ordered�	id_by_uidr�r/�sidr��n_imgs_total�n_with_imgs�
n_broken_refsr:�textsr��all_rows�head_countries�head_with_imgsr�r��statsr-r r�r!�main7s| 118$�119"�	�&r�__main__)r)r�r�)3�__doc__r)r}rD�urllib.requestrS�pathlibr�collectionsrr�datasetsrrrrr	r&�ARCHIVE_ROOTr�r�rJr�rhr3�parentr<�compile�120IMG_TAG_REr�r�intrr"r0r;r@rLr_rkrrr�r�r�r�r�r�r�r�r�__name__r r r r!�<module>sh121122123124
%"125126��+127T128�