stevengubkin/mathoverflow_text_arxiv_labels
Downloaded from https://archive.org/download/stackexchange Used TexSoup to replace all text in math environments with [UNK]. For instance the text: "The integral $\int_a^b f(x) \textrm{ d}x$ is easy to evaluate if..." was replaced with "The integral [UNK] is easy to evaluate if..." Note: There is still some "ascii math". For instance, people sometimes write things like f: X --> Y. This is retained. Concatenated title and body. Some of these are "answer" posts rather than "question" posts.… See the full description on the dataset page: https://huggingface.co/datasets/stevengubkin/mathoverflow_text_arxiv_labels.
Downloaded from https://archive.org/download/stackexchange
Used TexSoup to replace all text in math environments with [UNK]. For instance the text:
"The integral $\int_a^b f(x) \textrm{ d}x$ is easy to evaluate if..."
was replaced with
"The integral [UNK] is easy to evaluate if..."
Note: There is still some "ascii math". For instance, people sometimes write things like f: X --> Y. This is retained.
Concatenated title and body.
Some of these are "answer" posts rather than "question" posts. In the original data these are untagged. I tagged each "answer" post with the tags of the question they are responding to.
I only retained posts which used at least one of the 32 arxiv tags ('ac.commutative-algebra', 'ag.algebraic-geometry', ..., 'st.statistics').
I only retained posts which had >5 upvotes.
The train/valid/test split was accomplished using MultilabelStratifiedShuffleSplit. This does a better job of respecting multilabel co-occurance statistics than a purely random split.