colonelwatch/abstracts-embeddings

abstracts-embeddings This is the embeddings of the titles and abstracts of 110 million academic publications taken from the OpenAlex dataset as of January 1, 2025. The embeddings are generated with a Unix pipeline, chaining together the AWS CLI, gzip, oa_jsonl (a C parser tailored to the JSON Lines structure of the OpenAlex snapshot), and a Python embedding script. The source code of oa_jsonl and the Makefile which sets up the pipeline is available on Github, but the general… See the full description on the dataset page: https://huggingface.co/datasets/colonelwatch/abstracts-embeddings.