colonelwatch/abstracts-embeddings
abstracts-embeddings This is the embeddings of the titles and abstracts of 110 million academic publications taken from the OpenAlex dataset as of January 1, 2025. The embeddings are generated with a Unix pipeline, chaining together the AWS CLI, gzip, oa_jsonl (a C parser tailored to the JSON Lines structure of the OpenAlex snapshot), and a Python embedding script. The source code of oa_jsonl and the Makefile which sets up the pipeline is available on Github, but the general… See the full description on the dataset page: https://huggingface.co/datasets/colonelwatch/abstracts-embeddings.
Add sources up to March 30, including OpenAlex v2 update, and delete works that did not transition from v1 to v2
Do a one-time shuffle of the dataset so that it reflects insertion order, making it Xet friendly, and add sources up to Sept 30
Update the README with more precise instructions about how the embeddings were made
Drop old .gitattributes lines
Update documentation
Add OpenAlex checkpoints
Add Stella 1.5B embeddings
Clean up .gitignore
Clear the original embeddings
Rename idxs.txt to openalex_ids.txt
Drop old file references
Hide the reconstructed array file from git
Omnibus update 2023-05-10
OPQ64_256,IVF65536,PQ64 gets ~75% recall (10@10)
Regenerate using reproducible sampling
Add first indexing run
Add initial embeddings files
Add expected file types to .gitattributes
initial commit