hackersgame/Free_Language_Embeddings

1---2language:3- en4license: gpl-3.05tags:6- word-embeddings7- word2vec8- embeddings9- nlp10- free-software11- dfsg12datasets:13- wikimedia/wikipedia14- pg1915metrics:16- accuracy17model-index:18- name: fle-v3419  results:20  - task:21      type: word-analogy22      name: Word Analogy23    dataset:24      type: custom25      name: Google Analogy Test Set26    metrics:27    - type: accuracy28      value: 66.529      name: Overall Accuracy30    - type: accuracy31      value: 61.432      name: Semantic Accuracy33    - type: accuracy34      value: 69.235      name: Syntactic Accuracy36library_name: numpy37pipeline_tag: feature-extraction38---39 40# Free Language Embeddings (V34)41 42300-dimensional word vectors trained from scratch on ~2B tokens of freely-licensed text using a single RTX 3090.43 44**66.5% on Google analogies** — beating the original word2vec (61% on 6B tokens) by 5.5 points with 1/3 the data.45 46## Model Details47 48| | |49|---|---|50| **Architecture** | Dynamic masking word2vec skip-gram |51| **Dimensions** | 300 |52| **Vocabulary** | 100,000 whole words |53| **Training data** | ~2B tokens, all [DFSG-compliant](https://wiki.debian.org/DFSGLicenses) (see below) |54| **Training hardware** | Single NVIDIA RTX 3090 |55| **Training time** | ~4 days (2M steps) |56| **License** | GPL-3.0 |57| **Parameters** | 60M (30M target + 30M context embeddings) |58 59### Training Data60 61All training data meets the [Debian Free Software Guidelines](https://wiki.debian.org/DFSGLicenses) for redistribution, modification, and use. No web scrapes, no proprietary datasets.62 63| Source | Weight | License |64|--------|--------|---------|65| Wikipedia | 30% | CC BY-SA 3.0 |66| Project Gutenberg | 20% | Public domain |67| arXiv | 20% | Various open access |68| Stack Exchange | 16% | CC BY-SA 4.0 |69| US Government Publishing Office | 10% | Public domain (US gov) |70| RFCs | 2.5% | IETF Trust |71| Linux kernel docs, Arch Wiki, TLDP, GNU manuals, man pages | 1.5% | GPL/GFDL |72 73## Benchmark Results74 75| Model | Data | Google Analogies |76|-------|------|-----------------|77| **fle V34 (this model)** | **~2B tokens** | **66.5%** |78| word2vec (Mikolov 2013) | 6B tokens | 61.0% |79| GloVe (small) | 6B tokens | 71.0% |80| Google word2vec | 6B tokens | 72.7% |81| GloVe (Pennington 2014) | 840B tokens | 75.6% |82| FastText (Bojanowski 2017) | 16B tokens | 77.0% |83 84Breakdown: semantic 61.4%, syntactic 69.2%. Comparatives 91.7%, plurals 86.8%, capitals 82.6%.85 86## Quick Start87 88```bash89# Download90pip install huggingface_hub numpy91python -c "92from huggingface_hub import hf_hub_download93hf_hub_download('hackersgame/Free_Language_Embeddings', 'fle_v34.npz', local_dir='.')94hf_hub_download('hackersgame/Free_Language_Embeddings', 'fle.py', local_dir='.')95"96 97# Use98python fle.py king - man + woman99python fle.py --similar cat100python fle.py   # interactive mode101```102 103### Python API104 105```python106from fle import FLE107 108fle = FLE()                                  # loads fle_v34.npz109vec = fle["cat"]                             # 300d numpy array110fle.similar("cat", n=10)                     # nearest neighbors111fle.analogy("king", "man", "woman")          # king:man :: woman:?112fle.similarity("cat", "dog")                 # cosine similarity113fle.query("king - man + woman")              # vector arithmetic114```115 116## Examples117 118```119$ python fle.py king - man + woman120  → queen                0.7387121  → princess             0.6781122  → monarch              0.5546123 124$ python fle.py paris - france + germany125  → berlin               0.8209126  → vienna               0.7862127  → munich               0.7850128 129$ python fle.py --similar cat130  kitten               0.7168131  cats                  0.6849132  tabby                 0.6572133  dog                   0.5919134 135$ python fle.py ubuntu - debian + redhat136  centos               0.6261137  linux                0.6016138  rhel                 0.5949139 140$ python fle.py brain141  cerebral             0.6665142  cerebellum           0.6022143  nerves               0.5748144```145 146## What Makes This Different147 148- **Free as in freedom.** Every dataset is DFSG-compliant. Every weight is reproducible. GPL-3.0 licensed. The goal: word embeddings you could `apt install` from Debian main.149- **Dynamic masking.** Randomly masks context positions during training, forcing the model to extract signal from partial views. The result: geometry that crystallizes during cosine LR decay — analogies jump from 1.2% to 66.5% in the second half of training.150- **Whole-word vocabulary.** No subword tokenization. Subwords break word2vec geometry completely — they don't carry enough meaning individually for co-occurrence statistics to produce useful structure.151 152## Training153 154Trained with cosine learning rate schedule (3e-4 → 1e-6). The training curve shows a striking crystallization pattern: near-zero analogy accuracy for the first 50% of training, then rapid emergence of geometric structure as the learning rate decays.155 156Full training code and visualizations: [github.com/ruapotato/Free-Language-Embeddings](https://github.com/ruapotato/Free-Language-Embeddings)157 158## Interactive Visualizations159 160- [Embedding Spectrogram](https://ruapotato.github.io/Free-Language-Embeddings/spectrogram.html) — PCA waves, sine fits, cosine surfaces161- [3D Semantic Directions](https://ruapotato.github.io/Free-Language-Embeddings/semantic_3d.html) — See how semantic axes align in the learned geometry162- [Training Dashboard](https://ruapotato.github.io/Free-Language-Embeddings/dashboard.html) — Loss curves and training metrics163 164## Citation165 166```bibtex167@misc{hamner2026fle,168  title={Free Language Embeddings: Dynamic Masking Word2Vec on DFSG-Compliant Data},169  author={David Hamner},170  year={2026},171  url={https://github.com/ruapotato/Free-Language-Embeddings}172}173```174 175## License176 177GPL-3.0 — See [LICENSE](https://github.com/ruapotato/Free-Language-Embeddings/blob/main/LICENSE) for details.178 179Built by David Hamner.180