Modelpublic

Phazel/fa-floret-wiki-vectors

sourceHugging Facecc-by-sa-4.0updated 7d agoView on Hugging Face

1likes12downloads

Model Card

fafloretwiki_200k

Persian floret static vector table: 200,000 rows x 300 dimensions, floret mode, minn=maxn=5, hashcount=2, trained with floret-torch on the full Persian Wikipedia dump for 5 epochs. Vectors only, no pipeline components. This is the table used by the fa*newslg tier.

Install

bash

pip install https://huggingface.co/Phazel/fa-floret-wiki-vectors/resolve/main/fa_floret_wiki_200k-0.1.0-py3-none-any.whl

Use

python

import spacy

nlp = spacy.load("fa_floret_wiki_200k")
print(nlp.vocab.vectors.shape)
print(nlp("می‌رود")[0].has_vector)   # floret hashes subwords: always True

To train your own pipeline against it, unpack the wheel and point spaCy at the directory:

bash

pip download --no-deps -d . https://huggingface.co/Phazel/fa-floret-wiki-vectors/resolve/main/fa_floret_wiki_200k-0.1.0-py3-none-any.whl
python -m spacy train config.cfg --paths.vectors ./fa_floret_wiki_200k

The table

Property	Value
Rows	200,000
Dimensions	300
Mode	floret
`minn` / `maxn`	5 / 5
`hash_count`	2
Corpus	full Persian Wikipedia (`fawiki`) dump, extracted with WikiExtractor, sentence-tokenized with spaCy `blank("fa")`: 8,428,449 sentences, 190,781,621 tokens
Tokens	190,781,621 (measured)
Epochs	5
Trained with	floret-torch (GPU port of explosion/floret), cbow, on a Colab T4

Trained with:

python -m floret_torch.train --input fa.txt --output fa \
  --model cbow --mode floret --dim 300 --minn 5 --maxn 5 \
  --hashCount 2 --bucket 200000 --neg 10 --epoch 5 \
  --lr 0.05 --minCount 20 --batch 8192

Other packages in this family

Pipeline	Tier	LAS	ENTS_F	Wheel
`fa_dep_news_sm`	`sm`	85.15	-	7.9 MB
`fa_core_news_sm`	`sm`	85.15	71.87	13.5 MB
`fa_ent_news_sm`	`sm`	-	71.87	5.9 MB
`fa_dep_news_md`	`md`	86.34	-	62.6 MB
`fa_core_news_md`	`md`	86.34	74.71	68.5 MB
`fa_ent_news_md`	`md`	-	74.71	60.6 MB
`fa_dep_news_lg`	`lg`	86.60	-	229.3 MB
`fa_core_news_lg`	`lg`	86.60	75.94	235.2 MB
`fa_ent_news_lg`	`lg`	-	75.94	227.3 MB
`fa_core_news_trf`	`trf`	90.79	82.89	608.2 MB

Tiers: sm hash embeddings, no vectors, md 50k x 300d floret vectors, lg 200k x 300d floret vectors, trf fine-tuned transformer, GPU recommended.

Standalone vector tables, usable as --paths.vectors for your own training:

Vectors	Rows	Used by	Wheel
`fa_floret_400k`	50,000	`md` tier	54.5 MB
`fa_floret_full_wiki`	50,000	no shipped pipeline	54.9 MB
`fa_floret_wiki_200k` (this one)	200,000	`lg` tier	221.3 MB

Training scripts, configs and evaluation: <https://github.com/Fazel94/spacy-persian>.

Sources

Source	Author	Licence
fa_floret static vectors (200k rows x 300d, floret mode, full Persian Wikipedia dump, 5 epochs, trained with floret-torch)	Kiyarash Fazeli	CC BY-SA 4.0
Persian Wikipedia dump (fawiki), the text the vector table is trained on	Wikipedia contributors	CC BY-SA 4.0