Modelpublic

kabaros/ar-camel-ebooks-heavy-8.3M

sourceHugging Facecc-by-4.0updated 1mo agoView on Hugging Face

0likes143downloads

Model Card

BabyLM Arabic — Best Models per Dataset

Six GPT-2-style causal LMs (~17M params: n_embd=512, n_layer=4, n_head=8, n_ctx=512), each the best-performing tokeniser/seed combination for one of six ~8.3M-word Arabic corpus compositions, trained as part of an MSc dissertation on tokenisation and dataset composition for small-scale Arabic language models (University of Stirling). See the companion dataset cards for the five modified corpora (e.g. kabaros/ara-ebooks-heavy-8.3M) for composition details.

Grammaticality is measured via MultiBLiMP (minimal-pair judgment, mean of 3 seeds reported in the dissertation; the score below is this specific published checkpoint's individual run, the best of the 3 seeds trained for that dataset).

Dataset	Model dir	Tokeniser	Seed	MultiBLiMP
Original Data (`babylm-ara`)	`ar-camel-with-unigram-babylm-ara-seed43`	CAMeL + Unigram	43	77.20
8.3M ebooks-heavy	`ar-camel-custom-8.3M-wiki_0-movies_0-habibi_0-ebooks_100-seed44`	CAMeL (BPE)	44	80.33
8.3M ebooks w/ 5% habibi	`ar-camel-with-unigram-custom-8.3M-wiki_0-movies_0-habibi_5-ebooks_95-seed43`	CAMeL + Unigram	43	79.26
8.3M ebooks w/ 5% movies	`ar-camel-with-unigram-custom-8.3M-wiki_0-movies_5-habibi_0-ebooks_95`	CAMeL + Unigram	42 (default)	79.67
8.3M Impossible Man	`ar-farasa-with-unigram-custom-impossible-man-8.3M-fixed-20%-ebooks-42%-hindawi-38%`	Farasa + Unigram	42 (default)	79.26
8.3M Diverse	`ar-camel-with-unigram-diversity-8.3M-seed44`	CAMeL + Unigram	44	77.61

For context: the BabyLM-community/babylm-ara paper baseline (BPE, single run) scores 75.9; OLMo-2 (32B params, 6T tokens) scores 87% on the same task in Arabic.

Loading: the CAMeL/Farasa models use a custom PreTrainedTokenizerFast subclass (camel_tokenizer.py/farasa_tokenizer.py, included in each model's files) that applies morphological segmentation before every encode call. Load with trust_remote_code=True:

python

from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("kabaros/<repo-name>", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("kabaros/<repo-name>")