CoolFace
Modelpublic

LikC1606/chinese-babylm-cog-a197-strict-best

sourceHugging Faceotherupdated 3mo agoView on Hugging Face
0likes2.8kdownloads
Model Card

Chinese BabyLM Cog A197 Strict Best

Model Summary

chinese-babylm-cog-a197-strict-best is a custom BERT-based wrapper model for the NLPCC 2026 Chinese BabyLM shared task, optimized for the Cognitive Modeling (Cog) Track. It combines internal strict BERT branches, official-corpus static features, and MLM uncertainty features for brain-aligned fMRI evaluation.

Competition Compliance

RequirementStatus
From scratchSource weights from internal random-initialized strict lineage only
No pretrained checkpointNo official baseline or external pretrained checkpoint
No distillationNo teacher-model distillation
Data optionOfficial corpus (chinese-babylm-org/babylm-zho-100M)
Evaluation leakageNo CogBench stimulus text, labels, predictions, or item-level feedback used in training
No external resourcesNo pypinyin, IDS/CJKVI, radical tables, glyph/font resources, or inference-time rules
Frozen submissionSingle exported wrapper checkpoint

Architecture

FieldValue
Model typeCogAllLayerUncertaintyGate wrapper
ArchitectureCogAllLayerUncertaintyGateModel
Backendmlm
Hidden size1409
Max positions256
Vocabulary16000 (WordPiece, trained on official corpus)

Components

Word branch: A180 strict word BERT, layers [2,3,4] concat, RI scale 3.05 + log-frequency features.

Sentence branch: A130 + gamma -5.1 × (A158s0250 − A122s1500), layers [6,7,8] weighted [0, 0.98, 0.02] + raw384 window8 + log-frequency features.

Uncertainty: A129 step_01000 MLM features (logprob, surprisal, entropy, margin, probability), scale 1.25.

Hidden Evaluation

Evaluated with the final Chinese BabyLM pipeline (chinese-babylm/chinese-babylm-pipeline-final):

TaskScore
word_fmri0.5655
fmri0.1138

Usage

python
from transformers import AutoModel, AutoTokenizer

repo_id = "LikC1606/chinese-babylm-cog-a197-strict-best"
tokenizer = AutoTokenizer.from_pretrained(repo_id, trust_remote_code=True)
model = AutoModel.from_pretrained(repo_id, trust_remote_code=True, output_hidden_states=True)

inputs = tokenizer("春天的花开了。", return_tensors="pt")
outputs = model(**inputs, output_hidden_states=True)

Limitations

  • —Feature extraction model only, not for text generation.
  • —Requires trust_remote_code=True.
  • —Optimized for Cog track; not intended as a general NLU model.