AMFORGE/cofos_general_600m

1---2license: apache-2.03language:4- en5- fr6tags:7- text-generation8- multilingual9- foundation-model10- sparse-transformer11- amforge12- sparsemind13library_name: pytorch14pipeline_tag: text-generation15inference: false16---17 18# Cofos General 600M — Bilingual Foundation Model19 20**Cofos General 600M** is a 640M-parameter foundation language model trained from scratch on curated French and English web-scale text. It is the base model in the **Cofos General** family by **AMEFORGE**, built on the proprietary **SparseMind** architecture and designed to serve as a substrate for downstream specialization through fine-tuning.21 22This model is **not** intended as a standalone assistant. Its purpose is to provide a clean, bilingual, controllable foundation that downstream models (code assistants, personalized assistants, domain-specific tools) can build upon.23 24---25 26## Model Summary27 28| Field | Value |29|---|---|30| **Developer** | AMEFORGE |31| **Architecture** | SparseMind v15 (proprietary) |32| **Parameters** | 640M |33| **Context length** | 2048 tokens |34| **Vocabulary** | 32,000 (custom NexusBPE, multilingual) |35| **Languages** | French (50%), English (50%) |36| **Training data** | Public web-scale text (educational subsets) |37| **Model type** | Causal language model (base, no instruction tuning) |38| **License** | Apache 2.0 |39| **Status** | Active training |40 41---42 43## Intended Use44 45### Primary use cases46 47- **Foundation for fine-tuning** into specialized downstream models (code assistants, personalized assistants, domain experts)48- **Bilingual text-completion** in French and English where a small, controllable base is required49- **Research** on small bilingual foundation models, sparse architectures, and balanced cross-lingual representations50 51### Out-of-scope52 53This model is **not designed for**:54- Direct deployment as a user-facing assistant (it has no instruction tuning and no RLHF)55- Languages other than French and English56- Tasks requiring extensive factual knowledge or current information (training data has a cutoff and limited coverage)57- Safety-critical applications without additional alignment and filtering layers58- Long-context reasoning beyond 2048 tokens59 60If you want an instruction-following code assistant, look at downstream models in the Cofos family (forthcoming `cofos_general_code_600m`, `cofos_logo_600m`).61 62---63 64## Why a small bilingual foundation model?65 66The model landscape is dominated by either very large general-purpose models or specialized models built on English-only foundations. Cofos General 600M occupies a deliberate niche:67 681. **Balanced bilingual representation**: Trained 50/50 on French and English educational web text, providing native-quality coverage in both languages rather than the typical 95%+ English bias of comparable open models.692. **Small enough for on-device fine-tuning**: At 640M parameters, fine-tuning is tractable on a single high-end consumer GPU, making downstream specialization accessible.703. **Curated training data**: Trained on educational subsets of public web crawls rather than raw uncurated web text, reducing noise and improving the foundation's quality-per-token ratio.714. **Controllable substrate**: As the training data is documented and reproducible, downstream users know what their fine-tuned models inherited from the base.72 73---74 75## Performance76 77This is a **base model under active training**. Performance characteristics are reported as training progresses. Refer to the latest model card revision on the HuggingFace repository for current metrics.78 79The model is evaluated primarily on:80- Cross-entropy loss on held-out French and English validation sets81- Downstream task performance after fine-tuning (which is the intended use)82 83Direct zero-shot benchmark performance is not the design target. **A base model that is uninteresting standalone but excellent under fine-tuning is, by design, doing its job.**84 85---86 87## Usage88 89### Loading90 91```python92from huggingface_hub import hf_hub_download93import torch94 95checkpoint_path = hf_hub_download(repo_id="AMFORGE/cofos_general_600m", filename="cofos_model.pt")96tokenizer_path  = hf_hub_download(repo_id="AMFORGE/cofos_general_600m", filename="cofos_tokenizer.model")97```98 99Loading and inference require the AMEFORGE SparseMind runtime. The model architecture is proprietary; contact AMEFORGE for access to the runtime, or wait for the public inference utilities released with downstream models.100 101### Recommended workflow102 103The recommended usage is **not** direct generation but **fine-tuning** for a specific task. Typical pipeline:104 1051. Download this base model1062. Prepare a task-specific dataset1073. Fine-tune with standard transfer-learning hyperparameters (low learning rate, fresh optimizer, small number of epochs)1084. Deploy the fine-tuned variant109 110The forthcoming `cofos_general_code_600m` and `cofos_logo_600m` repos illustrate this workflow concretely.111 112---113 114## Training115 116Cofos General 600M is trained from scratch on a curated mix of public, openly-licensed web text:117- **English educational web text** (filtered for educational quality)118- **French web text** (multilingual web corpus, French subset)119 120Training is conducted on the AMEFORGE SparseMind training pipeline with periodic safety checkpointing to HuggingFace to ensure recoverability. Mixed sampling preserves a strict 50/50 ratio between French and English throughout training.121 122**Tokenizer**: [AMFORGE/cofos_general_tok](https://huggingface.co/AMFORGE/cofos_general_tok) — a 32,000-token custom SentencePiece model with multilingual byte fallback for full Unicode coverage and structural tokens reserved for downstream task formatting.123 124---125 126## Lineage127 128```129cofos_general_tok (tokenizer)130       ↓131cofos_general_600m (this model) — bilingual foundation132       ↓133cofos_general_code_600m (forthcoming) — instruction-tuned for code134       ↓135cofos_logo_600m (forthcoming) — personalized variant136```137 138Cofos General 600M is a **from-scratch base model**. It is not derived from any other published model.139 140---141 142## Limitations & Biases143 144- **No instruction tuning**: This is a raw base model. It will not naturally follow instructions, refuse harmful requests, or behave like an assistant. It is a text-completion engine.145- **Limited training data**: At ~3-6 billion tokens of training (compared to trillion-scale corpora for SOTA models), Cofos General 600M's knowledge breadth is much smaller than models like SmolLM2 or Qwen-0.5B. It is **not competitive on broad knowledge benchmarks** — this is by design, as breadth is sacrificed for tractable specialization.146- **Bias inheritance**: The model will reflect biases present in the FineWeb-Edu (English) and FineWeb-2 (French) training corpora. These are large public web corpora with all the typical biases of such sources.147- **No safety alignment**: Cofos General 600M has no RLHF, no refusal training, and no harm-prevention filtering. It should never be deployed in user-facing products without a downstream safety layer.148- **Capacity limits**: 640M parameters is small by modern standards. Complex multi-step reasoning and long-context coherence will be inferior to larger models. The intended remediation is task-specific fine-tuning, not direct use.149 150---151 152## Environmental Considerations153 154Cofos General 600M is intentionally small to minimize the compute footprint of training and to make downstream fine-tuning accessible to individual researchers and small teams. The model can be fine-tuned and deployed on a single consumer GPU.155 156---157 158## License159 160This model is released under the **Apache 2.0** license. You are free to use, modify, and redistribute it, including for commercial purposes, subject to the terms of the license.161 162Note: training data was sourced from publicly available datasets (FineWeb-Edu, FineWeb-2). Users redistributing this model or derivatives should ensure compliance with the original source licenses.163 164---165 166## Citation167 168If you use Cofos General 600M in your work, please cite:169 170```bibtex171@misc{cofos_general_600m_2026,172  title  = {Cofos General 600M: A Bilingual Foundation Model for Downstream Specialization},173  author = {{AMEFORGE}},174  year   = {2026},175  url    = {https://huggingface.co/AMFORGE/cofos_general_600m}176}177```178 179---180 181## Contact182 183For questions, collaborations, or access to the AMEFORGE SparseMind runtime:184 185- **Organization**: AMEFORGE186- **HuggingFace**: [@AMFORGE](https://huggingface.co/AMFORGE)187 188---189 190*Cofos General 600M is the foundation layer of the Cofos model family by AMEFORGE. See the [AMFORGE organization page](https://huggingface.co/AMFORGE) for downstream specialized models built on this base.*