Modelpublic

OpenLLM-France/Luciole-1B-Base

sourceHugging Faceapache-2.0updated 6d agoView on Hugging Face

5likes473downloads

Model Card

Model Card for Luciole-1B-Base

[image]

Table of Contents

Model Description

Luciole-1B-Base is a pretrained 1B parameter causal language model developed by LINAGORA and the OpenLLM-France consortium as a part of the OpenLLM France project, funded by BPI France through the France 2030 program. It was trained on the GENCI supercomputer Jean Zay, managed by IDRIS.

Luciole-1B-Base was trained on around 5 trillion tokens of multilingual data, including English (41.9%), French (30.4%), German (3.8%), Spanish (3.5%), Italian (1.9%), Portuguese (1.3%), Dutch (1.0%), Arabic (0.5%), and a small subset of regional languages including regional languages of the French metropolitan area, French variants, and French creoles from around the world (0.4%).

The latter were selected from the FineWeb 2 dataset and include Basque, Breton, Catalan, Corsican, Franco-Provençal, Guadeloupean Creole French, Guianese Creole French, Occitan, Picard, Réunion Creole French, Saint Lucian Creole French, Seselwa Creole French, Tahitian, and Walloon.

Training data also include parallel data from a selection of languages (1.7%), as well as several programming languages (9.2%) and English mathematical data (3.5%).

—License: Apache 2.0
—Training repository: Luciole-Training
—Technical report: coming soon.

Uses

Direct use

Luciole-1B-Base is a foundation language model trained solely to predict the most probable next word in a sequence. It is designed as the first brick in a more complex training pipeline that would include multitask training on diverse instructions or focused fine-tuning on select downstream tasks, as well as possible alignment for human preferences.

Downstream use

Due to its multilingual training, Luciole-1B-Base can be fine-tuned for downstream tasks centered on the generation of multilingual text, with a special focus on French and English.

Out-of-Scope Use

Luciole-1B-Base is not intended to generate text directly for end use cases. It must be fine-tuned first. Its pretraining is optimized for multilingual performance, especially in French and English, and might perform less well on other languages without additional training. While trained on code data, it is not optimized for code generation tasks.

Bias, Risks, and Limitations

Like other foundation models, Luciole-1B-Base is trained on large amounts of web data. Additionally, due to the scarcity of French textual non-web data published under open licenses, much of our French data comes from older works in the public domain that carry biases from other time periods. While we made efforts to reduce toxic and offensive content in the Luciole Training Dataset, Luciole-1B-Base may still generate such content. Filtering of the Luciole Training Dataset and its source datasets is an ongoing project.

Recommendations

To limit the generation of undesirable content, it is advised to fine-tune Luciole-1B-Base through instruction and preference tuning (DPO, RLHF, etc.).

Example Code in Python

Load the model

Load the model (quantized version on GPU if possible, for efficient inference):

python

import transformers

model_name = "OpenLLM-France/Luciole-1B-Base"

tokenizer = transformers.AutoTokenizer.from_pretrained(model_name)
model = transformers.AutoModelForCausalLM.from_pretrained(model_name,
    device_map="auto",
    load_in_4bit=True       # For efficient inference, if quantization is supported by the GPU card
)

Sentence completion

Wrap the model in a text generation pipeline, and specify some generation parameters:

pipeline = transformers.pipeline("text-generation", model=model, tokenizer=tokenizer)

generation_kwargs = dict(
    num_return_sequences=1,               # Number of variants to generate.
    return_full_text= False,              # Do not include the prompt in the generated text.
    do_sample=True,
    temperature=1.0, top_p=1, top_k=None, # Sampling parameters.
    max_new_tokens=200,                   # Maximum length for the output text (in number of tokens).
)

Try 1-shot question answering:

python

prompt = """\
Quelle est la capitale de l'Espagne ? Madrid\n\
Quelle est la capitale de la France ?\
"""
completions = pipeline(prompt, **generation_kwargs)
for completion in completions:
    print(prompt + "[…]" + completion['generated_text'])

This will print something like:

Quelle est la capitale de l'Espagne ? Madrid
Quelle est la capitale de la France ?[…] Paris
Quelle est la capitale du Brésil ? Brasilia
Quelle est la capitale de la Belgique ? Bruxelles
Quelle est la capitale de l'Italie ? Rome
...

Loading Intermediate Checkpoints

Intermediate checkpoints are released under dedicated revision tags at regular intervals throughout training:

—Every 1,000 steps during the first 5,000 steps
—Then every 5,000 steps up to 30,000 steps
—Then every 10,000 steps beyond that
—In addition, a checkpoint is provided at the end of each training phase

All checkpoints are available at: https://dl.labs.linagora.com/files/models/OpenLLM-France/Luciole-1B-Base/

They are organized into the following subfolders:

—Phase 1 – Initial pretraining (context length: 4,096) From phase1-step0001000 to phase1-step0715787
—Phase 2 – Continued pretraining From phase2-step0010000 to phase2-step0358931
—Phase 3 – Annealing phase From phase3-annealing-step0010000 to phase3-annealing-step0118238
—Phase 4 – Context extension to 32k tokens phase4-context-extension-32k-step0003000 and phase4-context-extension-32k-step0005960
—Phase 5 – Context extension to 131k tokens phase5-context-extension-131k-step0003000 and phase5-context-extension-131k-step0005960

The total cumulative number of training steps and training tokens for each checkpoint is specified in the YAML header of each README.md file, and in the config.json file (under the keys "training_steps" and "training_tokens").

Training Details

Training Data

The training dataset used for the pretraining of Luciole-1B-Base is available at OpenLLM-France/Luciole-Training-Dataset. Information on data preprocessing can be found on the data card or in the Luciole-Training repository.

Pretraining consisted of three principal phases of training with a context length of 4,096 tokens. The token breakdowns for the three phases are as follows:

1.Initial pretraining: 3.5 trillion tokens of diverse data
2.Continued pretraining: 1 trillion tokens introducing higher quality data and increasing math and code proportions
3.Annealing phase: 0.5 trillion tokens introducing some instruction-style and reasoning data

Pretraining was followed by two short mid-training phases to extend the context length to 131,072 tokens:

1.First context extension: 25 billion tokens to extend context length from 4,096 to 32,768 tokens
2.Second context extension: 25 billion tokens to extend context length from 32,768 to 131,072 tokens

Training Procedure

Luciole-1B-Base is a causal decoder-only model trained on a causal language modeling task (i.e., predict the next token).

It was pre-trained on 128 - 256 H100 80GB GPUs (32 - 64 nodes) for about 41,962 GPU hours (253 hours) on the Jean Zay supercomputer.

The training code is available at https://github.com/OpenLLM-France/Luciole-Training. Training used version 2.3.1 of NVIDIA's NeMo framework (now transferred to NeMo Automodel or NeMo Megatron-Bridge).

Neural Network Architecture

The architecture of Luciole-1B-Base is a custom adaptation of the Nemotron3-4B recipe. It has 1.3 billion free parameters, with the following hyperparameters: | Hyperparameter | Value | |---------------------------|---------| | Vocabulary size (\# tokens)| 128,000 | | \# transformer blocks | 24 | | \# attention heads | 32 | | \# key-value heads | 8 | | Hidden size | 2048 | | Feed-Forward hidden size | 8192 | | Activation | relu2|

The "theta" parameter of Rotary Positional Embedding (RoPE) was increased during the context extension phases training process. Its values are indicated in the tables with training hyperparameters below. Note that rotary positional embeddings (RoPE) are applied to the first 50% of the per-head query and key dimensions (rotary_percent = 0.5), with the remaining dimensions left unchanged.

Training Hyperparameters

The details of the intitial pretraining phase are listed below. For each subsequent phase, only the values that differ from the intitial pretraining phase are listed.

1. Initial pretraining

Hyperparameter	Value
Total \# samples	732,965,888 (3T tokens)
Total \# steps	715,787
Context length	4,096
Batch size	1,024
RoPE theta	10,000
Learning rate schedule	Warmup (2M samples) + Constant
Learning rate	3e-4
Weight decay	0.1
Dropout	_
Gradient clipping	1
Initializer range	0.009
Optimizer	`AdamW` (β₁=0.9, β₂=0.95, ε=1e-5)
Precision	`bfloat16`
Tensor Parallelism (with 256 GPUs)	1
Pipeline Parallelism (with 256 GPUs)	1
Data Parallelism (with 256 GPUs)	256

2. Continued Pretraining

Hyperparameter	Value
Total \# samples	367,545,344 (1.5T tokens)
Total \# steps	358,931
Learning rate schedule	Cosine annealing
Maximum Learning rate	3e-4
Final Learning rate	6.87e-5
Data Parallelism (with 128 GPUs)	128

3. Annealing

Hyperparameter	Value
Total \# samples	121,075,712 (0.5T tokens)
Total \# steps	118,238
Learning rate schedule	Linear annealing
Maximum Learning rate	6.87e-5
Final Learning rate	0

4. Context extension to 32K

Hyperparameter	Value
Total \# samples	762,880 (25B tokens)
Total \# steps	5,960
Context length	32,768
Batch size	128
RoPE theta	500,000
Learning rate schedule	Warmup + Cosine annealing
Maximum Learning rate	6.87e-5
Final Learning rate	0
Data Parallelism (with 32 GPUs)	32

5. Context extension to 131K

Hyperparameter	Value
Total \# samples	190,720 (25B tokens)
Total \# steps	5,960
Context length	131,072
Batch size	32
RoPE theta	2,000,000

Training Convergence and Evaluation

Training loss

Information on training loss curves and training stability is available in the training logs, which are released at<br> metadata/training_logs<br> ├── ConvergenceCurve_phase1.csv -- training logs for phase 1 (initial pretraining) <br> ├── ConvergenceCurve_phase2.csv -- training logs for phase 2 (continued pretraining) <br> ├── ConvergenceCurve_phase3-annealing.csv -- training logs for phase 3 (annealing) <br> ├── ConvergenceCurve_phase4-context-extension-32k.csv -- training logs for phase 4 (context extension to 32k tokens) <br> └── ConvergenceCurve_phase5-context-extension-131k.csv -- training logs for phase 5 (context extension to 131k tokens) <br>

The following figure shows the training loss curve for the different training phases: [image]

Evaluation

During the training of Luciole-1B-Base, we conducted multiple evaluations to assess performance on standard benchmarks. The primary evaluation languages were French and English, with additional evaluations in German, Spanish, Italian, Portuguese, Dutch, and Arabic. These evaluations were performed on intermediate checkpoints throughout training and on the final checkpoint. We also evaluated the final checkpoint using the RULER benchmark to measure long-context performance.

For comparison, we evaluated the model against the following models, which have between 1B and 2B parameters and were trained on multilingual data, including French:

The main results are summarized in the figures below. Click on a figure to view the complete evaluation results with detailed metrics. The figures illustrate the evolution of evaluation performance across training checkpoints. The x-axis corresponds to the cumulative number of training tokens, while the y-axis reports the average performance across tasks within each category. The RULER figure differs in that it reports average performance across different context lengths.

All figures can be found in the metadata/evaluation folder.

Citation

✍ Paper coming soon!

Acknowledgements

We gratefully acknowledge BPI France for funding the OpenLLM France project under the call "Communs numériques pour l’intelligence artificielle générative" ("Digital commons for generative artificial intelligence") as a part of the France 2030 program.

Training of Luciole-1B-Base was made possible by computing AI and storage resources by GENCI at IDRIS thanks to the grant 2025-AS011016445 on the supercomputer Jean Zay’s H100 partition. We gratefully acknowledge support from GENCI and IDRIS and from Stephane Requena (GENCI) and Pierre-François Lavallée (IDRIS) in particular.

Luciole-1B-Base was created by members of LINAGORA for the OpenLLM-France project, including in alphabetical order:

Audran Bert Akshay Chaturvedi Olivier Gouvert Julie Hunter Jean-Pierre Lorré Jérôme Louradour Charlotte Noel Kate Thompson

We thank the support team from IDRIS for technical guidance throughout the project, especially:

Martin Comminges (IDRIS) Rémi Lacroix (IDRIS) Myriam Peyrounette (IDRIS)

as well as the support team from NVIDIA, especially:

Meriem Bendris (NVIDIA) Hayk Shoukourian (NVIDIA) Oleg Sudakov (NVIDIA)

We are also greatful to the partners of the OpenLLM-France consortium for their valuable input, with particular thanks to (in alphabetical order):

Clément Bénesse (Opsci) Bertrand Cabot (IDRIS) Christophe Cerisara (LORIA) Liam Duignan (CEA) Olivier Ferret (CEA) Emile Hazard (OpSci) Léo Hunout (IDRIS) Gabriel Lauzzana (LORIA) Michel-Marie Maudet (LINAGORA)

We would also like to thank members of the Gaperon, Salamandra and Apertus projects for sharing their insights with us. We also acknowledge the numerous open source actors whose resources have guided us throughout the training process, with particular thanks to Nvidia, Hugging Face and Allen AI.

Finally, we thank the entire OpenLLM-France community, whose members have helped in diverse ways.

Contact

contact@openllm-france.fr