CMB-ClimateModernBERT/Merge_Soup_LRD

ClimateModernBERT · Soup

Part of ClimateModernBERT, a family of climate-domain encoders obtained by continued pretraining of ModernBERT-Base on climate text.

The manuscript's best model (76.3 average F1). Uniform average of the three single-source Phase-2 checkpoints; the component list was read from the repo's mergekit_config.yml and matches the paper's ℱ (FastText-filtered) component. Start here.

Republished from `sraj/Merge_Linear` under a name that matches the paper's notation. The weights are identical; the original repository remains available.

Training data

Continued pretraining used Academic + Synthetic + Climate Web from a 6.42B-token climate corpus:

Raw academic text is not redistributed: peer-reviewed articles are accessed under institutional publisher licenses, and news shards and handbooks were collected for non-commercial research use. The processing pipelines are released instead.

Training procedure

Two stages, following ModernBERT's own continued-pretraining recipe:

• —Phase 1 — context extension. 3 epochs, constant LR 3e-4, global batch 576, sequence length 8,192, MLM masking 30%, StableAdamW, BF16.
• —Phase 2 — LRD specialization. 3 further epochs on a 1 − √t decay schedule from LR 3e-4 with final LR factor 1e-3.

4× NVIDIA A100, MosaicML Composer. Final checkpoints are converted to HF Transformers format.

Usage

ModernBERT is native to transformers from 4.48 onward, so no trust_remote_code is required.

from transformers import AutoTokenizer, AutoModel

model_id = "CMB-ClimateModernBERT/Merge_Soup_LRD"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModel.from_pretrained(model_id)

inputs = tokenizer("Scope 1 and 2 emissions fell 12% against a 2019 baseline.", return_tensors="pt")
outputs = model(**inputs)   # outputs.last_hidden_state -> (batch, seq, 768)

For a downstream task, load AutoModelForSequenceClassification and fine-tune. The paper's recipe: LR 4e-5, effective batch 64, weight decay 0.01, up to 10 epochs with early stopping on validation F1, BF16 with fused AdamW.

Evaluation

This checkpoint reaches 76.3 average F1 across the nine benchmarks, as reported in the manuscript.

Evaluated on nine climate NLP benchmarks: Climate Detection, Climate Specificity, Commitments & Actions, Climate Sentiment, Net Zero & Reduction, TCFD Recommendations, WFB Nature, WXImpactBench, and ClimRetrieve. Binary tasks and ClimRetrieve report positive-class F1; multi-class and multi-label tasks report macro-F1. Scores are the mean over three fine-tuning seeds under a single shared hyperparameter configuration.

Reference points from the paper: the ModernBERT-Base stable-phase baseline reaches 73.5 average F1, and ClimateBERT reaches 72.1 under the same protocol.

This is the recommended ClimateModernBERT checkpoint for general use.

Intended use

Research on climate NLP: encoding climate text, and fine-tuning for classification, multi-label tagging, and retrieval over corporate disclosures, policy documents, scientific literature and climate news.

Limitations

• —English only, and built on a single encoder family (ModernBERT-Base).
• —Current climate NLP benchmarks are largely sentence- or passage-level, so the model's long-context capacity is not fully exercised by the reported evaluation.
• —Findings about corpus composition are demonstrated within climate NLP and should not be read as universal principles of domain adaptation.
• —Synthetic training data has task-dependent effects: it helps taxonomy- and framework-driven tasks while degrading performance on tasks requiring finer-grained discourse and commitment understanding.
• —The model is a masked language model, not an instruction-following system, and produces no calibrated factual guarantees about climate science.

Paper

Climate-ModernBERT: Revisiting Corpus Composition for Domain-Adaptive Continued Pretraining. Preprint manuscript, currently under review — no venue, DOI or arXiv identifier yet, and no citation to give. The PDF is hosted in the project repository.

• —Project website: https://climatemodernbert.github.io/ClimateModernBERT/
• —Code and pipelines: https://github.com/ClimateModernBERT/ClimateModernBERT
• —Full model catalog: https://github.com/ClimateModernBERT/ClimateModernBERT/blob/main/docs/model-inventory.md
• —Naming guide: https://github.com/ClimateModernBERT/ClimateModernBERT/blob/main/docs/model-naming.md

License

Not yet set on this repository. The upstream base model is ModernBERT-Base; check its terms, and those of the underlying corpora, before redistributing. No license is asserted here on the maintainers' behalf.