P0L3/CliReNER-EnvironmentalBERT-base

SpanMarker-EnvironmentalBERT for Climate Research NER

This model is a SpanMarker model fine-tuned for fine-grained Named Entity Recognition (NER) in the climate change research domain, extracting 28 distinct entity types. It utilizes the domain-specific ESGBERT/EnvironmentalBERT-base as the underlying encoder.

📌 Model Details

• —Model Type: SpanMarker
• —Encoder: ESGBERT/EnvironmentalBERT-base
• —Maximum Sequence Length: 512 tokens
• —Maximum Entity Length: 14 words
• —Language: English
• —License: cc-by-sa-4.0

Model Labels

🚀 Main Results (Selected Checkpoint)

This repository provides the best-performing checkpoint selected from 5 runs with different random seeds. While the internal training logs tracked performance on the validation split of CliReNER_silver, the final model selection and the metrics below are evaluated on the independent, expert-annotated CliReNER_gold dataset.

This checkpoint corresponds to the seed with the highest strict F1 on the gold evaluation set (Seed 3 - 3012).

📊 Results Across Seeds

We fine-tuned the model using 5 different random seeds to assess the stability and robustness of the architecture on the domain-specific text.

Summary:

• —F1: mean = 45.90, std = 1.02
• —Precision: mean = 50.34, std = 1.24
• —Recall: mean = 42.18, std = 0.93

Model Selection Strategy: The uploaded checkpoint is the single best seed (highest strict F1 on the gold dataset), ensuring strong real-world performance and high-fidelity alignment with domain-expert consensus.

📂 Dataset & Evaluation

• —Training Dataset: CliReNER_silver
• —Splits used: Stratified 80:10:10 ratio (Train/Validation/Test). The 80% split was used for training.
• —Evaluation Dataset: CliReNER_gold
• —Splits used: Evaluated on the combined 192 sentences (expert-annotated via Weighted Expert Voting).
• —Preprocessing:
• —Texts were tokenized using the tokenizer corresponding to the EnvironmentalBERT encoder.
• —The dataset utilizes a flat NER schema (nested entities are excluded, and overlapping entities are resolved to the most relevant span).
• —Metric Details:
• —F1 type: Strict F1 (Entity-level exact match).
• —Evaluation was performed ensuring entities match both the exact boundary span and the exact semantic label to be considered correct.

⚖️ Precision vs Recall Behavior

(Note to author: Describe the model’s tendency here based on your results. Example: "The model exhibits a balanced precision and recall profile.")

⚙️ Usage

Direct Use for Inference

Because this model was trained using the SpanMarker framework, it requires the span_marker library for inference.

pip install span_marker

from span_marker import SpanMarkerModel

# Download from the 🤗 Hub
model = SpanMarkerModel.from_pretrained("P0L3/CliReNER-EnvironmentalBERT-base")

# Run inference
text = "The volume of climate-related literature is expanding exponentially; publications indexed since 2020 already exceed the total output of the preceding decade by 11% (Pan et al. 2025)."
entities = model.predict(text)

for entity in entities:
    print(f"Entity: {entity['span']} | Label: {entity['label']} | Score: {entity['score']:.4f}")


# Entity: climate-related literature | Label: Intellectual Artefact | Score: 0.5150
# Entity: publications | Label: Intellectual Artefact | Score: 0.9278
# Entity: 2020 | Label: Time Period | Score: 0.8446
# Entity: total output | Label: Quantity | Score: 0.9485
# Entity: preceding decade | Label: Time Period | Score: 0.9777
# Entity: 11% | Label: Quantity | Score: 0.9928
# Entity: 2025 | Label: Time Period | Score: 0.8557

Downstream Use

You can easily continue fine-tuning this model on your own dataset.

<details><summary>Click to expand</summary>

from span_marker import SpanMarkerModel, Trainer
from datasets import load_dataset

# Download from the 🤗 Hub
model = SpanMarkerModel.from_pretrained("your-huggingface-username/your-model-name")

# Specify a Dataset with "tokens" and "ner_tags" columns
dataset = load_dataset("your_custom_dataset")

# Initialize a Trainer using the pretrained model & dataset
trainer = Trainer(
    model=model,
    train_dataset=dataset["train"],
    eval_dataset=dataset["validation"],
)
trainer.train()
trainer.save_model("span_marker_model_id-finetuned")

</details>

📉 Training Details

Training Set Metrics

Training Hyperparameters

• —learning_rate: 5e-05
• —train_batch_size: 8
• —eval_batch_size: 8
• —seed: 3012
• —gradient_accumulation_steps: 2
• —total_train_batch_size: 16
• —optimizer: adamw_torch with betas=(0.9,0.999) and epsilon=1e-08
• —lr_scheduler_type: linear
• —lr_scheduler_warmup_ratio: 0.1
• —num_epochs: 20

Training Results (CliReNER_silver Validation Split)

Framework Versions

• —Python: 3.10.19
• —SpanMarker: 1.7.0
• —Transformers: 4.50.0
• —PyTorch: 2.9.1
• —Datasets: 3.0.0
• —Tokenizers: 0.21.4

📚 Citation

If you use this model or the CliReNER datasets in your research, please cite the project:

@misc{poleksic2026named,
  author       = {Poleksić, Andrija and Martinčić-Ipšić, Sanda},
  title        = {Named Entity Recognition for Climate Change Research},
  year         = {2026},
  howpublished = {Research Square},
  note         = {Preprint}
}

Please also acknowledge the SpanMarker framework:

@software{Aarsen_SpanMarker,
    author = {Aarsen, Tom},
    license = {Apache-2.0},
    title = {{SpanMarker for Named Entity Recognition}},
    url = {https://github.com/tomaarsen/SpanMarkerNER}
}