Miladsaeedi70/smollm2-135m-scientific-sft-lora

SmolLM2-135M Scientific SFT LoRA

This repository contains a LoRA adapter fine-tuned from `HuggingFaceTB/SmolLM2-135M-Instruct` for concise scientific instruction following.

The model was trained on a custom dataset covering scientific question answering, summarization, concept explanation, method comparison, technical simplification, bullet generation, data analysis, and Python code generation.

This repository contains the LoRA adapter and tokenizer files, not a complete standalone copy of the base model. The adapter must be loaded together with the original SmolLM2-135M-Instruct model.

Training Summary

Dataset Overview

The training dataset contains eight scientific instruction-following tasks:

• —Scientific question answering (256)
• —Summarization (220)
• —Concept explanation (200)
• —Method comparison (200)
• —Technical simplification (200)
• —Bullet generation (200)
• —Data analysis (200)
• —Code generation (100)

The dataset focuses on scientific and technical domains including environmental science, air pollution, geospatial machine learning, computer vision, large language models, and scientific Python programming. Grouped train/validation/test splitting was used to reduce prompt leakage between data splits.

Model Details

Model Description

The model is a parameter-efficient scientific instruction-following adaptation of SmolLM2-135M-Instruct.

Supervised fine-tuning was performed using Low-Rank Adaptation across the model's linear layers. The training data emphasizes concise, structured, technically accurate responses in scientific and machine-learning domains.

The primary covered domains include:

• —Environmental science
• —Air pollution
• —Geospatial machine learning
• —Data science
• —Computer vision
• —Large language models
• —Scientific Python programming

Model characteristics:

• —Developed by: Milad Saeedi
• —Model type: Decoder-only causal language model with a LoRA adapter
• —Base model: HuggingFaceTB/SmolLM2-135M-Instruct
• —Fine-tuning method: Supervised fine-tuning with PEFT LoRA
• —Language: English
• —Pipeline: Text generation
• —Library: PEFT, Transformers, and TRL
• —License: Apache-2.0, following the base model license
• —Parameters updated: LoRA adapter parameters only

Model Sources

• —Base model: HuggingFaceTB/SmolLM2-135M-Instruct
• —Adapter repository: Miladsaeedi70/smollm2-135m-scientific-sft-lora

Uses

Direct Use

The adapter can be used for scientific instruction-following tasks such as:

• —Answering scientific questions
• —Explaining technical concepts
• —Simplifying technical language
• —Summarizing scientific text
• —Comparing scientific or machine-learning methods
• —Generating concise bullet points
• —Interpreting small data-analysis problems
• —Producing short Python examples
• —Drafting structured scientific responses

The model is intended mainly for experimentation, education, portfolio demonstrations, and lightweight scientific NLP applications.

Downstream Use

The adapter may be integrated into:

• —Scientific assistants
• —Educational applications
• —Retrieval-augmented generation systems
• —Scientific document-processing pipelines
• —Lightweight local language-model applications
• —Model-comparison demonstrations
• —Further preference optimization experiments such as GRPO

The adapter may also be merged with the base model for deployment, subject to the licenses and terms of both the base model and this repository.

Out-of-Scope Use

The model should not be relied upon as the sole source for:

• —Medical diagnosis or treatment
• —Legal advice
• —Safety-critical engineering decisions
• —Environmental regulatory decisions
• —High-stakes scientific conclusions
• —Fully autonomous code execution
• —Factual claims requiring verified and current evidence

The model was not trained to provide unrestricted general-purpose assistance across every field. It may perform poorly on topics that differ substantially from the training domains.

Bias, Risks, and Limitations

This adapter inherits limitations from the SmolLM2-135M-Instruct base model.

Because the base model is small, generated responses may:

• —Contain factual inaccuracies
• —Hallucinate scientific terminology or references
• —Oversimplify complex topics
• —Produce incomplete reasoning
• —Generate incorrect Python code
• —Repeat phrases
• —Misinterpret ambiguous prompts
• —Perform poorly on long-context inputs
• —Struggle with advanced mathematical reasoning
• —Produce plausible but unsupported scientific conclusions

The training dataset is custom and relatively small. It does not represent the full diversity of scientific disciplines, writing styles, institutions, geographic regions, or technical perspectives.

The evaluation primarily uses ROUGE, which measures lexical overlap with reference answers. Higher ROUGE scores do not guarantee factual correctness, scientific validity, originality, or safe downstream behavior.

Recommendations

Users should independently verify scientific claims, calculations, code, and citations produced by the model.

For higher-reliability applications, the model should be combined with:

• —Retrieval from trusted sources
• —Citation verification
• —Domain-specific validation
• —Human expert review
• —Structured output checks
• —Code testing and sandboxed execution

How to Get Started

Install the required packages:

pip install -U transformers peft accelerate torch