khazarai/datascience-RLHF

Model Card for Model ID

Model Details

This model is a fine-tuned version of Qwen3-1.7B using ORPO (Odds Ratio Preference Optimization), a reinforcement learning from human feedback (RLHF) method.

Model Description

<!-- Provide a longer summary of what this model is. -->

• —Base Model: Qwen3-1.7B
• —Fine-tuning Method: ORPO (RLHF alignment)
• —Dataset: ~1,000 data science–related preference samples (chosen vs. rejected responses).
• —Objective: Improve model’s ability to generate higher-quality, relevant, and well-structured responses in data science
• —Language(s) (NLP): English
• —License: MIT

Uses

Direct Use

• —Assisting in data science education (explanations of ML concepts, statistical methods, etc.).
• —Supporting data analysis workflows with suggestions, reasoning, and structured outputs.
• —Acting as a teaching assistant for coding/data-related queries.
• —Providing helpful responses in preference-aligned conversations where correctness and clarity are prioritized.

Bias, Risks, and Limitations

• —Hallucinations: May still produce incorrect or fabricated facts, code, or references.
• —Dataset Size: Fine-tuned on only 1K preference pairs, which limits generalization.
• —Domain Focus: Optimized for data science, but may underperform on other domains.
• —Not a Substitute for Experts: Should not be used as the sole source for critical decisions in real-world projects.
• —Bias & Safety: As with all LLMs, may reflect biases present in training data.

How to Get Started with the Model

Use the code below to get started with the model.

from huggingface_hub import login
from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel

login(token="")

tokenizer = AutoTokenizer.from_pretrained("unsloth/Qwen3-1.7B",)
base_model = AutoModelForCausalLM.from_pretrained(
    "unsloth/Qwen3-1.7B",
    device_map={"": 0}, token=""
)

model = PeftModel.from_pretrained(base_model,"Rustamshry/datascience-RLHF")


prompt = """Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.

### Instruction:
{}

### Input:
{}

### Response:
{}"""


inputs = tokenizer(
    [
        prompt.format(
            "You are an AI assistant that helps people find information",
            "What is the k-Means Clustering algorithm and what is it's purpose?", 
            "",  
        )
    ],
    return_tensors="pt",
).to("cuda")


from transformers import TextStreamer

text_streamer = TextStreamer(tokenizer)
_ = model.generate(**inputs, streamer=text_streamer, max_new_tokens=1800)

Framework versions

• —PEFT 0.17.1