cs-552-2026-Flash-McQueenS-and-TheKing/safety_model

safety_model — Qwen3-1.7B fine-tuned for safety multiple-choice (non-thinking)

This is the safety individual model for the CS-552 (Modern NLP, EPFL, Spring 2026) course project. It is a supervised fine-tune of Qwen/Qwen3-1.7B specialised for the safety benchmark, which is multiple-choice, scored pass@1.

The model answers in non-thinking mode: it emits a one-sentence justification and then the answer letter inside \boxed{}, with no <think> reasoning block. The rationale is given in Why non-thinking, and no CoT distillation.

Output contract. Every answer ends with the option letter wrapped in \boxed{...}, e.g. \boxed{C}. The boxing instruction and the non-thinking switch are baked into the tokenizer's chat_template.jinja, so they apply even when the evaluator calls apply_chat_template(messages, add_generation_prompt=True) with no extra arguments.

Training data

The full training set is released as a companion dataset: **cs-552-2026-Flash-McQueenS-and-TheKing/safety_sft_data**.

It contains 3,250 English multiple-choice items across the seven safety categories of SafetyBench (Zhang et al., 2024). Each example pairs a user message (a question with labelled options) with an assistant message (a one-sentence justification followed by the answer letter in \boxed{}). Four categories are derived from established public datasets; three smaller categories are LLM-generated to cover topics with no convenient public source.

Processing pipeline

• —Letter balancing. Within each category the correct option is shuffled per item so the answer key is not concentrated on one position. (The aggregate A/B skew is a structural consequence of most categories being 2- or 3-option, not a per-item bias.)
• —Synthetic validation. The three generated categories were filtered for validity, deduplicated, self-consistency-checked, and letter-balanced before inclusion.
• —Decontamination. Because four categories come from public datasets that the safety benchmark may also draw on, every training row was checked against the SafetyBench English test split using word 8-gram containment and sentence-embedding cosine similarity; near-duplicate rows were dropped.
• —Format. Each example is a single user→assistant turn. The user turn carries the question and labelled options; the assistant target is a one-sentence justification followed by \boxed{<letter>}.

Fine-tuning

The LoRA adapter was merged back into the base weights so this repository is a standalone, vLLM-loadable Qwen3 checkpoint (full model.safetensors, config.json, generation_config.json, and a tokenizer carrying chat_template.jinja).

Train / inference consistency

Each training example was built as apply_chat_template(user, add_generation_prompt=True, enable_thinking=False) + completion + <|im_end|>, with the prompt tokens masked from the loss. The model is therefore trained on exactly the prefix the evaluator produces — including the empty <think></think> block that Qwen3 emits in non-thinking mode — so there is no train/test format drift.

Generation config

Sampling defaults favour low variance, because the benchmark is pass@1: a single completion is scored, so determinism is worth more than diversity. We use a low temperature while keeping do_sample=true as the project requires. For a pass@k task the opposite choice would be appropriate.

Why non-thinking, and no CoT distillation

A reasonable first instinct is to distill chain-of-thought (CoT) from a stronger model and train the student to reason before answering. We deliberately did not do this, for four reasons specific to this task:

1. 1.The benchmark is not reasoning-intensive. The SafetyBench authors explicitly omit CoT-based evaluation, noting the benchmark is less reasoning-intensive than capability benchmarks such as MMLU. Safety MCQ is largely knowledge and norm-judgment, not multi-step deduction.
2. 2.CoT mostly helps on math and symbolic tasks. The meta-analysis of Sprague et al. (2024), "To CoT or not to CoT?", finds the large gains from CoT concentrate on math, logic, and symbolic reasoning, with little benefit on knowledge/judgment multiple-choice. On such tasks, directly emitting the answer is about as accurate as reasoning first.
3. 3.CoT can hurt small models on classification-style tasks, and a 1.7B model is firmly in the size range where this risk is real. Long reasoning traces also add variance to a single-shot (pass@1) prediction.
4. 4.Token budget and the boxing contract. Under a capped generation budget, a long <think> trace risks consuming the budget before reaching \boxed{} — a failure mode that scores ~0 despite a fluent answer. Non-thinking emits the justification and box immediately, so answer extraction is reliable and fast.

There is also a data-integrity argument: generating CoT with a teacher that already knows the gold label tends to produce post-hoc rationalisations rather than genuine reasoning, which is especially hazardous on bias/safety items (e.g. BBQ ambiguous-context questions) where a fluent justification can be built for a stereotyped wrong answer.

Note the model is not answer-only: the one-sentence justification before the box is itself a lightweight, in-format rationale. We keep that, and skip the heavier <think> block.

Intended use and limitations

• —Intended use. Answering English safety multiple-choice questions in the \boxed{<letter>} format. This is a research/coursework artifact.
• —Option-count coverage. Training items span 2–4 options; performance on items with many more options is less certain.
• —Category imbalance. The four public-derived categories (700 each) are far larger than the three synthetic ones (150 each), so the model is expected to be stronger on the former.
• —Not a deployable safety system. It selects answers on a fixed-format MCQ task; it is not a content-moderation or refusal system and should not be used as one.

How to use