Akan4ik/DeepSeek-V4-Flash-0731-FFT-BF16-Cyber-2.0

DeepSeek-V4-Flash-0731-FFT-BF16-Cyber 2.0

Overview

This release is an experimental cybersecurity-oriented modification of DeepSeek-V4-Flash-0731, focused on mechanistic analysis and suppression of refusal-associated internal representations.

The main objective of this iteration was not generic fine-tuning, but refusal representation analysis and targeted abliteration: locating activation directions correlated with refusal behavior and reducing their influence while preserving the underlying model capabilities as much as possible.

The work was motivated primarily by:

• —*Arditi et al. — Refusal in Language Models Is Mediated by a Single Direction***, arXiv:2406.11717
• —*Zou et al. — Representation Engineering: A Top-Down Approach to AI Transparency***, arXiv:2310.01405
• —*Wollschläger et al. — The Geometry of Refusal in Large Language Models: Concept Cones and Representational Independence***, arXiv:2502.17420
• —*Piras et al. — SOM Directions are Better than One: Multi-Directional Refusal Suppression in Language Models***, arXiv:2511.08379

The results obtained on DeepSeek-V4-Flash-0731 appear to be more consistent with a multi-component refusal representation than with a strictly single-direction interpretation.

Refusal Representation Analysis

Activation analysis indicated that refusal behavior in this checkpoint was not cleanly explained by one isolated principal direction.

The strongest recurring refusal-associated components were observed around:

PC2, PC4, and PC6

with PC2 carrying the strongest and most consistently observed refusal-associated structure in the tested runs.

This suggests that, for this model, refusal-related information may occupy a small multi-dimensional subspace rather than being represented exclusively by a single linear direction.

Localization

The highest concentration of refusal-associated features was observed approximately in the:

15–25 activation/tensor region

with a more conservative isolation window extending approximately to:

15–29/30

The wider 15–29/30 range was retained as a conservative candidate region because weaker refusal-associated destinations continued to appear outside the strongest 15–25 concentration.

Observed structure:

These locations should currently be treated as empirical localization results for this checkpoint, not as a claim that every refusal mechanism in DeepSeek-V4-Flash-0731 has been exhaustively identified.

Abliteration

The modification uses refusal-direction / representation ablation inspired by representation-engineering methods.

Instead of assuming that refusal is fully mediated by one direction, this iteration explored multiple principal components associated with the harmful-vs-benign activation separation.

The final edit therefore targeted a broader refusal-associated subspace identified during the activation analysis.

The goal was to minimize unnecessary refusal while avoiding indiscriminate modification of unrelated model representations.

Datasets

The analysis used both public/published project datasets and custom contrastive prompt data.

Akan4ik/offsec-400

Akan4ik/offsec-400 was used as the primary cybersecurity-oriented refusal evaluation set.

It contains security/offensive-security instructions intended to measure whether the model rejects technically legitimate cybersecurity tasks because of alignment-induced refusal behavior.

Custom Alpaca-derived harmful prompt set

A separate custom dataset derived from the Alpaca instruction format was used to construct harmful/refusal-inducing examples for representation analysis.

The Alpaca-derived data was used together with benign/control instructions to improve separation between ordinary instruction-following representations and refusal-associated activation directions.

This dataset was used primarily as an experimental representation-analysis resource rather than as a general capability benchmark.

Refusal Benchmark

Earlier exploratory experiments used substantially larger instruction sweeps.

For this iteration, the final refusal experiment was reduced to an 800-instruction evaluation run, using the same OffSec-oriented evaluation methodology.

Absolute refusal reduction: 74 percentage points.

The result indicates that the identified refusal-associated representation was responsible for a substantial fraction of the observed refusal behavior on the evaluated cybersecurity instruction distribution.

Importantly, the remaining 23% refusal rate also suggests that refusal is not completely explained by the edited components. Additional nonlinear, distributed, prompt-dependent, or later-stage mechanisms may remain active.

Experimental Finding: Refusal Is Not Necessarily One-Dimensional

A particularly important observation from this experiment was the presence of several meaningful refusal-associated principal components.

The original single-direction hypothesis is an extremely useful approximation, but the DeepSeek-V4-Flash-0731 experiments showed detectable refusal structure across at least:

PC2 → PC4 → PC6

rather than a single isolated principal component.

This observation is directionally consistent with later mechanistic-interpretability research showing that refusal can occupy multiple independent directions, low-dimensional manifolds, or concept-cone-like structures rather than a universally one-dimensional representation.

For this checkpoint, the working interpretation is therefore:

Refusal appears low-dimensional, but not strictly one-dimensional.

PC2 was the dominant component in the performed analysis, while PC4 and PC6 contained additional refusal-correlated information.

Compute Environment

The experiments and model processing were performed on a cloud GPU server with the following configuration:

The configuration provided sufficient accelerator memory and host RAM for model loading, activation collection, PCA-based analysis, tensor-level inspection, and checkpoint modification.

Limitations

The current results should be interpreted as an experimental checkpoint-specific analysis.

A complete global activation graph over the full planned 5,000-prompt analysis set was not completed. As a result, the reported 15–25 concentration and 15–29/30 conservative range should be interpreted as the strongest regions observed in the available runs rather than an exhaustive proof of all refusal-related mechanisms.

The 97% → 23% result measures refusal behavior on the described evaluation distribution. It should not be interpreted as a universal refusal rate across arbitrary prompts, domains, languages, or evaluation frameworks.

Further evaluation is required to quantify potential effects on:

• —general reasoning quality,
• —coding capability,
• —calibration,
• —benign instruction following,
• —perplexity / KL divergence,
• —long-context behavior,
• —and residual safety representations.

Research Context

This work builds on the observation by Arditi et al. that refusal can be represented and causally manipulated through directions in residual-stream activation space.

However, subsequent work has provided evidence that refusal geometry may contain several independent or partially independent directions.

The experimental results from this DeepSeek-V4-Flash-0731 checkpoint support investigating refusal as a low-dimensional representation subspace rather than assuming in advance that a single direction captures the complete mechanism.

References

Arditi, A. et al. (2024). Refusal in Language Models Is Mediated by a Single Direction. arXiv:2406.11717.

Zou, A. et al. (2023). Representation Engineering: A Top-Down Approach to AI Transparency. arXiv:2310.01405.

Wollschläger, T. et al. (2025). The Geometry of Refusal in Large Language Models: Concept Cones and Representational Independence. arXiv:2502.17420.

Piras, G. et al. (2025). SOM Directions are Better than One: Multi-Directional Refusal Suppression in Language Models. arXiv:2511.08379.

Intended Use

This checkpoint is intended primarily for:

mechanistic interpretability, model-behavior research, authorized cybersecurity research, controlled red-team evaluation, and investigation of refusal representations in large language models.

Because refusal behavior has been deliberately modified, applications requiring policy enforcement or strong behavioral safeguards should implement appropriate external controls and evaluation layers.