PTEZ/Wukomg2.2-Lite-SINQ-Q4_K_M-GGUF

Wukomg 2.2: Efficient Language Models via Adversarial Knowledge Distillation

Abstract

The Wukomg 2.2 series is a family of fully open-source dense language models, built upon the Qwen3.5-9B base model. We construct a large-scale fine-tuning corpus containing approximately 7.5M samples, sourced from the generated data of several advanced large language models as well as other public resources.

Our core innovation is the proposal and application of the knowledge adversarial distillation paradigm: two high-performance teacher models engage in multi‑round adversarial interactions and debates. Through this “dialogic hybridization” mechanism, training samples that encapsulate deep reasoning and diverse perspectives are automatically generated. This approach significantly improves the quality of knowledge distillation while greatly reducing the cost of acquiring high-quality data.

Experiments show that after fine‑tuning with adversarial distillation, Wukomg 2.2 achieves substantial gains in reasoning, knowledge coverage, code generation, and multi‑turn dialogue consistency, validating the effectiveness of this paradigm and its practical value in the open‑source ecosystem.

1. Introduction

State‑of‑the‑art large language models today are often trained with hundreds of billions or even trillions of parameters, making their inference costly and deployment challenging. Knowledge distillation offers a viable path to compress the capabilities of large teacher models into a smaller student model. However, in standard distillation workflows, the student learns only from the teacher’s one‑way output, which tends to inherit the teacher’s cognitive biases and struggles to cover edge cases in complex reasoning.

To address this issue, we propose a knowledge distillation method based on adversarial interactions between models – two independent, comparably capable large models engage in dynamic games such as debates, mutual questioning, and critical dialogues, generating high‑information‑density training trajectories. By injecting such data into the fine‑tuning process, the base model learns not only to imitate a single model’s behavior but also to internalize the processes of disagreement, synthesis, and reasoning across multiple experts. We term this entire approach knowledge adversarial distillation, and based on it we train the Wukomg 2.2 series models.

2. Method

2.1 Base Model and Training Data Composition

We select Qwen3.5‑9B as our base model, which already exhibits strong bilingual and reasoning capabilities. For instruction fine‑tuning and capability enhancement, we construct a mixed dataset of approximately 7.5 million samples, mainly from the following sources:

• —Multi‑model distillation corpus: High‑quality responses, solution steps, and explanations generated by several top closed‑source and open‑source large models, covering knowledge QA, coding, mathematics, creative writing, and many other domains.

• —Public and synthetic data: Cleaned public dialogues, technical documentation, and additional training samples produced by automated pipelines, ensuring broad domain coverage and stylistic diversity.

2.2 Knowledge Adversarial Distillation: The “Dialogic Hybridization” Generation Mechanism

The key technical innovation of this work lies in the training data generation method. Unlike traditional one‑way “teacher → student” distillation, we design the following adversarial generation protocol:

1. 1.Dual‑teacher interaction: Select two top‑tier large language models that differ in architecture or training strategy, serving as Teacher A and Teacher B.

1. 1.Multi‑round debate and cross‑questioning: Given a topic or open‑ended question, models A and B engage in a multi‑turn dialogue with clear roles. They take turns asking questions, pointing out flaws in each other’s reasoning, requesting evidence, or deliberately providing opposing viewpoints for adversarial exploration, forming a complete interaction trajectory.

1. 1.Trajectories as training samples: These interaction records naturally contain multi‑step reasoning, belief revision, contradiction resolution, and conditional explanations. Their information richness far exceeds that of a single model’s monologue. This process is vividly termed “dialogic hybridization” – fusing the knowledge of two models through adversarial interplay to produce superior synthetic data.

1. 1.Scalability and automation: This generation pipeline requires no manual annotation. It can run autonomously given a topic list or a small set of seed prompts, efficiently accumulating massive, diverse training samples with deep logical chains.

Through this knowledge adversarial distillation, the student model (Qwen3.5‑9B) is constantly exposed, during fine‑tuning, to the complete thinking processes of the teacher models when handling conflicting information and complex problems, thereby internalizing stronger generalization and reasoning abilities.

2.3 Training Details

We mix the “dialogic hybridization” data generated by adversarial distillation with the rest of the general high‑quality corpus, and fine‑tune the base model using standard supervised fine‑tuning. The training process is fully reproduced using open‑source frameworks, with no extra parameter tuning or architectural modifications, ensuring full transparency and reproducibility. The converged weights constitute the Wukomg 2.2 series models.

3. Experimental Evaluation

We compare the base model and Wukomg 2.2 on several commonly used benchmarks. The results are summarized below (higher is better for all metrics):

As seen, after knowledge adversarial distillation, the model’s performance improves most notably on tasks that demand strong reasoning (mathematics and programming) – a direct benefit of the deep logical unfolding emphasized by the multi‑model debate data. In addition, open‑domain dialogue quality and consistency are clearly enhanced; the model exhibits stronger topic tracking and opinion‑analysis capabilities over multiple turns. Furthermore, the entire data generation process is highly automated, allowing a significant performance leap with limited computational cost, demonstrating excellent distillation efficiency.

4. Conclusion

Wukomg 2.2 successfully achieves substantial performance breakthroughs at the 9B parameter scale via knowledge adversarial distillation and a large‑scale “dialogic hybridization” dataset. Without increasing model size or relying on expensive manual annotations, and using only a clever data generation strategy, the base model obtains considerable gains across multiple dimensions.

This work validates the excellent balance between efficiency and quality offered by adversarial interactive knowledge distillation, providing a practical reference path for more efficient model training in the future.