Mercity/Qwen3-8B-LaCo-30L

Qwen3-8B-LaCo-Pruned

This model is a layer-pruned version of Qwen3-8B-Base using the LaCo (Layer Collapse) structured pruning method.

Model Summary

Key Results

This model achieves 16.7% compression while retaining:

• —~90% of physical reasoning (PIQA)
• —~94% of commonsense reasoning (WinoGrande)
• —~79% of common sense completion (HellaSwag)
• —~41% of factual knowledge (MMLU)

This is a raw pruned model without post-training. Fine-tuning can further recover lost capabilities.

Benchmark Results (Pre-Training)

Note: All benchmarks below are evaluated on the pruned model without any post-training or fine-tuning. These results represent the raw performance after pruning only. Post-training is expected to improve these scores, particularly on knowledge-intensive tasks like MMLU.

Comparison with Original Qwen3-8B-Base

Original scores from [Qwen3 Technical Report](https://arxiv.org/abs/2505.09388)

Benchmark Interpretation

The "Knowledge Cliff"

Our experiments reveal a critical finding: factual knowledge collapses catastrophically between 16-22% compression.

While reasoning capabilities degrade gradually with compression, factual knowledge encoded in specific layers is lost abruptly when those layers are removed.

Intended Use

This model is suitable for:

• —Research on model compression and efficiency
• —Fine-tuning base for domain-specific applications
• —Inference optimization where speed/memory matters
• —Applications prioritizing reasoning over factual recall

Limitations

Important: This is a raw pruned model without post-training.

Pruning Details

LaCo Hyperparameters

Pruning Statistics

Usage

Basic Inference

from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "Mercity/Qwen3-8B-LaCo-Pruned"

tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype="auto",
    device_map="auto",
    trust_remote_code=True
)

# Text generation
prompt = "The process of photosynthesis"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=100, do_sample=True, temperature=0.7)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

With 4-bit Quantization (Further Compression)

from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig

quantization_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_compute_dtype="float16",
    bnb_4bit_quant_type="nf4",
)

model = AutoModelForCausalLM.from_pretrained(
    "Mercity/Qwen3-8B-LaCo-Pruned",
    quantization_config=quantization_config,
    device_map="auto",
    trust_remote_code=True
)

Recovery Recommendations

To improve factual knowledge after pruning:

LoRA Fine-tuning (Recommended)

from peft import LoraConfig, get_peft_model

lora_config = LoraConfig(
    r=32,
    lora_alpha=64,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj", 
                    "gate_proj", "up_proj", "down_proj"],
    lora_dropout=0.05,
)
model = get_peft_model(model, lora_config)
# Fine-tune on OpenOrca, Alpaca, or domain-specific data

Expected recovery: MMLU could reach 45-55% with fine-tuning.

Technical Specifications

Citation

If you use this model, please cite the original LaCo paper and Qwen3:

@article{yang2024laco,
  title={LaCo: Large Language Model Pruning via Layer Collapse},
  author={Yang, Yifei and Cao, Zouying and Zhao, Hai},
  journal={arXiv preprint arXiv:2402.11187},
  year={2024}
}

@misc{qwen3technicalreport,
  title={Qwen3 Technical Report},
  author={Qwen Team},
  year={2025},
  eprint={2505.09388},
  archivePrefix={arXiv},
  primaryClass={cs.CL},
  url={https://arxiv.org/abs/2505.09388}
}

References

• —LaCo Paper
• —LaCo Official Implementation
• —Qwen3 Technical Report
• —Qwen3-8B-Base

License

Apache 2.0 (same as base Qwen3 model)

Acknowledgments

• —Qwen Team for the excellent Qwen3-8B-Base model
• —LaCo authors for the pruning methodology
• —Hugging Face for model hosting