khawarizmiai/Khawarizmi-SPI-MLP-8B

Khawarizmi-SPI-MLP-8B

Model Overview

Khawarizmi-SPI-MLP-8B is a hybrid language model developed by Khawarizmi AI, leveraging the innovative Selective Parameter Interpolation on MLP Layers (SPI-MLP) algorithm. This model ingeniously combines the robust linguistic capabilities of Qwen3-8B with the advanced reasoning patterns of DeepSeek-R1. The fusion is specifically applied to the Multi-Layer Perceptron (MLP) layers, with a composition of 60% DeepSeek and 40% Qwen, while critically preserving Qwen's original attention and normalization layers. This unique architectural approach aims to deliver a model highly proficient in complex reasoning, code generation, and multilingual tasks, with a particular emphasis on Arabic-English understanding.

Technical Specifications

Based on the config.json and generation_config.json files, the Khawarizmi-SPI-MLP-8B model exhibits the following technical characteristics:

Architecture and Configuration

Generation Configuration

Merge Strategy

The Khawarizmi-SPI-MLP-8B model employs a sophisticated Selective Parameter Interpolation (SPI) strategy specifically targeting the MLP layers. This method allows for a nuanced integration of two distinct models: Qwen3-8B and DeepSeek-R1. The core idea is to selectively interpolate parameters within the MLP layers, achieving a blend that harnesses the strengths of both base models while maintaining the structural integrity of Qwen's attention and normalization layers. This approach ensures that the model benefits from DeepSeek-R1's reasoning capabilities without compromising Qwen3-8B's established linguistic prowess.

For each weight tensor $Wk$: $$ Wk^{\text{merged}} = \begin{cases} 0.6 \cdot Wk^{\text{(DeepSeek)}} + 0.4 \cdot Wk^{\text{(Qwen)}} & \text{if "mlp" in } k \\ W_k^{\text{(Qwen)}} & \text{otherwise} \end{cases} $$

How to Use

To utilize the Khawarizmi-SPI-MLP-8B model, follow the instructions below. Ensure you have the necessary dependencies installed.

Installation

First, install the required Python packages using pip:

pip install -q transformers accelerate safetensors sentencepiece torch

Model Loading and Inference

Once the dependencies are installed, you can load the model and tokenizer using the Hugging Face transformers library and perform text generation:

from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "khawarizmiai/Khawarizmi-SPI-MLP-8B"

tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype="auto",
    device_map="auto",
    trust_remote_code=True
)

# Example usage (you can expand on this with more detailed examples)
# prompt = "Write a short story about a robot learning to feel."
# input_ids = tokenizer(prompt, return_tensors="pt").to(model.device)
# generated_ids = model.generate(**input_ids, max_new_tokens=100)
# print(tokenizer.decode(generated_ids[0], skip_special_tokens=True))

Evaluation

While specific benchmark results for Khawarizmi-SPI-MLP-8B are not detailed in the provided files, the model's design, which integrates DeepSeek-R1's reasoning patterns, suggests a focus on improving performance in areas such as:

• —Reasoning: Enhanced logical reasoning and problem-solving capabilities.
• —Code Generation: Improved ability to generate accurate and efficient code.
• —Multilingual Tasks: Stronger performance in understanding and generating text in multiple languages, particularly Arabic and English.

Further evaluations would be necessary to quantify the model's performance across standard benchmarks (e.g., MMLU, GSM8K, HumanEval) to provide a comprehensive understanding of its capabilities.

Limitations

As with all large language models, Khawarizmi-SPI-MLP-8B may exhibit certain limitations inherent to current AI technology:

• —Hallucination: The model might generate factually incorrect or nonsensical information.
• —Bias: Potential biases present in the training data could be reflected in the model's outputs.
• —Lack of Common Sense: The model may occasionally lack human-like common sense reasoning, leading to unexpected or illogical responses.

Users are advised to exercise caution and verify critical information generated by the model.

License

The licensing information for Khawarizmi-SPI-MLP-8B is available in the LICENSE file within the repository. Users should refer to this file for detailed terms and conditions regarding the use and distribution of the model.

Citation

If you find Khawarizmi-SPI-MLP-8B useful in your research or applications, please consider citing it. A formal citation will be provided upon publication of the research paper detailing the SPI-MLP algorithm and the model's development.

Acknowledgements

We extend our gratitude to the open-source community and the developers of Qwen3-8B and DeepSeek-R1, whose foundational work has been instrumental in the creation of Khawarizmi-SPI-MLP-8B. Their contributions continue to drive innovation in the field of artificial intelligence.

Contact

For inquiries, collaborations, or feedback regarding Khawarizmi-SPI-MLP-8B, please reach out to the Khawarizmi AI team through the Hugging Face platform or official channels as they become available.

Disclaimer

Khawarizmi-SPI-MLP-8B is provided for research and experimental purposes. While efforts have been made to ensure its quality and performance, Khawarizmi AI does not guarantee its suitability for any specific application. Users are responsible for assessing the model's outputs and ensuring compliance with all applicable laws and regulations.

live