Kodjaoglanian/Athenas-Symbiote-9B-GGUF

Athenas-Symbiote-9B

<div align="center"> <img src="image.png" alt="Athenas-Symbiote Logo" width="300"/> </div>

Model Details

Athenas-Symbiote-9B is a large language model (LLM) fine-tuned for advanced reasoning, complex text interpretation, and legal/academic domain expertise, primarily optimized for Brazilian Portuguese (PT-BR).

Built upon the Qwen3.5-9B architecture, this model leverages Parameter-Efficient Fine-Tuning (PEFT) to adapt its robust multilingual baseline specifically to the nuances, terminology, and logical structures required by the Brazilian legal and educational systems.

📄 Technical Report: Available on Zenodo — full methodology, evaluation protocol, and per-edition analysis.

Model Description

• —Developed by: Bruno Kodjaoglanian Cardoso Tulux (Independent AI Researcher)
• —Model Type: Causal Language Model (Transformer)
• —Base Model: unsloth/Qwen3.5-9B-Base
• —Parameter Count: 9 Billion
• —Context Length: 32,768 tokens
• —Language(s) (NLP): Portuguese (Primary), English
• —License: Apache 2.0
• —Finetuning Approach: LoRA (Low-Rank Adaptation)

Evaluation & Benchmarks

The model was comprehensively evaluated using the lm-evaluation-harness framework. Inference was conducted on high-performance infrastructure (1x NVIDIA H100 80GB) utilizing native bfloat16 precision to ensure metric fidelity and prevent quantization degradation.

[Insert your consolidated performance bar/radar chart here]

National Benchmarks (Brazil)

Evaluations targeting the Brazilian cultural, legal, and educational framework.

Benchmark Datasets: - OAB Exams: `maritaca-ai/oab-bench` - ENEM Challenge: `maritaca-ai/enem`

ENEM Challenge — Detailed Breakdown by Exam

OAB Exams — Detailed Breakdown by Exam

Global Benchmarks (Translated / Multilingual)

Frontier evaluations assessing general intelligence and mathematical reasoning.

Note: Global benchmark evaluations are pending. Scores will be updated upon completion.

How to Get Started with the Model

Use the code below to get started with the model. As this is a base model fine-tuned with LoRA, there is no built-in chat template. The ChatML format is recommended for optimal inference with instruction-style prompts.

Requirements

pip install transformers accelerate bitsandbytes

Python Inference

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "Kodjaoglanian/Athenas-Symbiote-9B"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    device_map="auto",
    torch_dtype=torch.bfloat16
)

messages = [
    {"role": "system", "content": "Você é a Athenas, uma assistente virtual altamente inteligente e precisa."},
    {"role": "user", "content": "Explique a diferença entre dolo e culpa no código penal brasileiro."}
]

text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer([text], return_tensors="pt").to(model.device)

outputs = model.generate(
    **inputs, 
    max_new_tokens=512, 
    temperature=0.3,
    do_sample=True
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Training Details

Training Data

The model was fine-tuned on the `ClassiCC-Corpus/ClassiCC-PT` dataset, consumed in streaming mode with a strict quality filter (edu_score > 0.8). This heuristic ensured the model ingested only high-rigor educational, literary, and academic content. The training corpus includes:

• —Brazilian jurisprudence, legislation, and legal proceedings.
• —Academic literature and technical documentation in Portuguese.
• —Chain-of-Thought (CoT) reasoning structures.

Data Pipeline: Streaming → edu_score > 0.8 filter → EOS concatenation → Sequence packing (packing=True)

Training Pipeline

graph TD
    A["Load Qwen3.5-9B-Base<br/>(4-bit, Unsloth)"] --> B["Configure PEFT<br/>LoRA r=32, α=16, rsLoRA"]
    B --> C["Dataset: ClassiCC-PT<br/>(streaming, edu_score > 0.8)"]
    C --> D["SFTTrainer<br/>1,200 steps, LR 5e-5<br/>adamw_8bit, packing=True"]
    D --> E{"Training OK?"}
    E -- No --> F["Auto-retry<br/>(5x, resume checkpoint)"]
    F --> D
    E -- Yes --> G["Push to HF Hub<br/>(Safetensors + Tokenizer)"]
    G --> H["Evaluation<br/>lm-eval-harness<br/>H100, BF16, 3-shot"]

    style A fill:#1B3A5C,color:#fff
    style B fill:#1B3A5C,color:#fff
    style C fill:#1B3A5C,color:#fff
    style D fill:#1B3A5C,color:#fff
    style F fill:#C5A059,color:#fff
    style G fill:#C5A059,color:#fff
    style H fill:#C5A059,color:#fff

Training: Kaggle T4 (16GB) | Evaluation: Lightning AI H100 (80GB)

Training Procedure

The fine-tuning process prioritized structural knowledge retention and computational efficiency.

• —Hardware: 1x NVIDIA T4 (16GB VRAM) via Kaggle
• —Quantization: 4-bit (NF4 via bitsandbytes)
• —Precision: FP16 (T4 does not support BF16)
• —Framework: Unsloth + TRL + PEFT + bitsandbytes
• —Optimization: LoRA adapters applied to all linear projections
• —LoRA Rank: 32
• —LoRA Alpha: 16
• —Scaling Strategy: rsLoRA (use_rslora=True)
• —LoRA Dropout: 0.0
• —Target Modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
• —Learning Rate: 5e-5
• —LR Scheduler: Linear
• —Warmup Steps: 50
• —Training Steps: 1,200 (Dataset via Streaming)
• —Batch Size: 1 (Effective Batch Size: 8 via Gradient Accumulation)
• —Optimizer: adamw_8bit
• —Weight Decay: 0.01
• —Packing: True
• —Gradient Checkpointing: Unsloth
• —Seed: 3407
• —Max Sequence Length (Training): 512 tokens

Evaluation Hardware

• —Infrastructure: Lightning AI platform
• —GPU: 1x NVIDIA H100 (80GB VRAM)
• —Precision: bfloat16 (native, no quantization)
• —Framework: lm-evaluation-harness (EleutherAI)
• —Few-Shot: 3-shot
• —Bootstrap Iterations: 100,000
• —Execution: Sequential evaluation to isolate memory overhead.

Intended Use & Limitations

Intended Use

• —Primary Use Case: Research and application in Natural Language Processing (NLP) specifically for the Portuguese language.
• —Secondary Use Case: Automation of complex text analysis, summarization, and logical reasoning within legal and academic contexts (assisted by human review).

Out-of-Scope Use

The model is not designed to operate autonomously in high-stakes environments. It should not be used for:

• —Automated legal counsel or medical diagnosis without human-in-the-loop validation.
• —Generation of deceptive, malicious, or highly biased content.

Bias, Risks, and Limitations

Like all statistical language models, Athenas-Symbiote-9B is susceptible to generating hallucinations—plausible-sounding but factually incorrect information. Furthermore, the model may inherit and amplify biases present in its pre-training data and fine-tuning corpus. Users must exercise critical judgment and implement rigorous verification protocols before utilizing the model's outputs in production environments.

Citation

If you use this model in your research, please cite it as follows:

@misc{athenas_symbiote_9b,
  author       = {Bruno Kodjaoglanian Cardoso Tulux},
  title        = {Athenas-Symbiote-9B: A Parameter-Efficient Fine-Tuned LLM for Brazilian Portuguese Legal and Academic Domains},
  year         = {2026},
  publisher    = {Zenodo},
  doi          = {10.5281/zenodo.21418168},
  url          = {https://zenodo.org/records/21418168},
  note         = {Model weights: https://huggingface.co/Kodjaoglanian/Athenas-Symbiote-9B}
}