Modelpublic

jungter/potato-gang-gc

sourceHugging Faceupdated 7mo agoView on Hugging Face

0likes8downloads

Model Card

LFM2.5-1.2B-Base LoRA Fine-Tune (Checkpoint 19000)

LoRA adapter fine-tuned on curated Discord group chat data for conversational style transfer. Just a for fun project.

Model Details

Model Description

This is a LoRA adapter trained on top of LiquidAI/LFM2.5-1.2B-Base using the Unsloth + SFTTrainer pipeline. The training data was sourced from Discord message exports and processed through a multi-stage cleaning pipeline: heuristic-based filtering (bot removal, URL stripping, duplicate/echo detection, consecutive message merging, low-quality filtering), followed by LLM-based pair classification (Qwen3.5-35B-A3B via vLLM) to validate prompt-response relevance and conversation coherence.

—Developed by: Ted S
—Model type: Causal Language Model (LoRA adapter)
—Language(s): English
—License: Same as base model (LiquidAI/LFM2.5-1.2B-Base)
—Finetuned from model: LiquidAI/LFM2.5-1.2B-Base

Uses

Direct Use

Conversational text generation mimicking the style and tone of the Discord group used for training. Intended for personal/hobby use as a chatbot.

Out-of-Scope Use

Not intended for production applications, factual Q&A, or any use requiring reliable or safe outputs. This model adopts the casual, unfiltered style of the training data.

Training Details

Training Data

—Source: Discord group chat message export
—Preprocessing pipeline:
—Message normalization (bot filtering, content cleaning, timestamp parsing)
—Consecutive message merging (120s window)
—Echo/duplicate removal
—Low-quality message filtering (min 5 chars, min 2 words)
—Reply candidate collection (explicit Discord replies + adjacent message pairing)
—LLM-based classification using Qwen3.5-35B-A3B with structured JSON output to validate prompt-response pairs (confidence threshold: 0.65)
—Multi-turn conversation building + standalone pair extraction
—Format: Chat-template formatted text (user/assistant roles) with Unsloth packing

Training Procedure

Training Hyperparameters

Parameter	Value
Training regime	bf16 mixed precision
Base model	LiquidAI/LFM2.5-1.2B-Base
Method	LoRA (PEFT)
LoRA rank (r)	64
LoRA alpha	64
LoRA dropout	0
Target modules	qproj, kproj, vproj, oproj, gateproj, upproj, down_proj
Max sequence length	2048
Batch size	32–64 (adjusted mid-training)
Gradient accumulation	1–2 (effective batch: 64)
Learning rate	2e-5
LR scheduler	Cosine
Optimizer	AdamW (full precision)
Weight decay	0.01
Warmup steps	100
Packing	Enabled
Gradient checkpointing	Unsloth optimized
Eval split	5%
Checkpoint	Step 19000

Technical Specifications

Compute Infrastructure

Hardware

—Training GPU: NVIDIA B200, H100 (overkill but i didn't want to wait)
—Data cleaning GPU: NVIDIA H100 (vLLM inference for LLM-based pair classification)

Software

—Unsloth
—Transformers
—TRL (SFTTrainer)
—PEFT
—vLLM (data cleaning pipeline)

Framework Versions

—PEFT 0.18.1