ilovesushiandkimchiandmalaxiangguo/shuPT-xiaohongshu-Llama3-8B-Instruct

[image]

Model Card for Model ID

Introducing 🍠(shǔ)PT-Llama3-8B-Instruct ! A SFT (Supervised Fine-Tuning) model trained on Xiaohongshu data.

Model Details

• —Base Model: Meta-Llama-3-8B-Instruct
• —Architecture: LLaMA 3 with LoRA fine-tuning
• —Language: Chinese
• —Task: Instruction following and conversational responses
• —License: Same as base LLaMA 3 model.

Training Details

• —Training Data: 30k crawled from Xiaohongshu (小红书)
• —Framework: Hugging Face Transformers, PEFT, TRL

Training Data Details

• —Data source: 30k entries crawled from Xiaohongshu (小红书), specifically targeting 'city walk' related posts
• —Copyright: All content rights belong to Xiaohongshu
• —Usage restrictions: Academic/research purposes only
• —Format: Alpaca-style instruction tuning format
• —Known issues:
• —Incomplete conversation formatting in some samples
• —EOS token placement inconsistencies
• —Response length variations

Training Parameters

• —Learning rate: 2e-4
• —Weight decay: 0.001
• —Max gradient norm: 0.3
• —Warmup ratio: 0.03
• —LR scheduler: Cosine
• —Training precision: 4-bit quantization (QLoRA)
• —LoRA rank (r): 64
• —LoRA alpha: 16
• —LoRA dropout: 0.1

Intended Use

• —Chinese language instruction following
• —Conversational responses
• —Location-based recommendations
• —City navigation assistance
• —Cultural and historical information sharing

Limitations

• —Limited to Chinese language understanding and generation
• —Domain-specific knowledge biased towards Xiaohongshu content
• —Inherits base model limitations
• —May generate inconsistent responses due to temperature-based sampling

Performance

• —Shows improved performance on Chinese instruction following
• —Demonstrates strong capabilities in location-based recommendations

Ethical Considerations

• —Model inherits potential biases from Xiaohongshu data
• —Should be used in compliance with base model's usage policies
• —Content generation should be monitored for accuracy and appropriateness

Optimization Opportunities

• —Training parameters could be optimized for better performance
• —Data cleaning and formatting could be improved

Result

• —Question: 请推荐一下北京的city walk路线 (Please recommend some city walk routes in Beijing) [image]