Modelpublic

trillionlabs/Tri-7B-Base

sourceHugging Faceapache-2.0updated 1y agoView on Hugging Face

4likes125downloads

Model Card

Tri-7B-Base

Introduction

We present Tri-7B-Base, a foundation language model that serves as the pre-trained base for our Tri-7B model family. This model represents our commitment to efficient training while establishing a strong foundation for downstream fine-tuning and adaptation.

Key Features

—Foundation Architecture: State-of-the-art transformer architecture optimized for efficiency
—Multi-lingual Foundation: Pre-trained on diverse data in Korean, English, and Japanese
—Efficient Training: Optimized training methodology for computational efficiency

Model Specifications

Tri-7B-Base

—Type: Causal Language Model
—Training Stage: Pre-training
—Architecture: Transformer Decoder with RoPE, SwiGLU, RMSNorm
—Number of Parameters: 7.76B
—Number of Layers: 32
—Number of Attention Heads: 32
—Context Length: 4,096
—Vocab Size: 128,128

Use Cases

As a base model, Tri-7B-Base is designed to serve as a foundation for various downstream applications:

—Fine-tuning: Adapt to specific domains or tasks
—Instruction Tuning: Create chat or assistant models
—Domain Specialization: Customize for specific industries or use cases
—Research: Explore model behaviors and capabilities
—Language Generation: General text completion and generation tasks

Limitations

—Base Model Nature: This is a pre-trained base model without instruction tuning or alignment. For chat or assistant capabilities, consider fine-tuned variants.
—Language Support: The model is optimized for English, Korean, and Japanese. Usage with other languages may result in degraded performance.
—Knowledge Cutoff: The model's information is limited to data available up to February, 2025.
—Generation Quality: As a base model, outputs may require post-processing or filtering for production use cases.

License

This model is licensed under the Apache License 2.0.

Contact

For inquiries, please contact: info@trillionlabs.co