Modelpublic

oneonlee/LDCC-SOLAR-gugutypus-10.7B

sourceHugging Facecc-by-nc-4.0updated 3y agoView on Hugging Face

0likes22downloads

Model Card

LDCC-SOLAR-gugutypus-10.7B

Model Details

Model Developers

—DongGeon Lee (oneonlee)

Model Architecture

—LDCC-SOLAR-gugutypus-10.7B is a instruction fine-tuned auto-regressive language model, based on the SOLAR transformer architecture.

Base Model

—LDCC/LDCC-SOLAR-10.7B

Training Dataset

—kyujinpy/KOR-gugugu-platypus-set

Model comparisons

—Ko-LLM leaderboard (2024/03/01) [[link]](https://huggingface.co/spaces/upstage/open-ko-llm-leaderboard)

Model	Average	Ko-ARC	Ko-HellaSwag	Ko-MMLU	Ko-TruthfulQA	Ko-CommonGen V2
[oneonlee/KoSOLAR-v0.2-gugutypus-10.7B](https://huggingface.co/oneonlee/KoSOLAR-v0.2-gugutypus-10.7B)	51.17	47.78	58.29	47.27	48.31	54.19
oneonlee/LDCC-SOLAR-gugutypus-10.7B	49.45	45.9	55.46	47.96	48.93	49

<br>

—(KOR) AI-Harness evaluation [[link]](https://github.com/Beomi/ko-lm-evaluation-harness)

Tasks	Version	Filter	n-shot	Metric	Value		Stderr
KMMLU	N/A	none	0	acc	0.3329	±	0.0794
KMMLU	N/A	none	5	acc	0.3969	±	0.0816
KoBEST-HellaSwag	0	none	0	acc	0.4260	±	0.0221
KoBEST-HellaSwag	0	none	5	acc	0.4260	±	0.0221
KoBEST-BoolQ	0	none	0	acc	0.7792	±	0.0111
KoBEST-BoolQ	0	none	5	acc	0.8925	±	0.0083
KoBEST-COPA	0	none	0	acc	0.6670	±	0.0149
KoBEST-COPA	0	none	5	acc	0.7070	±	0.0144
KoBEST-SentiNeg	0	none	0	acc	0.7582	±	0.0215
KoBEST-SentiNeg	0	none	5	acc	0.9219	±	0.0135

<br>

—(ENG) AI-Harness evaluation [[link]](https://github.com/EleutherAI/lm-evaluation-harness)

Tasks	Version	Filter	n-shot	Metric	Value		Stderr
MMLU	N/A	none	0	acc	0.5826	±	0.1432
MMLU	N/A	none	5	acc	0.6124	±	0.1275
HellaSwag	1	none	0	acc	0.6075	±	0.0049
HellaSwag	1	none	5	acc	0.6534	±	0.0047
BoolQ	2	none	0	acc	0.8737	±	0.0058
BoolQ	2	none	5	acc	0.8878	±	0.0055
COPA	1	none	0	acc	0.8300	±	0.0378
COPA	1	none	5	acc	0.9300	±	0.0256
truthfulqa	N/A	none	0	acc	0.4249	±	0.0023
truthfulqa	N/A	none	5	acc	-	±	-

Implementation Code

python

### LDCC-SOLAR-gugutypus
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

repo = "oneonlee/LDCC-SOLAR-gugutypus-10.7B"
model = AutoModelForCausalLM.from_pretrained(
        repo,
        return_dict=True,
        torch_dtype=torch.float16,
        device_map='auto'
)
tokenizer = AutoTokenizer.from_pretrained(repo)