jjae/Midm-KCulture-2.0-Base-Instruct

1---2license: mit3language:4- ko5base_model:6- K-intelligence/Midm-2.0-Base-Instruct7tags:8- Korean9- Culture10---11 12 13# Midm-KCulture-2.0-Base-Instruct14- This model is fine-tuned from KT/Midm-2.0-Base-Instruct on the 'Korean Culture Q&A Corpus' using the LoRA (Low-Rank Adaptation) methodology.15 16## GitHub17Check out the full training code [here](https://github.com/dahlia52/KR-Culture-QA/tree/main).18 19## Training Hyperparameters20 21| Hyperparameter                | Value                         |22| :---------------------------- | :---------------------------- |23| **SFTConfig** |                               |24| `torch_dtype`                 | `bfloat16`                    |25| `seed`                        | `42`                          |26| `epoch`                       | `3`                           |27| `per_device_train_batch_size` | `2`                           |28| `per_device_eval_batch_size`  | `2`                           |29| `learning_rate`               | `0.0002`                      |30| `lr_scheduler_type`           | `"linear"`                    |31| `max_grad_norm`               | `1.0`                         |32| `neftune_noise_alpha`         | `None`                        |33| `gradient_accumulation_steps` | `1`                           |34| `gradient_checkpointing`      | `False`                       |35| `max_seq_length`              | `1024`                        |36| **LoraConfig** |                               |37| `r`                           | `16`                          |38| `lora_alpha`                  | `16`                          |39| `lora_dropout`                | `0.1`                         |40| `target_modules`              | `["q_proj", "v_proj"]`        |41 42## Usage43```python44from transformers import AutoModelForCausalLM, AutoTokenizer45model_name = "jjae/Midm-KCulture-2.0-Base-Instruct"46model = AutoModelForCausalLM.from_pretrained(47      model_name,48      torch_dtype=torch.bfloat16,49      trust_remote_code=True,50      device_map="auto")51tokenizer = AutoTokenizer.from_pretrained(model_name)52```53 54