unsloth/notebooks
36211
1{2 "cells": [3 {4 "cell_type": "markdown",5 "metadata": {6 "id": "o6RxqIirisTj"7 },8 "source": [9 "To run this, press \"Runtime\" and press \"Run all\" on a **free** Tesla T4 Google Colab instance!\n",10 "<div class=\"align-center\">\n",11 " <a href=\"https://github.com/unslothai/unsloth\"><img src=\"https://github.com/unslothai/unsloth/raw/main/images/unsloth%20new%20logo.png\" width=\"115\"></a>\n",12 " <a href=\"https://discord.gg/u54VK8m8tk\"><img src=\"https://github.com/unslothai/unsloth/raw/main/images/Discord.png\" width=\"145\"></a>\n",13 " <a href=\"https://ko-fi.com/unsloth\"><img src=\"https://github.com/unslothai/unsloth/raw/main/images/Kofi button.png\" width=\"145\"></a></a> Join our Discord if you need help!\n",14 "</div>\n",15 "\n",16 "To install Unsloth on your own computer, follow the installation instructions on our Github page [here](https://github.com/unslothai/unsloth#installation-instructions---conda).\n",17 "\n",18 "You will learn how to do [DPO data prep](#Data), and how to [train via `DPOTrainer`](#Train).\n",19 "To learn more about DPO, read TRL's [blog post](https://huggingface.co/blog/dpo-trl). We follow [Huggingface's Alignment Handbook](https://github.com/huggingface/alignment-handbook) to replicate [Zephyr](https://huggingface.co/HuggingFaceH4/zephyr-7b-beta)."20 ]21 },22 {23 "cell_type": "code",24 "execution_count": null,25 "metadata": {26 "id": "2eSvM9zX_2d3"27 },28 "outputs": [],29 "source": [30 "%%capture\n",31 "# Installs Unsloth, Xformers (Flash Attention) and all other packages!\n",32 "!pip install unsloth\n",33 "# Get latest Unsloth\n",34 "!pip uninstall unsloth -y && pip install --upgrade --no-cache-dir \"unsloth[colab-new] @ git+https://github.com/unslothai/unsloth.git\""35 ]36 },37 {38 "cell_type": "markdown",39 "metadata": {40 "id": "QYds3fcii6gC"41 },42 "source": [43 "* We support Llama, Mistral, CodeLlama, TinyLlama, Vicuna, Open Hermes etc\n",44 "* And Yi, Qwen ([llamafied](https://huggingface.co/models?sort=trending&search=qwen+llama)), Deepseek, all Llama, Mistral derived archs.\n",45 "* We support 16bit LoRA or 4bit QLoRA. Both 2x faster.\n",46 "* `max_seq_length` can be set to anything, since we do automatic RoPE Scaling via [kaiokendev's](https://kaiokendev.github.io/til) method.\n",47 "* With [PR 26037](https://github.com/huggingface/transformers/pull/26037), we support downloading 4bit models **4x faster**! [Our repo](https://huggingface.co/unsloth) has Llama, Mistral 4bit models.\n",48 "* DPO requires a model already trained by SFT on a similar dataset that is used for DPO. We use `HuggingFaceH4/mistral-7b-sft-beta` as the SFT model. Use this [notebook](https://colab.research.google.com/drive/1Dyauq4kTZoLewQ1cApceUQVNcnnNTzg_?usp=sharing) first to train a SFT model.\n",49 "* [**NEW**] We make Gemma 6 trillion tokens **2.5x faster**! See our [Gemma notebook](https://colab.research.google.com/drive/10NbwlsRChbma1v55m8LAPYG15uQv6HLo?usp=sharing)"50 ]51 },52 {53 "cell_type": "code",54 "execution_count": null,55 "metadata": {56 "colab": {57 "base_uri": "https://localhost:8080/"58 },59 "id": "E8-BWi7MzkRz",60 "outputId": "0626186f-6e70-47b7-db59-b8e7591d0de4"61 },62 "outputs": [63 {64 "output_type": "stream",65 "name": "stderr",66 "text": [67 "/usr/local/lib/python3.10/dist-packages/unsloth/__init__.py:67: UserWarning: CUDA is not linked properly.\n",68 "We shall run `ldconfig /usr/lib64-nvidia` to try to fix it.\n",69 " warnings.warn(\n"70 ]71 }72 ],73 "source": [74 "# One must patch the DPO Trainer first!\n",75 "from unsloth import PatchDPOTrainer\n",76 "PatchDPOTrainer()"77 ]78 },79 {80 "cell_type": "code",81 "execution_count": null,82 "metadata": {83 "id": "QmUBVEnvCDJv",84 "colab": {85 "base_uri": "https://localhost:8080/",86 "height": 491,87 "referenced_widgets": [88 "4c7a9e6327fc4b839f1e2eee705e1574",89 "904fef1ac7404cc8a33e65ffe8938c8b",90 "f7562580501845e9a67ab9a8b4a96ba0",91 "96f9847627504ae7abe7461e4c0b3cff",92 "5165aa69bef148af9717d90cc5a870b1",93 "e6dfcf0456824556ad09fe7945674a14",94 "80568bca0c604eaaab2933fc76da2bf0",95 "079284acd5c6454eb03d8876a62e6f12",96 "1d917d774780421fbdca680009a706ac",97 "342cf2a9d6674b4d94fdbaf3b63b9b79",98 "16ace0251b954bc08a33eaf3b2a47216",99 "2cb8afe9c6e2441fa4e232146ecd08b4",100 "bc745fa3da114007af35ea15dbd7ab7d",101 "127ed424448840cea8b14fc9997b9cde",102 "803f5904c00d4cde8729226139e3c258",103 "4a273a69979b466dbf1c987552101fbf",104 "68893d4db1f74557932d8b24e0f72820",105 "3ccdc55cf6384695967eaf79325b1aef",106 "859bc87dfda746819298e213e4e0b067",107 "32478ec41d044773b70eba24e2ac0b43",108 "5aeb33ba799149c1ad6e580a2a1c0a51",109 "a2e31ba35dcf41c79c72dfd431da781f",110 "f7c089469828416aa89ade0cfa5a04cc",111 "3a52a2ec426b4fc2aaad23c7d8ce6d2b",112 "c298081665e844539b8d366325dbdab5",113 "65b1731e23364d4ea0bb292550d4a403",114 "5f6fd4c19b34458aa8f2019a25bfd605",115 "33dda465ac2f42fb8846171a974193ac",116 "bf6bc9099d1a475884be6f15115f01ad",117 "cdc411110d1b405ea12342b57fd8de30",118 "84d445570b31490fb658b08f819dd6a7",119 "ceb3830da80146da8fbe6423d25fd8a3",120 "d58599afce36484c8508c16beba13e36",121 "d43eea508a4d496e9460f01931659b52",122 "b146c0bbd15644678198977d63edebb7",123 "aa0323eef053447b9836931535984912",124 "effec7acec3d4b5384bcf481e3263b6c",125 "03ec8d28c0f34116b42e4e0dc65e3dd3",126 "9fd5df9115cf4deb8a36daf4a654bfc7",127 "cd6e07c3984a4e0e920d87c6ebd6ad1c",128 "b0831169530b402cac31b1e60484fbb2",129 "ccabf636e15e471cb82bb431b5ab4b5c",130 "e4120afe895742848cf756fd58cb42f8",131 "fe6eabd38a614113b36bafd1f8ce6920",132 "9b12628a46c546aebe2c749085ba2e61",133 "63d48b48f4ce449e85a55477df0b6967",134 "068c416eaaed476eacffb8f2f343e08f",135 "9fd3fe23f14b4403a18ec7d73b64dc6a",136 "cf639313cba7442893fdf880d368616d",137 "4bd0cf4d31604194a7761ea0345cc0a4",138 "783ad9a7097e4fcdb768c8ecb1f21227",139 "2557c3090a8d4760ae844fb518e1b555",140 "b508872df72d45beb26a625c7499cc8b",141 "cbc6fe3cc12a4d79ba08b67f23e5ce01",142 "d0d340fa13a54bcf9f1130cc2fc09cc8",143 "5d7fce052f8e4937adaaa670be8c1ae4",144 "77939b191e82425da0a5908fdbca2a2d",145 "30dc6079591b4908a57f5e285ebeda68",146 "960bd84fc3fe4d8eb7e79ecba0f3edf6",147 "bb7c7c1866af4641bc07d1ff0a5cd069",148 "0dc6aefaa24043768d656f5a61d79545",149 "d27056ab29e04c448fc62f9f18cbb82d",150 "926319902fe84c5095e0b5bd4b3e8187",151 "9f660e1b7b9c4ae7922d51353783acf9",152 "82d00942a39e465c8a6f869ef97cc250",153 "b9e2b829291247619a7b37c9f544d395",154 "3f6ccae685e34d05be9fde5538dc48cf",155 "3df8638084104cb7a0530a96ec6201af",156 "06d1fd85a45541609d11ec7f0bd3b444",157 "a9d05a8c3aaf4fe0a2f061c3eeb48b44",158 "8095a44a62804d6d86c24d62b9e1a1c1",159 "f61b85b63d5147db9f1040f30f4a3034",160 "e152d696872242ab8e0bc00ce9b50142",161 "623e912cfe404e69a64992fbb1c1bc47",162 "69f9e4bf820047658699d55d84d682c0",163 "a8306ce933304694aba188495743c2f4",164 "1cd708f679de4d319af26797aecca94d"165 ]166 },167 "outputId": "f04a3ea4-0d0f-424d-8265-0fb99fd17ef3"168 },169 "outputs": [170 {171 "output_type": "stream",172 "name": "stderr",173 "text": [174 "/usr/local/lib/python3.10/dist-packages/huggingface_hub/utils/_token.py:72: UserWarning: \n",175 "The secret `HF_TOKEN` does not exist in your Colab secrets.\n",176 "To authenticate with the Hugging Face Hub, create a token in your settings tab (https://huggingface.co/settings/tokens), set it as secret in your Google Colab and restart your session.\n",177 "You will be able to reuse this secret in all of your notebooks.\n",178 "Please note that authentication is recommended but still optional to access public models or datasets.\n",179 " warnings.warn(\n"180 ]181 },182 {183 "output_type": "display_data",184 "data": {185 "text/plain": [186 "config.json: 0%| | 0.00/1.04k [00:00<?, ?B/s]"187 ],188 "application/vnd.jupyter.widget-view+json": {189 "version_major": 2,190 "version_minor": 0,191 "model_id": "4c7a9e6327fc4b839f1e2eee705e1574"192 }193 },194 "metadata": {}195 },196 {197 "output_type": "stream",198 "name": "stderr",199 "text": [200 "==((====))== Unsloth: Fast Mistral patching release 2024.1\n",201 " \\\\ /| GPU: Tesla T4. Max memory: 14.748 GB\n",202 "O^O/ \\_/ \\ CUDA capability = 7.5. Xformers = 0.0.22.post7. FA = False.\n",203 "\\ / Pytorch version: 2.1.0+cu121. CUDA Toolkit = 12.1\n",204 " \"-____-\" bfloat16 = FALSE. Platform = Linux\n",205 "\n",206 "You passed `quantization_config` to `from_pretrained` but the model you're loading already has a `quantization_config` attribute. The `quantization_config` attribute will be overwritten with the one you passed to `from_pretrained`.\n"207 ]208 },209 {210 "output_type": "display_data",211 "data": {212 "text/plain": [213 "model.safetensors: 0%| | 0.00/4.13G [00:00<?, ?B/s]"214 ],215 "application/vnd.jupyter.widget-view+json": {216 "version_major": 2,217 "version_minor": 0,218 "model_id": "2cb8afe9c6e2441fa4e232146ecd08b4"219 }220 },221 "metadata": {}222 },223 {224 "output_type": "display_data",225 "data": {226 "text/plain": [227 "generation_config.json: 0%| | 0.00/116 [00:00<?, ?B/s]"228 ],229 "application/vnd.jupyter.widget-view+json": {230 "version_major": 2,231 "version_minor": 0,232 "model_id": "f7c089469828416aa89ade0cfa5a04cc"233 }234 },235 "metadata": {}236 },237 {238 "output_type": "display_data",239 "data": {240 "text/plain": [241 "tokenizer_config.json: 0%| | 0.00/1.48k [00:00<?, ?B/s]"242 ],243 "application/vnd.jupyter.widget-view+json": {244 "version_major": 2,245 "version_minor": 0,246 "model_id": "d43eea508a4d496e9460f01931659b52"247 }248 },249 "metadata": {}250 },251 {252 "output_type": "display_data",253 "data": {254 "text/plain": [255 "tokenizer.model: 0%| | 0.00/493k [00:00<?, ?B/s]"256 ],257 "application/vnd.jupyter.widget-view+json": {258 "version_major": 2,259 "version_minor": 0,260 "model_id": "9b12628a46c546aebe2c749085ba2e61"261 }262 },263 "metadata": {}264 },265 {266 "output_type": "display_data",267 "data": {268 "text/plain": [269 "tokenizer.json: 0%| | 0.00/1.80M [00:00<?, ?B/s]"270 ],271 "application/vnd.jupyter.widget-view+json": {272 "version_major": 2,273 "version_minor": 0,274 "model_id": "5d7fce052f8e4937adaaa670be8c1ae4"275 }276 },277 "metadata": {}278 },279 {280 "output_type": "display_data",281 "data": {282 "text/plain": [283 "special_tokens_map.json: 0%| | 0.00/624 [00:00<?, ?B/s]"284 ],285 "application/vnd.jupyter.widget-view+json": {286 "version_major": 2,287 "version_minor": 0,288 "model_id": "3f6ccae685e34d05be9fde5538dc48cf"289 }290 },291 "metadata": {}292 }293 ],294 "source": [295 "from unsloth import FastLanguageModel\n",296 "import torch\n",297 "max_seq_length = 4096 # Choose any! We auto support RoPE Scaling internally!\n",298 "dtype = None # None for auto detection. Float16 for Tesla T4, V100, Bfloat16 for Ampere+\n",299 "load_in_4bit = True # Use 4bit quantization to reduce memory usage. Can be False.\n",300 "\n",301 "model, tokenizer = FastLanguageModel.from_pretrained(\n",302 " model_name = \"unsloth/zephyr-sft-bnb-4bit\", # Choose ANY! eg mistralai/Mistral-7B-Instruct-v0.2\n",303 " max_seq_length = max_seq_length,\n",304 " dtype = dtype,\n",305 " load_in_4bit = load_in_4bit,\n",306 " # token = \"hf_...\", # use one if using gated models like meta-llama/Llama-2-7b-hf\n",307 ")"308 ]309 },310 {311 "cell_type": "code",312 "execution_count": null,313 "metadata": {314 "cellView": "form",315 "id": "AqkY_wHdKyOl"316 },317 "outputs": [],318 "source": [319 "#@title Alignment Handbook utils\n",320 "import os\n",321 "import re\n",322 "from typing import List, Literal, Optional\n",323 "\n",324 "from datasets import DatasetDict, concatenate_datasets, load_dataset, load_from_disk\n",325 "from datasets.builder import DatasetGenerationError\n",326 "\n",327 "\n",328 "DEFAULT_CHAT_TEMPLATE = \"{% for message in messages %}\\n{% if message['role'] == 'user' %}\\n{{ '<|user|>\\n' + message['content'] + eos_token }}\\n{% elif message['role'] == 'system' %}\\n{{ '<|system|>\\n' + message['content'] + eos_token }}\\n{% elif message['role'] == 'assistant' %}\\n{{ '<|assistant|>\\n' + message['content'] + eos_token }}\\n{% endif %}\\n{% if loop.last and add_generation_prompt %}\\n{{ '<|assistant|>' }}\\n{% endif %}\\n{% endfor %}\"\n",329 "\n",330 "\n",331 "def apply_chat_template(\n",332 " example, tokenizer, task: Literal[\"sft\", \"generation\", \"rm\", \"dpo\"] = \"sft\", assistant_prefix=\"<|assistant|>\\n\"\n",333 "):\n",334 " def _strip_prefix(s, pattern):\n",335 " # Use re.escape to escape any special characters in the pattern\n",336 " return re.sub(f\"^{re.escape(pattern)}\", \"\", s)\n",337 "\n",338 " if task in [\"sft\", \"generation\"]:\n",339 " messages = example[\"messages\"]\n",340 " # We add an empty system message if there is none\n",341 " if messages[0][\"role\"] != \"system\":\n",342 " messages.insert(0, {\"role\": \"system\", \"content\": \"\"})\n",343 " example[\"text\"] = tokenizer.apply_chat_template(\n",344 " messages, tokenize=False, add_generation_prompt=True if task == \"generation\" else False\n",345 " )\n",346 " elif task == \"rm\":\n",347 " if all(k in example.keys() for k in (\"chosen\", \"rejected\")):\n",348 " chosen_messages = example[\"chosen\"]\n",349 " rejected_messages = example[\"rejected\"]\n",350 " # We add an empty system message if there is none\n",351 " if chosen_messages[0][\"role\"] != \"system\":\n",352 " chosen_messages.insert(0, {\"role\": \"system\", \"content\": \"\"})\n",353 " if rejected_messages[0][\"role\"] != \"system\":\n",354 " rejected_messages.insert(0, {\"role\": \"system\", \"content\": \"\"})\n",355 " example[\"text_chosen\"] = tokenizer.apply_chat_template(chosen_messages, tokenize=False)\n",356 " example[\"text_rejected\"] = tokenizer.apply_chat_template(rejected_messages, tokenize=False)\n",357 " else:\n",358 " raise ValueError(\n",359 " f\"Could not format example as dialogue for `rm` task! Require `[chosen, rejected]` keys but found {list(example.keys())}\"\n",360 " )\n",361 " elif task == \"dpo\":\n",362 " if all(k in example.keys() for k in (\"chosen\", \"rejected\")):\n",363 " # Compared to reward modeling, we filter out the prompt, so the text is everything after the last assistant token\n",364 " prompt_messages = [[msg for msg in example[\"chosen\"] if msg[\"role\"] == \"user\"][0]]\n",365 " # Insert system message\n",366 " if example[\"chosen\"][0][\"role\"] != \"system\":\n",367 " prompt_messages.insert(0, {\"role\": \"system\", \"content\": \"\"})\n",368 " else:\n",369 " prompt_messages.insert(0, example[\"chosen\"][0])\n",370 " # TODO: handle case where chosen/rejected also have system messages\n",371 " chosen_messages = example[\"chosen\"][1:]\n",372 " rejected_messages = example[\"rejected\"][1:]\n",373 " example[\"text_chosen\"] = tokenizer.apply_chat_template(chosen_messages, tokenize=False)\n",374 " example[\"text_rejected\"] = tokenizer.apply_chat_template(rejected_messages, tokenize=False)\n",375 " example[\"text_prompt\"] = tokenizer.apply_chat_template(\n",376 " prompt_messages, tokenize=False, add_generation_prompt=True\n",377 " )\n",378 " example[\"text_chosen\"] = _strip_prefix(example[\"text_chosen\"], assistant_prefix)\n",379 " example[\"text_rejected\"] = _strip_prefix(example[\"text_rejected\"], assistant_prefix)\n",380 " else:\n",381 " raise ValueError(\n",382 " f\"Could not format example as dialogue for `dpo` task! Require `[chosen, rejected]` keys but found {list(example.keys())}\"\n",383 " )\n",384 " else:\n",385 " raise ValueError(\n",386 " f\"Task {task} not supported, please ensure that the provided task is one of {['sft', 'generation', 'rm', 'dpo']}\"\n",387 " )\n",388 " return example\n",389 "\n",390 "\n",391 "def get_datasets(\n",392 " data_config: dict,\n",393 " splits: List[str] = [\"train\", \"test\"],\n",394 " shuffle: bool = True,\n",395 ") -> DatasetDict:\n",396 " \"\"\"\n",397 " Loads one or more datasets with varying training set proportions.\n",398 "\n",399 " Args:\n",400 " data_config (`DataArguments` or `dict`):\n",401 " Dataset configuration and split proportions.\n",402 " splits (`List[str]`, *optional*, defaults to `['train', 'test']`):\n",403 " Dataset splits to load and mix. Assumes the splits exist in all datasets and have a `train_` or `test_` prefix.\n",404 " shuffle (`bool`, *optional*, defaults to `True`):\n",405 " Whether to shuffle the training and testing/validation data.\n",406 "\n",407 " Returns\n",408 " [`DatasetDict`]: The dataset dictionary containing the loaded datasets.\n",409 " \"\"\"\n",410 "\n",411 " if type(data_config) is dict:\n",412 " # Structure of the input is:\n",413 " # dataset_mixer = {\n",414 " # \"dataset1\": 0.5,\n",415 " # \"dataset1\": 0.3,\n",416 " # \"dataset1\": 0.2,\n",417 " # }\n",418 " dataset_mixer = data_config\n",419 " else:\n",420 " raise ValueError(f\"Data config {data_config} not recognized.\")\n",421 "\n",422 " raw_datasets = mix_datasets(dataset_mixer, splits=splits, shuffle=shuffle)\n",423 " return raw_datasets\n",424 "\n",425 "\n",426 "def mix_datasets(dataset_mixer: dict, splits: Optional[List[str]] = None, shuffle=True) -> DatasetDict:\n",427 " \"\"\"\n",428 " Loads and mixes datasets according to proportions specified in `dataset_mixer`.\n",429 "\n",430 " Args:\n",431 " dataset_mixer (`dict`):\n",432 " Dictionary containing the dataset names and their training proportions. By default, all test proportions are 1.\n",433 " splits (Optional[List[str]], *optional*, defaults to `None`):\n",434 " Dataset splits to load and mix. Assumes the splits exist in all datasets and have a `train_` or `test_` prefix.\n",435 " shuffle (`bool`, *optional*, defaults to `True`):\n",436 " Whether to shuffle the training and testing/validation data.\n",437 " \"\"\"\n",438 " raw_datasets = DatasetDict()\n",439 " raw_train_datasets = []\n",440 " raw_val_datasets = []\n",441 " fracs = []\n",442 " for ds, frac in dataset_mixer.items():\n",443 " fracs.append(frac)\n",444 " for split in splits:\n",445 " try:\n",446 " # Try first if dataset on a Hub repo\n",447 " dataset = load_dataset(ds, split=split)\n",448 " except DatasetGenerationError:\n",449 " # If not, check local dataset\n",450 " dataset = load_from_disk(os.path.join(ds, split))\n",451 "\n",452 " if \"train\" in split:\n",453 " raw_train_datasets.append(dataset)\n",454 " elif \"test\" in split:\n",455 " raw_val_datasets.append(dataset)\n",456 " else:\n",457 " raise ValueError(f\"Split type {split} not recognized as one of test or train.\")\n",458 "\n",459 " if any(frac < 0 for frac in fracs):\n",460 " raise ValueError(\"Dataset fractions cannot be negative.\")\n",461 "\n",462 " if len(raw_train_datasets) > 0:\n",463 " train_subsets = []\n",464 " for dataset, frac in zip(raw_train_datasets, fracs):\n",465 " train_subset = dataset.select(range(int(frac * len(dataset))))\n",466 " train_subsets.append(train_subset)\n",467 " if shuffle:\n",468 " raw_datasets[\"train\"] = concatenate_datasets(train_subsets).shuffle(seed=42)\n",469 " else:\n",470 " raw_datasets[\"train\"] = concatenate_datasets(train_subsets)\n",471 " # No subsampling for test datasets to enable fair comparison across models\n",472 " if len(raw_val_datasets) > 0:\n",473 " if shuffle:\n",474 " raw_datasets[\"test\"] = concatenate_datasets(raw_val_datasets).shuffle(seed=42)\n",475 " else:\n",476 " raw_datasets[\"test\"] = concatenate_datasets(raw_val_datasets)\n",477 "\n",478 " if len(raw_datasets) == 0:\n",479 " raise ValueError(\n",480 " f\"Dataset {dataset_mixer} not recognized with split {split}. Check the dataset has been correctly formatted.\"\n",481 " )\n",482 "\n",483 " return raw_datasets"484 ]485 },486 {487 "cell_type": "markdown",488 "metadata": {489 "id": "EQ-Cp2V6kDcr"490 },491 "source": [492 "<a name=\"Data\"></a>\n",493 "### Data Prep\n",494 "We follow Huggingface's [Alignment Handbook](https://github.com/huggingface/alignment-handbook) for [Zephyr](https://huggingface.co/HuggingFaceH4/zephyr-7b-beta) and use the [Ultra Feedback dataset](https://huggingface.co/datasets/HuggingFaceH4/ultrafeedback_binarized), and sample 0.5% of it to speed things up. You can sample the full dataset for a full run."495 ]496 },497 {498 "cell_type": "code",499 "execution_count": null,500 "metadata": {501 "colab": {502 "base_uri": "https://localhost:8080/",503 "height": 553,504 "referenced_widgets": [505 "5e875e8a442a41e8afef4ad494ef6897",506 "7adc769f384346b3a5835c73542a2d62",507 "fccd2f68b05d40c9b359133fbf383b1c",508 "2a197f7c6c4746e1a7fded1e0539bf29",509 "37a2eb8b91ed42a1be411ecb9079a082",510 "833097685a694d1e9cd5aa016b6b7d73",511 "4e48c220077d4546a51c02df9397a084",512 "f6200f86a4084c5f978f62481100b7a3",513 "530cd47150094e7fb925eb24ec071b5e",514 "9f6197721b90446091bf8b97de656d4b",515 "35edd118e5e54d19b87aa7e966e3c9fe",516 "3b258a09c7e0462e9d9fefa6746ebb9f",517 "ade70be3dc49443581a295e89faa65c6",518 "ad03a2ff7f974fa3a0f1be2dee56c9a0",519 "727d7d710957497c8bb91d52a04a00f3",520 "347ae8c1baf54004953d0eb9325f39f8",521 "6b419485aabe49c6bed15b5bf1316e43",522 "9eabdceae6484ba7b543ed1fa4f65141",523 "6cce73a8bdda4852968c807354a6d212",524 "7953ace8fec34f358506307a3efb208d",525 "ba6f8296f866419384de64f6e2cc727e",526 "d3fcca8f0d8c41a3b78bd2470542e950",527 "243999bf48174e72ae981effb01117ac",528 "77e6abd454c94559825201f34d5de044",529 "7522d496e9dd41828a4daddb9dd37883",530 "13b208926d4a4bae854e057bc9913100",531 "78ab252184ac46eba3e82c076f6b914a",532 "80308f7c18b244a698614fa86bc4c564",533 "9473389066664ed4aa2089c78adbb20d",534 "4a6807afa14049c781c7549467b85e1c",535 "4cc5ba58138d47129b776b04bc2d6646",536 "90b49ecfaf4a44409b0b405271c9797c",537 "9032335e102f465ea2e1adcc47462a8c",538 "6730cd77f7fa416b97f20fc5f5660488",539 "5a40fa44b76f44f88a24835c1bf00e56",540 "57f9a1fd43024e8bbe7c0df0867c8e12",541 "8f6c6f07bd254516ae4af2f80d2492f2",542 "5c28f5682c37476384f6a985d42e8360",543 "900f8248c6074ea5a93ef9372da4e958",544 "39a4f06a16f149789507232e4e28acb6",545 "c558b2bb11274910888b08138d9ee195",546 "3635323c2caf4f7ab2034306cdac61cb",547 "d855ad7e63ae45089832de406a3b00e6",548 "bf25365768a0441e8e4121ea1966971d",549 "757089cbb90f4d538e4a98a9ecee7dc6",550 "d28fb551bdac4910a04bb39b500b9228",551 "82cc0b43ea6d4ff4bddce1a1536fef62",552 "379878ecc7e44b448425c4155e106477",553 "5f8ada6bd56b4d79861a8f2bc36b7785",554 "3456b1a43ca742e3bd5bcf695a7b548e",555 "56da48c3277342b3ba6841d25a387202",556 "007871aa8b7f4de7bb0f96ebe08e5283",557 "0611164c2c6f4778adcbda713521155b",558 "a4199d34e668433a98b0fed23ab84541",559 "2b850e92569c4236a683b9df2de49775",560 "97a28a558b2a4ef997fb35547ef2f51d",561 "3760dc92a7b64410865257af9f95782c",562 "2189b92750854b1f9eaa70dc23d16c55",563 "37944b0ab46c4de7acb5e06b91f5da52",564 "461aeb76622449b6a0bb7a794ea16ced",565 "e98afd2ed7b44211932dc55a2a0c3e3a",566 "dfb40aa450f844e28319893415573906",567 "841f91867ff5489d92dfb1bbbe76ca70",568 "4631d92c2f3a44589478a981a4143836",569 "e6c183829cfb45e389348c0d5a1d943d",570 "feefb2db48a84e7f9808accd7fd68df7",571 "e10b53d12a134bc6a674bc568e1ab7bf",572 "80cbef017c5041aab8fe651fe5e5f0a0",573 "53d29032d6a64f958dffaced43b46f58",574 "1093b380f208425aadf0085c0aebed80",575 "4032d6814ad44a2f86c2f3b9cbb7afad",576 "35b0f9e4188f4c22bd2ec62fb8c7e228",577 "3243cc3a35e742048fb4d62f1c448752",578 "df9f065bf72b44f289a57855e5f38620",579 "7303bc3b53094dcd8fd4966f18622f20",580 "0003c40814494b279a4f21e422723caa",581 "124aae7bb4a94223b8ae7f6b8d20961a",582 "65ea6829bd484590a9e99f07bbadf52a",583 "404791f1b4034ed9ad68752480b55731",584 "84610b3317ec4cad9bfe284ab48f3a0f",585 "2ef3a65aadc24bd085f95acf4c3de5f5",586 "a818ce650c9c43deb8f674dd9e08c1ef",587 "3d786f96440a4a76b1b09f80c3d1c0e9",588 "ee568de931f1427f9fb252f24a6bdb01",589 "a77440b9604140a78d99cca3e9845a1f",590 "56021b483de347999853740bce5b4066",591 "c1e6987ebc5f40ce90d4c94056c74333",592 "7631d285c62c4c848930eef7390ef58b",593 "03b41c21d7634543a1b29eaadbf1cd40",594 "ff122787d9174a718b5d3223f53edf22",595 "70896fbd2fab4db29322fc78e2cd84c5",596 "4673095a06fc47b581f46754117e83fe",597 "435cd70149224c18aeff12ad5d2926b8",598 "1aa72a1d1d0d4ce4b36b10c8ecf021a8",599 "a004e9208a5f441e9e4e60cc1f6d7aa9",600 "1407812065e84df28e3cbfce5bc26fc9",601 "a86aecff032345c5a3f176851052bcf0",602 "cc8d776ab3b4482ba8a2cab6bce3f310",603 "5052e40d06ed4eefa2f6df4be951aa3d",604 "0c7cd9bca9994ee4abc817035db4f115",605 "7f622c1b4fef4ebb8c290cfc164f3a95",606 "e05e528fd80548c1b7147d1756d980c9",607 "073e4cf5a7f944048d507076589bcf7e",608 "1794dcbd585747adb0530f76ec4d5be8",609 "31f8708ee9444a82923272f09da1fe91",610 "2fad7fdcdb574f75bf994a363f8c31c9",611 "59086cae36714c9bb971cf48e9d3f78b",612 "0715885acccf4440aed4955f4aaff0f2",613 "6adc4e60532f4558ade289ad526aa87d",614 "25634e71ebbf4a41baf59a8cb8540bf3",615 "1d8bd9a5ec064157a2099a23621ccdad",616 "16ca05cd0a404e36b737bae31235fac6",617 "f4065d3723d649bfa930624fca67ddd9",618 "92a0e4161e0a4d7e88aaf764f4df92b7",619 "6deaa7e61cf443e99dce5869e1281c0c",620 "05d95e4de7b84da282818e23c8c22345",621 "04ac7654040b4002a1f359c4cd000cc5",622 "4fd685420f4e41edb9d9c97e6ac89585",623 "1a319e2196e3465989cbf69e736fbd3e",624 "a70c5ea79cba4f4bac16c1bebe5aa0b5",625 "dd2a3af3a6144a9ba59fb130090c0283",626 "baa1b968a1de4d029730f6a5a2301101",627 "4d080de8a7dc4809abafd654f1f62691",628 "16708105f96343c482ea0df18d42cbf3",629 "47fb8b7a47eb47179ccb0e2e1b313fbd",630 "f9fb3e9d512944f295b81b249119d025",631 "ee9b1c6c792143da997c0b563a95ffce",632 "7bc84a52af5d4b1398b0ef212155e857",633 "4ad4cbf343684ded9a8fc5f8274a261c",634 "e5d62477070e475c93ededb4de412063",635 "9e26503e85234dda982f4a731dacabbf",636 "c0e6950f66d84e5091051fbe173b2b3f",637 "b6c12557e5814d4db2ca39372d44a167",638 "59d72ddb22ee4efeb86af48c83dd8110",639 "ae4e761cd74642788667953acf5fd0aa",640 "40904407149e4ceea2f3bdf769aba0d1",641 "da08e39e0cbd43b68dd1058954a3d93f",642 "f1b1def618354651b0c7662b055d7cb2",643 "02672649160d44c3a21f0dbb4f1142c2",644 "c19e821b4a7f458085202f1d2bd148dc",645 "86215ff3b19748859771c04a654a92a6",646 "853df198b21147f390d68af8043610b1",647 "86af0415540c41658b4f125e3f372c52",648 "e6ed1837c16b47d3913f123a7c09d833",649 "f9af1d0e95b846e7923b0fbf48df969d",650 "7c58d359b4934b8c8b99b9aa13abb661",651 "d22351f5e6844faca41027ee44ead93e",652 "162e0a2a77b743868d3e6c53f2220b14",653 "22f984300d814049bb2cfa3c68c6461a",654 "1dc30290e4c4478695246cef4b785563",655 "da199afde5ed47cdad0801ce45cf34a7",656 "76cc2729929048d0a9ea919735fea53e",657 "2ab8ed5174004d60ba0ecfc7505e3c54",658 "061f919ac1c74f078b5c909ff889e894",659 "e63c3a8c021b4d30b0c93b99a7dba155",660 "45cc7c0a9c664353ba11a5049c0353b6",661 "514e5d566f734683bec2319b987c16dd",662 "21687939a40a491395c37bec223c37fa",663 "440101772f084da5b3e5724fb28c52d1",664 "32f7c34cd24c4d83976c96ee0cdc05cf",665 "57e3acfbf42c41d2b2a3774247e046b6",666 "bd508aa803f94073af7a91332e738668",667 "ddd131d700884af596f2b77c921f48c5",668 "8c785cb20818402a829e54ff3467e03f",669 "3d17ac67bb8a4fc9b4709f89de589fcd"670 ]671 },672 "id": "r6bUnxe6N3pf",673 "outputId": "ff62990d-3e46-4a6d-826f-d4533b596a98"674 },675 "outputs": [676 {677 "output_type": "display_data",678 "data": {679 "text/plain": [680 "Downloading readme: 0%| | 0.00/5.98k [00:00<?, ?B/s]"681 ],682 "application/vnd.jupyter.widget-view+json": {683 "version_major": 2,684 "version_minor": 0,685 "model_id": "5e875e8a442a41e8afef4ad494ef6897"686 }687 },688 "metadata": {}689 },690 {691 "output_type": "display_data",692 "data": {693 "text/plain": [694 "Downloading data: 0%| | 0.00/222M [00:00<?, ?B/s]"695 ],696 "application/vnd.jupyter.widget-view+json": {697 "version_major": 2,698 "version_minor": 0,699 "model_id": "3b258a09c7e0462e9d9fefa6746ebb9f"700 }701 },702 "metadata": {}703 },704 {705 "output_type": "display_data",706 "data": {707 "text/plain": [708 "Downloading data: 0%| | 0.00/3.50M [00:00<?, ?B/s]"709 ],710 "application/vnd.jupyter.widget-view+json": {711 "version_major": 2,712 "version_minor": 0,713 "model_id": "243999bf48174e72ae981effb01117ac"714 }715 },716 "metadata": {}717 },718 {719 "output_type": "display_data",720 "data": {721 "text/plain": [722 "Downloading data: 0%| | 0.00/180M [00:00<?, ?B/s]"723 ],724 "application/vnd.jupyter.widget-view+json": {725 "version_major": 2,726 "version_minor": 0,727 "model_id": "6730cd77f7fa416b97f20fc5f5660488"728 }729 },730 "metadata": {}731 },732 {733 "output_type": "display_data",734 "data": {735 "text/plain": [736 "Downloading data: 0%| | 0.00/2.84M [00:00<?, ?B/s]"737 ],738 "application/vnd.jupyter.widget-view+json": {739 "version_major": 2,740 "version_minor": 0,741 "model_id": "757089cbb90f4d538e4a98a9ecee7dc6"742 }743 },744 "metadata": {}745 },746 {747 "output_type": "display_data",748 "data": {749 "text/plain": [750 "Downloading data: 0%| | 0.00/222M [00:00<?, ?B/s]"751 ],752 "application/vnd.jupyter.widget-view+json": {753 "version_major": 2,754 "version_minor": 0,755 "model_id": "97a28a558b2a4ef997fb35547ef2f51d"756 }757 },758 "metadata": {}759 },760 {761 "output_type": "display_data",762 "data": {763 "text/plain": [764 "Downloading data: 0%| | 0.00/7.12M [00:00<?, ?B/s]"765 ],766 "application/vnd.jupyter.widget-view+json": {767 "version_major": 2,768 "version_minor": 0,769 "model_id": "e10b53d12a134bc6a674bc568e1ab7bf"770 }771 },772 "metadata": {}773 },774 {775 "output_type": "display_data",776 "data": {777 "text/plain": [778 "Generating train_sft split: 0%| | 0/61966 [00:00<?, ? examples/s]"779 ],780 "application/vnd.jupyter.widget-view+json": {781 "version_major": 2,782 "version_minor": 0,783 "model_id": "65ea6829bd484590a9e99f07bbadf52a"784 }785 },786 "metadata": {}787 },788 {789 "output_type": "display_data",790 "data": {791 "text/plain": [792 "Generating test_sft split: 0%| | 0/1000 [00:00<?, ? examples/s]"793 ],794 "application/vnd.jupyter.widget-view+json": {795 "version_major": 2,796 "version_minor": 0,797 "model_id": "03b41c21d7634543a1b29eaadbf1cd40"798 }799 },800 "metadata": {}801 },802 {803 "output_type": "display_data",804 "data": {805 "text/plain": [806 "Generating train_gen split: 0%| | 0/61966 [00:00<?, ? examples/s]"807 ],808 "application/vnd.jupyter.widget-view+json": {809 "version_major": 2,810 "version_minor": 0,811 "model_id": "0c7cd9bca9994ee4abc817035db4f115"812 }813 },814 "metadata": {}815 },816 {817 "output_type": "display_data",818 "data": {819 "text/plain": [820 "Generating test_gen split: 0%| | 0/1000 [00:00<?, ? examples/s]"821 ],822 "application/vnd.jupyter.widget-view+json": {823 "version_major": 2,824 "version_minor": 0,825 "model_id": "1d8bd9a5ec064157a2099a23621ccdad"826 }827 },828 "metadata": {}829 },830 {831 "output_type": "display_data",832 "data": {833 "text/plain": [834 "Generating train_prefs split: 0%| | 0/61966 [00:00<?, ? examples/s]"835 ],836 "application/vnd.jupyter.widget-view+json": {837 "version_major": 2,838 "version_minor": 0,839 "model_id": "baa1b968a1de4d029730f6a5a2301101"840 }841 },842 "metadata": {}843 },844 {845 "output_type": "display_data",846 "data": {847 "text/plain": [848 "Generating test_prefs split: 0%| | 0/2000 [00:00<?, ? examples/s]"849 ],850 "application/vnd.jupyter.widget-view+json": {851 "version_major": 2,852 "version_minor": 0,853 "model_id": "b6c12557e5814d4db2ca39372d44a167"854 }855 },856 "metadata": {}857 },858 {859 "output_type": "display_data",860 "data": {861 "text/plain": [862 "Formatting comparisons with prompt template (num_proc=12): 0%| | 0/309 [00:00<?, ? examples/s]"863 ],864 "application/vnd.jupyter.widget-view+json": {865 "version_major": 2,866 "version_minor": 0,867 "model_id": "e6ed1837c16b47d3913f123a7c09d833"868 }869 },870 "metadata": {}871 },872 {873 "output_type": "display_data",874 "data": {875 "text/plain": [876 "Formatting comparisons with prompt template (num_proc=12): 0%| | 0/2000 [00:00<?, ? examples/s]"877 ],878 "application/vnd.jupyter.widget-view+json": {879 "version_major": 2,880 "version_minor": 0,881 "model_id": "e63c3a8c021b4d30b0c93b99a7dba155"882 }883 },884 "metadata": {}885 }886 ],887 "source": [888 "raw_datasets = get_datasets(\n",889 " {\"HuggingFaceH4/ultrafeedback_binarized\" : 0.005}, # 0.5% sampled\n",890 " splits = [\"train_prefs\", \"test_prefs\"],\n",891 ")\n",892 "column_names = list(raw_datasets[\"train\"].features)\n",893 "\n",894 "raw_datasets = raw_datasets.map(\n",895 " apply_chat_template,\n",896 " fn_kwargs = {\"tokenizer\": tokenizer, \"task\": \"dpo\"},\n",897 " num_proc = 12,\n",898 " remove_columns = column_names,\n",899 " desc = \"Formatting comparisons with prompt template\",\n",900 ")\n",901 "\n",902 "# Replace column names with what TRL needs, text_chosen -> chosen and text_rejected -> rejected\n",903 "for split in [\"train\", \"test\"]:\n",904 " raw_datasets[split] = raw_datasets[split].rename_columns(\n",905 " {\"text_prompt\": \"prompt\", \"text_chosen\": \"chosen\", \"text_rejected\": \"rejected\"}\n",906 " )"907 ]908 },909 {910 "cell_type": "markdown",911 "metadata": {912 "id": "7AxUmeAGkjDd"913 },914 "source": [915 "We shall print a random item from the dataset"916 ]917 },918 {919 "cell_type": "code",920 "execution_count": null,921 "metadata": {922 "colab": {923 "base_uri": "https://localhost:8080/"924 },925 "id": "oF63zQqNlNJC",926 "outputId": "5e4e2858-9ac2-4023-f554-13bfeec367a8"927 },928 "outputs": [929 {930 "output_type": "stream",931 "name": "stdout",932 "text": [933 "('<|system|>\\n'\n",934 " '</s>\\n'\n",935 " '<|user|>\\n'\n",936 " 'List two natural resources which was made in the factory.</s>\\n'\n",937 " '<|assistant|>\\n')\n",938 "('Natural resources are not made in factories. Natural resources are materials '\n",939 " 'and substances that occur naturally on Earth, such as water, minerals, '\n",940 " 'forests, and fossil fuels. Factories typically produce man-made materials or '\n",941 " 'process natural resources into finished products.</s>\\n')\n",942 "(\"I'm sorry, but it seems there might be some confusion in your question as \"\n",943 " 'natural resources are typically sourced from the earth or sea, and not made '\n",944 " 'in a factory. However, factories often use natural resources to create '\n",945 " 'various products. Two examples of natural resources that factories may use '\n",946 " 'are crude oil and iron ore. Crude oil is refined to produce various '\n",947 " 'petroleum products, such as gasoline and plastics, while iron ore is refined '\n",948 " 'to create steel, which is used in the construction industry, vehicle '\n",949 " 'manufacturing, and more. Does this help clarify things?</s>\\n')\n"950 ]951 }952 ],953 "source": [954 "import pprint\n",955 "row = raw_datasets[\"train\"][8]\n",956 "pprint.pprint(row[\"prompt\"])\n",957 "pprint.pprint(row[\"chosen\"])\n",958 "pprint.pprint(row[\"rejected\"])"959 ]960 },961 {962 "cell_type": "markdown",963 "metadata": {964 "id": "86wyNoeMj-Ph"965 },966 "source": [967 "We now add LoRA adapters so we only need to update 1 to 10% of all parameters!"968 ]969 },970 {971 "cell_type": "code",972 "execution_count": null,973 "metadata": {974 "colab": {975 "base_uri": "https://localhost:8080/"976 },977 "id": "6bZsfBuZDeCL",978 "outputId": "677aef23-d549-44d3-aace-9efda302d1e8"979 },980 "outputs": [981 {982 "output_type": "stream",983 "name": "stderr",984 "text": [985 "Unsloth 2024.1 patched 32 layers with 32 QKV layers, 32 O layers and 32 MLP layers.\n"986 ]987 }988 ],989 "source": [990 "model = FastLanguageModel.get_peft_model(\n",991 " model,\n",992 " r = 64, # Choose any number > 0 ! Suggested 8, 16, 32, 64, 128\n",993 " target_modules = [\"q_proj\", \"k_proj\", \"v_proj\", \"o_proj\",\n",994 " \"gate_proj\", \"up_proj\", \"down_proj\",],\n",995 " lora_alpha = 64,\n",996 " lora_dropout = 0, # Currently only supports dropout = 0\n",997 " bias = \"none\", # Currently only supports bias = \"none\"\n",998 " use_gradient_checkpointing = True,\n",999 " random_state = 3407,\n",1000 " use_rslora = False, # We support rank stabilized LoRA\n",1001 " loftq_config = None, # And LoftQ\n",1002 ")"1003 ]1004 },1005 {1006 "cell_type": "markdown",1007 "metadata": {1008 "id": "-kyd_iyz7DUM"1009 },1010 "source": [1011 "<a name=\"Train\"></a>\n",1012 "### Train the DPO model\n",1013 "Now let's use Huggingface TRL's `DPOTrainer`! More docs here: [TRL DPO docs](https://huggingface.co/docs/trl/dpo_trainer). We do 3 epochs on 0.5% of the dataset to speed things up."1014 ]1015 },1016 {1017 "cell_type": "code",1018 "execution_count": null,1019 "metadata": {1020 "id": "v-2BFpDWzo1K"1021 },1022 "outputs": [],1023 "source": [1024 "# One must patch the DPO Trainer first!\n",1025 "from unsloth import PatchDPOTrainer\n",1026 "PatchDPOTrainer()"1027 ]1028 },1029 {1030 "cell_type": "code",1031 "execution_count": null,1032 "metadata": {1033 "colab": {1034 "base_uri": "https://localhost:8080/",1035 "height": 104,1036 "referenced_widgets": [1037 "1410214ec6fd4e19a200b5e03464ceee",1038 "0ba04e31d310422f9efaef634afb1296",1039 "6ac7582965d341c383c806ae3a8b513a",1040 "971371cfb1bd447e94f9f2e38f9f50c8",1041 "3d4dbe5814d24b7dae1a7462a33e7f40",1042 "6b4743fee43b40638d8f2df36fa9fed9",1043 "77f706dc09eb43d7aa4522f8a0b36142",1044 "175c7bb9e8c94b8daa3287dd773b6837",1045 "a10418a202e14a96893e24267bc194c9",1046 "8138d9f175df4a28b2e58e9c852cabb8",1047 "43b39a8abcb94847bc21ec16fa373ed9"1048 ]1049 },1050 "id": "QtoqUw80QDV0",1051 "outputId": "40764ccc-6502-4be1-fb4f-e386f5460147"1052 },1053 "outputs": [1054 {1055 "output_type": "stream",1056 "name": "stderr",1057 "text": [1058 "/usr/local/lib/python3.10/dist-packages/trl/trainer/dpo_trainer.py:294: UserWarning: When using DPODataCollatorWithPadding, you should set `remove_unused_columns=False` in your TrainingArguments we have set it for you, but you should do it yourself in the future.\n",1059 " warnings.warn(\n"1060 ]1061 },1062 {1063 "output_type": "display_data",1064 "data": {1065 "text/plain": [1066 "Map: 0%| | 0/309 [00:00<?, ? examples/s]"1067 ],1068 "application/vnd.jupyter.widget-view+json": {1069 "version_major": 2,1070 "version_minor": 0,1071 "model_id": "1410214ec6fd4e19a200b5e03464ceee"1072 }1073 },1074 "metadata": {}1075 }1076 ],1077 "source": [1078 "from transformers import TrainingArguments\n",1079 "from trl import DPOTrainer\n",1080 "\n",1081 "dpo_trainer = DPOTrainer(\n",1082 " model = model,\n",1083 " ref_model = None,\n",1084 " args = TrainingArguments(\n",1085 " per_device_train_batch_size = 2,\n",1086 " gradient_accumulation_steps = 4,\n",1087 " warmup_ratio = 0.1,\n",1088 " num_train_epochs = 3,\n",1089 " learning_rate = 5e-6,\n",1090 " fp16 = not torch.cuda.is_bf16_supported(),\n",1091 " bf16 = torch.cuda.is_bf16_supported(),\n",1092 " logging_steps = 1,\n",1093 " optim = \"adamw_8bit\",\n",1094 " weight_decay = 0.0,\n",1095 " lr_scheduler_type = \"linear\",\n",1096 " seed = 42,\n",1097 " output_dir = \"outputs\",\n",1098 " ),\n",1099 " beta = 0.1,\n",1100 " train_dataset = raw_datasets[\"train\"],\n",1101 " # eval_dataset = raw_datasets[\"test\"],\n",1102 " tokenizer = tokenizer,\n",1103 " max_length = 1024,\n",1104 " max_prompt_length = 512,\n",1105 ")"1106 ]1107 },1108 {1109 "cell_type": "code",1110 "execution_count": null,1111 "metadata": {1112 "colab": {1113 "base_uri": "https://localhost:8080/",1114 "height": 10001115 },1116 "id": "EWGFqAo5Q2me",1117 "outputId": "1bcba65a-8253-4a49-a180-fde8e26ad4b5"1118 },1119 "outputs": [1120 {1121 "output_type": "stream",1122 "name": "stderr",1123 "text": [1124 "Unsloth: `use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`\n",1125 "Could not estimate the number of tokens of the input, floating-point operations will not be computed\n"1126 ]1127 },1128 {1129 "output_type": "display_data",1130 "data": {1131 "text/plain": [1132 "<IPython.core.display.HTML object>"1133 ],1134 "text/html": [1135 "\n",1136 " <div>\n",1137 " \n",1138 " <progress value='114' max='114' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",1139 " [114/114 1:06:24, Epoch 2/3]\n",1140 " </div>\n",1141 " <table border=\"1\" class=\"dataframe\">\n",1142 " <thead>\n",1143 " <tr style=\"text-align: left;\">\n",1144 " <th>Step</th>\n",1145 " <th>Training Loss</th>\n",1146 " <th>rewards / chosen</th>\n",1147 " <th>rewards / rejected</th>\n",1148 " <th>rewards / accuracies</th>\n",1149 " <th>rewards / margins</th>\n",1150 " <th>logps / rejected</th>\n",1151 " <th>logps / chosen</th>\n",1152 " <th>logits / rejected</th>\n",1153 " <th>logits / chosen</th>\n",1154 " </tr>\n",1155 " </thead>\n",1156 " <tbody>\n",1157 " <tr>\n",1158 " <td>1</td>\n",1159 " <td>0.693100</td>\n",1160 " <td>0.000000</td>\n",1161 " <td>0.000000</td>\n",1162 " <td>0.000000</td>\n",1163 " <td>0.000000</td>\n",1164 " <td>-297.338806</td>\n",1165 " <td>-218.968842</td>\n",1166 " <td>-2.758142</td>\n",1167 " <td>-2.924523</td>\n",1168 " </tr>\n",1169 " <tr>\n",1170 " <td>2</td>\n",1171 " <td>0.693100</td>\n",1172 " <td>0.000000</td>\n",1173 " <td>0.000000</td>\n",1174 " <td>0.000000</td>\n",1175 " <td>0.000000</td>\n",1176 " <td>-237.602417</td>\n",1177 " <td>-217.613892</td>\n",1178 " <td>-2.731790</td>\n",1179 " <td>-2.913610</td>\n",1180 " </tr>\n",1181 " <tr>\n",1182 " <td>3</td>\n",1183 " <td>0.692200</td>\n",1184 " <td>0.001937</td>\n",1185 " <td>-0.000008</td>\n",1186 " <td>0.625000</td>\n",1187 " <td>0.001945</td>\n",1188 " <td>-172.792877</td>\n",1189 " <td>-202.709259</td>\n",1190 " <td>-2.464616</td>\n",1191 " <td>-2.728198</td>\n",1192 " </tr>\n",1193 " <tr>\n",1194 " <td>4</td>\n",1195 " <td>0.692700</td>\n",1196 " <td>0.000855</td>\n",1197 " <td>-0.000130</td>\n",1198 " <td>0.750000</td>\n",1199 " <td>0.000985</td>\n",1200 " <td>-117.745728</td>\n",