CoolFace
Modelpublic

hlee131/Llama-3-OOD-Reflection-DPO

sourceHugging Facellama3.1updated 1y agoView on Hugging Face
0likes4downloads
Model Card

<!-- This model card has been generated automatically according to the information the Trainer had access to. You should probably proofread and complete it, then remove this comment. -->

reflection_dpo.pt

This model is a fine-tuned version of meta-llama/Llama-3.1-8B-Instruct on an unknown dataset. It achieves the following results on the evaluation set:

  • —Logits/chosen: -0.9495
  • —Logits/rejected: -0.9444
  • —Logps/chosen: -75.1085
  • —Logps/rejected: -153.4672
  • —Loss: 0.2019
  • —Rewards/accuracies: 0.9303
  • —Rewards/chosen: -3.3185
  • —Rewards/margins: 8.4758
  • —Rewards/rejected: -11.7943

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • —learning_rate: 0.0001
  • —trainbatchsize: 8
  • —evalbatchsize: 8
  • —seed: 42
  • —optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • —lrschedulertype: linear
  • —training_steps: 500

Training results

Training LossEpochStepLogits/chosenLogits/rejectedLogps/chosenLogps/rejectedValidation LossRewards/accuraciesRewards/chosenRewards/marginsRewards/rejected
0.44960.003210-0.7050-0.6918-44.4013-46.50390.43520.8096-0.24780.8502-1.0980
0.35220.006420-0.7595-0.7500-46.6875-56.60430.33750.8405-0.47641.6316-2.1080
0.2220.009630-0.7949-0.7888-46.7048-61.94100.29810.8520-0.47822.1635-2.6417
0.13720.012840-0.8280-0.8277-43.7714-64.46510.27270.8635-0.18482.7093-2.8941
0.31710.016050-0.8491-0.8518-43.7808-70.72190.25610.8685-0.18583.3340-3.5198
0.23950.019260-0.8561-0.8587-44.3195-74.50110.23920.8894-0.23963.6581-3.8977
0.19570.022470-0.8841-0.8821-46.2857-79.31550.22250.8958-0.43633.9429-4.3792
0.37340.025680-0.9419-0.9412-48.2650-82.64140.21880.8973-0.63424.0776-4.7118
0.19940.028890-0.9986-1.0003-50.5246-88.40250.22370.8980-0.86024.4277-5.2879
0.1420.0320100-1.0438-1.0429-60.2203-104.67380.24390.8901-1.82975.0853-6.9150
0.06370.0352110-1.0523-1.0494-64.6031-112.83950.26230.8901-2.26805.4636-7.7316
0.39940.0384120-1.0401-1.0326-67.1203-119.28530.25860.8930-2.51975.8564-8.3761
0.49630.0416130-1.0260-1.0195-63.6645-114.91730.24430.8944-2.17415.7652-7.9393
0.1840.0448140-0.9956-0.9886-64.3210-116.18680.24400.8994-2.23985.8265-8.0663
0.45480.0480150-0.9475-0.9363-66.2748-120.65040.24540.9016-2.43526.0775-8.5127
0.36720.0512160-0.9220-0.9124-62.2740-113.50570.21400.9059-2.03515.7631-7.7982
0.17020.0544170-0.9642-0.9637-53.7962-101.30660.18820.9167-1.18735.3910-6.5783
0.49430.0576180-0.9846-0.9897-49.7029-94.75730.18480.9131-0.77805.1454-5.9233
0.41570.0608190-0.9938-0.9997-49.2515-92.85100.17960.9159-0.73294.9999-5.7327
0.17730.0640200-1.0428-1.0499-50.9003-96.49290.18290.9159-0.89775.1992-6.0969
0.0610.0672210-1.0744-1.0813-53.2245-101.36530.18940.9124-1.13015.4540-6.5841
0.25280.0704220-1.0751-1.0807-55.2461-106.93780.18690.9174-1.33235.8091-7.1414
0.12330.0736230-1.0647-1.0694-58.8487-115.01160.19220.9217-1.69266.2562-7.9488
0.1020.0768240-1.0603-1.0651-60.8948-118.73470.19280.9203-1.89726.4239-8.3211
0.33240.0800250-1.0533-1.0557-63.6693-125.61550.19040.9224-2.17466.8345-9.0092
0.0060.0832260-1.0321-1.0304-72.4092-138.86710.20700.9195-3.04867.2857-10.3343
0.0880.0864270-1.0164-1.0114-76.2556-146.47410.21870.9203-3.43337.6618-11.0950
0.13460.0896280-0.9962-0.9865-80.2022-149.61630.23050.9188-3.82797.5813-11.4092
0.45960.0928290-0.9952-0.9845-79.7203-148.67090.22910.9210-3.77977.5350-11.3147
0.30190.0960300-1.0053-0.9956-78.3724-147.59430.22220.9203-3.64497.5621-11.2071
0.07080.0992310-1.0132-1.0058-75.9911-145.45090.21330.9188-3.40687.5859-10.9927
0.03710.1024320-1.0142-1.0081-79.0266-152.03270.22910.9217-3.71047.9405-11.6509
0.33830.1056330-0.9949-0.9877-83.4509-159.52450.24520.9167-4.15288.2473-12.4001
1.10150.1088340-0.9688-0.9596-84.9845-163.43420.25080.9210-4.30618.4849-12.7910
0.20880.1120350-0.9577-0.9474-83.8031-160.27320.24600.9181-4.18808.2869-12.4749
0.35550.1152360-0.9630-0.9531-80.5315-156.92310.23140.9210-3.86088.2791-12.1399
0.1970.1184370-0.9738-0.9651-78.5307-154.62400.22150.9246-3.66088.2493-11.9100
0.59490.1216380-0.9840-0.9770-75.5271-151.07300.20790.9260-3.36048.1945-11.5549
0.32720.1248390-0.9869-0.9810-74.7702-150.49240.20080.9332-3.28478.2121-11.4969
0.06130.1280400-0.9777-0.9725-75.3109-151.86700.19950.9325-3.33888.2955-11.6343
0.34680.1312410-0.9809-0.9769-73.3059-148.06930.19490.9289-3.13838.1163-11.2545
0.42020.1344420-0.9788-0.9744-73.1894-148.32770.19450.9274-3.12668.1537-11.2804
0.21380.1376430-0.9738-0.9687-73.5879-149.23240.19470.9289-3.16658.2044-11.3709
0.4750.1408440-0.9682-0.9636-73.3113-149.38200.19590.9303-3.13888.2470-11.3858
0.00140.1440450-0.9672-0.9625-73.3271-149.40550.19760.9296-3.14048.2478-11.3882
0.02160.1472460-0.9631-0.9585-73.6283-150.24990.19940.9303-3.17058.3021-11.4726
0.15170.1504470-0.9563-0.9517-74.2343-151.64510.20180.9303-3.23118.3810-11.6121
0.37190.1536480-0.9516-0.9466-74.7146-152.70810.20160.9303-3.27928.4393-11.7184
0.11760.1567490-0.9502-0.9450-75.0118-153.31250.20200.9296-3.30898.4700-11.7789
0.23330.1599500-0.9495-0.9444-75.1085-153.46720.20190.9303-3.31858.4758-11.7943

Framework versions

  • —PEFT 0.12.0
  • —Transformers 4.44.2
  • —Pytorch 2.4.0+cu124
  • —Datasets 2.21.0
  • —Tokenizers 0.19.1