Datasetpublic

odyn-network/lora-hyperparameter-benchmark-v1

Odyn benchmark: LoRA fine-tuning hyperparameter configs (V1) Curated benchmark of real, cited LoRA and QLoRA fine-tuning configurations for validating a hyperparameter advisor. Each row is a published or measured supervised (SFT) LoRA config with its hyperparameters (learning rate, LoRA rank/alpha/dropout, epochs, batch, sequence length, gradient checkpointing), the dataset it trained on, and per-field provenance. Schema Column Type Description id… See the full description on the dataset page: https://huggingface.co/datasets/odyn-network/lora-hyperparameter-benchmark-v1.

sourceHugging Faceapache-2.0updated 2mo agoView on Hugging Face

0likes11downloads

Dataset Card

Odyn benchmark: LoRA fine-tuning hyperparameter configs (V1)

Curated benchmark of real, cited LoRA and QLoRA fine-tuning configurations for validating a hyperparameter advisor. Each row is a published or measured supervised (SFT) LoRA config with its hyperparameters (learning rate, LoRA rank/alpha/dropout, epochs, batch, sequence length, gradient checkpointing), the dataset it trained on, and per-field provenance.

Schema

Column	Type	Description
id	string	Unique row id
model	string	Base model name
modelsizeb	float	Model size (billions of parameters)
base_precision	string	Training precision: full, 8bit, 4bit, awq-4bit, gptq-4bit, aqlm-2bit
lora_rank	int	LoRA rank
lora_alpha	int	LoRA alpha as stated by the source
loraalphaeffective	float	Alpha after resolution (value the run effectively used)
loraalphaprovenance	string	Origin of the alpha value (stated, framework_default, derived)
lora_dropout	float	LoRA dropout as stated
loradropouteffective	float	Dropout after resolution
loradropoutprovenance	string	Origin of the dropout value
learning_rate	float	Learning rate
num_epochs	float	Training epochs (n/a where step-based)
batch_size	int	Per-device batch size
grad_accum	int	Gradient accumulation steps
seq_len	int	Sequence length / cutoff
gradient_checkpointing	bool	GC enabled
gradientcheckpointingprovenance	string	Origin of the GC value
dataset_samples	int	Training samples the run used
dataset	string	Dataset id (NR if the source did not disclose it)
cite	string	Human-readable citation
source_url	string	Link to primary source

Conventions: NR means not recorded or unrecoverable from the source. A stated value is the raw number the source gives. An effective value is what the run would actually use after normalization, and the matching provenance column records how it was determined (stated, frameworkdefault, or derived). The advisorwarnings and advisor_suggestions columns hold the advisor output for the row, so the false-positive check is reproducible from the file alone (an accepted config should carry no warnings).

Provenance and recovery

Values were recovered from primary sources only: framework example configs, Hugging Face model cards, adapterconfig.json, allresults.json and trainerstate.json, trainingargs.bin (inspected as a pickle, not executed), the HF datasets-server API for row counts, and published notebooks or write-ups. Anything not stated or safely derivable is left NR rather than guessed.

Sources

Rows cite framework examples and recipes (LlamaFactory, TRL, Axolotl, Unsloth, PEFT), Hugging Face model cards and notebooks, and public fine-tuning write-ups. See cite and source_url per row.

Usage

python

from datasets import load_dataset

ds = load_dataset("odyn-network/benchmark-finetune-lora-configs-v2", split="train")
print(ds[0]["model"], ds[0]["lora_rank"], ds[0]["learning_rate"])