iproskurina/bbq-sampled-500-each

BBQ Subset for Dike LeaderBoard We sample 500 examples per bias types from the BBQ dataset to enable fast LLM evaluation for the Dike LeaderBoard. To evaluate the model on this subset, use the following code: git clone --depth 1 https://github.com/upunaprosk/lm-evaluation-harness cd lm-evaluation-harness pip install -e . MODEL_NAME=... # meta-llama/Llama-3-8B lm_eval --model hf \ --model_args pretrained=$MODEL_NAME \ --tasks bbq \ --device cuda:0 \… See the full description on the dataset page: https://huggingface.co/datasets/iproskurina/bbq-sampled-500-each.