soldni/jeopardy
Jeopardy questions from Mosaic Gauntlet Sourced from https://github.com/mosaicml/llm-foundry/blob/main/scripts/eval/local_data/world_knowledge/jeopardy_all.jsonl Description: Jeopardy consists of 2,117 Jeopardy questions separated into 5 categories: Literature, American History, World History, Word Origins, and Science. The model is expected to give the exact correct response to the question. It was custom curated by MosaicML from a larger Jeopardy set available on Huggingface.… See the full description on the dataset page: https://huggingface.co/datasets/soldni/jeopardy.
Jeopardy questions from Mosaic Gauntlet
Sourced from https://github.com/mosaicml/llm-foundry/blob/main/scripts/eval/localdata/worldknowledge/jeopardy_all.jsonl
Description: Jeopardy consists of 2,117 Jeopardy questions separated into 5 categories: Literature, American History, World History, Word Origins, and Science. The model is expected to give the exact correct response to the question. It was custom curated by MosaicML from a larger Jeopardy set available on Huggingface.
How to use
from datasets import load_dataset
dataset = load_dataset("soldni/jeopardy", "mosaicml_gauntlet")
model = ...
tokenizer = ...
# Given context, try to predict the continuation
for row in dataset:
input_ids = tokenizer(row['context'], return_tensors='pt').to(model.device)
outputs = model.generate(input_ids, max_new_tokens=100)
decoded = tokenizer.decode(outputs[0], skip_special_tokens=True)
correct = row['continuation'] in decoded
print("Gold:", row['continuation'])
print("Pred:", decoded)
print("Correct?", correct)
print("----")