metrum-ai/prompt-library

Metrum AI Prompt Library

A prompt library for LLM inference workload and performance benchmarking, prepared for use with metrum-ai/bench-cli. It contains 593,730 records with prompt text, intended lengths, token buckets, and reasoning labels. It contains no reference answers.

The full configuration preserves all source records, including repeated prompts and their distinct workload targets. Prompts may appear duplicated, with only target_output_length differing. These variants are intentional: the consuming application should append the intended output word count as a hint to the LLM, for example:

request_prompt = (
    row["prompt"]
    + f"\n\nPlease aim for approximately {row['target_output_length']} words in your response."
)

Thus the same base prompt can exercise different requested output lengths. The hint guides generation; it does not guarantee an exact response length. Keep the stored prompt unchanged and append the hint when constructing the request. Include the appended hint when measuring the rendered input length.

The sample provides 2,960 records, ten from each of 296 observed input-bucket/output-bucket/reasoning groups. It is a balanced smoke-test sample, not a frequency-representative sample of the full library. The train split name is a loading convention, not a recommendation to train on this benchmark library.

Load

from datasets import load_dataset

# For reproducible benchmarking, replace main with a commit SHA from the Files tab.
revision = "main"
ds = load_dataset("metrum-ai/prompt-library", "full",
                  split="train", revision=revision, streaming=True)
print(next(iter(ds)))

sample = load_dataset("metrum-ai/prompt-library", "sample",
                      split="train", revision=revision)

The full configuration uses 12 Zstandard-compressed Parquet shards with at most 50,000 rows per shard. Both configurations use explicit file mappings, so loading them does not also load the raw JSONL or prompt-only sample export. See Hugging Face loading documentation.

Fields

All numeric metadata is retained unchanged. prompt_length and actual_words differ in 568,784 records. For actual input words, compute len(prompt.split()) using Unicode whitespace splitting. For model-specific token lengths, count using an identified tokenizer and state whether system/chat formatting is included. Do not infer a universal tokens-per-word ratio.

Token targets currently equal their corresponding buckets in every record, but that does not establish that they equal actual token counts. Target input tokens range from 32 to 7,000, and target output tokens from 32 to 2,048. See manifest.json for all group counts and numeric statistics.

Use with bench-cli

Get the CLI, installation instructions, and usage documentation from [metrum-ai/bench-cli on GitHub](https://github.com/metrum-ai/bench-cli).

The inspected CLI accepts local or HTTP(S) JSONL with a string prompt field, extracts only that field, and uses a global --max-tokens value. Consequently, the raw source and the prompt-only sample.jsonl already satisfy its current loader contract. The CLI does not yet use this dataset's per-row length targets or append the word-count hint. A consuming application or the future bench-cli integration must construct that hint from each full dataset row; the prompt-only sample export does not contain the metadata. Its existing loader also reads the full input into memory; start with the small sample.

Example for the existing CLI, using an endpoint/model you supply:

metrum-ai-bench-llm \
  --scenario prompt-library-smoke \
  --url "$ENDPOINT_URL" --model "$MODEL" \
  --prompts sample.jsonl --mode chat \
  --num-requests 100 --concurrency 1 --max-tokens 128 \
  --seed 42 --data-log results.jsonl

This example uses a uniform token cap; it does not target a particular mix. Download sample.jsonl from the dataset's Files tab first, and configure any endpoint authentication as required by your bench-cli version.

BENCH_CLI_HANDOFF.md is an implementation brief for an agent in the bench-cli repository. It specifies pinned dataset loading, joint ISL/OSL mean or median selection, words or tokens as explicit units, per-request budgets, deterministic schedules, and requested-versus-observed statistics. These features require subsequent changes to bench-cli.

Source preservation and reproducibility

• —raw/prompt-library-new.jsonl is the exact original 2,028,539,356-byte JSONL file, retained without changes.
• —Parquet preserves all nine fields and source row order; the only value transformation is normalization of 15,872 string reasoning labels to booleans.
• —Distinct rows are retained even when their prompt text matches. There are 526,912 distinct prompt strings, identified during preparation by SHA-256.
• —The sample ranks each source row by SHA256("42:<one-based-source-line>") within its normalized group, takes the ten lowest ranks, and restores source order. No prompt text or numeric metadata is rewritten.
• —sample-index.json maps each sample row to its one-based source line. Full-row zero-based ordinals are source lines minus one.
• —manifest.json records source checksum, counts, and group distributions. checksums.sha256 covers the uploaded package files except itself and Hub-generated metadata.

See VALIDATION.md for the preparation and compatibility checks.

Rebuild into a new output directory using Python 3.12:

python -m pip install -r requirements.txt
python scripts/prepare_dataset.py raw/prompt-library-new.jsonl ../rebuilt
python scripts/validate_dataset.py ../rebuilt
python -m unittest discover -s scripts -p 'test_*.py'

The preparation script streams the source and writes bounded-size batches. The validator compares every source record with the Parquet output and independently reconstructs the sample selection. Use scripts/publish_dataset.py to upload an explicitly allowlisted package; it never uploads credential files.

Provenance, intended use, and license

The source file was supplied by the dataset publisher. Its original authors, generation process, source collection, and tokenizer are not documented in the provided material. The owner confirmed the meaning of target_output_length and selected Apache-2.0 for publication; see LICENSE.

This is a performance-workload library, not an answer-quality benchmark. Reasoning labels and requested lengths have not been independently validated. There has been no content-safety, privacy, representativeness, or factual-quality audit of the prompt corpus. Generated responses may stop well before token limits, and actual output lengths must be measured separately from targets.