open-athena/exp_rpt_crosscodeeval-csharp-v4-qwen3.5-122b-131k-opencode-traces
Agent trace dataset Decoding the literal token IDs The prompt_token_ids / completion_token_ids / logprobs columns are the verbatim tokens the serving engine emitted, stored PER AGENT STEP as a list-of-lists (one inner list per turn). To turn them back into text you MUST use the exact tokenizer the model was served with — a generic same-family tokenizer will decode word tokens to garbage. Served model / tokenizer source: Qwen/Qwen3.5-122B-A10B-FP8 from transformers… See the full description on the dataset page: https://huggingface.co/datasets/open-athena/exp_rpt_crosscodeeval-csharp-v4-qwen3.5-122b-131k-opencode-traces.
Agent trace dataset
Decoding the literal token IDs
The prompt_token_ids / completion_token_ids / logprobs columns are the verbatim tokens the serving engine emitted, stored PER AGENT STEP as a list-of-lists (one inner list per turn). To turn them back into text you MUST use the exact tokenizer the model was served with — a generic same-family tokenizer will decode word tokens to garbage.
Served model / tokenizer source: Qwen/Qwen3.5-122B-A10B-FP8
from transformers import AutoTokenizer
# Use the served model's own tokenizer (pull from the ref above; if it is a
# gs:// mirror, copy tokenizer.json/tokenizer_config.json/vocab.json/merges.txt
# locally first and point AutoTokenizer at that dir).
tok = AutoTokenizer.from_pretrained("Qwen/Qwen3.5-122B-A10B-FP8")
# token_ids are list-of-lists (one list per turn) — decode each turn:
text = [tok.decode(turn, skip_special_tokens=False) for turn in completion_token_ids]Engine-reported served model name: 2714893308785853
See tokenizer_provenance.json for a machine-readable version.