open-athena/exp_rpt_crosscodeeval-java-qwen3.5-122b-131k-opencode-traces
Agent trace dataset Decoding the literal token IDs The prompt_token_ids / completion_token_ids / logprobs columns are the verbatim tokens the serving engine emitted, stored PER AGENT STEP as a list-of-lists (one inner list per turn). To turn them back into text you MUST use the exact tokenizer the model was served with — a generic same-family tokenizer will decode word tokens to garbage. Served model / tokenizer source: Qwen/Qwen3.5-122B-A10B-FP8 from transformers… See the full description on the dataset page: https://huggingface.co/datasets/open-athena/exp_rpt_crosscodeeval-java-qwen3.5-122b-131k-opencode-traces.
0110
1---2tags:3- agent-traces4- literal-tokens5---6 7# Agent trace dataset8 9## Decoding the literal token IDs10 11The `prompt_token_ids` / `completion_token_ids` / `logprobs` columns are the12verbatim tokens the serving engine emitted, stored PER AGENT STEP as a13list-of-lists (one inner list per turn). To turn them back into text you MUST14use the exact tokenizer the model was served with — a generic same-family15tokenizer will decode word tokens to garbage.16 17Served model / tokenizer source: `Qwen/Qwen3.5-122B-A10B-FP8`18 19```python20from transformers import AutoTokenizer21# Use the served model's own tokenizer (pull from the ref above; if it is a22# gs:// mirror, copy tokenizer.json/tokenizer_config.json/vocab.json/merges.txt23# locally first and point AutoTokenizer at that dir).24tok = AutoTokenizer.from_pretrained("Qwen/Qwen3.5-122B-A10B-FP8")25# token_ids are list-of-lists (one list per turn) — decode each turn:26text = [tok.decode(turn, skip_special_tokens=False) for turn in completion_token_ids]27```28 29Engine-reported served model name: `1500772956264735`30 31See `tokenizer_provenance.json` for a machine-readable version.32