open-athena/exp_rpt_crosscodeeval-java-qwen3.5-122b-131k-opencode-traces
Agent trace dataset Decoding the literal token IDs The prompt_token_ids / completion_token_ids / logprobs columns are the verbatim tokens the serving engine emitted, stored PER AGENT STEP as a list-of-lists (one inner list per turn). To turn them back into text you MUST use the exact tokenizer the model was served with — a generic same-family tokenizer will decode word tokens to garbage. Served model / tokenizer source: Qwen/Qwen3.5-122B-A10B-FP8 from transformers… See the full description on the dataset page: https://huggingface.co/datasets/open-athena/exp_rpt_crosscodeeval-java-qwen3.5-122b-131k-opencode-traces.
0103
1{2 "schema": "tokenizer_provenance/v1",3 "served_model": "Qwen/Qwen3.5-122B-A10B-FP8",4 "served_model_name_observed": "1500772956264735",5 "literal_columns": [6 "prompt_token_ids",7 "completion_token_ids",8 "logprobs"9 ],10 "literal_columns_shape": "list-of-lists (one inner list per agent step/turn)",11 "decode_with": "transformers.AutoTokenizer.from_pretrained(served_model)",12 "note": "Decode with the EXACT served-model tokenizer; a same-family tokenizer decodes word tokens to garbage. Use skip_special_tokens=False to keep <|im_end|>/<tool_call>/<think> markers."13}14