DataOrchestra/Orchestrator

1---2license: apache-2.03base_model: Qwen/Qwen3-1.7B-Base4language:5- en6library_name: transformers7pipeline_tag: text-generation8tags:9- pretraining-data10- data-curation11- data-cleaning12- dataorchestra13---14 15# DataOrchestra — Orchestrator Model16 17## Model Details18 19| | |20| --- | --- |21| Base model | [`Qwen/Qwen3-1.7B-Base`](https://huggingface.co/Qwen/Qwen3-1.7B-Base) |22| Role | Orchestrator (plan generator) |23| Input | one pretraining-data chunk (≤ 1024 Qwen3 tokens) |24| Output | a flat JSON plan (`decision` + NP / SR / PA) |25| Inference mode | non-thinking, greedy decoding |26 27 28## Usage29 30The wire format is a one-line system prompt plus the raw chunk wrapped in `[DOC]` / `[/DOC]`. The model responds with a single JSON plan.31 32```python33import json34from transformers import AutoModelForCausalLM, AutoTokenizer35 36MODEL = "DataOrchestra/Orchestrator"37tokenizer = AutoTokenizer.from_pretrained(MODEL)38model = AutoModelForCausalLM.from_pretrained(MODEL, torch_dtype="auto", device_map="auto")39 40SYSTEM_PROMPT = "You are an excellent orchestrator for pretraining data cleaning."41 42 43def plan_for_chunk(chunk: str) -> dict:44    messages = [45        {"role": "system", "content": SYSTEM_PROMPT},46        {"role": "user", "content": f"[DOC]\n{chunk}\n[/DOC]"},47    ]48    text = tokenizer.apply_chat_template(49        messages,50        tokenize=False,51        add_generation_prompt=True,52        enable_thinking=False,           # orchestrator runs non-thinking53    )54    inputs = tokenizer(text, return_tensors="pt").to(model.device)55    generated = model.generate(56        **inputs,57        max_new_tokens=1024,58        do_sample=False,                 # greedy: temperature 0.0 / top_p 1.059    )60    response = tokenizer.decode(61        generated[0][inputs.input_ids.shape[1]:], skip_special_tokens=True62    )63    return json.loads(response)64 65 66chunk = (67    "Home | About | Contact\n\n"68    "The Pythagorean theorem states that a^2 + b^2 = c^2 for a right triangle. "69    "It is one of the most fundamental results in geometry.\n\n"70    "Click here to subscribe to our newsletter!"71)72print(json.dumps(plan_for_chunk(chunk), indent=2, ensure_ascii=False))73```74 75### Serving with vLLM76 77For high-throughput curation, serve the model with an OpenAI-compatible endpoint:78 79```bash80vllm serve DataOrchestra/Orchestrator --served-model-name DataOrchestra-Orchestrator --trust-remote-code81```82 83```python84from openai import OpenAI85 86client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="EMPTY")87resp = client.chat.completions.create(88    model="DataOrchestra-Orchestrator",89    messages=[90        {"role": "system", "content": "You are an excellent orchestrator for pretraining data cleaning."},91        {"role": "user", "content": "[DOC]\n<your chunk here>\n[/DOC]"},92    ],93    temperature=0.0,94    max_tokens=1024,95    extra_body={"chat_template_kwargs": {"enable_thinking": False}},96)97print(resp.choices[0].message.content)98```99 100## Output Schema101 102The orchestrator returns a flat plan JSON:103 104```json105{106  "decision": "clean",107  "noise_pruning": true,108  "surface_rectification": "Remove the navigation header and the newsletter call-to-action; keep the statement of the theorem.",109  "pedagogical_augmentation": "Add an intuitive explanation of why a^2 + b^2 = c^2 holds, with a worked example."110}111```112 113| Field | Type | Meaning |114| --- | --- | --- |115| `decision` | `"drop"` \| `"untouch"` \| `"clean"` | top-level gate; only `clean` triggers the stages below |116| `noise_pruning` | `bool` | run the NP tool model (whole-line `remove_lines` edits) |117| `surface_rectification` | `str` \| `null` | if a string, run SR with this chunk-specific instruction; `null` skips |118| `pedagogical_augmentation` | `str` \| `null` | if a string, run PA with this chunk-specific instruction; `null` skips |119 120For `drop` / `untouch` decisions, all three stage fields are inert.121 122 123## Citation124 125If you find this work useful, please cite:126 127```bibtex128@article{dataorchestra2026,129  title   = {DataOrchestra: Learning to Orchestrate Per-Example Curation of Pretraining Data},130  author  = {Huang, Zhen and Wang, Yikun and Xia, Shijie and Liu, Pengfei},131  year    = {2026},132  journal = {arXiv preprint arXiv:2607.24717}133}134```135