wallfacers/weft-lineage-extractor-1.5b
weft-lineage-extractor-1.5b
## ⚠️ RESEARCH ARTIFACT — a NEGATIVE RESULT about synthetic-only training. Not a production tool. ### ✅ Resolved: real-corpus training fixes this. If you want a usable lineage extractor, use [weft-lineage-extractor-3b](https://huggingface.co/wallfacers/weft-lineage-extractor-3b) — same task, trained on real scripts, real precision 0.33 → 0.64, memorization leak gone.
A 1.5B model LoRA-fine-tuned only on synthetic ETL scripts to extract table-level data lineage. On its synthetic held-out set it looks near-perfect (precision 0.995). On real GitHub ETL scripts it collapses (precision 0.27), and a large share of its mistakes are verbatim table names memorized from the synthetic training pool (22–40% of hallucinations, depending on language). It is published so the failure — a systematic pathology of synthetic-only training — is reproducible and citable, and so the real-corpus resolution (3B) has a baseline.
Takeaway: synthetic-benchmark scores for structured-extraction models can be severely optimistic. A model can ace a held-out synthetic split by memorizing the generator's vocabulary, then emit those memorized names on real, out-of-distribution inputs.
- Base: Qwen/Qwen2.5-Coder-1.5B-Instruct
- Training data: 10,000 synthetic ETL scripts (Python/Shell, 9 structural forms) — no real scripts in training.
- Companion artifacts: 0.5B / 3B scale points, a Scala/Java (JVM) variant, and the real-corpus 3B resolution.
The headline: synthetic looks great, real does not
Same model, table-level metrics, identical extraction convention ("Convention A": label a table only if its literal name appears in an executable read/write statement; ignore dynamic names, file paths, temp views, comments, config-driven jobs).
Four-way comparison on the real Python/Shell set (n=139, non-empty gold 59):
Why it fails: memorization leak
A hallucination = a predicted table name that is neither in the gold nor literally present in the script. We check how many are verbatim names from the synthetic training pool, or share its shape (schema.schema_base_suffix, e.g. dws.dws_member_point_di).
Given a real script it cannot parse, the model falls back to reciting training table names. This is the negative result, and it is gold-independent.
Scale & cross-language
Memorization leak shrinks monotonically with model size (a capacity problem), but direction confusion does not improve with scale and the failure reproduces across languages. "More synthetic data" does not close the gap — real training data does (bottom row).
Intended use
- ✅ Reproducing / studying the synthetic-only-training memorization-leak failure.
- ✅ A baseline for abstention, real-data augmentation, or leak-mitigation research.
- ❌ Not for production lineage — use weft-lineage-extractor-3b instead.
Prompt format & quick start
System prompt (exact — must match training verbatim):
You are a data lineage extractor for ETL scripts. Given a PYTHON or SHELL task
script, output ONLY a JSON object {"reads": [...], "writes": [...]} where each
item is {"table": str, "columns": [str] or null}. Rules: include a table only if
its literal name appears in the script text; ignore dynamically-built table names,
commented-out SQL, and SQL that is merely printed or logged; if nothing is read or
written, output {"reads": [], "writes": []}.import json, re, torch
from transformers import AutoModelForCausalLM, AutoTokenizer
MODEL = "wallfacers/weft-lineage-extractor-1.5b"
tok = AutoTokenizer.from_pretrained(MODEL)
model = AutoModelForCausalLM.from_pretrained(MODEL, torch_dtype=torch.bfloat16, device_map="auto").eval()
SYSTEM = ("You are a data lineage extractor for ETL scripts. Given a PYTHON or SHELL task "
"script, output ONLY a JSON object {\"reads\": [...], \"writes\": [...]} where each "
"item is {\"table\": str, \"columns\": [str] or null}. Rules: include a table only if "
"its literal name appears in the script text; ignore dynamically-built table names, "
"commented-out SQL, and SQL that is merely printed or logged; if nothing is read or "
"written, output {\"reads\": [], \"writes\": []}.")
def extract(task_type, script, max_new_tokens=256):
msgs = [{"role": "system", "content": SYSTEM},
{"role": "user", "content": f"task_type: {task_type}\nscript:\n{script}"}]
inp = tok.apply_chat_template(msgs, add_generation_prompt=True, tokenize=True,
return_dict=True, return_tensors="pt").to(model.device)
with torch.no_grad():
out = model.generate(**inp, max_new_tokens=max_new_tokens, do_sample=False,
pad_token_id=tok.pad_token_id or tok.eos_token_id)
raw = tok.decode(out[0][inp["input_ids"].shape[1]:], skip_special_tokens=True).strip()
m = re.search(r"\{.*\}", raw, re.DOTALL)
return json.loads(m.group(0)) if m else {"reads": [], "writes": []}
print(extract("PYTHON", 'cur.execute("SELECT * FROM orders WHERE status = \'pending\'")'))
# -> {"reads": [{"table": "orders", "columns": null}], "writes": []}Training
Limitations & honest disclosures
- Not a production tool. Real-world precision ~0.27; direction ~coin-flip. Use the 3B real-corpus model.
- Literal-only by design: dynamic names, commented/logged SQL, temp views, config-driven jobs are out of scope.
- Evaluation gold is human-adjudicated under Convention A; real sets are small (Python/Shell n=139; JVM n=141). The leak metric is gold-independent (verbatim 40.4%→40.8% on JVM under full re-adjudication).
- Column-level output exists in the schema but is best-effort; evaluated claims are table-level.
Links & citation
- Real-corpus resolution: weft-lineage-extractor-3b
- Dataset (synthetic + eval/leak reports): wallfacers/weft-script-lineage-synth
- Platform: Weft (data-weave)
@misc{weft-lineage-negresult-2026,
author = {{Weft Contributors}},
title = {{Synthetic-only training induces memorization leak in small
models for ETL data-lineage extraction: a negative result}},
year = 2026,
publisher = {{Hugging Face}},
howpublished = {{\url{https://huggingface.co/wallfacers/weft-lineage-extractor-1.5b}}},
}