uv-scripts/object-detection

1---2name: detection-bootstrap3description: Bootstrap an object-detection dataset and a small trained detector from images that have NO labels — zero-shot label with Falcon-Perception, validate, convert, then fine-tune a compact Apache-licensed model, all on Hugging Face Jobs. Runs fully autonomously or with human review checkpoints. Use when you have an image collection and want a detector but no annotations exist.4---5 6# Bootstrap a detector from unlabeled images7 8The loop: **zero-shot teacher labels → validate → convert → train a small student → evaluate → publish.**9Every step is a self-contained UV script from10[`uv-scripts/object-detection`](https://huggingface.co/datasets/uv-scripts/object-detection) on the11Hugging Face Hub, or a `hf jobs` command. `--help` works on every script.12 13## Pick your path14 15Five decisions cover most runs; each routes into the numbered steps below.16 171. **Where are the images?** Dataset repo → `falcon-perception.py`. Bucket → `falcon-perception-bucket.py`18   (reads `.jpg`/`.jpeg`/`.png` only — convert JPEG 2000 / TIFF first).192. **Transport to the trainer**: build canonical `train.parquet` / `validation.parquet` with20   `embed-bucket-images.py` (images embedded, gold excluded and asserted). Trainers that take HF21   datasets read the parquet directly — including straight off a bucket; trainers that want a COCO22   directory tree get one **generated in-job** with `materialize-coco.py` — onto a bucket mount if23   more than one job will train on it (a complete tree is reused, not rebuilt). Never hand-assemble24   or upload directory trees: a tree generated from the parquet cannot have missing-image25   mismatches. Run `smoke-test.py` first (free, local, ~30 s): it proves these plumbing scripts26   still produce correct output before a paid job depends on them.273. **Boxes or masks?** Boxes → the D-FINE default in step 5. Masks → an RF-DETR-Seg-style trainer via28   `materialize-coco.py` (RLE masks carried through and resized from the teacher's inference frame to the29   image frame).304. **Human available?** Show step-1 previews and do the step-6 gold slice. Headless → numeric proxies31   and say **unreviewed**.325. **After the first student**: run the step-6 loop — student over the teacher-empty pages at a low33   threshold, VLM pre-triage, retrain on the corrections.34 35## Check if a human in the loop36 37You can use the approach outlined in this skill with or without a human in the loop.38 39- **With a human in the loop** (better models): show them the step-1 previews — "is the teacher boxing40  the right things?" is the highest-value question, and its fix is the cheapest (a better query and a re-run of the teacher). Then train on a small slice first (500–1k images) and show 20 rendered41  predictions before spending on the full corpus. If corrections are worth collecting at volume, run a42  review pass with `review-detections.py` (keyboard accept/reject in the browser — quick mode for43  whole-image verdicts in random order with quotable rates, boxes mode for per-box rejects; pushes a44  `review` column), fold corrections in and retrain. Diff the corrected set against the first pass (`diff-hf-datasets.py`) to measure how45  good the zero-shot pass actually was.46- **Autonomously** (headless): don't pause for review — use the numeric proxies, and say **unreviewed**47  in the final report and model card.48 49## 1. Sense-check the class name before spending GPU money (free)50 51Falcon-Perception queries are **class names, not instructions** ("photograph" works; "the photographs,52excluding captions" returns nothing), and **one class per run** (combined queries collapse — run per53class and merge on `image_id`). Model details: `hf models card tiiuae/Falcon-Perception`.54 55Check cheaply on 3 images before any full pass. The teacher (Falcon-Perception) is a **0.6B model,561.3 GB download** — it runs on a CUDA GPU (fast), Apple Silicon (MLX backend auto-selected, about 6 s/image),57or plain CPU (slow, but fine for 3 images). Run the check wherever is practical for you:58 59```60# locally, if your machine can:61uv run https://huggingface.co/datasets/uv-scripts/object-detection/raw/main/falcon-perception.py \62  --dataset <USER>/<IMAGES> --limit 3 --query photograph --preview63 64# or the same check as a small job (previews don't persist on Jobs — push a tiny dataset instead).65# l4x1 is the cheapest flavor that fits the engine (see step 2's flavor rule):66hf jobs uv run --flavor l4x1 --secrets HF_TOKEN \67  https://huggingface.co/datasets/uv-scripts/object-detection/raw/main/falcon-perception.py \68  --dataset <USER>/<IMAGES> --limit 3 --query photograph --out <USER>/<NAME>-check --private69```70 71Judge the result before scaling up:72- **If you can view images**, look at the rendered previews (or the pushed check dataset) — are the73  right things boxed? `render-detections.py` renders any dataset in this schema and **pixel-verifies74  its own output** (a page with instances whose render equals the source exits nonzero — silent75  blank-overlay bugs are real and have been shown to humans as "done").76- **If you can't**, compare instance counts across candidate queries (`stats-hf-dataset.py` below works77  on a pushed check dataset): near-zero instances/image means the class name is wrong for this material —78  try a synonym (`photograph` / `illustration` / `figure` / `cartoon`). Suspiciously many (more than about 10/image)79  *can* mean the query is matching layout blocks — but dense plates genuinely carry 10–20 figures,80  so counts are a fallback signal only; previews are the judge.81- Measured on real material, previews judged:82 83  | material | worked | partial | dud |84  |---|---|---|---|85  | historic newspaper pages (b/w scans) | `photograph`, `illustration` | | |86  | book / encyclopaedia plates | `illustration` (incl. dense multi-figure plates) | `caption` (good on true plates, grabs whole text columns on text-heavy pages) | `figure` (0 hits on the same pages) |87- **No vision at all?** A vision-capable subagent can judge the previews if you can spawn one;88  otherwise tell the user the check ran unviewed.89 90(Falcon-Perception has a custom architecture, so it can't be served as an OpenAI-compatible endpoint —91iterate via the batch script. If you swap in a teacher that vLLM can serve, a temporary hot server on92Jobs is the faster way to iterate on queries: see93[Serve Models on Jobs](https://huggingface.co/docs/hub/jobs-serving).)94 95## 2. Teacher pass on Jobs96 97```98hf jobs uv run --flavor a10g-large --secrets HF_TOKEN --timeout 2h \99  https://huggingface.co/datasets/uv-scripts/object-detection/raw/main/falcon-perception.py \100  --dataset <USER>/<IMAGES> --query photograph --out <USER>/<NAME>-photograph --private101```102 103- Sizing: expect a few images per second, not tens — the pass is decode-bound, so a bigger GPU104  changes little; to go faster, shard the file list across several jobs writing to the same output105  bucket. `--limit` on either script caps a run.106- Flavor rule (all three failures measured): the engine needs a **24 GB-VRAM GPU** (16 GB T4s107  CUDA-OOM during prefill) and **more than 15 GB host RAM** (the engine sizes itself from the GPU108  and ignores host RAM, so `t4-small` and `a10g-small` are OOMKilled before the first image).109  `hf jobs hardware --json` lists every flavor's `ram`, accelerator and price — `l4x1` is the110  cheapest fit (fine for the step-1 check); `a10g-large` is faster for a corpus pass.111- One job per class (step 1's rule). Every run labels its boxes `category` 0 in a single-name112  `ClassLabel`, so a naive concat collapses the classes — renumber each run to its index in a113  combined `ClassLabel` when merging. Rows align on `image_id` (every run contains every image):114 115  ```python116  from datasets import ClassLabel, Sequence, load_dataset117 118  names = ["illustration", "map"]119  parts = [load_dataset(f"<USER>/<NAME>-{n}", split="train") for n in names]120  extra = [dict(zip(ds["image_id"], ds["objects"])) for ds in parts[1:]]121 122  def merge(row):123      o = {k: list(v) for k, v in row["objects"].items()}124      for i, run in enumerate(extra, start=1):125          r = run[row["image_id"]]126          o["bbox"] += r["bbox"]; o["area"] += r["area"]127          o["rectangularity"] += r["rectangularity"]128          o["category"] += [i] * len(r["bbox"])129      return {"objects": o, "n_instances": len(o["bbox"])}130 131  feats = parts[0].features.copy()132  feats["objects"]["category"] = Sequence(ClassLabel(names=names))133  merged = parts[0].map(merge, features=feats)134  ```135 136  (`masks_rle` concatenates the same way if you need the masks.)137- Output schema: `objects.bbox` in **YOLO format** (normalized center x, y, w, h), `objects.category`138  (a `ClassLabel` named after the query), `objects.area`, `objects.rectangularity`, plus `image`,139  `image_id`, `width`, `height`.140- There are **no confidence scores** (the model has none). `rectangularity` (mask area ÷ box area) is the141  triage proxy: values near 0 are usually junk, 0.785 is a circle, 1.0 a full rectangle.142- Submit with `--detach` (returns the job id immediately), then block on completion with143  `hf jobs wait <id> [<id> ...] --timeout 2h` — it exits 0 only if every job succeeded, so it144  chains cleanly into the next step. `hf jobs logs <id>` / `hf jobs inspect <id>` for progress and errors.145- A job can sit in SCHEDULING while the flavor queue drains — that is a queue, not a failure.146  **Don't resubmit**: a second copy racing to the same `--out` just doubles the bill. If you do147  switch (`hf jobs hardware` for alternatives), cancel the queued copy first (`hf jobs cancel <id>`).148- For images in a [storage bucket](https://huggingface.co/docs/hub/storage-buckets) instead of a149  dataset, use `falcon-perception-bucket.py` — it writes resumable parquet parts back to a bucket150  (kill and re-run the same command; done keys are skipped). It reads `.jpg` / `.jpeg` / `.png` only —151  convert JPEG 2000 or TIFF scans first, or it will silently find zero images:152 153  ```154  hf jobs uv run --flavor a10g-large --secrets HF_TOKEN --timeout 2h --detach \155    https://huggingface.co/datasets/uv-scripts/object-detection/raw/main/falcon-perception-bucket.py \156    --src <namespace>/<bucket> --prefix <path/under/bucket> \157    --out <namespace>/<out-bucket> --query illustration158  ```159 160  Publish once at the end so the parts feed the rest of this loop (parquet stores `category` as161  bare ints; the cast attaches the class name):162 163  ```python164  from datasets import ClassLabel, Image, Sequence, load_dataset165  ds = load_dataset("parquet", data_files="hf://buckets/<namespace>/<out-bucket>/part-*.parquet",166                    split="train")167  feats = ds.features.copy()168  feats["objects"]["category"] = Sequence(ClassLabel(names=[ds[0]["query"]]))169  if "image" in feats:  # parts written with --embed-images: make the bytes a decodable Image column170      feats["image"] = Image()171  ds.cast(feats).push_to_hub("<namespace>/<dataset>")172  ```173 174  The bucket path's output is **annotations-only** by default — there is no `image` column, and the175  step-5 trainer and `review-detections.py` both need embedded images. `embed-bucket-images.py` (same176  repo) joins the bytes back in, drops the teacher's error rows, excludes and asserts the gold slice,177  splits train/validation, and writes the final schema exactly once — to a dataset repo, or as `train.parquet`/`validation.parquet`178  in a bucket. (`--embed-images` on the teacher pass writes the bytes into the parts instead — storage179  is cheap, and the join step then skips its re-fetch; the cost is a copy of the corpus in the output180  bucket.) Trainers that want a COCO directory tree get one generated from that parquet by181  `materialize-coco.py` — once, onto a bucket mount if several jobs will train on it — never182  hand-assemble or upload directory trees.183 184## 3. Validate the labels (free, local)185 186```187uv run https://huggingface.co/datasets/uv-scripts/object-detection/raw/main/validate-hf-dataset.py \188  <USER>/<NAME>-photograph --bbox-format yolo189uv run https://huggingface.co/datasets/uv-scripts/object-detection/raw/main/stats-hf-dataset.py \190  <USER>/<NAME>-photograph --bbox-format yolo191```192 193Expect **VALID** with 0 out-of-bounds and 0 zero-area boxes. `W001` warnings on empty images are normal194and worth keeping as training signal — but treat them as **unverified negatives**: zero-shot teachers195miss real instances on a meaningful fraction of "empty" pages (a third, on one measured corpus). The196step-6 loop is how you find and flip them.197Drop obvious junk before training: degenerate slivers (extreme aspect ratio + tiny area) and near-duplicate198boxes (IoU > 0.9 within one image).199 200## 4. Convert YOLO → COCO for training (free, local)201 202Trainers expect COCO `xywh` pixels; the teacher emits YOLO normalized. One command:203 204```205uv run https://huggingface.co/datasets/uv-scripts/object-detection/raw/main/convert-hf-dataset.py \206  <USER>/<NAME>-photograph <USER>/<NAME>-coco --from yolo --to coco_xywh207```208 209## 5. Train a small detector210 211A known-good default: fine-tune212[`ustc-community/dfine-small-coco`](https://huggingface.co/ustc-community/dfine-small-coco)213(D-FINE small, 10.4M params, Apache-2.0, in `transformers`) on the step-4 COCO dataset —214800 images, 30 epochs, `t4-medium`, about 48 minutes (`hf jobs hardware` shows current prices). Training needs only a T4:215step 2's 24 GB-VRAM rule is the teacher's engine, not the student's.216 217The [**`huggingface-vision-trainer`**](https://github.com/huggingface/skills/tree/main/skills/huggingface-vision-trainer)218skill runs the training end to end (dataset validation,219augmentation, mAP eval, Hub persistence) — install it with `hf skills add huggingface-vision-trainer`220if you don't have it, and follow its object-detection path with the `<USER>/<NAME>-coco` dataset and221the settings above. Hold out the validation split — and the step-6 gold slice — BEFORE training,222and never train on either. Write checkpoints **continuously to the synced `/data` mount**, not `/tmp`223or a local output dir: Jobs can be SIGTERM'd at any time (node reclaim, requeue), anything outside the224mount dies with the job, and durable checkpoints are also what make stopping at a plateau safe.225 226Other trainers work — the dataset is plain COCO. [RT-DETRv2](https://huggingface.co/PekingU/rtdetr_v2_r18vd)227is a comparable compact Apache-2.0 pick; [RF-DETR](https://github.com/roboflow/rf-detr) (Apache-2.0,228DINOv2 backbone) is a good starter, and its Seg variant can learn from the teacher's `masks_rle`229masks. Check the license fits the use — `hf models card <id>` shows it; flag restrictive licenses230(e.g. ultralytics/YOLO is AGPL) to the user rather than deciding for them. Explore further:231[transformers object-detection models](https://huggingface.co/models?pipeline_tag=object-detection&library=transformers&sort=trending) ·232[ultralytics-library models](https://huggingface.co/models?library=ultralytics).233 234Decode `masks_rle` like this — each RLE lives in its own frame, which never matches the235recorded width/height:236 237```python238import json, numpy as np239from PIL import Image240from pycocotools import mask as mask_utils241 242for rle in json.loads(row["masks_rle"]):243    seg = mask_utils.decode({**rle, "counts": rle["counts"].encode()})  # frame = rle["size"]244    if seg.shape != (row["height"], row["width"]):245        seg = np.asarray(Image.fromarray(seg).resize((row["width"], row["height"]), Image.NEAREST))246```247 248## 6. Evaluate honestly249 250- Report mAP on the held-out slice. Be clear about what it measures: **agreement with the teacher**,251  not accuracy against human truth — no human labels exist in this loop unless you make some (next252  bullet).253- **Gold slice** (with a human in the loop): hold out about 100 random images BEFORE training — keyed254  on a **stable image id** that is identical in every dataset you build (path prefixes from different255  runs silently break the match) — and **assert the exclusion** before submitting any training job:256  train count = total − gold, overlap = 0. Then have257  the human verify every box on them with `review-detections.py --mode boxes --order random`, then258  correct any misses (the tool flags them with M; drawing the missing boxes is manual for now).259  Then report TWO numbers: mAP vs teacher labels AND mAP vs the human gold. They260  differ, and the gap is the finding — in the validation run of this skill: 0.84 vs teacher labels261  but 0.44 vs human gold, both mAP@50 on held-out pages. That gap is the teacher's systematic262  divergence from human annotators, which teacher-agreement alone cannot see.263- The student can at best match its teacher (measured on a comparable loop: student 97.4% vs teacher264  95.0% human-acceptable on the same sample). The point of distilling is **throughput and cost**265  (10–100× cheaper per image than the teacher), not accuracy gains.266- Evaluate with the model card's decode contract, and write that contract INTO the card (input267  padding, score handling — with one class use the raw logit/sigmoid, never softmax). This is268  load-bearing: a standard decode against a padded-square model measured 0.03 mAP where the269  documented decode measured 10× higher. (Evaluating locally on Apple Silicon: pass the trainer's270  eval a CPU device — the COCO eval path uses float64, which MPS lacks.)271- Spot-check 20 or so predictions visually before calling it done — or, if running without a human and you272  cannot view images, state prominently in the report that the model is **unreviewed**.273- It can make sense to run this process in a loop: predict → review (a human, or a vision-capable274  agent, via `review-detections.py`) → retrain on the corrections → review again, until the acceptance275  rate stops improving. Two things make the loop cheap: point the student at the **teacher-empty pages276  at a low threshold** first (that is where the teacher's false negatives concentrate, and flipping277  them from negative to positive is the biggest training-signal win), and **pre-triage candidates with278  a VLM judge** (one crop per instance, mask highlighted) so the human only reviews the uncertain279  residue rather than every candidate.280 281## 7. Publish with honest provenance282 283Push the model and dataset — ask the user whether public or private; if you can't ask, default to284private and say so. Build each dataset's **final schema in memory and push once** — never stage an285intermediate push to the repo you will publish. A second push with different columns leaves the repo's286stored features stale and `load_dataset` fails with a cast error; if the schema must change, push to a287new repo id. The cards must state: labels are **zero-shot weak labels** from Falcon-Perception288(name the script + date), which filters ran, and that **recall is unmeasured** unless you measured it289against an independent source. Say what the model is for and what it was trained on. A model trained290this way is a first pass. The review loop above is how it gets better.291