build-small-hackathon/puck

1# Puck backlog2 3Ideas not yet built. Each entry: what, why, and where it slots into the architecture4(engine = pure policy/data, ui = presentation, modes/sim = orchestration).5 6## Gestures — sprite dances/reactions keyed to event type7 8**What:** Puck performs a short gesture matched to *what* he's reporting, not just9flying + speaking. A build passing → a little hop/spin of pride. Tests failing or a10permission block → an urgent shake/alarm wiggle. Discord noise he ignored → a dismissive11shrug. Claude finished → a satisfied stretch. Stale tab → a curious head-tilt. Sleep →12a yawn. The gesture plays during the flight-to + bubble beat.13 14**Why:** On a busy real desktop the sprite is glanceable; a gesture lets you read the15*kind* of news pre-attentively, before the words. It's the single highest charm-per-line16addition left, and it makes the overlay feel like a creature rather than a notifier.17 18**Where it slots in:**19- `engine/` — a pure `gestureFor(event | decision)` map: `EventDef.id`/`source` →20  gesture name. Data, not logic; pin a couple of mappings with a test. Could also key21  off the existing tier/mischief so high-mischief gestures are more theatrical.22- `ui/Sprite.tsx` — a `gesture?: GestureName` prop adding a CSS class (`gesture-hop`,23  `gesture-shake`, `gesture-shrug`, …); animations on `.puck-bob`/`.puck-face` only24  (compositor-friendly transforms, like the existing flap/bob). Clear the class on25  animationend so it re-triggers.26- `modes/sim/SimApp.tsx` — set the gesture alongside the existing `flyTo`/`setBubble`27  in `fireEvent`; clears when the surface resolves.28 29**Notes:** keep gestures to transform/opacity so they stay GPU-composited (the whole30sprite layer is). Reuse the per-form structure in `SpriteBody` — gestures should read31on all four forms (mossling/wisp/gremlin/moth), so animate the wrapper, not form-specific32parts. Pairs naturally with the existing mood tint + alert ring.33 34**Color pulse (same feature, cheapest channel):** transient-tint the alert ring + glow35to a per-event-type hue while a surface is pending — red/urgent for failures &36permission blocks, gold/success for completions, grey/dim for ignored noise. The ring37and `.puck-glow` already read CSS vars (`--accent`, `--glow`), so this is a single38`--alert-hue` override set from the same `engine` map that picks the gesture — one39`{ gesture, hue }` lookup feeds both. Even shipping the color pulse *alone* (before40gesture animation work) is a real readability win. Don't fight the existing mood tint41(`--puck-body` etc.): pulse the ring/glow, leave the body color to mood, so "what kind42of news" (hue) and "how Puck feels lately" (mood) stay separate signals.43 44## Ambient quips — Puck reacts to what you're doing45 46**What:** Occasionally Puck mutters a quip or non-sequitur riffing on your current47context — the active app, a window title, the shape of what you're typing. Not help,48not a summary: flavor. "Still in the auth thicket, I see." / "That's a lot of tabs for49one small human." / a non-sequitur about the dock breathing. Low frequency, skippable,50never twice about the same thing.51 52**Why:** This is the difference between a notifier that lives in the corner and a53familiar that *shares the room*. It's the most "alive" feature on the list — and the54most dangerous, so it ships last and most carefully.55 56**Privacy is the feature, not a caveat** (design doc §17.3–17.4 are the law here):57- **Local inference ONLY — hard gate.** Context never leaves the machine. If the brain58  is the cloud path (Modal/ZeroGPU), this feature is *disabled*, full stop, not degraded.59  Enforce in code: the quip path checks the resolved brain is localhost and refuses60  otherwise — a loud guard, not a setting.61- **Opt-in, off by default.** A distinct toggle from notifications; the permission copy62  says plainly what's read and that it stays local.63- **Ephemeral, never persisted, never a trace, never training data.** Context for a quip64  is read, used for one generation, dropped. It must not touch the memory garden, the65  trace export, or localStorage.66- **Redact before the model sees it.** Strip obvious secrets (password fields, token-shaped67  strings, anything in a field marked sensitive). Prefer *abstractions* over content —68  "a long terminal command", "a code file", "a messaging app" — over the literal text.69  Quoting back verbatim is the creepy line; stay on the abstract side of it.70- **Never about people or private content.** App categories and your own activity, yes;71  the contents of a DM, an email body, a name on screen — no.72 73**Where it slots in:**74- The desktop watcher (future, §9.2: NSWorkspace frontmost app, optional AX/screen) is the75  context source — same daemon `/events` path, a new low-priority `context` event kind, or76  a separate local-only endpoint that never queues.77- `engine` decides *whether* to quip (rare; respects annoyance budget / presence, reuses78  the interruption-taste machinery so an annoying quip trains Puck quieter).79- The quip generation uses the local brain with a tight "one playful aside, ≤12 words,80  never quote the user" prompt; surfaces through the existing bubble channel.81 82**Ship order:** after the desktop watcher exists and after a privacy pass. Until then it's83sim-only flavor at most (riffing on the fake desktop, where there's nothing real to leak).84 85## Phototaxis — a fairy drawn to stimuli (whimsical wander)86 87**What:** Puck should behave like a moth/fairy — *pulled* toward activity rather than88drifting at random. Flits toward motion, lingers near what's lively, chases the89occasional shiny thing, then loses interest.90 91**Why:** The wander is the sprite's resting personality — it's on screen far more than92any bubble. Uniform-random reads as a screensaver; attraction reads as *alive and93curious*. Highest charm-per-effort of the ambient ideas.94 95**Buildable now — zero new permissions (do this part first):**96Replace the uniform-random wander target in `SimApp` with a weighted pull toward salient97points we already have:98- the **cursor** (occasional gentle follow / curious approach, then retreat — never99  clingy; respects `presence`),100- the **last event location** (he lingers where something just happened),101- in the overlay, the **focused window** rect (future: NSWorkspace frontmost-app102  position via the daemon — he hangs near where you're working, patrols where you're not).103Keep it a *gradient*, not a leash: weighted-random pick among attractors + noise, so it104stays unpredictable. Lives in the engine as a pure `pickWanderTarget(attractors, rng)`;105the loop already exists. Tune against `presence` (low = aloof, high = follows the action).106 107**Sensor-gated — backlog, same privacy rules as ambient quips (local-only, ephemeral):**108- **Screen color / motion / "flashing lights":** needs ScreenCaptureKit — the transparent109  overlay can't see what's beneath it. A coarse, downsampled brightness/motion map (NOT110  readable content) could let him drift toward an area that just changed (a video started,111  a notification flashed). Local-only, never stored, abstractions not pixels.112- **Audio reactivity:** mic is a hard no by default; "system audio is playing / its level"113  is lighter but still opt-in + local-only. A bass-thump bob or a turn-toward-the-sound114  would be delightful but ships last, behind the same gate as quips.115 116**Smell test for all of it:** attraction must stay *cute*, never *surveillant*. He reacts117to the shape of activity (something moved, something's loud), never to its content.118 119## Take-me-there — Puck knows where the activity is, and ferries you to it120 121**What:** On a real notification, Puck should point you at the *actual* window that needs122you — fly to it if it's on this Space, beckon "follow me" if it's on another — and123**clicking him navigates there** (focuses the app, macOS brings its Space forward).124 125**Current behavior (the gap):** wire events arrive with `target: null` (the sim's targets126were fake windows), so in the overlay `fireEvent` flies Puck to a *random* screen point.127He has no idea where the source app lives — we never gave him real-window awareness.128This is the design doc's Phase 4 ("patrol the desktops you abandoned" presumes knowing129where they are).130 131**Phase 1 — click-to-activate (high value, no Space geometry needed):**132- Event carries a **locator**: source app bundle id / pid / window title. The Claude hook133  already has `TERM_PROGRAM`, `cwd`, and the calling pid available; `puck-run` knows its134  terminal. Add an optional `locator` to the wire schema (`{bundleId?, pid?, title?}`).135- Rust command `activate_target(locator)` -> `NSRunningApplication(bundleIdentifier:).activate`136  (or AX focus by pid/title). macOS switches to that app's Space automatically.137- Frontend: clicking Puck while a located surface is pending calls it. This alone delivers138  "Puck lit up -> click -> you're where the thing is," across Spaces, without knowing which.139 140**Phase 2 — same-Space vs other-Space (the fiddly bit):**141- Determine if the target window is on the *current* Space: `CGWindowListCopyWindowInfo`142  with `kCGWindowListOptionOnScreenOnly` lists on-screen windows; absent target -> elsewhere.143  Robust Space identity needs the semi-private CGSSpace APIs — fragile, optional.144- Same Space -> fly to the window's screen rect (window bounds from CGWindowList).145  Other Space -> a "come hither" beckon toward the screen edge (pairs with the gesture146  entry), and the click teleports.147 148**Notes:** locator is metadata, not content — bundle id + window title, never window149*contents* (anti-creep). Activation is an explicit user click (navigation, not an150autonomous action), so it stays inside the safety tiers. Depends on: gesture vocabulary151(beckon) and the future native window watcher (NSWorkspace frontmost / CGWindowList).152 153## Camouflage — Puck adapts to what he's floating over154 155**What:** When Puck drifts over text or busy content, he reacts to his surroundings like156a chameleon/glass-wisp — goes translucent, refracts, or (dream version) "mirrors" the157texture behind him onto his own body. Blends, then pops back when he moves to empty space.158 159**Why:** Sells "he's really *in* your desktop, not pasted on top." A creature that160responds to its background reads as inhabiting the space. Pairs with the wisp form161(already glass-like).162 163**Cheap / free now (no sensing):**164- **Shy fade:** lower sprite opacity while stationary over the busy center of the screen,165  restore when wandering to the margins — pure CSS/opacity on the existing wander state.166  Reads as "blending in" without literally seeing anything.167- **Refraction (maybe free, needs a WKWebView test):** a `backdrop-filter` glass body on168  the sprite. In the transparent overlay this *might* sample the real desktop behind the169  window (same open question as the speech-bubble blur) — if it does, a refractive/distort170  body gives instant chameleon shimmer with zero screen-capture. Test in the Tauri overlay171  before committing to it.172 173**Dream version — literal mirror (screen-capture gated):**174- Sample the screen region directly under the sprite (ScreenCaptureKit), downsample, and175  paint it onto his body as living camouflage. Stunning, but it's the same hard gate as176  ambient quips / phototaxis sensors: local-only, ephemeral, opt-in, never stored, never177  leaves the machine. Texture/color only — never treated as readable content.178 179**Where it slots in:** `ui/Sprite.tsx` (a `camouflage` intensity prop on `.puck-body`),180driven by `modes/sim` from sprite position vs. screen regions. The shy-fade is a18120-minute add; refraction is a test-then-maybe; the literal mirror waits for the screen182watcher + privacy pass.183 184## Real app icons — known apps, OS-extracted where possible185 186**What:** Replace the glyph characters (✳ ◍ ✉ …) in the sim windows/dock and feed187source-markers with real app icons — Claude, ChatGPT, Chrome, Gmail, Mail, Discord,188Terminal, etc. Looks dramatically more legit, especially in the overlay.189 190**Pulling the *actual* OS icon (the good version):**191- macOS: `NSWorkspace.shared.icon(forFile: "/Applications/Foo.app")` returns the real192  icon for any installed `.app` bundle — a Rust/Tauri command can extract → PNG → hand to193  the webview. So Chrome, Mail, Discord, Slack, the host terminal: real icons, free, always194  current.195- **The limit you called:** a terminal *binary* (claude, codex) has no bundle and no icon.196  Fall back to the **host terminal's** icon (iTerm/Terminal/Ghostty/Warp — which the Claude197  hook can report via `TERM_PROGRAM`), or a curated Puck-styled glyph.198- **Web apps with no native app** (ChatGPT, Gmail as a tab): no bundle to extract from —199  these need a small **curated bundled icon set** (a dozen SVGs/PNGs).200 201**So: hybrid.** OS extraction where a bundle exists (Rust command, overlay only), curated202bundled set for web-apps + terminal-binary fallbacks (works everywhere incl. the Space/sim).203 204**Where it slots in:**205- `ui/Desktop.tsx` (`WIN_DEFS` icons, dock glyphs) and the feed `source` marker — currently206  single glyph chars; swap for an `<Icon source=…>` that prefers OS-extracted, falls back to207  bundled, falls back to glyph.208- Overlay-only Rust command `app_icon(bundleId|path) -> png` for the extraction half.209- Engine `SOURCES`/`EventDef` already carry source identity; add an optional `bundleId` hint.210 211**Cheap first step (no native):** ship the curated bundled icon set + source→icon map; use212glyph only as last resort. The OS-extraction half is an overlay enhancement on top.213 214## Local vision — the private, free path for continuous perception215 216**What:** Run Puck's eyes on-device instead of (or alongside) Modal, so real-screen217perception is private and continuous vision costs ~nothing. Brain seam already supports218it: point PUCK_VISION_URL at a local OpenAI-compatible server.219 220**Capability is NOT the blocker** (verified 2026-06-07): screen-reading is OCR + light221"what's notable" reasoning — small VLMs excel at it. Options:222- **Holotron-12B local** via llama.cpp + mmproj — llama.cpp merged Nemotron-Nano-12B-v2-VL223  (PR #19547). `convert_hf_to_gguf.py --mmproj` → vision projector → `llama-server`. Same224  model as cloud, same capability, ~24GB on the 48GB Mac, free. NB: Ollama can't load the225  mmproj — must use raw llama-server. Keeps the Nemotron Quest tie.226- **Qwen2.5-VL-7B local** — ~6GB, 95.7 DocVQA, fast; ideal for *continuous ambient* (every227  45s forever on the M4 Max). Loses the Nemotron tie, plenty for screen-reading.228- MiniCPM-V 2.6 (~5.5GB), Moondream2 (1.9B, CPU) as even-lighter fallbacks.229 230**Recommendation:** Holotron for the showcase + Quest (cloud now → local llama-server231later as the private path); Qwen2.5-VL-7B-local as the cheap continuous engine. The232visionMode "Continuous" tier (built, currently same cloud path) should switch to a local233PUCK_VISION_URL when this lands.234 235**Where it slots in:** zero engine/frontend change — it's a runtime: spin up236`llama-server --mmproj` (or vLLM/mlx when fixed) and set PUCK_VISION_URL. Plus the237real-screen capture (ScreenCaptureKit, overlay) to feed it actual pixels instead of the238sim snapshot.239