Inside LookFor

Voice in, on-device. The mic button uses expo-speech-recognition over Apple’s SFSpeechRecognizer with on-device recognition required — audio never leaves the phone, matching the app’s “Data Not Collected” privacy label. While the recogniser session is open, haptic feedback is muted (iOS silences haptics during audio capture), and the session is closed before guide feedback resumes. Typing works identically; voice is just another way to fill the same query field.

The composer is the conversational trick. src/finder/queryComposer.ts is pure TypeScript with simple, predictable rules — no LLM in v1:

Because refinement just rewrites a string, and the next stage re-encodes any changed string, “it’s blue-green” updates the search within 1–2 frames. The refined text matters twice: it steers the guide loop and it is the thing boxes are re-ranked against at the end.

Tokenize. CLIP’s text tower doesn’t read letters; it reads byte-pair-encoded tokens. LookFor ships its own TypeScript BPE tokenizer (src/ml/clipTokenizer.ts) that reproduces the reference open_clip tokenizer exactly — special SOT/EOT tokens, lowercasing, a 77-token context.

Embed with a prompt ensemble. The query isn’t encoded bare. Five templates (“a photo of a {q}”, “a {q} on a table”, “a {q} on the floor”, “a close-up of a {q}”, “a {q} in a room”) are each encoded, then averaged and L2-normalised into one unit vector t. Averaging washes out template-specific quirks, leaving the meaning — a standard zero-shot CLIP trick from the original paper. A fixed negative set (“a room”, “furniture”, “a wall”, “a floor”, “background”, “a person”) is encoded once at startup; these give the softmax something to compare against, so an empty wall can’t score “warm”.

Text encoding runs once per query change, so its latency (~hundreds of ms) never gates the frame loop.

Frames stay native. react-native-vision-camera delivers preview-sized RGB frames to a frame-processor plugin as native buffers. Pixels never cross the JS bridge — only 512-float embeddings and small box arrays do. A scheduler enforces cadence (Battery / Balanced / Max = 2 / 4 / max-achievable guide calls per second), pauses when the accelerometer says the phone is still, and backs off automatically on thermal pressure.

The rotating-cell economy. Scoring the full frame plus all nine 3×3 cells every frame would cost ~10 × 48 ms ≈ 480 ms — too slow on the A13. Instead each frame encodes the full frame + 2–3 rotating cells (~150–200 ms per batched Core ML call), cycling so every cell refreshes about every 3 frames (~0.5 s). A CosCache keeps the last score per cell and merges each frame’s partial results; the guide only reacts once it holds a full 10-crop snapshot. stepFinder recovers each crop’s index from its rect, so a frame’s cache-merge always matches the plan that actually produced its embeddings.

Why a native Swift module exists at all. The M0 spike measured the alternatives:

The Swift module (modules/lookfor-ml) also does crop + resize + CLIP mean/std normalisation natively, in one batched call, and exposes thermalState() so the scheduler can downshift cadence before iOS throttles the whole app.

Cosine similarity is the whole trick. CLIP was trained contrastively so that an image and a text that describe the same thing land near each other on a unit sphere. For each crop embedding e, the app computes cos(e, t) and cos(e, negᵢ) for the six negatives. Raw CLIP cosines live in a narrow band (roughly 0.15–0.35 — nothing near 1.0), which is why the next two steps exist.

Softmax with temperature τ turns “is the target cosine meaningfully above the background cosines?” into a probability. Small τ sharpens the contrast; τ is tuned offline in the replay harness against recorded frames. An absolute cosine floor (~0.20–0.25) additionally suppresses warmth when even the best cosine is background-level — softmax alone would happily declare a winner among six losers.

Warmth is a smoothed, debounced signal. The instantaneous max p_target is noisy frame to frame, so warmth is an exponential moving average (α ≈ 0.3) with hysteresis at the thresholds — it takes sustained evidence to cross into warm, and slightly more absence to fall back out, so the ring never flickers. Calibrated from real device data: warmOn = 0.45. The measured bands that justify it: real objects scored warmth 0.46–0.57 on device; background scenes stayed ≤ 0.24. The GuideAggregator also reports the best 3×3 cell (the heat blob) and a direction cue (“try left”) from where that cell sits.

Phases: idle → searching → warm → found → lost, in src/finder/stateMachine.ts — pure TypeScript, no React, no native calls. The per-frame heartbeat stepFinder (src/finder/finderStep.ts) wires it: score crops → merge into the cache → update the aggregator → feed the machine → collect cues. Because the whole decision layer is pure, it runs in Jest with fake embeddings (162 tests), and scripts/replay.ts replays recorded frame sequences offline — time-to-find and false-positive rate gate every release.

One owner for all feedback. A single FeedbackController consumes state-machine events and drives everything sensory: ring colour (cold blue → hot orange), tone pitch and haptic tempo rising with warmth, speech only on state changes (“getting warmer”, “try left”, “lost it” — never a running commentary), and queued VoiceOver announcements. One owner means modalities never fight — the chime can’t talk over the speech, haptics pause during voice capture.

Open-vocabulary detection. A classic detector has a fixed class list baked in. YOLOE’s text-prompt variant instead takes class embeddings as a runtime input — LookFor exports it un-fused so the model signature is (image, classEmbeds[k,D]) → boxes. That single export decision is what makes the app open-vocabulary: new classes cost an embedding lookup, not a retrain.

The vocabulary table. ~400 household nouns are pre-embedded offline, twice: once with YOLOE’s own text tower (vocab.yoloe.bin — must match its head) and once with the app’s permissive CLIP tower (vocab.guide.bin — used to map your query to its nearest nouns at runtime). Per confirm pass, the class set = top-8 vocab nouns nearest the query noun, plus the raw query embedding. So “book” also hunts as “notebook”, “magazine”, “folder” — near-synonyms the detector head understands.

Re-rank: the division of labour. YOLOE proposes candidate boxes (confidence ≥ 0.25) but only knows nouns. Each candidate is cropped, pushed through the CLIP image tower, and scored by cosine against the full refined query embedding t. YOLOE finds the books; CLIP picks the blue-green one. The first real on-device find: warmth 0.57 triggered the confirm loop, YOLOE boxed the mug, re-rank scored p = 0.61 against foundScore = 0.55.

k-of-n before celebrating. A box must beat the found threshold in 3 of the last 5 confirm frames before the app declares found. Single-frame spikes — a glint, a motion blur, one lucky crop — get filtered out. Same debouncing idea as the EMA, applied as a vote instead of a smoother.

Found is a moment, not a state to hover in: box + label, one haptic, a chime, speech (“Found: blue-green book, upper left” — the position comes from where the box sits on screen), and a freeze-frame card with Keep looking / Done. Scanning pauses so the phone stops burning battery on a solved problem.

Lost handles the physical world: if the box disappears for more than 2 seconds (you turned, something occluded it), the app drops to lost, pins a “last seen here” marker at the box’s final screen position, and keeps the guide loop running. Re-acquiring the object snaps straight back to found.

Cold start. The 42 MB binary ships with no models. First launch downloads ~320 MB (CLIP image tower ~168 MB fp16, text tower ~120 MB, detector ~24 MB, vocab < 2 MB) from GitHub Releases — resumable, SHA-256-verified, cellular only with an explicit toggle. The app becomes usable in guide-only (hot/cold, no boxes) as soon as the guide model lands; the detector arrives after. Every downloaded model also passes an on-device parity check against desktop reference embeddings before it’s trusted.