FoundYou · Durgesh Gowda C

Brief

Picture a mall with twenty cameras and zero coordination between them: someone walks past Camera 3, ducks behind a pillar, and reappears on Camera 7 two seconds later — and as far as the system is concerned, that's a brand-new person. FoundYou is the fix. It tracks people across an arbitrary number of camera feeds and gives each one a persistent global ID that survives camera handoffs, temporary occlusion, and leaving the frame entirely. Each camera runs YOLO26 + ByteTrack independently in its own thread; a central GlobalReIDManager resolves those per-camera local tracks into global identities using OSNet-512 appearance embeddings.

Identity resolution runs as a three-step cascade per new tracklet: match against currently active global tracks, attempt FAISS-accelerated reacquisition against archived tracks, or spawn a new global ID if neither succeeds — in that order, because guessing wrong here means confidently tracking the wrong person.

Don’t take my word for it — the receipts are on GitHub

My Contribution

Designed and built the full pipeline solo: per-camera YOLO26 + ByteTrack workers, the OSNet-512 embedding extractor, and the GlobalReIDManager that ties them together.

Implemented a robust gallery-distance metric — median cosine distance per camera, averaged across cameras — instead of nearest-frame matching, so one blurry or anomalous frame can no longer trigger a false match.

Added an optional face-detection pre-filter (OpenCV DNN, batched across all crops in a single forward pass) that vetoes body-embedding matches when face distance disagrees, with an InsightFace/dlib embedding path for sharper ranking when available.

Built a post-hoc merge pass that periodically collapses two active tracks that are really the same person — the case where ByteTrack drops an ID before a track has enough frames to be matched on first appearance.

Wrote a runtime IDValidator, a read-only watchdog that flags three failure modes on a rolling streak basis: SWAP (two tracks on different cameras that are probably one person wearing two IDs), DUPLICATE (fragmentation the merge pass hasn’t caught yet), and UNSTABLE (a gallery contaminated by more than one person).

Built a full offline evaluation harness (MOTA, MOTP, IDF1, Rank-k, mAP, CMC, throughput) plus a synthetic benchmark mode that needs no camera or GPU, so the matching thresholds can be tuned by ablation rather than guesswork.

System Architecture

Per-camera workers

Each CameraWorker runs YOLO26 + ByteTrack in its own thread, producing local (camera, track_id) pairs and batched OSNet + face features.

Step A — match active

A new tracklet is first compared against all currently active global tracks via robust gallery distance, with an optional face-distance veto.

Step B — reacquire

On no match, FAISS runs a coarse inner-product search over archived (no-longer-visible) tracks, then verifies with the full robust-distance check — this is how someone who left and came back gets their old ID.

Step C — new track

If both steps fail, a fresh GlobalTrack and global ID are created.

Maintenance loop

A periodic pass archives idle tracks, runs the merge pass on fragmented duplicates, rebuilds the FAISS index, and sweeps the IDValidator for SWAP/DUPLICATE/UNSTABLE issues.

Technical Challenges

Single-frame false matches

Matching against the single nearest stored frame is vulnerable to one blurry or coincidentally-similar frame. Solved with the median-distance-per-camera metric, which is far more resistant to outliers than a min() over the gallery.

Cold-start fragmentation

A person’s gallery is too small to match reliably in their first few frames, so if ByteTrack drops the local ID in that window, two global tracks get created for one person. The periodic merge pass reconciles this once both galleries are mature enough to compare.

Thread-safety of cv2.dnn

OpenCV’s DNN module isn’t thread-safe, so each CameraWorker instantiates its own FaceDetector rather than sharing one across threads.

Takeaways

The hardest part of FoundYou wasn't the detector or the embedding network — both are well-understood off-the-shelf components you can mostly trust out of the box. It was the identity-resolution logic sitting between them: deciding, under noise and occlusion, whether two observations are the same person without ever being able to check the ground truth at runtime. Building the IDValidator (a read-only watchdog that flags trouble instead of hiding it) — rather than tuning the thresholds once and hoping — was the single change that turned "seems to work in the demo" into a system I'd actually stand behind.