Where is My Mug? Training-Free Framework for Personalized Image Segmentation

Abstract

Personalized Image Segmentation (PIS) aims to segment the specific object in a target image that matches a user-provided reference. Unlike category-based segmentation or few-shot segmentation, PIS must discriminate between multiple visually similar instances, a requirement central to deployed applications such as augmented reality, personalized content editing, robotics manipulation, inventory tracking, and fine-grained visual retrieval. We show that the strong numbers reported by existing PIS methods are largely an artifact of single-instance benchmarks: when evaluated on realistic scenes with several same-category look-alikes, prior point-prompt methods collapse, because the correct instance is frequently never surfaced as a prompt. We trace this to a recall bottleneck rather than a mask-quality one, and address it with a training-free, modular system. The pipeline first generates high-recall object proposals using an off-the-shelf class-agnostic detector, then matches each proposal to the reference using foundation-model features (e.g., BLIP-2), and finally refines the selected box with SAM. For robust matching we propose top-K Sinkhorn, a patch-level distance that scores each candidate only on its most confidently matched patches; trimming the Sinkhorn dual potentials recovers the robustness of partial optimal transport at more than 100× lower cost, keeping it inside the per-proposal candidate loop. To evaluate this regime, we introduce PerMultiSeg, a realistic multi-instance benchmark. Across PerMultiSeg, PerSeg, PerMIS, and DAVIS, our system surpasses PerSAM, P2SAM, and PDM by up to ∼48 mIoU while matching their runtime and using fewer parameters, establishing a strong, deployable training-free baseline for real-world personalized segmentation.

Publication
Winter Conference on Applications of Computer Vision (WACV), 2027
Khoi Nguyen
Khoi Nguyen
AI Research Scientist

My research interests include Computer Vision

Related