Keypointguided placement for jewellery and apparel
Inpaintingrather than flat overlay compositing
Open sourcewith Docker packaging
The problem
Virtual try-on needs the generated item to sit in the right place, follow the body, and still look photographic — naive image generation fails all three.
Approach
- Used Stable Diffusion 2 inpainting so the generated item is composited into the original photo rather than overlaid.
- Tracked facial and hand keypoints with MediaPipe to position the item before generation.
- Applied OpenCV spatial alignment and xFormers memory optimisation to keep the pipeline usable.
Stack
- Diffusers
- Stable Diffusion 2
- MediaPipe
- OpenCV
- Appwrite
- xFormers
- Docker
What these numbers do and do not prove
This is an open-source experiment. Quality depends heavily on the input photo and the item being fitted.