SlideShare a Scribd company logo

SelfieCap

•
•
J
Jennifer Rubinovitz

This document describes the development of SelfieCap, an app that uses computer vision and facial tracking to allow users to control 3D animations in real-time. The developers originally wanted to create an app for automating comic drawing but faced challenges with human pose estimation and drawing abilities. They pivoted to focus on facial motion capture without markers by using constrained local models and a point distribution model to track facial keypoints. Significant effort went into integrating computer vision, 3D graphics engines, and video streaming on iOS. The result allows video streams to be enhanced with real-time 3D graphics controlled by a user's facial movements.

Related slideshows

Types of animation
Types of animationTypes of animation
Types of animation
 
Overview of Computer Vision For Footwear Industry
Overview of Computer Vision For Footwear IndustryOverview of Computer Vision For Footwear Industry
Overview of Computer Vision For Footwear Industry
 
Animation
AnimationAnimation
Animation
 
1 of 20
Download to read offline
SelfieCap By Bin Shen, Jennifer Rubinovitz, Michał Wolski
DRAW 1. People love comics! 2. Most people can’t draw 3. We could try to use computer vision to automate the process Originally wanted to draw comics
Proof Momentcam 10,000,000 - 50,000,000 installs Cartoon Camera 10,000,000 - 50,000,000 installs Otaku Camera 1,000,000 - 5,000,000 installs
Ran into some issues 1. State of the art in human pose estimation: http://www.vision.ee.ethz.ch/~hpedemo/
More issues 2. We couldn’t find a single API to do the drawing for us 3. So we tried to do the drawing ourselves
Turns out we can’t draw either 4. We used Cocos2D to draw and animate this guy =>
Pivot Motion capture without the markers
Face Tracking
How it works ● Learn a model for each keypoint on the face ○ considering small patches in the image ● Then constrain their arrangement using a point distribution model
Constrained Local Models
Point Distribution Model image point scaling parameter rotation matrix translationbasis of variationsmean location of object point non-rigid motion parameter
Global shape prior ● The mean point locations are computed by taking the average of their locations in a labeled dataset ● The basis of variation can be found by running PCA on the covariance matrix of the points PCA Components Demo
Graphics Turns out to be pretty hard: ● Had to find a good graphics engine for iOS ● Then learn how to use it (ended up picking Cocos3D)
Model & Rigging More things to learn: ● Using Maya ● Rigging Models Then we had some issues porting them to iOS
Model & Rigging ● Maya => .pod => iOS using PVRGeoPOD converter ● Most of the advanced maya functionality didn’t work on iOS ○ could only use joints
Model & Rigging ● Normal directions ● Smooth meshes ● Texturing ● Texture mapping ● Skinning(weight painting)
iOS Application ● Facetracker ○ iOS integration ○ OpenCV ● Graphics ○ Cocos3d ○ Model display ○ Model tweak ● Video streaming ○ AVFoundation
Graphics on iOS Video Stream + Face Tracking + 3D Graphics =>
Possible Improvements ● Use animation blending instead of key points ○ smoother control ● Integrate more things into 3D ○ Interaction ● Performance tuning
Demo!

More Related Content

SelfieCap

  • 1. SelfieCap By Bin Shen, Jennifer Rubinovitz, Michał Wolski
  • 2. DRAW 1. People love comics! 2. Most people can’t draw 3. We could try to use computer vision to automate the process Originally wanted to draw comics
  • 3. Proof Momentcam 10,000,000 - 50,000,000 installs Cartoon Camera 10,000,000 - 50,000,000 installs Otaku Camera 1,000,000 - 5,000,000 installs
  • 4. Ran into some issues 1. State of the art in human pose estimation: http://www.vision.ee.ethz.ch/~hpedemo/
  • 5. More issues 2. We couldn’t find a single API to do the drawing for us 3. So we tried to do the drawing ourselves
  • 6. Turns out we can’t draw either 4. We used Cocos2D to draw and animate this guy =>
  • 9. How it works ● Learn a model for each keypoint on the face ○ considering small patches in the image ● Then constrain their arrangement using a point distribution model
  • 11. Point Distribution Model image point scaling parameter rotation matrix translationbasis of variationsmean location of object point non-rigid motion parameter
  • 12. Global shape prior ● The mean point locations are computed by taking the average of their locations in a labeled dataset ● The basis of variation can be found by running PCA on the covariance matrix of the points PCA Components Demo
  • 13. Graphics Turns out to be pretty hard: ● Had to find a good graphics engine for iOS ● Then learn how to use it (ended up picking Cocos3D)
  • 14. Model & Rigging More things to learn: ● Using Maya ● Rigging Models Then we had some issues porting them to iOS
  • 15. Model & Rigging ● Maya => .pod => iOS using PVRGeoPOD converter ● Most of the advanced maya functionality didn’t work on iOS ○ could only use joints
  • 16. Model & Rigging ● Normal directions ● Smooth meshes ● Texturing ● Texture mapping ● Skinning(weight painting)
  • 17. iOS Application ● Facetracker ○ iOS integration ○ OpenCV ● Graphics ○ Cocos3d ○ Model display ○ Model tweak ● Video streaming ○ AVFoundation
  • 18. Graphics on iOS Video Stream + Face Tracking + 3D Graphics =>
  • 19. Possible Improvements ● Use animation blending instead of key points ○ smoother control ● Integrate more things into 3D ○ Interaction ● Performance tuning
  • 20. Demo!