Smartphone Interaction with Registered Displays (Touch Projection)

In this project, we use a camera-equipped mobile phone to view a PC screen (or public display) and we can control many interactions between the two devices. The essence of the idea is that, after estimating the planar projective mapping (homography) between the image on a mobile and a (possibly distant) display, any 'click' on the mobile's touch screen can be projected onto the correct corresponding point on the distant display. Of course, a wireless communication link is required so that the mobile can send its image to the display's PC, along with the point 'clicked' on its touch screen. Our idea has been reportred in a 'New Scientist' article in January 2008. This report made the top ten articles in the ACM TechNews alert, and it has been reported in many press articles and on many websites around the world.

More recent work by Boring et al (CHI 2010) has coined the term Touch Projection for this idea, as a user's touch on the mobile's screen is projected to another device's screen, wherever the particular interaction is occuring (see Touch projector website).

Video demonstrations

• Photo montage exercise. In this interaction task, photos have to be translated, scaled and rotated on a screen, using the mobile phone.
• Drawing exercise. The user has to draw the outline of a house with a mobile phone.
• "Write-through" exercise on a PDA. A circle, square, triangle and diamond are drawn around the markers on the PDA and appear in the correct places on the PC screen. This video was catured using a mobile phone, so apologies for the poor quality footage.

Publications.

The key publication is number 2 below. Paper 1 describes a new markerless registration scheme, based on scale invariant features such as SURF and SIFT. Paper 3 is the original publication on the `display registration' idea.

1. Smartphone Interaction with Registered Displays
   N. E. Pears, P. Olivier and D. Jackson.
   IEEE Pervasive Computing, vol 8, no 2 (April-June 2009), pp 14-21.
   Special issue on Smarter Phones.
   [PDF] [DOI][BibTeX]
   
   
2. Display Registration for Device Interaction
   N. E. Pears, P. L. Olivier and D. Jackson
   3rd Int. Conf. on Computer Vision Theory and Applications 2008 (VISAPP'08), 22 - 25 March, 2008, Funchal, Madeira, Portugal.
   [PDF] [BibTeX]
   
   
3. Mobile Device and Intelligent Display Interaction via Scale-Invariant Image Feature Matching.
   L. Herbert, N. E. Pears, D. Jackson and P. Olivier
   1st Int. Conf. Pervasive and Embedded Computing and Communication Systems (PECCS 2011)
   [PDF] [BibTeX]
   
   

BACK to Nick Pears' Research Projects page.