Future Interfaces Lab Pairs Phones To Big Touch Displays Using Just The Phone Camera

December 20, 2016 by Dave Haynes

Finding a seamless link between digital signage/interactive displays and smartphones has so far been something of a bridge too far – the problems being a need for certain technologies to be included or installed on the phone. A new system from the Future Interfaces Group research lab at Carnegie Mellon University takes a run at the challenge, with interesting results.

As reported in the MIT Technology Review, the CapCam system “allows users to pair mobile devices with touch screens by pressing them together. Touch screens can track devices as they move across their surface, while the devices extract data by reading a flashing color pattern.”

“CapCam is meant to be an alternative to manual pairing technologies like Bluetooth, without the specific hardware requirements of options like NFC (the technology behind Apple Pay). As long as a relatively modern mobile device has a camera, it should be able to be paired with another touch-screen device via CapCam. That opens up easily pairing phones and tablets with public touch screens, or transferring data between two phones by touching them together.”

The devices pair via the touch screen, which senses and (I guess) triangulates the shape of the smartphone and flashed a unique color pattern that represents a pairing code and instructions. The phone picks the light code up using its built-in camera, and established a wireless connection. The phone would need a specific app installed.

The system, continues the Tech Review piece, “could come in handy in spaces like malls, where visitors commonly find information at large public kiosks. Carnegie Mellon human-computer interaction researcher Robert Xiao used a subway station as an example; New York has digital screens where visitors can pull up a recommended itinerary for their ride.”

“If you have CapCam, you just press your phone to the screen and the itinerary you picked out is downloaded to your phone and is in Google Maps,” Xiao says.

Here’s a video produced by Carnegie Mellon showing CapCam in use. The research team is now looking for a commercial partner.

I’m not sure this removes any of the real friction around making phones and digital displays best friends forever. You still need an app – the big issue with beacon adoption and usage rates. You can already do much of what’s described using NFC taps without whatever coding or hardware adjustments (if any) might be needed for the displays. And you can get a link to a map using now old school tech like QR codes, which, like CapCam, need an app installed for scanning.

There are also apps out there that pair media players for screens with phones, and throw controls to the phone, to take over the screen.

You can play virtual air hockey, as in the video, but how many people want to use their $700 device like that?

There are also security questions, of course, around pairing. Do you want to hold up your phone to a public device and let a display pair up with the thing that holds a pile of personal and financial information? The video suggests this would all be secure and some of the interactions are just one way, but …

That stated, there’s a lot of interesting and useful stuff going on here. I like the spatial awareness aspect of this – best seen in the video where the user holds the phone up to a particular image on a screen to get that image’s information. The file sharing is also pretty slick.  With tech like NFC tags, you typically see one NFC tag on the side of the screen and I think the most exotic thing it can do is serve dynamic URLs based on what content is playing on the screen (Capital Networks and Real Digital Media have demo’d this, and maybe others).