Mobile screen-spy software recreates passwords

When one types text messages, emails and login credentials on one’s virtual iPhone or Android keyboard, larger bubbles containing each character pop up for a second while the individual keys are being pressed.

But handy as the feature undoubtedly is for those users who have trouble deciphering the smaller characters on the keyboard, it can also be misused for spying on the users’ activity.

When typing in confidential information on the phone, the more paranoid among us usually surreptitiously check if someone is behind or near us and can read what we write. But, according to a team of researchers from the University of North Carolina at Chapel Hill, that might not be enough to thwart resolute spies.

To prove their point, they have developed a program called iSpy, which can identify the text a user has typed in by analyzing video footage of the screen while he was doing it. And what’s worse, the program can even extract that information from the footage of the screen as it is reflected in a window or one’s sunglasses.

To do this, the spy doesn’t have to have telescopic lenses or high-end equipment – smartphone cameras are more than enough.

“We then use a number of computer vision techniques to process the recorded video, identifying, for each frame, potential keys that were pressed,” explain the researchers. “This visual detection, coupled with a language model, enables us to achieve surprisingly accurate sentence retrieval results, even under challenging real-world scenarios.”

According to the New Scientist, the spy has to be rather close to the person he’s spying on to record the input with a phone camera – standing within 3 meters from the target is enough if he can record the screen directly. If he can record the action with a digital SLR camera, he can be up to 60 meters away from the target. In ideal conditions, over 90 percent of the letters are identified correctly.

The percentage drops somewhat when the spy can only record a reflection, because the screen image is smaller and blurrier. But, the spy can do a relatively good job of it if he’s using an SRL camera and is located no more than 12 meters away.

But, there are fairly simple solutions to this type of attack. For one, you can disable the visual key press confirmation mechanism, and reducing the brightness of the screen can also help.

“Our ability to reconstruct text typed on virtual keyboards from compromising reflections underscores the need to continually reevaluate our preconceptions of privacy – or the lack thereof – in modern society,” is the point that the researchers are trying to make. “Even cryptography and secure devices are of little use when, across the aisle, someone who appears to be reading email on their phone is in fact surreptitiously recording every character we type.”

For more details about the research, download the paper.