Apple Glass could put you anywhere in the world

Apple is considering a system that will allow the wearer of Apple Glass or another Apple-made headset to choose the background that will appear when someone is standing in front of them. The technology is similar to the “green-screen” used on television shows and movies that make it look as though an actor is somewhere else rather than standing in front of a screen on a studio lot.

Unlike the 2D system used by Skype and Zoom, Apple’s patent application envisions the use of a 3D system that lets the Apple Glass user replace the background of a subject while he or she is moving and walking around. As Apple’s patent application notes, “Low latency is a key element required to achieve great immersion (often called “Presence”). The color keying processing which is embedded in the Head-mounted display (HMD) detects the selected range of color and mixes in the virtual content coming from the HMD itself or from an external source such as a connected computer. Virtual content can also be composited on top of the color-keyed area to allow mixed-reality content occluding real-life objects.”

The patent application was filed in February and published today by the U.S. Patent and Trademark Office (USPTO). It is titled “Low latency chroma keying embedded in a head-mounted display for mixed reality.” The low latency used with this system is important because it allows the processing to be done at a speed that prevents users from getting ill when viewing the fake backdrop stutter and lag. And the low latency is made possible by embedding the processor inside the HMD.

Interestingly, the current thought is that at least in the early days of Apple Glass, users will need to pair the wearable with an iPhone and allow the handset’s processor to do the heavy lifting. That is what Apple first did with the Apple Watch; the latter originally required the processor from a user’s iPhone to run most features before Apple released a standalone version of the timepiece. As a result, the technology mentioned in the patent might not be employed on Apple Glass until the third-generation of the AR glasses or later.

The patent application explains that “The system achieves mixed reality by having a user look at a display through wide-angle lenses (with a field-of-view of 110 degrees or more). Two cameras located behind the HMD display capture the environment from viewpoints located a few centimeters in front of each eye.”