Apple is considering a system that will allow the wearer of Apple Glass or another Apple-made headset to choose the background that will appear when someone is standing in front of them. The technology is similar to the “green-screen” used on television shows and movies that make it look as though an actor is somewhere else rather than standing in front of a screen on a studio lot.
Unlike the 2D system used by Skype and Zoom, Apple’s patent application envisions the use of a 3D system that lets the Apple Glass user replace the background of a subject while he or she is moving and walking around. As Apple’s patent application notes, “Low latency is a key element required to achieve great immersion (often called “Presence”). The color keying processing which is embedded in the Head-mounted display (HMD) detects the selected range of color and mixes in the virtual content coming from the HMD itself or from an external source such as a connected computer. Virtual content can also be composited on top of the color-keyed area to allow mixed-reality content occluding real-life objects.”
The patent application was filed in February and published today by the U.S. Patent and Trademark Office (USPTO). It is titled “Low latency chroma keying embedded in a head-mounted display for mixed reality.” The low latency used with this system is important because it allows the processing to be done at a speed that prevents users from getting ill when viewing the fake backdrop stutter and lag. And the low latency is made possible by embedding the processor inside the HMD.
Interestingly, the current thought is that at least in the early days of Apple Glass, users will need to pair the wearable with an iPhone and allow the handset’s processor to do the heavy lifting. That is what Apple first did with the Apple Watch; the latter originally required the processor from a user’s iPhone to run most features before Apple released a standalone version of the timepiece. As a result, the technology mentioned in the patent might not be employed on Apple Glass until the third-generation of the AR glasses or later.
The patent application explains that “The system achieves mixed reality by having a user look at a display through wide-angle lenses (with a field-of-view of 110 degrees or more). Two cameras located behind the HMD display capture the environment from viewpoints located a few centimeters in front of each eye.”
If you don’t mind the patent jargon, Apple says, “While chroma keying is usually a post-production effect or done live with dedicated equipment/software, it can be applied in the context of augmented reality (AR) headsets, also referred to as mixed reality (MR), where virtual elements are composited in real-time into the real-world environment seen by the user through the head-mounted display. One example is a crane simulator where the controls would be real objects and where a color screen would be replaced by a virtual crane and a virtual environment. Another example would be a social MR experience using a color screen that covers the whole field of view and only the user’s hands and/or body are not replaced by virtual content (unless their color is the same as the screen). This example application is sometimes classified as virtual reality (VR) but for simplicity, we here consider it as augmented reality (AR).”
As with any patent application filed by the company, or even with a patent received by the tech giant, there is no guarantee that Apple will actually use the patented technology. Apple applies for and receives a large number of patients each year and while some are for innovative technologies actually employed by the firm, some ideas are just patented to protect Apple from the competition.
