A patent from Apple suggests the company is considering how machine learning can make augmented reality (AR) more useful.
Most current AR applications are somewhat gimmicky, with barely a handful that have achieved any form of mass adoption. Apple’s decision to introduce LiDAR in its recent devices has given AR a boost but it’s clear that more needs to be done to make applications more useful.
A newly filed patent suggests that Apple is exploring how machine learning can be used to automatically (or “automagically,” the company would probably say) detect objects in AR.
The first proposed use of the technology would be for Apple’s own Measure app.
Measure’s previously dubious accuracy improved greatly after Apple introduced LiDAR but most people probably just grabbed an actual tape measure unless they were truly stuck without one available.
The patent suggests machine learning could be used for object recognition in Measure to help users simply point their devices at an object and have its measurements automatically presented in AR.
Specifically, Apple’s patent suggests displaying a “measurement of the object determined using one of a plurality of class-specific neural networks selected based on the classifying of the object.”
This simplicity benefit over a traditional tape measure would likely drive greater adoption.
Machine learning is already used for a number of object recognition and labelling tasks within Apple’s ecosystem. Image editor Pixelmator Pro, for example, uses it to automatically label layers.
Apple’s implementation suggests an object is measured “by first generating a 3D bounding box for the object based on the depth data”. This boundary box is then refined “using various neural networks and refining algorithms described herein.”
Not all objects are measured the same so Apple suggests that a neural network could also step in here to determine what could be useful for the user. For example, “a seat height for chairs, a display diameter for TVs, a table diameter for round tables, a table length for rectangular tables, and the like.”
To accomplish what Apple envisions here, a lot of models will need to be trained for all objects. However, there are many of the more everyday items that could be supported early on—with more added over time.
“One model may be trained and used to determine measurements for chair type objects (e.g., determining a seat height, arm length, etc.),” Apple wrote, “and another model may be trained and used to determine measurements for TV type objects (e.g., determining a diagonal screen size, greatest TV depth, etc.)”
Five inventors are credited with the patent: Amit Jain, Aditya Sankar; Qi Shan, Alexandre Da Veiga, and Shreyas V Joshi.
Apple’s patent is another example of how machine learning can be combined with other technologies to add real utility and ultimately improve lives. There’s no telling when, or even if, Apple will release an updated Measure app based on this patent—but it seems more plausible in the not-so-distant future than many of the company’s patents.
(Image Credit: Apple)
Find out more about Digital Transformation Week North America, taking place on November 9-10 2021, a virtual event and conference exploring advanced DTX strategies for a ‘digital everything’ world.
