Smart glasses, augmented reality and time-of-flight imaging

By Edmund Neo and Rolf Weber

Augmented Reality (AR) is frequently discussed to deliver exciting new vision-related products for consumer and industrial applications. A popular AR implementation uses AR glasses. While the global smart augmented reality glasses market was only 255,600 units in 2020, market research predicts robust growth and a volume of 8.8 million units by 2026 [1]. The key to successful fitting and a realistic user movement experience is seamless and accurate identification of goggle position and orientation. Time-of-flight (ToF) imaging is poised to play an important role in this market acceptance. In fact, combined with AR, it could create the next big thing after smartphones and smartwatches.

AR today

While virtual reality (VR) is popular among gamers, AR could become ubiquitous as anyone could use it just like a smart phone and smart watch. Today, several AR glasses are available, including:

  • Break Eyeglasses 3 smart glasses with 3D videos, 3D photos, 3D effects and more ($375)
  • Vuzix Blade Enhanced Smart Glasses ($799.99 with optional prescription)
  • Facebook and Ray Ban stories (from $299)
  • magic jump lightweight and portable computer (Magic Leap 2 coming soon)
  • Varjo XR-3 mixed reality headset
  • Lenovo ThinkReality A3 Smart Glasses ($1499.99)
  • Xiaomi Smart Glasses (announced September 14, 2021)

Common features of today’s AR glasses include:

  • Acts as display and control for apps, smartphone and/or TV on the go
  • Wi-Fi/Bluetooth connection
  • Microphone/Speakers (taking calls, listening to music)
  • RGB camera (recording, conference call)
  • Iris recognition as password for glasses

Other, less common features include:

  • OLED screen (video call, simple augmented reality)
  • Laser projecting an image into the eye
  • Simultaneous Inertial Measurement Unit (IMU) Locating and Mapping (SLAM)
  • 3D-ToF
  • Manual tracking
  • Thought tracking

Many people may remember Google Glass, which was initially released on May 15, 2014. It had privacy issues, including privacy issues and the unattractive appearance of bezels based on the volume required to house technology and devices. batteries for adequate life. Other issues included weight, processing power requirements, and cost.

While improvements are made to address previously recognized issues, the addition of time-of-flight can provide additional functionality to differentiate an interesting concept from a must-have product in the future. With improved sensitivity from ToF imagers, lower power consumption from more capable microcontrollers, designer sunglasses with AR and ToF functionality are already on offer for a few hundred dollars.

Potential Augmented Reality Use Cases

There are several compelling use cases for AR glasses. Since they can double as a display, they can do everything a phone can do with the added convenience of not having to hold the phone and losing the use of one or both hands. With a ToF imager, night vision could be added to goggles. A thermal camera can give you heat detection capability. Today’s compelling use cases for augmented reality include object recognition and object explanation. With object recognition, an object (or a person) can be cut out of the scene and placed in front of a blurred (bokeh effect) or artificial background. AR has also proven its use in learning or displaying repair manuals/instructions for service personnel, police, fire, security and other first responders.


Fig. 1: 3D-ToF sensing adds depth and position sensing information to AR.

While some applications don’t need position sensing, its addition makes AR glasses more appealing and desirable to the consumer. Other features could include Zoom vision, photo taking and video recording of objects of interest, even while driving (without the driver distraction aspect), so the user has a recording and can reference the situation before an accident.

When an ongoing incident, such as a car dangerously approaching the vehicle, even from behind, occurs, the glasses could capture that event and provide the required legal evidence to confront the offending driver. Also in the vehicle, AR glasses can help drivers see text without looking down while driving. With a voice-to-text feature, they can see the text directly in front of them through the glasses. This could help solve the problem that despite all the great Bluetooth and hands-free speakers on the market, drivers are still trying to text while driving.

Perhaps one of the most compelling and recently reported use cases involves ToF detection to detect hidden spy cameras. [2]. Applications include locations such as hotel rooms, temporary rentals, restrooms, etc. Using ToF detection, researchers designed and implemented an application to automatically detect and locate hidden cameras in real time. With the threat to individual privacy being a global issue, this could be a killer app.

With all possible additions, AR glasses should be a great platform for the future. This takes AR Glass far beyond gaming and applies them to normal life situations. In a few years, the phone form factor could be gone – replaced by the smartwatch or the pocket key fob with the display on the AR glasses. While some people don’t like wearing glasses, sunglasses seem to overcome that hurdle. Glasses that adjust to sunlight and then automatically adjust to indoor vision requirements (another feature) could solve this problem.

The value of depth data

For those not ready to consider and embrace the potential capabilities that 3D-ToF can bring to augmented reality, consider this. The “never say never” aspect of 3D-ToF can be countered and even validated by Elon Musk’s longstanding stance against LiDAR as a required or even useful sensor in Tesla vehicles for advanced automation systems. driver (ADAS) and autonomous driving. His strident opinion was that radar and cameras would be enough and that Tesla didn’t need the 3D sensing provided by LiDAR.

While others, including Waymo and many potential suppliers of LiDAR sensors and systems, have moved towards the implementation and viability of LiDAR for vehicle use, Musk has continued to be noted for his reluctance to pursue technology. It was even put in a position to have a war on LiDAR [3]. This year, Musk apparently resolved the conflict and partnered with Luminar, a LiDAR provider, to test and develop LiDAR technology. [4]. This shows that regardless of the established position, once a compelling reason or argument is presented, decisions to change direction can be made.

In AR glasses, the non-visible illumination source, the illuminator in the ToF approach, could be used in a night vision function. The RGB camera which is there for normal viewing/camera purposes does not work in the dark. With 3D-ToF complementing the camera, the user can walk in the dark and use the other features of the AR glasses.

Another important feature of AR is object recognition. RGB cameras do this by edge detection. It works with proper lighting conditions and sufficient contrast between objects. With its depth information, 3D-ToF provides a more robust solution to these two limitations of RGB cameras.

Wanted: Visionaries to see the future

With rapidly improving processing power, lower power consumption and longer battery life, now is the time to consider the interesting applications for AR glasses. Superman or a “kitchen sink” class of AR glasses could be the next big thing to entice users.

Some of the possibilities have been presented here and hopefully will further inspire and even provide the impetus to implement them. Now is the time to innovate for the AR Glass visionaries of today and, more importantly, tomorrow, to consider how 3D-ToF can differentiate their products for future customers.

The references

[1] https://www.strategyr.com/market-report-smart-augmented-reality-ar-glasses-forecasts-global-industry-analysts-inc.asp

[2] S. Sami et al., “LAPD: Hidden Spy Camera Detection Using Smartphone Time-of-Flight Sensors,” https://dl.acm.org/doi/10.1145/3485730.3485941

[3] Brad Templeton, “Elon Musk’s War on LIDAR: Who’s Right and Why Do They Think That?” https://www.forbes.com/sites/bradtempleton/2019/05/06/elon-musks-war-on-lidar-who-is-right-and-why-do-they-think-that/?sh =13d6f17f2a3b

[4] https://www.theverge.com/2021/5/24/22451404/tesla-luminar-lidar-elon-musk-autonomous-vehicles

Edmund Neo is a senior product manager for power and sensor systems at Infineon Technologies.

Rolf Weber is a senior engineer for time-of-flight applications at Infineon Technologies.

Sarah C. Figueiredo