Augmented Reality

In 2010, augmented reality was just starting to come out of research labs and be demo’d in a limited form in mobile phones. Our mobile phone engineering team asked me to develop a point of view on interface-driven hardware requirements for future augmented reality UIs. In this engagement, we had to construct how even to measure UX quality for virtual interfaces (as there was little understanding of this across the industry at the time) and then to simulate future applications to try to answer some foundational questions on performance requirements from engineering and product planning. I ended up presenting this work at the very first ARE (Augmented Reality Event) industry confernce in 2011.

01 Background and Goals

In an augmented reality interface, generally speaking the phone senses what it is being pointed at, and projects a virtual object that is overlaid on the user’s view of reality, whether it be through a phoen screen or a set of AR goggles. After a few phone manufacturers expressed interest in support of AR, the 2014 mobile chipset development team had an important problems for me to solve: would the “sense where Im pointing” function be adequately addressed by the compass and gyroscope embedded in the phone, or woudl they need to invest in visual recognition technology (e.g., depth cameras, scene recognition) to provide an adequate user experience? And how would we even measure the performance of the device as compared to what benmchmarks?

02 Methods

Phase 1 - How to Measure? Over the previous couple of years I had been working with Intel Labs to network with academic labs at institutions like Columbia University, Georgia Tech, and TU Graz to understand the rapid development of augmented reality. When I posed to the academics the experience measurement problem, many of them suggested we look at projects sponsored by DARPA in adjacent technologies such as heads up displays for fighter jets. Not only was there a well understood body of work on HCI challenges, but DARPA had been leveraging it to publish new requirements for test projects in the emerging field of AR. So, after some desk research of publicly available Department of Defence papers, we had a head start on our metrics.

Phase 2 - Simulating Future Interfaces The next stage was an experiment which I designed simulating an AR-based wayfinding application where we could test user experience quality at different levels of hardware performance. We developed a simple application that would throw up informational annotations over a video scene set in a riverside park in Portland. We then vartied the performance of the app in areas such as

  • Lag

  • Accuracy of alignment of virtual annotation onto real object

  • Recognition delay

  • Issues arising from information density

The tested were done in the actual park using a handheld device. In subsequent phases of this research the HCI team developed additional experiments for applications such as gaming.

03 Insights

In the end we developed a matrix of future augmented reality use cases and the performance requirements we projected for each. The key finding we delivered to the product team described the threshold beyond which directional sensors on their own would be inadequate for a good user experience, and for which computer vision would be needed to insure things like accurate alignment of the virtual object onto the video feed of “reality”.

04 Actionability

This was an advanced set of findings for 2010…I gave a talk on our research at the first Augmented Reality Expo conference, and my HCI team published an academic paper. Within Intel, the research had two impacts. First it set the product team down the path of optimizing the chipset for visual recognition and reaching out to some of the academic contacts we’d made to collaborate on this. Second, it impacted our product plans around “sensor fusion”, ie the ability for the vision system to work in tandem with the directional sensors to improve the speed and accuracy of recognition.

05 My Learnings

  • When tackling a research problem that is totally new, someone out there has probably solved it already but in a different context.

  • The value of networking with academic researchers when developing new to the world products.