How Augmented Reality Can Change Airplane Maintenance and Piloting Forever

A Boeing 737 scheduled for a cabin modification sits in an aircraft hangar, seating and panels removed. A maintenance engineer wearing an augmented reality (AR) headset with a visor that stretches across his field of vision can immediately access all the information needed to route and install new looms and equipment. A virtual model of what he should see once the modification has been completed overlaps what he’s actually looking at. Step by step, each phase of the modification is described, visualized with any necessary tooling called out. Any mistakes are quickly identified before the next task is started. The use of the AR headset ensures quality is built in as this complex task proceeds, and accurate recording of work completed facilitates a comprehensive and smooth handover should the modification run across multiple shifts.

Meanwhile, across the airport a pilot sits in a Cessna 172 getting ready for her pre-flight checks. She too slips on an AR headset, which begins to visually guide her through her checklist. Once she’s all set and on the runway, she clears her view so that what remains are the visual cues necessary for take-off and any warning messages that should be accounted for. Despite the dark and rainy conditions, enhanced vision systems overlay a crisp image of the surrounding areas and enable a confident, on time departure.

Welcome to the near future of augmented reality and avionics.

Why Augmented Reality, Why Now

Virtual reality may be all you’re hearing about lately, but thanks to AR’s broader industrial applications, it’s more likely to be the medium that will rewrite our digital futures. By all estimations, the market value of AR will be worth $90 billion by 2020 — three times that of virtual reality. And that money has an effect on the smart glasses that will drive the changes to come.

“It’s accelerating the process of getting them ready for all of these difficult use-cases that are coming to the market,” says Vincent Higgins, CEO of Optech4D, which specializes in industrial applications of AR and VR.

What’s also accelerating the process is good technological timing. “We’ve got the convergence of mobile computing power, display technology, sensors — all of the things that you need to really make the technology work are now starting to come into play,” says Barry Po, Senior Director of Product and Business Development at NGRAIN, another maker of industrial AR. The effect of that convergence is that, the better the technology works, the more it’s going to become dependable enough for industries to start seeing the benefits of AR.

The aviation industry is especially poised to benefit from the rise of augmented reality. In fact, AR has already made its first steps into piloting and maintenance. Companies like Optech4D and NGRAIN, among many others, have begun previewing how aviation will undergo a major AR upgrade within the next five years.

The Potential of Augmented Flying

The arrival of AR headsets brings one immediate upgrade to the aviation industry: A shift from the forward looking-only head-up displays in commercial airline cockpits, to the omni-directional abilities of head-mounted displays (HMD) like Microsoft HoloLens. Now pilots will have the ability to look wherever they choose, at any time, and have digital information appear naturally in their field of vision. What they’ll be able to do with that is being spearheaded by innovators like Ákos Maróy and Franz Viertler.

Ákos Maróy is the CEO of Aero Glass, a Hungarian company (with a San Diego outpost) currently developing smart glasses which pilots would use not only to augment take-off and landing, but flight time too. With a headset on, a pilot could check their flight path, receive hazard warnings or be shown distances to objects, such as nearby aircraft. Franz Viertler, a system engineer at ESG in Germany, is working on something similar, albeit more specialized: Helping helicopter pilots to fly in low-visibility weather conditions with AR that projects digital topographical information (mountains, houses) in the pilot’s field of view, so they can “see” what’s ahead.

Both entrepreneurs share an ultimate goal, one that will influence the augmented piloting road ahead; that is that AR-assisted flying will empower pilots to make safer and more informed decisions, especially in challenging circumstances. “The bottom line is safety,” says Maróy.

The AR Piloting Challenges Ahead

Safety may be the ultimate goal, but there’s still work to be done before it gets there and is (potentially) widely adopted. “The performance is already there, but you have to ensure the reliability and the safety of the system itself,” Viertler says. In this case, that means improvements in the real-time information the HMD draws from to plot a pilot’s course. What if a week-old database update doesn’t show a construction crane that was set up the day before a pilot flies?

And in Maróy’s case, potential use on commercial airlines means standardization, which the company is working on with SAE, Maróy says. The company is seeking a “minimum set of requirements in terms of quality, reliability and interoperability with existing procedures and systems onboard” — but the goal isn’t just to conform to standards, but also potentially innovate them. “We really are trying to play around with things and explore new approaches,” he says.

There are still user-experience design questions that need to be finessed through that exploratory process: How will AR info appear in different lighting environments? Do existing aviation colors and symbology need to be adopted or evolved? How can visual clutter be avoided so as not to obscure a pilot’s view?

That’s why both entrepreneurs say there’s still a good deal of testing to be done. But while their expectations are reasonable, there is still optimism. “It will take more than three to five years to really have the systems where we want to have them, but in the next few years I think we’ll already bring them into reality and demonstrate that it is working there,” says Viertler.

The arrival of AR headsets brings one immediate upgrade to the aviation industry: A shift from the forward looking-only head-up displays in commercial airline cockpits, to the omni-directional abilities of head-mounted displays (HMD).

Augmenting Aircraft Maintenance and Inspections

One of the appeals of adapting AR to piloting is making a complicated job a little easier. NGRAIN’s Barry Po believes the same mission should be applied to airplane maintenance. “We have to stop thinking of aerospace engineers and technicians as tradespeople and start thinking about them a little bit more like they’re fighter pilots. They’ve got incredibly complex jobs where sometimes they don’t have a whole lot of time to make a decision,” he says.

He’s not the only one who believes AR can make that responsibility a little easier to handle. Several companies are currently developing AR options to streamline aircraft maintenance in ways that don’t just benefit the workers. “We think maintenance is something that is going to have a pretty dramatic return on investment for airlines and aircraft manufacturers, while pretty clearly demonstrating the power of the technology,” says Mark Cheben, the global marketing director of EON Reality.

Like many of his colleagues, Cheben stresses that the main ROI of augmented reality is hands-free movability and real-time information, but the technology is being used in a variety of ways — often in conjunction with artificial intelligence, the Internet of Things, sensors, 3D scanning and video conferencing. Optech4D, for example, has what it calls “The Safe Beacon,” which is a sensor that can be attached to a specific piece of equipment in a plane that, when approached with an AR-enabled tablet or headset, will trigger information about the equipment.

The company also has something called “Remote Expert,” a form of teleconferencing that uses smart glasses’ front-facing cameras to enable an expert at headquarters to see through an operator’s glasses and push through information — documents, annotations, e.g. — to guide the inspection or maintenance process.

As for NGRAIN, it has technology that uses artificial intelligence and 3D graphics to scan environments and conduct visual inspections that could identify scratches, dents and other anomalies that need looking after.

Many of these applications are still in their early stages, but nonetheless they foreshadow the immense potential ahead. Po says NGRAIN’s AR could scan, identify and track problems throughout an aircraft’s life — all with just smart glasses — and help airlines and manufacturers make more informed decisions about significant overhauls. It would enable, as he puts it, “proactive, not reactive, maintenance.”

Will Augmented Reality and Aviation Take Off?

The promise of such a fluid, streamlined, AR-enabled future — in both piloting and maintenance — is tantalizing, but the technology, while promising, still has a ways to go.

What’s more, airline companies may lean towards a wait-and-see approach to gauge whether the ROI of augmented reality is enough to embrace widespread, or even partial, adoption. But those taking these early steps will see advances in the next few years. “The hardware is increasingly going to get to a point where it is mature enough to be deployed in a manufacturing setting or in a real-world setting. That’s not necessarily a place where it is today for every scenario, but one would expect that in short order that’s where it will be,” says Po.

In other words, it may be worth second-guessing an inclination to write off augmented reality and aviation as hype. “People need to understand that it’s not just a cool technology,” says Higgins of Optech4D. “Big companies don’t buy ‘cool.’ They buy things that have a return on their investment.” If even a fraction of the ambition of these early explorers are realized, it’s not hard to imagine companies will be looking to buy.