A new reality: AR changes how technicians work and learn

Keeping up with modern truck maintenance is a towering task. There are sensors, cameras, advancements in diesel engine technology, advanced transmissions, just to name a few. And as trucks get more high-tech, it can be hard for seasoned technicians to keep up and even harder for novices to know where to start.

Augmented reality (AR), virtual reality (VR) and other reality-bending technologies are changing not only how technicians are trained but how they work. Matt Johnston, commercial solutions head of Design Interactive, said these new technologies have created “the ability to see what I see: If you have a phone and you point your camera at the engine block or a tire, I’ll be able to see what you’re looking at.”

A look at how AR-enabled goggles create interactive menus for technicians.

These features have been built into expensive AR headsets for years, but now there are more entry-level applications available on smartphones that interact with objects—from showing where all the sensors are hiding behind a bumper to overlaying step-by-step guides into a technician’s field of view. Using a smartphone app, a user can point a phone at equipment and see an augmented overlay of supplemental icons and other information on the phone screen that updates as the phone’s camera moves. And if that technician wants another set of eyes on the engine he or she is working on, an expert in another state can log in and virtually climb under the hood.

Design Interactive is creating a virtual classroom-style program to train technicians. Johnston said it combines point-of-view video training with AR that is available to live stream to technicians anywhere.

Technicians can use AR to inspect equipment before ordering it.

“All of the technicians that need to be trained can all watch that. But instead of just seeing a video stream, they’re looking at the augmented content as well,” he said. “So that instructor has now created a procedure. That procedure has images, three-dimensional interactive holograms, whatever the content might be. And they’re live-streaming the ‘how-to’ as they’re interacting with it.

“When they’re done,” he continued, “anybody within that organization who has that software can now download that content, and not a single one of those individuals had to travel. Nobody had to ship any equipment or sample parts or training aids. And digitally, you’re able to share that over and over again without having to do any extra work.”

The adoption of this technology has accelerated during the COVID-19 pandemic because AR is a socially distanced reality. With AR, fleets don’t need to have all of their knowledgeable workers in the same garage — let alone the same state. With AR goggles, such as Microsoft’s mixed reality HoloLens, a skilled technician can see exactly what the technician wearing the goggles sees in real time.

“I can put the goggles on and call a more experienced technician who can see everything I see and then guide me through the repair,” Alexandre Georgetti, director of manufacturing strategy for Eaton Vehicle Group, told FleetOwner. “You, as the more experienced one, can even draw circles and arrows on your screen, and everything you do I can see through my goggles.”

Modernizing maintenance

The technology of a modern heavy-duty truck has driven past what the educational system is using to teach the next generation of technicians. How do fleets prepare students and the technicians of tomorrow for an industry that schools are struggling to keep with today? One way is to extend reality.

Kenneth Calhoun, fleet optimization manager for Altec Group in Little Rock, Ark., has been on a mission to make this an actual reality. “I’ve been touring a number of campuses across the country and invariably what you find is we have these very well-preserved diesel museums all over the country,” he told FleetOwner. “Very few of our educational institutions are operating with equipment that is relevant to what’s out in our industry today.”

Calhoun, who was the 2019 chairman of the American Trucking Associations’ (ATA) Technology & Maintenance Council (TMC) said that few schools have the appetite for spending the money to modernize the truck equipment for the next generation of technicians. But these schools can embrace AR and VR to help set up the students for future success as fleet technicians.

“If you can supplement that through AR and VR so that the students can come away with at least an understanding of what’s out there—and then use their antiques to begin to understand torque sequences and more complex assembly, and some of the basic things—students can come out with a foundation that fleets can build on,” Calhoun explained.

This virtual hands-on learning is shaking up the old-school thought that “if you are going to learn something, you have to sit down and be quiet,” according to Beth Rutter, North American director of industry and customer engagement at Tradiebot Industries. “There are three different types of learners: visual, auditory and kinesthetic. Guess what kind of learning is typically attracted to the repair industry?”

It’s the kinesthetic—or tactile—learner, she said. “All of a sudden, the demographic that’s attracted to collision repair and working on vehicles—whether they be automotive or heavy-duty diesel—get to learn in an environment that actually speaks to them. They get it way faster than the people who have to sit down, write stuff out, and review instruction.”

Creating these visualizations for tactile learners is huge, Calhoun said. “They are going to come out of that with the understanding of what printed text on a page doesn’t accomplish for that type of learner.”

Augmented and virtual reality can give students and veteran technicians the same training experience as working on an actual vehicle, according to Marlo Brooke, CEO and founder of Avatar Partners. She told FleetOwner her company discovered that technicians have the highest memory retention through mixed reality. These days, this is a big asset as many skilled technicians are retiring, and the next generation has less experience.

“We’ve found the ability to literally take a Level 1 technician with zero to one month of experience on the job and enable that technician to train at the level of a senior technician,” Brooke explained.

Thanks to the reality-bending technology, the times to troubleshoot vehicle problems are reduced by up to 75%, she noted. “Essentially, that’s because when the technician uses the augmented reality experience when performing the maintenance task, they essentially have X-ray vision into the vehicle or into the engine, for example,” Brooke said. “It really allows them to hone in very, very quickly and find the problem.”

And while novice technicians certainly get a boost from AR, experienced technicians’ work is sped up by 25%, she added. This gives fleets facing attrition the ability not only to train novices to become experts but also to increase their experts’ skills.

“The fleet industry needs these experts to actually become more skilled,” she said. “This tool works very effectively in that manner to improve both the younger generation as well as skilled technicians.”

Augmented reality vs. virtual reality

Virtual reality (VR) takes users out of where they are and creates a new reality somewhere else. "VR is when you put on the headset and you're immersed in an environment that is outside yourself," according to Beth Rutter, North American director of industry and customer engagement at Tradiebot Industries. "You've taken yourself out of your office or classroom and put yourself into a spray booth or a flight simulator."

Augmented reality (AR) is an interface between your current reality and virtual reality, she said. “You’re standing in a real place in real-time, but you’re overlaying something that’s not with you into that environment.”

A simple version of augmented reality is the QR code, which is usually a square-shaped matrix barcode. “Most people are familiar with QR codes,” Rutter said. “That’s an augmented reality where you can take a picture of it and call up information.”

For vehicle maintenance, the potential of AR grows, particularly as vehicles become more complicated, noted Rutter, who works in the collision industry. With the right AR program, a technician can point his or her phone at a truck that needs to be repaired and then see specific repair procedures pop up on the screen.

Think of a rental car company, Rutter said, which has many vehicles on the road, several of which get into small fender-benders. “Back in the day, that was no big deal,” she recalled. “They would just send them over to the local body shop. They popped the bumper off, stuck some putty on it, shot some color on it, and sent it back at the end of the day.”

Those days are over, though. “We can’t do that anymore because there are sensors in that bumper,” Rutter continued. “And those sensors are attached to a lot of other things. Taking that bumper off now compromises a whole bunch of other systems in that vehicle. The AR component normalizes that process.”

A modern body shop technician would pull out a phone, take a picture of the vehicle, and run it through a repair smartphone app that calls up information from the cloud. “Now, you can look at the vehicle through the camera on your phone—in the app—and it populates with information on the vehicle. You’ve got little icons stuck on various parts of the car as you walk around the vehicle. The AR changes as you walk.”

And when the technician working on that bumper touches the icon, it shows where the clips are to remove the bumper, the tools to use, and the repair procedures. “It might also tell you that you need to disconnect the battery or other related but not necessarily known operations right on your phone,” Rutter added. “From a document standpoint, that’s huge.”