Posey's Tips & Tricks

Test-Driving Microsoft HoloLens in Zero Gravity

Despite all of HoloLens' unique uses, Brien never expected to be wearing it while floating around weightless. That's exactly how he spent this morning -- with mixed results.

• By Brien Posey
• 10/29/2018

Almost exactly two years ago, I was given the chance to use a Microsoft HoloLens device for the very first time.

Since then, I have had a number of opportunities to work with HoloLens, and even now I am still amazed by the device and what it can do. To say that I am anxiously awaiting the release of HoloLens 2, which will reportedly debut by mid-2019, would be an understatement.

Despite all of HoloLens' unique uses, I never expected to be wearing the device while floating around weightless. Even so, that is exactly how I spent this morning.

Earlier this week, I traveled to Ottawa for yet another series of parabolic (zero gravity) flights. Even though I will be the first to admit that being weightless is super-fun, these aren't tourist flights: Those of us on the flight crew were required to perform numerous scientific experiments for the Canadian Space Agency and for various universities and other organizations.

One of the experiments being flown was developed by my friend and colleague Anima Patil-Sabale, who had the idea that an astronaut on an EVA (a.k.a. a spacewalk) might have an easier time completing their task if they could access real-time visual data through the HoloLens. She decided to put this idea to the test on one of the parabolic flights -- and that meant that I got to wear the HoloLens in zero gravity.

My role during this flight campaign was that of test director for the last flight. As test director, my job was to oversee all of the experiments being performed on the aircraft. Patil-Sabale suggested that my job might be easier if we were to display my checklist and my notes within HoloLens.

There were two perceived benefits to doing this. First, because HoloLens uses a clear display, I would be able to use it like a heads-up display in an aircraft or a car. Digital information is superimposed over the real world. In the context of my parabolic flight, this would mean that I would never have to take my eyes off of the rest of the crew while reading the checklist.

The other perceived benefit was that the HoloLens might reduce the chances of being incapacitated by motion sickness. They don't call the aircraft used for parabolic flights "the vomit comet" for nothing. Motion sickness is an ever-present issue and the problem is made worse by head movements. Repeatedly looking down at a checklist could increase the chances of getting sick on the flight. Projecting the checklist through HoloLens was intended to reduce the need for head movement, thereby leading to a more comfortable flight.

Going in, I thought that Patil-Sabale had a great idea. I especially loved the part about not getting sick. As I started thinking about how the HoloLens works, however, I began to wonder if we were actually going to be able to make the experiment work.

HoloLens uses a special camera to create a 3-D map of the space around you. The device then allows you to anchor holograms to real-world locations. You can actually place a hologram in a specific location, come back an hour later, and the hologram will still be in the spot where you placed it. That being the case, I couldn't help but wonder if a jet traveling hundreds of miles per hour would simply leave the checklist hologram behind at the airport.

The day before the flight, I suggested to Patil-Sabale that we should try out HoloLens in a moving vehicle to see what happens. I drove the car and she rode in the back seat wearing the HoloLens. When the vehicle was in motion, HoloLens kept trying to remap our space (I guess it could see out the windows). Whenever we would stop, however, the checklist would appear in a fixed location within the display. This gave me hope that the HoloLens would function on the jet.

For the flight, I was only authorized to wear the HoloLens during the first three parabolas. Patil-Sabale and I are hoping to do more research on the device in the future, but the three parabolas did give me a good sense of how the device would work in space.

Before I tell you what happened, I need to explain how parabolic flights work. Each parabola begins with a steep climb at 2G. The G forces glue you to your seat and make things feel a lot heavier than they really are; even lifting your arms requires effort. At the top of the climb, the aircraft pitches down and descends at the same speed as gravity (9.81 M/S^2). It is during this descent that everything on board becomes weightless.

With that said, the HoloLens clearly was not designed for use under G load. During the 2G portion of the parabolas, the on-screen checklist could often be seen spinning. Sometimes it would fade to green, then red, and disappear completely before reappearing a few seconds later. The checklist would also sometimes spin the opposite direction.

Over the last few years, I have flown on a number of parabolic flights and have become sick on about half of them. Wearing the HoloLens during the 2G portion of the flight took what was already a potentially nauseating experience and made it far worse. I became nauseous by the second parabola because the visuals that I was seeing through HoloLens did not match up to what I was seeing and feeling in the real world. For example, the HoloLens made it seem as though I was spinning upside down, but I could clearly see and feel that the plane was not rolling.

It is worth noting, however, that the HoloLens may not always behave in this way while under G load. I was flying in a rear-facing seat, and there is a good chance that the HoloLens would have behaved differently if I had been facing forward. The HoloLens' behavior with regard to the orientation of the G force is something that will need to be researched in the future.

As horrible as the experience of wearing the HoloLens during the hyper-gravity portion of the parabola might have been, the HoloLens worked surprisingly well in zero gravity. It kept my checklist right where I needed it. The experience was not 100 percent perfect, but it was certainly good enough that I would consider using the HoloLens in zero gravity to be practical.

On a side note, I was a little bit concerned that the HoloLens would float off of my head, especially given that I had to wear it over a set of aviation-style headphones. Surprisingly, the HoloLens did not budge.

Perhaps the best testament to the HoloLens' ability to cope with zero gravity is something that happened during the first parabola. Several items that had been stowed in a seat-back pocket in front of me floated out and I suddenly found myself having to catch half a dozen objects before they could float away. This involved lots of head movement and fast motion, and yet the HoloLens never lost track of the position of my checklist, nor did it try to come off of my head. I was really impressed.

Unfortunately, I did not get to spend as much time using HoloLens in zero gravity as I would have liked. Even so, the initial testing results were promising, and I think that there is a realistic chance that the device would work in space. Significantly more research needs to be done, but I am optimistic of the device's potential. If anyone at Microsoft would be willing to contribute a couple of HoloLens devices to support future micro-gravity research, it would be immensely helpful and greatly appreciated. The next time around, I will simply close my eyes during the hyper G portion of the parabola so that I don't have to endure the nausea-inducing spinning windows.