Posey's Tips & Tricks

Lessons from Putting Microsoft Mixed Reality to Work, Part 1

Brien uses Microsoft's mixed-reality technology to take some 360-degree videos, and comes away with a couple of pointers.

• By Brien Posey
• 07/24/2018

• Read Part 2 here.

Several months ago, while working with Microsoft's Mixed Reality Portal, I stumbled onto a couple of apps that immerse you within 360-degree video.

If you aren't familiar with the concept of 360-degree video, it is essentially a spherical video format that places the viewer in the center of the sphere. By using a mixed-reality headset, it is possible to look up, down, left, right and even behind you as the video plays. Not only does the video surround you in every direction, but it is also rendered in 3-D, which makes the experience all the more real.

When well-done, the experience is extraordinary. It really gives you a sense of being there, and it is easy to get so caught up in the experience that you forget where you are really at. It is as though you become part of the video. One of the better introductory 360-degree video apps is Amaze 3D Video.

At around the same time that I began experimenting with 360-degree video in a mixed-reality environment, I found out that I had been selected to participate in a descent, landing and post-landing simulation onboard an Orion space capsule. The simulation required the flight crew (myself included) to put on a spacesuit, get strapped into the capsule and work through a simulated re-entry, splash-down and eventual egress from the capsule.

One of the things that made this particular simulation so unique was that the splash-down portion was real -- well, sort of. During the re-entry phase of the simulation, a crane picked up the capsule (with the crew inside) and carried it to the middle of a large swimming pool. When the simulation reached the point of splash-down, the crane lowered the capsule into the water. From inside, I could feel the jolt and hear the water splashing on the sides of the vehicle.

Normally, when a space capsule splashes down in the ocean, the astronauts stay inside of the capsule until a recovery team arrives. After all, it's safer to be inside a sealed capsule than it is to be outside in the open ocean. In our simulation, however, various emergencies prompted the need for an evacuation, either through the top hatch or through the side hatch. I won't bore you with all of the technical details, but let's just say that swimming becomes a lot more awkward when you are wearing a spacesuit, and climbing into a life raft while wearing a spacesuit that has taken on water is extraordinarily difficult to do.

Throughout my commercial astronautics career, I have (whenever possible) recorded videos of the training exercises because I like being able to show my family what I do. Given the uniqueness of the Orion simulation, however, I thought that it might be cool to record the entire experience in 360 degrees, and then replay it in a Microsoft mixed-reality environment. But while I was successful in my efforts, I found that creating the video was far more challenging than I expected it to be. As such, I wanted to share with you some of the lessons that I learned along the way.

First off, camera selection is everything. It is critically important to match the camera to what it is that you are going to be recording. In my case, I was going to be recording video inside of a space capsule (although I mounted a second 360-degree camera on the outside of the video, and had a third 360-degree camera in the water). The reason why that little detail is so important is because every 360-degree camera has a minimum focal length. That means that there is a minimum distance at which the items in the frame will be in focus. Although the Orion capsule is far more spacious inside than other capsules such as Apollo or Soyuz, I still had to find a camera that had a relatively short focal length. In case you are wondering, I went with the GoPro Fusion. The fact that the Fusion is also waterproof was definitely a bonus in this situation. It is worth noting, however, that the Fusion is not a 3-D camera.

Another lesson learned is that the camera's resolution matters -- a lot. I would even go so far as to say that getting the highest possible pixel density is essential to having a good experience. There are several reasons for this. The first reason why resolution is so important is because of the difference between a standard camera and a 360-degree camera. A normal 4K video camera can record 3,840 horizontal pixels by 2,160 vertical pixels. Those pixels are displayed on a flat screen. In the case of a 360-degree 4K camera, however, those same 3,840 horizontal pixels are distributed across a 360-degree sphere. In other words, each degree of arc has a mere 10.7 horizontal pixels. Vertical resolution can be even worse. If a 360-degree camera has 2,160 vertical pixels spread around 360 degrees of arc, then there are only going to be six pixels per vertical degree.  

The other reason why pixel density is so critical is because pixels get farther apart as objects get farther from a camera. Remember, a 360-degree camera records spherical video, and as the size of a sphere increases, the distance between two points on the sphere also increases.

Let me give you a really ridiculous example. If you take a round globe and measure the distance between Florida and California, you will probably find the distance to be a few inches. Let's pretend, for the sake of example, that it takes a thousand pixels to cover those few inches with a good amount of detail. Now, consider that the globe is nothing more than a representation of a much larger sphere -- the earth. If you measure the distance between those same two points on the actual earth, you will find that they are thousands of miles apart. Imagine how much detail you would lose if you had to span those same thousand pixels across thousands of miles.

Yes, this is a ridiculous example, but think of the example as it applies to spherical video. The larger the sphere becomes, the greater the distance between pixels. As such, objects that are close to the camera tend to have a higher pixel density than objects that are farther away from the camera.

Let me show you an example. The image shown in Figure 1 is a screen-capture from 360-degree video recorded in the pool during the simulation. Notice in the figure that the space capsule is nice and crisp (at least as consumer-grade 360 video goes). Now take a look at how much less crisp I appear (sitting on top of the capsule). I am farther away from the camera than the side of the capsule. Other objects in the frame become increasingly fuzzy with distance.

To see a more extreme example, check out Figure 2. This image was captured using the same camera as the previous images. Notice that the more distant trees tend to lack detail because the amount of geographic space that has to be covered by each pixel increases with distance.

One more reason why pixel density is so important: Microsoft Mixed Reality headsets tend to have a comparatively high resolution. My Samsung HMD Odyssey headset, for example, has a resolution of 1440x1600 for each eye. At this resolution, any imperfections in the 360-degree video become very noticeable.

The lesson here is to use the highest-resolution 360-degree camera possible. Even so, my experiences onboard the capsule taught me several other things about displaying 360-degree video in a Microsoft Mixed Reality headset. I will tell you what else I have learned in Part 2 here. In the meantime, if you are curious to see some of the 360-degree video from the Orion simulation, there are a few samples on YouTube. Here is what the experience was like from inside the capsule and here is a clip from a camera that was mounted on the outside of the capsule.