Virtual Reality. Augmented Reality. 360 degree video. What does it all mean? Is it another shiny techno distraction (think 3D TV, QR codes, Second Life) or is it the shiny future of storytelling?

I’ll explain, but first you may be asking, what exactly is VR/AR/360? Let’s have a little primer before we dive into the philosophical underpinnings of using these new storytelling methods.

To start, terms. While they may be used interchangeably, they definitely have different meanings. What they have in common is that they provide a partially or completely immersive experience to the participant.

  • VR = Virtual Reality
  • AR = Augmented Realtiy
  • 360 = 360 degree (spherical) video

There is one other term you may hear bandied about – MR or Mixed Reality. A few years ago, the terms MR and AR had specific meanings, but today they are basically a distinction without a difference. I won’t go into the history of the terms for this article, but I will say the term Mixed Reality has more or less been absorbed into the term Augmented Reality.

 

Virtual Reality

The idea of sensory immersion to the degree that it alters a viewer’s perception of reality is as old as moving pictures. Who remembers Sensorama? Although the term Virtual Reality has been around for a couple decades now, it is buzzing around many industries as of late. The idea that technology can transport our minds to another place, like the Star Trek Holodeck, until recently existed only in the realm of science fiction. However modern developments may give us the technology sooner than many might think.

Virtual Reality as we know it today, is the experience of an immersive world using special goggles, generally called a VR headset. There are two broad categories of headsets – tethered and mobile.

Tethered means the headset is connected to another device, such as a computer or video game console, by a cable, which feeds the picture and sound to the headset. Tethered headsets offer the highest quality experience due to their use of dedicated hardware and software. Sony PlayStation VR , Oculus Rift, and HTC Vive are examples of popular tethered headsets.

A mobile headset depends on a mobile phone to drive the picture and sound. Mobile headsets come in many styles and require the viewer to insert a mobile device, typically a smartphone, into the headset, along with headphones. The viewer then uses an app on the device to access the content. A few mobile VR headsets are designed to work with specific devices, such as Samsung’s Gear VR (designed to work with the Galaxy phone), or the Google Daydream (designed to work with Android phones), but there are hundreds of generic designs that can be used with any phone.

A true Virtual Reality experience is created inside the device that powers the headset. The experience is generated in real time as the user interacts with the software, much like a computer game. When a participant moves his or her head, the VR software  moves the scene with it, adjusting the point of view and re-positioning the audio sound field in sync with the head movements. This gives the sensation that the viewer is actually in the environment.

 

Augmented Reality

AR is similar to VR in that it projects software-generated virtual images into the participant’s field of view in real time and reacts to head movement in the same way. The difference however, is that augmented reality keeps open a window into the real world – images are overlaid into the real world and react with the real world as the participant moves and interacts with the content. AR is projected onto wearable devices that are generally referred to as smart glasses, although you can also view AR content on the display of a phone or tablet without any special hardware. Smartglasses project imagery directly onto the lens glass in a variety of ways, but the effect is basically the same.

Here is a fun example of AR in action.

There are many models of smart glasses, including Microsoft’s’ HoloLens, Sony SmartEyeGlass, Epson Moverio, and many others, including the ill-fated Google Glass. The consumer version of Google Glass was discontinued years ago, but has been reintroduced as an enterprise product. It is rumored that Apple is going to introduce a pair of smart glasses at some point in the future. Whether Apple rolls out a product or not, they recently introduced a very impressive AR toolkit for iOS app developers, effectively putting the tools for creating AR apps in the hands of Apple’s millions of eager developers.

The most famous current example of AR is the Pokemon Go game, which took the world by storm last summer and continues to be one of the most popular games in the world.

360 Degree Video

360 refers to linear video footage recorded with special cameras, or created by software to play back inside a spherical environment. Multiple physical or virtual cameras are arranged in a circular array and special software is used to stitch together the footage into a flattened sphere that is then unflattened inside a 360 player. A viewer may use a smartphone screen or a headset to view a 360 video, and like the other formats, the video moves with your head motion, giving the sensation of immersion. 360 degree video can be found on websites, such as YouTube and Vimeo, as well as within dedicated apps like Jaunt and Within.

360 video is a passive format therefore it does not afford the same level of interactivity found in VR and AR, where the content can be altered by input from the participant.

 

Ambisonic Audio

One infrequently discussed, but still important aspect of VR/AR/360, is audio. Most of the audio in the world is recorded and played back in a stereo format, consisting of two channels containing audio for the left and right ears. Stereo simulates the way sound behaves in the real world by allowing a listener to localize sounds in a horizontal plane through a pair of speakers or the two sides of a pair of headphones. Surround sound takes this a step farther by utilizing  more channels, adding multiple front, rear, and center sound sources. Surround sound formats are designed to expand the sound field, which gives the aural illusion of 3-dimensional sound, but this is not true spherical sound since it relies on the listener’s head to be fixed in space. Surround sound is achieved in movie theaters and home theaters using arrays of speakers placed in a pattern around the room. The effect can be dramatic but it is not exactly convenient to use with a smart phone or headset.

Luckily, there is a format that enables true 3-dimensional sound to be experienced in headphones: Ambisonic audio.

The Ambisonic recording format has actually been around since the 70’s, but has never come into widespread use, although the rising popularity of all the tech we are discussing here will certainly inject some momentum into its use. The format has survived due to nerdy hi-fi hobbyists and sound engineers experimenting for their own personal gratification, but I believe it will become much more important as AR/VR/360 gain widespread adoption.

The format encodes four channels of audio that represent sound coming from all directions. Ambisonic audio needs to be decoded in order to be played back correctly. In the context of VR/AR/360, as the playback mechanism decodes the audio, it alters the sound mix so that the sound source localization changes with your head position. Think of it like this. If you were in a movie theater and heard a sound source coming from the right side of the screen, say a small jazz combo playing on stage in a noisy club, your brain perceives the sound as coming from the right. If you turn your head away from the sound source, the music doesn’t change, even though in a real club with a real jazz combo, it would sound totally different as you move your head or move around the room. With ambisonic recording, the jazz combo, the room acoustics, the conversations around you, are all recorded and encoded. Then when played back in a 360-degree environment, the decoder remixes the audio in real time based on your head position, resulting in a realistically immersive experience.

 

Now, back to the original question…

Are they a fad or will they become a fixture in modern storytelling?

My opinion is that VR/AR/360 are here to stay. And I mean all three of the formats, because they all have distinct and viable uses. Many of the biggest companies in the world – Facebook, Google, Apple, Microsoft, Samsung, etc. have invested way too much money into these technologies for them not to take off. Yes, there are particular issues and idiosyncrasies with each format but they will be overcome. The idea that the public will someday be walking around the streets of our cities wearing huge goggles, immersed in some computer generated playground and completely tuned out of reality, unlikely. On the other hand, 30 years ago, television was beamed through the air and received by an antenna, while phones were hard wired into your house. Today, my kids don’t even know where live broadcast, Apple TV and YouTube begin or end, let alone care if they see it on a TV, phone or iPad. Tomorrow, physical wires may be as hard to find as a pay phone is today. The future might even look like this!

The point is, things change, sometimes very fast. Because much of this technology is in its infancy, I don’t think we can even imagine all the ways it will be harnessed in the future. Of course many will speculate and experiment – there will be winners and losers, but this new medium has the power to radically change the way we tell stories.

In what ways you might ask? Stay tuned…