The Audience Enwrapped: Prototyping 360 Video as an Immersive Story Form

The virtual reality (VR) showcase held behind the festival main stage felt like a scene from William Gibson’s prophetic novel Neuromancer. In the dimly lit digital den filled with trendy twenty-somethings, attendees waited in line to hack their minds with headgear displaying digital fantasies. This scene is becoming increasingly more frequent around the world, whether at the pioneer VR festival in Brisbane I recently attended; at SXSW in Austin; or the Ars Electronica Festival in Austria; and it’s only the beginning of a new arc of storytelling innovation.

Emerging disruptive screen technologies offer exciting opportunities for writers, filmmakers, designers and communicators to immerse their audiences within imaginative storyworlds. The introduction of new mediums can be highly democratising for storytellers looking to reach and engage audiences, at least for those who act fast enough.

Computer games have experimented with connected, branching, looping and interactive narratives for several decades, and the internet enables digital stories to be dynamically modified by local, variable contexts. Mobile technology makes instant contextualisation both portable and practical, while augmented and virtual reality allow for the perceptible dimensions of objective space and time to be digitally manipulated, warped and remixed. The convergence of technology and society, what Mark Deuze describes as a liquid Media Life connected through social media, makes sharing these stories almost frictionless.

Collectively, storytellers and creative practitioners are trying to keep up with the possibilities as they change at a rapid pace. In August this year I attended the inaugural Australian Virtual Reality Film Festival. It was illuminating to see professional creatives and filmmakers experimenting within this technology-based art form.

Attendees at the inaugural 2017 Australian Virtual Reality Film Festival in Brisbane

Among the stories showcased at the VR festival included Oscar Raby’s Assent, a Unity-based environment that deals with politics and personal biography, as well as 360 prison system documentary Step to the Line, which received the backing of Facebook co-founder and CEO Mark Zuckerberg as part of its 2017 premiere at the Tribeca Film Festival.

My Doctorate of Creative Industries project will test these fertile media fields in a brand context. I selected 360 video as the medium to deliver a brand story, due to its ability to immerse audiences and its low barrier of entry. As a pilot site I targeted a Queensland University of Technology (QUT) Open Day to deploy the story, in order to drive production and maximise the potential audience.

By wrapping a story around attendee’s heads, I discovered that storytelling has to compete with a novel, mind-bending and out-of-body experience for the audience’s attention.    

The immersion of words, audio and video

Each medium allows for different degrees of immersion, and words can be the ultimate immerors. Through words, writers can gift readers the sight of a thousand lives set across a thousand storyworlds. Readers can feel what it’s like to march on icy arctic snow in the mukluks of an eskimo; roar across the narrow sea as a rebellious queen riding an ancient wyrm; or perhaps emerge screaming into a toxic alien dimension, new life discovering abstract shapes, harsh sounds and strange sensations. Writers create limitless opportunities with their words, which active minds take full advantage of through 100 billion firing neurons. Words can be paused, re-read, processed and contemplated. Through words, our minds can temporarily escape to an alternative existence lightly tethered to this one.

Audio provides a different experience from words. Audio fills the spatial volume of lived life. While audio moves inexorably forward at its own pace, forcing the audience to keep up, it does allow for discussion, debate, and the contemplation of ideas. Think about the back and forth of talkback radio, or Dylan Thomas’ mystical narration of Under Milk Wood. Needing only to engage a single sense, the mind can articulate ideas as the sound is perceived and interpreted. The mind has time and breathing space to think.

This is unlike the combined visual and audio stimuli of video content. Marshall Mcluhan described video as an attention-grabbing ‘hot’ medium. For Neil Postman, video is pure somatic entertainment. With videos, what you see is what you get. This is particularly the case with modern Hollywood blockbusters that fill the screen with vibrant colours and fast-moving blurry shapes, belligerently hammering the audience with a cacophony of noise. Cinematic production values became so overpowering, the cinematic Dogme 95 Manifesto was born in the 90s, encouraging filmmakers to purposefully reduce production and leave gaps in the visuals.

Regardless of how the screen is filled, there is a special kind of magic that takes place when viewing video, akin to experiencing an oneiric, lucid dream. Instantaneously shifting the camera view from one vantage to another erases the tyranny of distance and reveals previously impossible perspectives. The camera placement and editor’s cut provide the magic that hacks the brain. It is a mental teleportation trick. The downside is that as the story progresses, the mind has little time to dwell and revel in the empty space that creativity craves and words provide. The director is always in control, not the audience. Videos can be visually splendid, but pale in comparison to what you are capable of conjuring with your own amazing, vivid imagination.

But how does it feel to inhabit someone else’s mind while inhabiting a video? Can 360 video compete with the immersion of words by masquerading as real life?

Mediated experiential immersion

James Dillon is a content strategist and Doctor of Creative Industries candidate at Queensland University of Technology, Australia

360 is a relatively new form of video production. To capture 360 degrees of video, multiple camera lenses are placed evenly around a spherical object. These lenses simultaneously capture overlapping frames of video that need to be stitched together. Some cameras immediately autostitch the content (my preference), while others need to be manually stitched (not fun!). The ideal way in which to view 360 is through comfortable but tethered headgear such as an Occulus Rift or the HTC Vive. It can also be viewed on mobile phones or computers in a limited way. Products such as the Google Cardboard are a cheap way in which to emulate a headgear experience at a higher volume, although they lack the comfort and clarity of dedicated headgear.

Importantly, working in 360 provides new story design challenges. While hundreds, or perhaps thousands of 360 videos have been produced and shared in the past few years, I’ve discovered few that contain a structured story, at least outside of VR film festivals. Most 360 stories rely heavily on the novelty of the medium, visual spectacle, emotional moments, or are focused on location tours. Discovery VR has a variety of example genres and themes to consider, ranging from a theatrical mystery to journalistic features.

Collaborating with a 360 video producer as well as a scriptwriter, we set out to create an authentic brand story centred on a student’s artistic journey. We soon found there were unique challenges imposed by this immersive medium.

The dos and don’ts of 360
Firstly and most obviously, there are now 360 degrees across X and Y axis’ from which the audience can observe. The audience is now inhabiting the story, but there is no guarantee that the audience will look where you want them to. There is also no certainty that what the viewer is listening to will match what they are seeing. Therefore, most of the hard work takes place before filming begins.

Scenes need to be ‘blocked’ out in the same manner as theatre production. Locations have to be carefully chosen to be as visually stimulating as possible. Scenes can be sparse with central elements of interest to help arrest the eye, but they can’t be too devoid of interest. Otherwise, what’s the point of using 360? Transitions between scenes should be smooth, and there can’t be too much quick camera movement as it can be disorienting. The audience’s attention needs to be purposefully directed. How close the characters are to the camera, and the height of the camera, have a drastic affect on the viewing experience.

A 360 scene from episode 2 of Journey to Ars

There’s no close-ups or clever edits to get you out of trouble, which every 2D video editor has relied on in their career. The sneaky jump cut, the overlay, the fade. Instead, new techniques and mindsets are required. Jessica Brillhart describes the experience of an established editor incorrectly editing a 360 video, provoking the need to create a new professional method of visualisation, that of the world-within-a-world, layers of experience.

I should mention that I now refer to normal video as 2D video in order to distinguish from 360 video. I’m not sure what terms others use, but I’m sure there will be a few competing concepts that emerge. Such is the change in frame of reference that comes from a new art form.

One of the greatest challenges our team faced was narration. Does the character acknowledge and talk directly to the camera; do the characters converse while the viewer observes; is the scene mute and the story driven by narration; or is it a mix of the three? For our story, we chose for the narrator to observe the scene alongside the audience, commenting on what they saw, from which we dipped into scene audio from time to time.

Our initial draft had too many short scenes and too much narration. Every spare second of the draft was filled with the voice of the narrator. The words were formal and abstract, discussing high falutin ideas such as artificial intelligence and the nature of art. We found the audience often had to make a choice: listen to the audio or look around at the visuals. Doing both became exhausting for some, impossible for others. The narration of our final version was drastically simplified, and whenever possible, we tried to have the narrative reference the scene being observed. We discovered that by using 360, we almost had to contract into a childlike state of storytelling in order to help the viewer acclimatise to the added dimension. This was not the medium in which to assault the viewer with a barrage of narration.

Providing the immersive experience
And so with the final draft completed just in time for our campus Open Day, prospective university students and parents were able to step into the mind of a student dancer. The three-minute 360 brand story, Journey to Ars, the first of a three part series, is narrated by postgraduate student and performer Jacob Watton. The story drops the viewer into several locations at QUT and the Australian city of Brisbane, including an interactive display centre, our robotics workshop, and by the Brisbane river. We used Occulus Rift gear tethered to an Alienware laptop to present the story, cleaning the headgear with wet wipes in between viewers.

Some Open Day participants had to be encouraged to try the headgear, while others pushed their way forward to put it on. Reactions were diverse, but overwhelmingly positive. They described the experience as “cool”, “amazing”, “surreal”, “trippy”, “weird”, “crazy” and “spectacular”. Many came out of the headgear with a wide-eyed, distant look. For those precious three minutes, they had forgotten about their existing world and had been taken elsewhere. Many complained that the commotion of the people around them intruded into the experience. As the number one complaint from the test, I was happy with the result.

In general, I found that the technology itself was the focus for the audience, particularly for those who had not experienced 360 video headgear before. As time goes by, audiences will be less wowed by the technology and demand more from their immersive 360 experience.

The first episode of the creative experimentation is available on Facebook or YouTube. The Facebook post has had the most organic reach of our Faculty’s account in 2017, reaching more than 10,000 people.

The story’s project team also included QUT School of Creative Practice academic Cody Jarrett, as well as 360 video producer and IVD graduate Jessie Hughes. We have taken the learnings from the first episode to inform production of thesecond episode, centred on QUT’s Robotronica festival event on 20 August. The third episode will be filmed at Ars Electronica in Linz, Austria, in September.