Live theatre performance in XR (extended reality), which includes both Virtual Reality (VR) and Augmented Reality (AR), is a rapidly growing field of digital drama that sits at the convergence of immersive theatre and interactive games design. While a number of theatre companies have adapted their existing work for VR, (notably the Royal Shakespeare Company with Dream Online in March 2021), our research project, Mrs Nemo XR, aims to create new works of musical theatre specifically written, designed and built to utilise the interactive and immersive features of this exciting new performance medium.

Mrs Nemo XR

Our initial project, a short-form immersive musical — which we term an ‘Immersical®’— is inspired by an episode from Jules Verne’s classic Victorian adventure story, Twenty Thousand Leagues under the Sea (Verne, 1, featuring the attack of Captain Nemo’s Nautilus submarine by a giant sea creature, the mythical ‘Kraken’. Adding a contemporary twist to the tale, the story is retold from the perspective of a mysterious female character in a Victorian bath chair, who appears at the outset to be Nemo’s wife, but who is later revealed to be a mermaid, one of the many marine creatures that Captain Nemo has collected for the undersea museum which we use as the setting for the show. 

Although we draw on Verne’s original story for the undersea environment and basic plotline, we have devised our own narrative to tell the tale from an alternate viewpoint; that of one of the sea creatures whom Nemo has captured. As the audience members join the show, they find themselves embodied as individual avatars of deep-sea divers, who have been transported into the submarine, and invited by the curator (Mrs Nemo) to explore the gallery displaying the museum artifacts. As Mrs Nemo tells her own tale through improvised dialogue and song, the audience can interact with the actor by talking to her avatar and examining in 3D some of the items in the museum. 

VR Platform

Staged on the browser-based online social VR platform Mozilla Hubs, accessible  either via desktop computer or VR Headset, Mrs Nemo XR is performed by a solo actor working remotely from home, singing live to pre-recorded music tracks. We are limited to one live singer, due to audio latency issues across the network which prevent multiple performers singing in sync. The show also has a short run-time of ten minutes due to concerns of how prolonged use of VR may adversely affect actors and audience members, especially new users of the technology.

Performances 

Our initial trial run of twelve performances with a live audience began in October 2021 and was staged in collaboration with an informal collective of virtual theatre-makers from around the world under the title OnBoard XR. While the whole event shared an underwater-themed virtual environment, each team designed their own show, including the set, props, avatars, and individual backstage cueing system. This article details the technical, design and production challenges we encountered while making Mrs Nemo XR, with specific reference to the narrative design and scenography of the production.

Performer singing in a VR Headset, while moving her own avatar in VR

Creative Team

This virtual theatre production, as with a typical physical theatre company, has a creative team composed of Producer, Writer, Director, Designer, 3D Artist and Performer; all theatre practitioners and creative technologists, several of us conducting research at Digital Creativity Labs, at the University of York, UK. Throughout the four-week production period in the autumn of 2021, team members worked remotely from different locations across the UK and the USA. Pre-production meetings, rehearsals and performances all took place online, managed through a combination of VR, video conferencing and text and audio messaging. Many of the sessions were intense early morning or late-night events, due to the complexity of distance working across time zones.

Interactive Narrative Design

Mrs Nemo XR, aims to engage the audience with the narrative, while giving them the freedom to explore the immersive submarine environment and interact with the performer and each other. Without a traditional framing device, such as a proscenium arch or cinema screen, we rely on alternate ways of directing audience focus, such as guided direction and environmental storytelling. Our audience, who were represented by avatars of deep-sea divers, were actively encouraged to be part of the story-world, through improvised dialogue and narrative song, and the way in which the actor directed their attention to the interactive props, such as the toy Kraken that could be picked up, moved and resized, and scenic effects like the sudden and dramatic appearance of the giant sea monster outside the viewing window of the sub.

Technical Challenges

One of the main technical issues we encountered when using Mozilla Hubs with our Meta Quest 2 (previously the Oculus Quest 2 VR headsets), is the limited audio capability afforded by this form of Web-based VR (a VR platform that is easily accessible on multiple devices via the Internet). While we did have the capability for spatial audio, we elected instead to have our performer’s voice override this feature, so that she could be clearly audible, regardless of position in relation to the listener. Since we knew that audio and visual latency would be problematic, our actor ran the music tracks from her laptop, whilst performing in a VR Headset and manipulating her own avatar, thus syncing all her actions at the point of transmission. 

The scenic cues, including the change of avatar for Mrs Nemo for the final reveal that she is a mermaid, and the attack of the Kraken, were also subject to delay of operation across the internet, and so were run remotely by the backstage crew to avoid distracting the performer by the desynchronised scene changes. This also enabled the production team to make live adjustments during the show when latency of operation necessitated the removal of used props from previous scenes or by ‘respawning’ the placement of set pieces that had entered the scene at the incorrect size or location.  

Scenography 

The inspiration for set design, props, avatars, and interactive artifacts were both derived from Verne’s original story, and from the writer’s scripted dialogue and lyrics. From the book it was clear that while the Nautilus did have a library and salon with a large viewing window, there was no dedicated space for storing or displaying the Captain’s undersea collection. This led to the idea that the salon should become a museum, its walls filled with books and artifacts relevant to the story. Using Mozilla Hubs’ custom editor, Spoke, which allows developers to create 3D worlds with their own designs or pre-made assets, we built the submarine interior, and using a bespoke plugin Stage Management System, we were able to program cues to make scenic objects, props and avatars appear, animate, and change size and position within the scene. 

Tech rehearsals in Zoom and Hubs showing the cues of the stage management system

Working in Mozilla Hubs

In order to perform well across a wide-range of devices, Hubs has certain operating limitations, such as the inability to move or ‘spawn’ objects easily, and a maximum scene size of 16MB, which resulted in our immersical taking on a low-poly (an art style where the number of polygons in a 3D model are reduced to give a low quality appearance to the resulting graphics. This style has the benefit of being efficient and easy to optimise)

style of graphics. Because of internet latency, we could not accurately predict how the scene would appear on the various viewing devices that would be subject to differing connections, internet speeds and locations. For example, we aimed to trigger the Kraken’s appearance outside the viewing window, as a shocking reveal to the audience, especially for those in VR who would see a gigantic monster attach itself to the side of the submarine. However,  the physical distance between the stage manager in York (UK), the performer in New York (US) and the worldwide audience, meant that the significant delay between cue and action led to some audience members, depending on the efficacy of their own devices and connection, experiencing a slight delay between what the actor was saying and what the scene was showing. 

Set Design and Layout

Our initial design plan for Mrs Nemo XR was to divide the submarine into smaller clustered areas to direct audience movement, (see Figure 3) to where the action in the scene was taking place. We particularly wanted to guide the audience towards the window to provide a good view of the Kraken’s sudden appearance outside the submarine. During rehearsals however, we realised that the audience tended to cluster around the performer and follow her around the scene. This led us to strip back some of the central blocked-out areas to provide a clear line of sight throughout the submarine interior, so that the performer and the action could be seen at all times.

Preliminary design of the submarine interior

Sub interior with Captain Nemo’s portrait on the wall

The eye of the Kraken at the window

Audience member as diver with props

Conclusion 

In spite of the technical limitations of the Mozilla Hubs platform, in particular the external cueing system, the low-poly graphics, and the audio/visual latency experienced, the show was well received and we felt we had achieved our central aim of creating our first fully functioning immersical. Through the rehearsal process and by observing the performances, we established methods of directing audience focus and promoting their engagement with the narrative, in three key non-verbal ways. Firstly, through the use of scenic devices like the viewing window, to frame the attack of the Kraken, and secondly, in spite of internet latency, we were able to sync actions to the music, cueing movement of scenery and interactive props at appropriate times. Thirdly, we found that  guided direction from the performer (e.g. gestures, movement, following her gaze and use of interactive props), supported audience attention and engagement with the story-world. 

Our research continues with a new iteration of the show, where we will be using a different VR platform, and extending the length of the performance, working with innovative techniques for reducing audio latency. We also intend to offer greater user agency and interactivity, and explore different styles of narrative design including non-linear and object-based storytelling. We see a great future for live performance in XR, particularly in terms of ease of access and affordability, and look forward to our next collaboration within the rapidly expanding community of VR theatre-makers and their virtual audiences.

Bibliography

RSC Dream online. Audience of the Future Live. (n.d.). Dream. [online]
RSC Dream online. Latest Press Releases | Royal Shakespeare Company.
Mrs Nemo XR – performance video #OnBoardXR [SHOW3.3] Live Short Series in WebVR.
Verne, J. Twenty Thousand Leagues Under the Sea. (first published in English, 1872). Project Gutenberg.

Credits

Mrs Nemo XR Creative Team: Director: David Gochfeld, Writer: Mary Stewart-David, Designer: Daniel Lock, Exec Producer: Cristobal Catalan, 3D Artist: Guy Schofield, Performer: Vivian Belosky
OnBoard Stage Management System developers: Roman Miletitch, David Gochfeld, Clemence Debaig, Michael Morran.
Digital Creativity Labs & Department of Theatre, Film, TV and Interactive Media, University of York

I love when I am working on a project that involves binaural sound with someone who hasn’t heard binaural sound before. Handing them a pair of headphones, playing them something and seeing them physically react by touching their head to make sure that the invisible electric razor buzzing near their ears isn’t really giving them a new hairstyle or turning, confused, to find the person speaking behind them and then bursting into surprised laughter when they realise there is no one there. They always seem to have that moment of confirmation, that visual or tactile check to make sure that what they are experiencing is an illusion and nothing more, and that they shouldn’t, though they are inclined to, believe their ears after all.

Recorded using two microphones in the ears of a real person or a fake ‘binaural head’ – that is, a fake human head shaped out of wood or another material – and played back through headphones, binaural sound replicates the way our ears really hear and thus allows our minds to spatially orientate the sound in relation to the listener. For example, a bee buzzes around your head, a car whizzes by, someone whispers into your ear causing your skin to prickle. It’s more than recorded sound, somehow. It feels real and live, like it’s happening around you in the physical world.

Aaron Hussain recording with Binaural Head he built for Sleight of Hand Theatre (Photo by Ellie Chadwick)

 In my work as a theatre practitioner experimenting with form and technologies, I’ve long been fascinated with theatrical uses of binaural sound. Relatively easy to produce, and even easier for audiences to access (even in a pandemic as all you need is a pair of headphones), binaural sound is arguably the simplest yet most effective form of technological immersion out there.

 But what immediately comes to mind if someone says ‘immersive technologies’? Perhaps Virtual or Augmented Reality, complex motion capture set-ups, or haptic feedback gloves. While these are all exciting tools that can create incredibly immersive experiences, leading us into another world and altering our sense of reality for a time, binaural sound’s ability to trick the brain into believing it is hearing real-time, three-dimensional, live sound is – I would say – just as effective at producing a powerful sense of alternate reality for audiences. Potentially even more effective, since it relies largely on the power of our own brains and imaginations in order to ‘fill the gaps’, and as we know from genres such as Horror Film, what isn’t shown is often more powerful (in Horror’s case, scarier) than what we see. Also, as neurologist Seth Horowitz points out, ‘one of the most important drivers of emotion is sound. And the reason it’s so important is because it works underneath our cognitive radar.’

 This kind of unconscious stimulation combined with the power to imagine makes for a powerful immersive tool. The power of imagination is, of course, intrinsic to the theatre, where audience and performer take part in a ritual contract whereby disbelief is suspended for a while and they buy into another reality. But in binaural sound, the other reality is arguably more than a conscious suspense of disbelief, it’s a deeper illusion that plays tricks on your mind. In its most effective moments, it can make the listener react completely naturally and instinctively to a given stimulation.

 In a performance I attended of David Rosenberg and Glenn Neath’s Ring at Battersea Arts Centre in 2014 (a piece set entirely in the dark, using binaural sound throughout) there was a moment where a character in the recording whispered in the collective ear of the audience, and several audience members screamed in genuine surprise. The effect was so strong that it was as if you could physically feel the character’s breath on your neck. The audience’s sense of truth was so distorted that such a moment felt entirely real.here was only one way to be separated from this new sense of reality and that was to remove your headphones and confirm, or realise, that there was no one there whispering or lurking in the darkness.

The physical nature of binaural sound is arguably what makes it so effective an illusion. In my academic research, I term this sensation ‘audiosomatic’, in that the technology is able to produce a reaction in the body that is highly physical. In fact, there are hundreds of videos online using binaural sound to aid relaxation by inducing a physical reaction; an autonomous sensory meridian response (ASMR), the experience of a static-like tingling sensation on the skin, often the back of the neck.

Since the work of creating the illusion is completed in the brain itself, utilising the way that we experience sound in everyday life, binaural becomes a particularly powerful tool in relation to the idea of creating a sense of ‘truth’ in fake perception i.e., in storytelling. Possibly this is why it is being used more frequently by experimental theatre practitioners.

Of course, during the pandemic, binaural and spatial sound are great tools for immersing audiences in stories while in their own homes. Various theatre companies and makers have been utilising binaural, often alongside other digital content, to reach audiences at a distance, such as Limbik Theatre with their audio-visual piece The Garden, Darkfield with their Darkfield Radio strand of work, and my company Sleight of Hand with projects Ergo Sum (an interactive experience about neurodiversity and mental health) and Silhouettes (an R&D project about the life and work of Ana Mendieta).

Oliver Church in Ergo Sum (Photo by Thom Buttery)

But while the technology lends itself well to these recent circumstances, it is not primarily the pandemic and social distancing that has provoked the desire to utilise binaural sound in theatrical storytelling. Our Ergo Sum online project has been adapted from a theatre show where audiences wore headphones and experienced haptic sensations and projection-mapped visuals to tell the characters’ stories in a live auditorium with performers. Darkfield, founded by Ring creators Rosenberg and Neath, began with binaural experiences in dark theatre spaces.

A particularly lauded example of binaural theatre from pre-Covid is The Encounter (2016) by Complicité. This one-man show takes audience members on an incredible journey through the Amazon rainforest using a combination of live and recorded binaural sound. The effect of the sound design on the imagination of the audience members is so visceral and ‘real’-feeling, that a family member of mine who had experienced the performance early in its run was surprised some months later when coming across the trailer online, since in her mind’s eye she had remembered the set as having trees and being much more naturalistic than a bare stage littered with wires. She had visualised a depiction of the Amazonian forest setting so vividly that she had half-forgotten it was created primarily through the immersive sound design and the power of imagination.

There was a similarly vivid impact on the imagination created by Ring, in which, as mentioned above, listeners hear a story told in binaural while sitting in complete darkness and wearing headphones. The audience sat in several rows on either side of the room, facing each other across a traverse stage. A character with crutches welcomed us in before the lights went down, his live audio melding seamlessly into a binaural recording which felt like a continuation of the live sound; the noise of his crutches walking around the room, the live rustling of the other audience members blending perfectly into a recording of the same. Once it was dark, the character asked us to rearrange our chairs.

Some nervous laughter was heard as everyone considered how to do this in the darkness. Then, everyone began to tentatively move around, chairs scraping and people negotiating obstacles in the dark, and a whisper in my ear told me that I was ok and I could stay put. The sounds around me revealed that other audience members had managed to move into a circle and they began talking, and the story commenced from there.

Of course, none of this was really happening. In reality, each audience member heard the same sounds, each person in the room was told they could stay where they were, and each one felt the circle being arranged around them. In my imagination, however, the other real-life audience members became the characters in the story. The whole room was rearranged and I was convincingly transported into a parallel, yet totally different, space and reality. It was so convincing that the belief manifested itself viscerally, in physical sensation. There were moments that sent a tingling shiver down my spine, and other times where the audio and utterly-dark environment led to the feeling of being able to almost see something: an experience verging on visual hallucination.

It is likely due to this illusionistic thrill and the way that binaural taps into the power of imagination that theatrical and artistic ventures utilising binaural technology are growing in number. They move beyond theatrical buildings, as well, into the streets. For example, Rosenberg’s Monument (2015) was an immersive piece taking place in London’s streets, combining binaural with live action integrating with the hubbub of the general public around you. Lavinia Greenlaw’s Audio Obscura (2011) took place at Kings Cross St Pancras, where audience members would wear headphones while wandering around the train station, hearing stories in the crowds.

Reviewer Kirsty McQuire reported that the experience of Audio Obscura was ‘like a kind of hypersensitive trance, induced by […] atmospheric sound’, saying that it took a while to come out of this heightened state of mind fostered by the work. McQuire also commented that the piece encouraged the projection of empathy onto the scenes witnessed, as you imagined the real-life people around you being the ones whose stories you could hear snatches of. Again, each audience member felt themselves placed more at the centre of the experience than is perhaps usual; an experience that was in large part created by their own interpretation of and response to the sensory stimuli around them. 

In 2019, The National Theatre gave binaural a try with the innovative show Anna, a tense political thriller where audiences wore headphones which gave them intimate access to the scenes, hearing from the character Anna’s perspective. Whispered conversations, overhearing snatches of information, and moving auditorily from room to room in the set enhanced the thriller experience by revealing information piece by piece. This gave a sense of intimacy and the sense of experiencing the storyline’s ups and downs from the perspective of the character at the centre of it all. In a similar manner, our production of Ergo Sum used binaural to give audiences a sense of being “inside the heads” of neurodiverse characters, including one who experiences autistic sensory overload, another struggling with flashbacks and trauma, and one who is having schizophrenic auditory hallucinations. By presenting the world from their perspective, the show promotes a sense of intimacy with the narratives and helps foster empathy and understanding.

As immersive technologies and experiences continue to grow in popularity, in a time when people are feeling more and more disconnected from each other with a lack of community in many areas, the polarising effects of social media, and recently social distancing taking its toll on collective mental health, it makes sense that audiences are gravitating towards illusory sensorial experiences that have the ability to transport and reconnect.

Whether sound or smell or touch, we relate to the sensorial on a deep and personal level.sensorially-immersive experience which works underneath our ‘cognitive radar’ encourages a particularly distinct feeling of emotional truth and an instinctive understanding, which – illusion or not – can impact us in much more powerful, personal, and memorable ways than any intellectual comprehension or logical truth. The memories of the Amazonian rainforest from The Encounter, of whispers and the moving chair circle in Ring, and of neurodiverse perspectives of the world in Ergo Sum, all are stored in the body via sensorial stimulation and so have a particular resonance or ‘ring of truth’ to them. 

In a way, there is little choice but to believe your ears with binaural sound, since its very workings depend on an illusion produced in your own brain. Even hallucinations have their own reality and in the performance of Ring on some level the chairs do move.

Examining immersive tools such as this lead us to consider new questions around the oft-discussed topic of the nature of truth when it comes to theatrical performance and storytelling. When does an illusion, representation or simulation of something become as, or more, real-feeling than actuality? What is the benefit of such experiences? What real effects, such as true empathy or a heightened emotion, can synthetic or pretend environments and experiences produce?

Recent developments in the field of cognitive science discuss the idea that narrative does not only represent but also constitutes reality. We can access truth and a sense of realness only through narrative understanding and, therefore, the act of understanding itself is our reality.  So, when the boundary between illusion and reality becomes blurred within the context of a story, as the popularity and power of theatre itself attests to, this is arguably when deeper understanding of a subject can be nurtured, and discoveries about the self and our emotional truth (both individual and collective) can be made; all via the embracing of a constructed environment and a good illusion.

I love location-based promenade theatre. I feel immersed in the narrative, surrounded by actors, music and lights. One of my favourite things to do is see if I can break my own suspension of disbelief. When I am aware that I’m not fully engaged with the theatre, and I am in that state of 4th wall curiosity, I actively explore the space and look for the cracks in the set and the gaps in the telling.

The best performances I’ve experienced have rewarded that exploration with small treasures for my wandering eyes. Well placed props, lighting and narrative hooks from eagle-eyed actors catch me exploring and successfully reel me back into the story.

VR Live performance: Tender Claws’ ‘The Under Presents Tempest’

Writing for XR requires writing a detail for every moment the audience exists within the narrative. It’s world-building for people like me who are always looking around, actively discovering their place in a story, instead of just watching it unfold. 

XR means ‘Extended Realities’. It’s a catch-all term that can describe experiences that use virtual or augmented technologies to extend our reality. It can also encompass experiences such as promenade theatre, which use various technologies and techniques to deepen our immersion. Writing for XR is still an experimental endeavour, rules are grafted from other industries and new lessons emerge with each new project.  

Here are 5 tips that may help you write stories for experiences where the audience is always present.

1. Build what feels like an entire world

Build a whole world, not necessarily in minute detail but in the sense of the audience’s perspective, their relationship to the space and characters around them and the wider contexts or laws of the world. The core narrative you’re writing for the experience is just a pathway into the story world for the audience and needs to feel like a living thing.

Ask yourself as many questions as you can about the world you’re building to help you.

For example; When the audience turns their head and they see a fridge behind them, are there any post-it notes on it? Can they open the fridge? Can they open it easily? How can you make the character and contents of the fridge a part of the narrative for the audience? Is there food in there or a penguin dimension? What does the penguin say if the audience prods him? 

2. Be present in the process

Just as your audience, who will be actively present in your story, you also need to be actively present in its creation. 

Narrative isn’t a silo in this production process and you shouldn’t expect the story you’re writing to be the only source material. XR is sometimes a full-body, multi-sensory experience where narrative will be processed by more than the audience’s eyes and ears.

Familiarise yourself with the delivery medium and the technology. Communicate with those in the project who are responsible for the UX or ‘user experience’; they’re the ones who will wrestle the technology to the will and whims of your narrative. Through their work, you’ll be able to understand the nuances of interactivity that will enable your story to be experienced. 

Make time to share perspectives and interpretations and decide how those experiences might influence the project. (Photo by Scott Graham)

3. Treat the interactivity as a scaffold for the narrative.

Once you understand how the audience will be encouraged to experience your story, then you can construct an effective narrative within that scaffold of interactivity. 

Be careful not to do the opposite and build the narrative around the interactivity as the best XR experiences have narrative at the core. Some XR experiences lean on interactivity to drive engagement but don’t allow it to lead the experience. Remember to keep the interactivity as the support.

A good way to make sure you’re keeping the narrative at the core is to always ensure that everyone (audience and makers) are approaching a ‘problem’ of interactivity from a narrative context and that the solution makes narrative sense. Back to the penguin in the fridge. I would try to interact with in by prodding it with my controller, which starts to eject me from the experience. If, however, the narrative asks how I would try to communicate with the penguin and the answer is that I ‘flap’ at him, moving my arms up and down, my controller would become invisible to me and the reward would be the penguin leading me into his cool ice cave. 

4. States of play

XR experiences are unlike games. In games, people are almost always interacting and their experience is almost always active. XR is more nuanced and audiences can sometimes switch, mid experience, between active and passive states of play. 

Depending on the kind of experience your team is building, the core ‘intended’ state of play for the audience can be defined in a way that denotes their active or passivity by describing them as ‘users’, ‘participants’, ‘witnesses’, ‘agents’ or ‘guests’ etc. 

Knowing this core tenant of interactivity is a useful tool for you to help structure your story amongst the interactivity. As a ‘guest’ in the penguins ice cave, an audience will have different narrative contexts for their engagement than an audience who might be described as ‘explorers’. 

Despite  a core tenant for us makers, the audience may not be privy to this but, even if they are, people will switch their state of play. Inexplicably, a person might become passive in an experience where the story is structured around them being active and vice versa. My advice for this phenomenon is to try and write in moments of reprieve for those people who might switch but, when in doubt, just write the fullest experience you can.

There is, at present, no standardised system of planning for XR experiences and many projects reinvent the wheel. You should seek advice from those who have gone before. (Photo by Startaê Team)

Understanding at which points people in an experience will likely switch from passive to active states of play is hard but it usually coincides with the level of intended immersion. 

You can try and force a switch of states. Games do this all the time by inserting cinematic sequences in between periods of gameplay but it’s hard to get right in XR experiences.

5. Levels of immersion

Many XR experiences involve multiple senses and need a certain level of immersion for people to feel the most engaged and allow themselves to solely focus on the story rather than the fact they’re wearing a VR headset, for example. We are always looking to create artificial flow states for those present in our experiences.

A tool to help you distil your XR writing into a flow state guide is a simple visualisation to map out the main beats or acts of the narrative onto a line that ramps, plateaus and slopes down again. The line represents the ideal immersion level for an audience throughout an experience.

The Ramp & Slope of the immersive experience
(Harrison Willmott)

People may always be in states of flow and so, when we want them to engage in XR experiences, we can guide their transition by gradually easing them into it. It’s a tough, careful and controlled process, which is why pushing them up a ramp seems like a fitting metaphor.

Once they’re in immersive flow, it’s relatively easy to keep them there until we want them to stop but we can’t just eject them from the experience as that can ruin the whole thing. The slope is a gradual, careful and considerate exit from immersive flow that enables the audience to properly process and appreciate what they’ve just been a part of.

You can assign the beats or acts of your narrative to any point on the immersive ramp and slope as though you were chronologically planning out a traditional story. The difference with this is that you can use the level of the line to help you determine the narrative’s immersiveness and from there you can extract more details about how your narrative may be experienced at each stage or act. 

Typically, at the start, the narrative may begin at the first moment an audience hears about an experience, where the immersive level is at the lowest. As you begin to guide the audience up the ramp, either through marketing of the experience or perhaps this is during a tutorial or loading screen, the immersiveness is still slowly rising and your writing will need to be delicate and intriguing, designed to entice and convert a person from curious to engaged. 

As you ascend the audience over the precipice and into the plateau of full immersion, you can really begin to flex your writing, surrounding them in a rich world of wonder and brilliance. As you prepare them for the gradual and controlled descent down the slope and out of immersion, perhaps this is an excellent time for an epilogue or fun little extras. This kind of thinking when it comes to writing feels as though you are writing the whole campaign for an experience but the experience for the audience isn’t just the bit where they’re in the headset or in the theatre. An XR audience begins their immersive journey for an experience from the moment they hear about it all the way to when they’re sharing their experience with friends and family afterwards. You should try to extend your writing to be alongside them for their whole immersive journey.

Even if you just take on one of the five tips I’ve written above, you will be well on your way to writing an exceptional XR experience. 

Twitter

The Writing Platform will feature a series of articles on virtual, augmented and mixed reality between July and September 2021. Get in touch if you’d like to propose an article. 

Democracy is starting to feel uncomfortable. In 2016, we saw the rise of the ‘silent majority’ with Trump, Brexit, and the return of One Nation. In the most recent Australian election, all of our polling was proven wrong with another surprise victory going to the major right-wing party.

It doesn’t matter how often this same plot twist happens – each time I feel personally shocked. There are so few people in my immediate community that share these ideologies, so it always feels so absurd that they can achieve power. But of course, this is the nature of our 21st century echo chambers.

 In such a noisy world, we curate our communities so tightly that we get used to the sound of our shared opinions. It’s easy to be lulled into a false sense of security when everyone you know denounces the alt-right.

It has been suggested that the silent majority is a consequence of generations being taught that it is rude to discuss politics at the dinner table. Maybe it’s not so much that democracy is not working for us, but rather that we are not using democracy in the way that we should. Even if we disagree with electoral outcomes, it would be great if we weren’t always so surprised!

This feeling became our starting point for a new project by Counterpilot, the transmedia performance collective that I direct. All of our work centres around the audience experience, with interactive technologies and gamified narratives. In this case, we wanted to create a petri dish of democratic dissent; a context where strangers could be invited to debate and disagree with each other without having anywhere to hide. Our challenge though, was to make the experience apolitical – it was to be about the feeling of political systems, without any political bias.

The result was Crunch Time:

Here, twelve participants sit around a projector-mapped dining table and share an elaborate five-course meal with two unique twists. The first is that they must democratically vote on each ingredient that will be used in the preparation of the meal. The second is that the ingredients will be cooked not by a trained chef, but by a special guest whom each night is co-opted from a position of public leadership.

 Our guest chefs typically included politicians, artistic directors and CEOs from within the local community. Their attempt at cooking in response to the diners’ choices was filmed live and streamed to the table, from a nearby kitchen space. Though their role was performative in nature, the guest chefs served as participants in their own right, reading from an automatic teleprompter and navigating the circumstances of the work with as much freshness as the other participants.

There’s a particular look some people give me whenever I explain a project like Crunch Time. They raise an eyebrow, cock their head to the side. “But how is it theatre?”

Of course from a distance, the project looks more like a dinner party than a piece of traditional theatre. Depending on an individual’s own frame of reference, I’ve also heard it being described as like an episode of reality TV, a video game, or a piece of installation art. But I maintain that theatre is in the DNA of this work.

I turn to Chapple & Kattenbelt (2006) for support here – where the notion of intermediality is used to position the audience as an integral part of the theatrical form. The audience acts as its own site of meaning with each member bringing their own lived experiences and cultural baggage with which to influence their reception of the work as meaning-makers. But they also become themselves one of the many media – as much a part of the work as the lighting, the video design, or the written script.

The bodies and minds of the audience members in the room become part of the confluence of media that makes performance. Intermediality is concerned with the space between where all of these components collide. While other formats like film (or in fact, traditional dramatic theatre) may seek to seamlessly integrate their components, contemporary performance thrives on empowering the spaces between, where reality and fiction can co-exist and where a constellation of forms crash against each other in time and space.

For another perspective on the argument, Peter Brook (1968) famously claimed that the DNA of theatre was comprised of three simple ingredients – an actor, an audience, and a space . In Crunch Time, the audience are simultaneously enrolled as actors. Throughout the experience, they are led to watch each other through a performative lens, with the understanding that the actions people take during this experience are both real and also un-real, provoked by the heightened nature of the evening.

The third ingredient, the space, becomes a powerful tool in enabling this mode of engagement. In this context, the space refers not just to the physical location, but the dramaturgical reality – the circumstances, rules and medial infrastructure that play host and give structure to the evening’s experiences.

There’s something hokey that tends to happen when participants are invited to improvise with a performer from within a fictional frame. They become self-aware, and they ‘perform’ with a layer of comfortable artifice, often making choices that disregard their own personal impulses, instead assuming that the exchange is merely a tool for moving the story forward.

When participants interact with fixed media though, there is seemingly less artifice. The media doesn’t pretend to be anything it’s not, and so the participant is also invited to respond as their real self. The moment does not demand any particular falsehood in order to move forward.

This performance dynamic is inspired by a style of automatic instructional theatre known as autoteatro, first defined by Sylvia Mercuriali & Ant Hampton. In autoteatro performance works, participants follow instructions, acting as both performer and observer for themselves in stylised or unfamiliar circumstances. In this way, Hampton & Mercuriali suggest that an autoteatro work acts as a “’trigger’ for a subsequently self-generating performance” (n.d.).

Being made to operate in this instructional context means that the participants must navigate the porous boundaries between the real world and their self-generated arts experience. Such ambiguity prompts a level of self-awareness, as the participants become conscious of the part that they are playing. In this way, participants become audience to themselves as actors, as much as they witness the other actors around them.

The work develops a uniquely liminal position in this genuine-but-manufactured zone of aestheticised reality. On one hand, nothing matters because it’s just a game and in gameplay we know not to take things too seriously. But on the other hand, the currency of the evening is established as food, which though trivial still feels strangely personal when it’s going to end up in your mouth.

And so Crunch Time is theatrical because it depends on this particular balance. It also depends entirely on the audience, innately trusting them to be interesting and to develop a genuine throughline in conversation with the work.

Crunch Time runs for two and a half hours, throughout which there are lots of fixed elements and carefully designed stimulus. But there is also space. Space for the audience to be surprising, to distract themselves with their own tangents, and to unpack their own roles in the moment. The balance between open and closed elements becomes a rhythmic dance like sex or music – riding the waves of give and take in order to craft the best possible circumstances for impact.

And in response, our audiences delivered time and time again. Like the sub-plot about a vegetarian diner who campaigned for one course to be meat-free, securing promises that were quickly abandoned as soon as bacon was available to be voted on. Or the diner who lunged across the table to knock someone else’s token out of the way, manipulating the outcome of a vote and feeling immediate shame when the significance of this act was pointed out by other diners. Or the conspirators who rallied the whole table to abstain from a vote in the hopes that they could bypass a set of ingredients that nobody wanted (spoiler: the system doesn’t work this way, and they ended up with cloves).

Giving so much control to the audience is both high-risk and high-reward. But maybe it also says something broader about culture too. That in a true democracy, we need to leave space for everyone to be heard.