In September 2019, five months before the pandemic, I moved to Toronto to begin a PhD at York University. I had been to the city before, and as I was a new international student  I thought I would make new social connections in school. However, in February 2020 the world transformed into its virtual ‘metaverse’ form, and I realised I would not be able to make friends and deepen any new relationships through Zoom, Teams or Slack. The loneliness hit me hard and even though leaving Canada was an option I was worried that I would not be able to return to finish my studies, so I stayed.

Looking back at the two years through a linear order of events, it seemed to me as though the time had been a vacuum. I could not put events into a linear narrative and I could not remember many things that happened in those two years. As a community as well as individually, we went through a repeated experience of fear, uncertainty, and freedom limitation. Negative emotions, such as boredom and anxiety, influence our long-term memory. Moreover, numerous studies have reported that ‘human cognitive processes are affected by emotions, including attention, learning and memory, and reasoning’ (Chai M. Tyng et al., 2017, p. 1454).

So, I wondered whether one of the post-pandemic consequences could be trauma (and possibly a collective trauma). Research on trauma shows that “the social environment does not have a direct and static impact but is mediated by emotional experience, the way it is lived through, interpreted, and processed on the basis of social, personal, and situational resources (today often termed as potential for resilience)” (Busch & McNamara, p.324). Each person experiences the hardships of life differently and some are more resilient than others. I was stuck, anxious and could not focus so I decided to do the one thing that satisfies me: making things. One of these ‘things’ was an interactive experience dealing with my own pandemic trauma. I decided to experiment on myself and find healing methods through interactive VR. If we take into account that the mind finds tools and technologies in the world in order to expand cognitive space constantly. And, if thinking is feeling and feeling is thinking, our emotions co-expand into this space. If we take creation into the equation and if “making is thinking is feeling” (Gauntlett, 2018), I am persuaded that emerging media are the best form for interactive collaboration between humans and machines.

The Making

The experience was created through the AI Storytelling Project as part of the Immersive Storytelling Lab (ISLab) at York University in Toronto. During my job there as an XR creator, I started to explore how to use interactive and immersive media; such as mixed, virtual and augmented reality (XR) and co-creation with the machines (Loveless, 2020; Wolozin, Uricchio and Cizek, 2020; Guzman & Lewis, 2020) for post-pandemic experiences. Specifically I focused on Natural language processing (NLP), a subdivision of artificial intelligence often used for different aspects of human-machine communication (e.g. speech recognition, text generation, speech-to-text and text-to-speech transformation, etc). Thanks to the cooperation between ISLaband the NLPsoftware-based storytelling platform Charisma.AI, I was given access to its Beta version, and I started exploring different options for an immersive and conversational experience.

My methodology during the creation of the virtual reality piece Home Is the World VR took the form of creation-as-research, where “creation is required in order for research to emerge” (Chapman & Sawchuk, 2012, p. 19). Via this VR piece, I investigated the relationship between technology, creation and the human condition, or as Guzman and Lewis explain, “how people understand AI in relation to themselves and themselves in relation to AI” (2020, p. 77).

I followed in the footsteps of the rich tradition of human-machine interaction, elaborated by Sherry Turkle when the personal computer was being adopted into everyday spaces. She described it as a “metaphysical machine,” a concept which led to the study of AI as a challenge to existing conceptualizations of the nature of humans” (Guzman & Lewis, 2020, p. 80). Emerging technologies may eventually push past the boundaries of human communication, meaning that human-AI communication may change how we communicate with each other as humans and with other entities.

Before I start describing the process of designing Home Is the World VR, I want to give a short synopsis to introduce the technology and the main idea of the piece. The VR experience is aimed at headsets with a passthrough option (i.e. Oculus Quest, HP Omnicept Reverb 2 etc.) –  a feature that allows users to step outside their view in VR to see a real-time view of their surroundings. Passthrough uses the sensors on the headset to approximate what one would see if they were able to look directly through the front of their headset. Hence, the interface is created by an ephemeral space between virtual reality and the user’s physical environment where they converse with the AI character and are asked to answer its questions and to follow a series of sensory-led tasks to induce pleasant memories before the Covid-19 pandemic. Such tasks include smelling coffee beans, conjuring the smell of an early summer morning, the sound of walking on snow, the taste of a delicious dessert, the touch of a plant etc. Then, the Charisma.AI software records a user’s spoken memories of the pandemic and generates a personal pandemic story for each user, individually. The twist: it re-tells the experience somewhat differently. It re-contextualizes the user’s story in a world full of positive news.

Project workflow

 Through Charisma.AI’s dialogue engine, and Google Cloud NLP services, players and audiences can meet with virtual characters, converse with them, and change the story. The AI character is created through the Charisma.AI platform and Google Cloud Speech-to-Text service. Charisma.AI uses the language of storytelling, with built-in features such as emotion, memory, scenes and subplots (Charisma, 2017 – 2021). However, it is not GPT3 based and works with a machine learning dialogue engine, which I found exciting when designing a semi-scripted interactive piece.

My research and ideation of the project started with the following questions. What if we could go back in time and re-shape our memories through an interactive AI storytelling experience? What happens when a human and AI re-create memories together? What if the experience played with the idea of the flexibility of memory and its ability of re-constitution? The connecting points in these questions were the ability of language to be performative, making sense of the world by creating stories, putting events and experiences into linear narratives, and the flexibility of human mind. And it seemed as though Charisma.AI, as a storytelling platform using such NLP modes as intents and sentiment, could not only be used as a tool to make stories but also as the content itself.

The saying ‘think before you speak’ has meaning in neuroscience – the formation of words acts as a delaying function, giving the brain time to deal with information input and retrieval. Traumatic experiences leave such an emotional mark on our brains that, when recalling certain events, we tend to be overcome by emotions and are not able to express what is happening to us through language. It is not by accident that trauma is often dealt with by speech therapy and storytelling. Remembering traumatic events through language and storytelling helps us to deal with  them.

Interestingly, there is a connection to sociolinguistics here, specifically to  J.L. Austin’s theory of performative speech acts. In his book How To Do Things with Words, Austin establishes performative utterances, which not only describe a given reality, but also change the social reality they are describing (1962). The theory was later elaborated by Judith Butler in her book Gender Trouble (first published in 1990), where she introduces gender as a discursive and performative practice.

According to van der Kolk, “Traumatic experiences are exceptional because these intensely emotional events are not encoded into the ongoing narrative states” (2014). The traumatic experience is recorded as separate and dissociated from other life events, and, thus, it takes on a timeless and alien quality. In healing trauma, language is crucial.

John J. Ratey claims that when a subject tries to recall a traumatic experience they are overcome with emotion and are not able to express it in words. They are ‘dumb struck’ for a variety of reasons, one of them being that an important part of our brain responsible for emotion amygdala ‘overreacts’ while another part responsible for language and speech, the so called “Broca’s area shuts down” (2001, p. 210).

Further, Ratey claims that “the formation and recall of a memory is dependent on the environment, mood and gestalt at the time the memory is formed or retrieved” (2001, p. 208). Each memory is created from a vast interconnected network of pieces in our brain such as language, emotions, beliefs. And our daily experiences alter these connections and, therefore, we remember things differently in different phases of our lives.

Apart from neuroscience and linguistics, the theoretical approach framing the project is affect theory, which helps to disclose the ways technology intersects with our limited proximal senses, rhythm and sense of motion and embodiment. When affect is processed through cognition – once the signal from the amygdala (limbic system) reaches the prefrontal cortex in our brains – it becomes an emotion. It occurs through cognitive action and relations between agents (humans, non-humans, things, environment).

These cognitive actions are inherently performative, such as language, bodily and facial gestures, or tone of voice. Sara Ahmed (2014) calls these relations “contact”. She argues that already when we feel that something is good or bad, it involves “reading the contact we have with objects in a certain way” (Ahmed, 2014, p. 6). Contact involves a process of reading, attribution of significance, it “involves also the histories that come before the subject” (Ahmed, 2014, p. 6). Emotions are, thus, culturally and linguistically constructed performative actualizations of affect. Ahmed posits emotion as cultural construct through language (2014) as when we name and perform our emotions they become solidified and transmitted to others.

This process of emotion transmission can be also called “affect contagion” (a perfect example is sentiment contagion in social media through text). Plus, according to Brian Massumi, emotions are culturally and linguistically situated, “emotions are embedded in the arbitrariness of language and gestural code (including face) involving cognition, and through which we assign these qualities, and which carry meanings in order to be communicated” (1995, p. 89).

Stage Two: The Design

Scripting in CharismaAI

The main aim of the piece is to trigger positive emotions via sensual memory associations and language. When reframing the memories through emotions, I applied methods in trauma research, using questions and sensory led tasks that connect past, present and future through emotion and affect (Damasio, 2005; Van der Kolk, 2014; Busch & McNamara, 2020; Bloomaert et al., 2007).

Upon entering the virtual world, users encounter an omnipresent AI character, which converses with them and gives them a series of sensory-led tasks to induce their memories and emotions associated with the pre-pandemic world and apply them in the ‘now’. First, the user is asked by the AI character to remember a situation during the Covid-19 pandemic. Second, they are prompted to answer questions such as: what sign are you? Where did you spend your pandemic time? Were there other people? In what month did the significant event happen? After the AI character collects all necessary information, the software generates a positive event, which happened in the same month in 2020 or 2021. These generated pieces of text come from a corpus of news articles about positive events during the pandemic. Here are some examples:

“In March 2020, we learned that a group of dogs trained to protect rhinos from poachers have saved 45 rhinos in South Africa. The dogs, including beagles and bloodhounds, among other breeds, were trained from birth to track down poachers alongside humans in Greater Kruger National Park.”

“June 2020: The Supreme Court rules that no one can be fired for being gay or transgender, and Beyonce releases Black Parade. Yaas, Queen B!”

An example from my Miro board of the corpus of  positive news events in 2020

Apart from the language, different senses, such as smell, touch, sound, vision and taste, are directly connected to our emotions (e.g. the sense of smell has direct relation to our limbic system, which is why it triggers memories and emotions without any cognitive interference). Therefore, in the second part of the experience, the user is asked to follow sensory-led tasks to deepen the positive emotions. The next image presents different sensory tasks which were brainstormed during a workshop with several participants.

Things to feel with

Blueprinting in UE4 with CharismaAI plugin

In the finale, the AI character summarises the collected information about the user. However, it replaces the negative event with a positive one. And it asks: do you feel better?

Prototyping

Stage Three: User Testing

User testing is crucial when making interactive and immersive experiences (e.g. for improving the user design). Moreover, it is vital to check that the VR experience doesn’t cause any physical or mental issues in the participants. When I tested the prototype in our lab, some participants felt as though it was a ‘confessional’ experience. They saw it as a safe space where they could talk to a scripted AI character who tried to make them feel better about the pandemic by taking them to other emotional landscapes.

On the other hand, when I did an embodied proof of concept exercise with participants at a real-life conference this year, some told me they did not want to go back to the pandemic years. Does this prove or denounce the healing effect of the experience, and the traumatic element of the pandemic?

Busch and McNamara claim that:

“Avoidance of painful intrusions as a measure of self-protection is only one of the reasons that makes it difficult to share a traumatic experience with others. There are many other reasons such as: speaking about what happened can be subject to interdiction, social taboo, or shame; a common ground of experience or knowledge is missing; one does not want to burden others with one’s own pain; the violation and the suffering have not yet been socially acknowledged” (2020, p. 329).

Whereas one user may go through an intimate immersive and interactive experience with an AI character, which not only lends them their ear but also gives a social acknowledgment of the pandemic as traumatic. For another user, it may be a reiteration of the painful memories which they are not ready to share.

Conclusion

A new research question needs to be proposed: When does a VR experience from the first-person perspective heal and when does it re-traumatize? I have experienced far too many re-traumatizing VR pieces in the past (e.g., being sexually, physically, and verbally assaulted in VR can re-traumatize a user if they experienced a similar situation in real life, and if they have no agency or control over the situation in VR). My VR piece aims to stand in opposition to such XR works. It re-iterates the negative memories into positive pandemic stories. And I hope that it can transform the negative emotions into positive ones through language, conversation, and sensory triggers. The iterative process of creation-as-research doesn’t end with a final build of an experience. My work on this project is to be continued.

References

Ahmed, S. (2014). The Cultural Politics of Emotion. Edinburgh University Press.

Austin, J.L. (1975). How to Do Things with Words. Clarendon Press.

Blommaert, J., M. Bock, and K. McCormick. (2007). ‘Narrative inequality in the TRC hearings’
in C. Anthonissen and J. Blommaert (eds): Discourse and Human Rights Violations, pp. 33–63.
John Benjamins Publishing.

Busch, B., & McNamara, T. (2020). Language and Trauma: An Introduction. Applied Linguistics, 41(3), 323–333. https://doi.org/10.1093/applin/amaa002

Butler, J. (2006). Gender Trouble. Routledge Classics. New York: Routledge. 

Chapman, O. B., & Sawchuk, K. (2012). Research-Creation: Intervention, Analysis and “Family Resemblances.” Canadian Journal of Communication, 37(1), Article 1.

Charisma — Storytelling powered by artificial intelligence (no date). Available at: https://charisma.ai/ (Accessed: 15 December 2021).

Damasio, A. (2005). Descartes’ Error: Emotion, Reason, and the Human Brain. Penguin.

Gauntlett, D. (2018). Making is Connecting (Second). Polity Press.

Guzman, A. L., & Lewis, S. C. (2020). Artificial intelligence and communication: A Human–Machine Communication research agenda. New Media & Society, 22(1), 70–86.

Loveless, N. (2019). How to Make Art at the End of the World. Duke University Press.

Massumi, B. (1995). The Autonomy of Affect. Cultural Critique, 31, 83–109.

Milgram, P. and Kishino, F. (1994). ‘A Taxonomy of Mixed Reality Visual Displays’, IEICE Trans. Information Systems, E77-D, no. 12, pp. 1321–1329.

OpenAI (no date) OpenAI. Available at: https://openai.com/ (Accessed: 15 December 2021).

Ratey, John J. (2001). A User’s Guide to the Brain. New York: Random House.

Tyng, C.M. et al. (2017). ‘The Influences of Emotion on Learning and Memory’, Frontiers in Psychology, 8, p. 1454. doi:10.3389/fpsyg.2017.01454.

Van der Kolk, Bessel (2014). The Body Keeps the Score. Viking.

Wolozin S., Uricchio W., Cizek K. (2020). Collective Wisdom. Massachusetts: MIT Press.

Live theatre performance in XR (extended reality), which includes both Virtual Reality (VR) and Augmented Reality (AR), is a rapidly growing field of digital drama that sits at the convergence of immersive theatre and interactive games design. While a number of theatre companies have adapted their existing work for VR, (notably the Royal Shakespeare Company with Dream Online in March 2021), our research project, Mrs Nemo XR, aims to create new works of musical theatre specifically written, designed and built to utilise the interactive and immersive features of this exciting new performance medium.

Mrs Nemo XR

Our initial project, a short-form immersive musical — which we term an ‘Immersical®’— is inspired by an episode from Jules Verne’s classic Victorian adventure story, Twenty Thousand Leagues under the Sea (Verne, 1, featuring the attack of Captain Nemo’s Nautilus submarine by a giant sea creature, the mythical ‘Kraken’. Adding a contemporary twist to the tale, the story is retold from the perspective of a mysterious female character in a Victorian bath chair, who appears at the outset to be Nemo’s wife, but who is later revealed to be a mermaid, one of the many marine creatures that Captain Nemo has collected for the undersea museum which we use as the setting for the show. 

Although we draw on Verne’s original story for the undersea environment and basic plotline, we have devised our own narrative to tell the tale from an alternate viewpoint; that of one of the sea creatures whom Nemo has captured. As the audience members join the show, they find themselves embodied as individual avatars of deep-sea divers, who have been transported into the submarine, and invited by the curator (Mrs Nemo) to explore the gallery displaying the museum artifacts. As Mrs Nemo tells her own tale through improvised dialogue and song, the audience can interact with the actor by talking to her avatar and examining in 3D some of the items in the museum. 

VR Platform

Staged on the browser-based online social VR platform Mozilla Hubs, accessible  either via desktop computer or VR Headset, Mrs Nemo XR is performed by a solo actor working remotely from home, singing live to pre-recorded music tracks. We are limited to one live singer, due to audio latency issues across the network which prevent multiple performers singing in sync. The show also has a short run-time of ten minutes due to concerns of how prolonged use of VR may adversely affect actors and audience members, especially new users of the technology.

Performances 

Our initial trial run of twelve performances with a live audience began in October 2021 and was staged in collaboration with an informal collective of virtual theatre-makers from around the world under the title OnBoard XR. While the whole event shared an underwater-themed virtual environment, each team designed their own show, including the set, props, avatars, and individual backstage cueing system. This article details the technical, design and production challenges we encountered while making Mrs Nemo XR, with specific reference to the narrative design and scenography of the production.

Performer singing in a VR Headset, while moving her own avatar in VR

Creative Team

This virtual theatre production, as with a typical physical theatre company, has a creative team composed of Producer, Writer, Director, Designer, 3D Artist and Performer; all theatre practitioners and creative technologists, several of us conducting research at Digital Creativity Labs, at the University of York, UK. Throughout the four-week production period in the autumn of 2021, team members worked remotely from different locations across the UK and the USA. Pre-production meetings, rehearsals and performances all took place online, managed through a combination of VR, video conferencing and text and audio messaging. Many of the sessions were intense early morning or late-night events, due to the complexity of distance working across time zones.

Interactive Narrative Design

Mrs Nemo XR, aims to engage the audience with the narrative, while giving them the freedom to explore the immersive submarine environment and interact with the performer and each other. Without a traditional framing device, such as a proscenium arch or cinema screen, we rely on alternate ways of directing audience focus, such as guided direction and environmental storytelling. Our audience, who were represented by avatars of deep-sea divers, were actively encouraged to be part of the story-world, through improvised dialogue and narrative song, and the way in which the actor directed their attention to the interactive props, such as the toy Kraken that could be picked up, moved and resized, and scenic effects like the sudden and dramatic appearance of the giant sea monster outside the viewing window of the sub.

Technical Challenges

One of the main technical issues we encountered when using Mozilla Hubs with our Meta Quest 2 (previously the Oculus Quest 2 VR headsets), is the limited audio capability afforded by this form of Web-based VR (a VR platform that is easily accessible on multiple devices via the Internet). While we did have the capability for spatial audio, we elected instead to have our performer’s voice override this feature, so that she could be clearly audible, regardless of position in relation to the listener. Since we knew that audio and visual latency would be problematic, our actor ran the music tracks from her laptop, whilst performing in a VR Headset and manipulating her own avatar, thus syncing all her actions at the point of transmission. 

The scenic cues, including the change of avatar for Mrs Nemo for the final reveal that she is a mermaid, and the attack of the Kraken, were also subject to delay of operation across the internet, and so were run remotely by the backstage crew to avoid distracting the performer by the desynchronised scene changes. This also enabled the production team to make live adjustments during the show when latency of operation necessitated the removal of used props from previous scenes or by ‘respawning’ the placement of set pieces that had entered the scene at the incorrect size or location.  

Scenography 

The inspiration for set design, props, avatars, and interactive artifacts were both derived from Verne’s original story, and from the writer’s scripted dialogue and lyrics. From the book it was clear that while the Nautilus did have a library and salon with a large viewing window, there was no dedicated space for storing or displaying the Captain’s undersea collection. This led to the idea that the salon should become a museum, its walls filled with books and artifacts relevant to the story. Using Mozilla Hubs’ custom editor, Spoke, which allows developers to create 3D worlds with their own designs or pre-made assets, we built the submarine interior, and using a bespoke plugin Stage Management System, we were able to program cues to make scenic objects, props and avatars appear, animate, and change size and position within the scene. 

Tech rehearsals in Zoom and Hubs showing the cues of the stage management system

Working in Mozilla Hubs

In order to perform well across a wide-range of devices, Hubs has certain operating limitations, such as the inability to move or ‘spawn’ objects easily, and a maximum scene size of 16MB, which resulted in our immersical taking on a low-poly (an art style where the number of polygons in a 3D model are reduced to give a low quality appearance to the resulting graphics. This style has the benefit of being efficient and easy to optimise)

style of graphics. Because of internet latency, we could not accurately predict how the scene would appear on the various viewing devices that would be subject to differing connections, internet speeds and locations. For example, we aimed to trigger the Kraken’s appearance outside the viewing window, as a shocking reveal to the audience, especially for those in VR who would see a gigantic monster attach itself to the side of the submarine. However,  the physical distance between the stage manager in York (UK), the performer in New York (US) and the worldwide audience, meant that the significant delay between cue and action led to some audience members, depending on the efficacy of their own devices and connection, experiencing a slight delay between what the actor was saying and what the scene was showing. 

Set Design and Layout

Our initial design plan for Mrs Nemo XR was to divide the submarine into smaller clustered areas to direct audience movement, (see Figure 3) to where the action in the scene was taking place. We particularly wanted to guide the audience towards the window to provide a good view of the Kraken’s sudden appearance outside the submarine. During rehearsals however, we realised that the audience tended to cluster around the performer and follow her around the scene. This led us to strip back some of the central blocked-out areas to provide a clear line of sight throughout the submarine interior, so that the performer and the action could be seen at all times.

Preliminary design of the submarine interior

Sub interior with Captain Nemo’s portrait on the wall

The eye of the Kraken at the window

Audience member as diver with props

Conclusion 

In spite of the technical limitations of the Mozilla Hubs platform, in particular the external cueing system, the low-poly graphics, and the audio/visual latency experienced, the show was well received and we felt we had achieved our central aim of creating our first fully functioning immersical. Through the rehearsal process and by observing the performances, we established methods of directing audience focus and promoting their engagement with the narrative, in three key non-verbal ways. Firstly, through the use of scenic devices like the viewing window, to frame the attack of the Kraken, and secondly, in spite of internet latency, we were able to sync actions to the music, cueing movement of scenery and interactive props at appropriate times. Thirdly, we found that  guided direction from the performer (e.g. gestures, movement, following her gaze and use of interactive props), supported audience attention and engagement with the story-world. 

Our research continues with a new iteration of the show, where we will be using a different VR platform, and extending the length of the performance, working with innovative techniques for reducing audio latency. We also intend to offer greater user agency and interactivity, and explore different styles of narrative design including non-linear and object-based storytelling. We see a great future for live performance in XR, particularly in terms of ease of access and affordability, and look forward to our next collaboration within the rapidly expanding community of VR theatre-makers and their virtual audiences.

Bibliography

RSC Dream online. Audience of the Future Live. (n.d.). Dream. [online]
RSC Dream online. Latest Press Releases | Royal Shakespeare Company.
Mrs Nemo XR – performance video #OnBoardXR [SHOW3.3] Live Short Series in WebVR.
Verne, J. Twenty Thousand Leagues Under the Sea. (first published in English, 1872). Project Gutenberg.

Credits

Mrs Nemo XR Creative Team: Director: David Gochfeld, Writer: Mary Stewart-David, Designer: Daniel Lock, Exec Producer: Cristobal Catalan, 3D Artist: Guy Schofield, Performer: Vivian Belosky
OnBoard Stage Management System developers: Roman Miletitch, David Gochfeld, Clemence Debaig, Michael Morran.
Digital Creativity Labs & Department of Theatre, Film, TV and Interactive Media, University of York

Weronika Lewandowska and Agnieszka Przybyszewska in conversation: on creating poetry in VR

“You are not supposed to call it a subject, but an avatar. There’s no reality being portrayed, no setting, but a simulation!” That is what Polish poets from the Rozdzielczość Chleba group, experimenting with new technologies, proclaimed. Imagine, then, that instead of reading a poem and recreating a lyrical situation in your imagination, all of a sudden you simply become its subject. You can hear, you can see and you can move. You can feel how your surroundings affect you, and you can see how you affect your surroundings. The “here and now” of the speaking “I” becomes your “here and now”, the literary work is no longer an “artefact” but an “event”, one in which you participate with your whole body and all your senses.

Now, don’t imagine it. Just put on your VR headset, get your hands on the controllers and immerse yourself in “Nightsss”, a VR work directed by Weronika Lewandowska and Sandra Frydrysiak. Yes, virtual reality can be a literary platform, too. Yes, you can experience poetry in VR. “Nightsss” VR (the original Polish title is “Noccc”) debuted at the Sundance Film Festival in early 2021 and has been presented at many festivals around the world (recently, it premiered in the UK) and is an extraordinary piece of work. However, it dovetails the realm of new-media activities sometimes referred to as VR or XR literature, including a long tradition of poems that are, in a way, written or re-written into VR experience (from Samantha Gorman’s “Canticle”, created using the CAVE environment, which also integrated poetic experience in VR with dance, to Andy Campbell’s “Water Cave”, in which the author applies the first-person point of view in the experience of landscape co-created with the use of typography). It is not the only romance between literature and VR with a Polish touch either (for instance, Anna Nacher co-created one of Mez Breeze’s VR collaborations). 

On the other hand, neither Weronika Lewandowska nor her poem “Nightsss”, which is (as the directors put it) the “narrative axis” of the VR experience, appeared “out of nowhere”. The work on “Nightsss” VR was preceded by years of artistic practice, creative violation of the boundaries of the poetic language and blazing trails on Polish and international spoken word stages, as well as her involvement in the creation of multimedia poetry publications and work on performances (also using VR) as part of artistic residencies and scholarships abroad.

The interview presented here is a record of conversations between The Writing Platform editor  Agnieszka Przybyszewska and Weronika Lewandowska about some aspects of creating poetic experience in VR.

“Nightsss” VR directed by Weronika Lewandowska and Sandra Frydrysiak (trailer). Check the whole “Nightsss” VR team here

Agnieszka Przybyszewska: An average fan of the art of words would not say that virtual reality and literature  have much in common. And, yet, we are here to talk about the “Nightsss” VR, which I would describe as poetry in VR, or VR poetry. Could you tell us what a “reader” who ventures into putting on goggles to experience a poem should expect?

 Weronika Lewandowska: “Nightsss” VR immerses you in an interactive and computer-generated space of images, words, sounds. It is an immersive piece of work, an experience of spatial metaphors, which come into being not only between words and meanings, but also between virtual representations of various elements written in the script, then put into motion and action in a virtual environment.

AP: All right, but most people would ask “where is the poetry here?” and “how will I find myself in all this?”

Weronika Lewandowska’s “Nightsss” as a spoken poem

WL: Everything is constructed around the axis of the “Nightsss” text, which was created sixteen years ago. It is a spoken word poem that I wrote with an international poetry slam in mind and presented many times before foreign audiences. My works created at that time took into account the spatial experience of a poem, the impressionistic quality resulting from the manner in which the body performs the text. I wanted to bring not only the words but also the performative nature of spoken word poetry into VR. Therefore, by immersing yourself in the virtual reality of “Nightsss/Noccc,” you can expect to “enter” the corporeality of the person performing the text, but also to experience the poem broken down into different elements in space, to spatialize the text through sound, visual, interactive actions. The text of “Nightsss” is about a real situation (the experience of love, intimacy, contact with nature and man that preceded the creation of the poem), which came back to me while writing the script. “Nightsss” VR also makes use of this memory, which can be seen in how the virtual world is constructed.

AP: Your explanation will probably reassure  those sceptical about VR poetry. Still, it is probably worth pointing out one more thing. Shortly after the Sundance Film Festival premiere, comments were voiced that your experience was visual poetry. The common understanding is that visual poems are made up of letters. Yet, in “Nightsss” VR, there is no typography at all. Your poem functions as a text to be listened to, the viewer is immersed in a visual soundscape. What did you want to achieve in this way?

WL: In VR, the poem functions not only as a text to be listened to, but also as an interactive audio element linked to visual change and the immersant’s activity. Through the movement of the hands, one layer of the vocal can be deformed. Thus, the person immersed in the virtual world has some extent of agency – they can deform the sound environment with a visual action linked to the movement of the body in space. They also immerse themselves in a very specific domain of impressions. Images activate certain areas of memory. So do sounds. The different impressions you get – from memory and from what you are experiencing at a given moment – mix together.

AP: Is this the very moment you meant when you and Sandra talked about “mind hacking” in one of your interviews, about creating new memories in your audience?

WL: VR experience, which engages you on various levels of perception, is able to “detach you from yourself” for a moment, from various developed ways of acting and perceiving. This is what I see as a mind-hacking experience. Charles Davis talked about the de-automatization of perception. For her (and for me too), VR changes the logic of perception and shatters habits, thanks to which we are able to change (because we allow into ourselves a different path of sensory experience). Virtual space reopens us, and this “fresh” experience gives us a new memory.

“Nightsss” by Weronika Lewandowska and plan.kton

AP: All your artistic activities seem to me to be related not only to space, but also to the de-automatization you mentioned, including the  de-automatization of reception. You offer slam poetry to the audience rather than traditional book volumes. On top of that, you work with plan.kton on audiovisual poetic performances. I’m also thinking of the video for “Nightsss”, which arranges the space with typography. It all seems like a test of which medium will allow you to best convey this spatial experience. Why did you decide that “Nightsss” VR should be a six degrees of freedom (6DoF) experience and not the more “classic” cinematic VR?

WL: You know, for me, cinematic VR is something I have already, let’s say, tested, if only by projecting visuals onto the space around me. Also, I wanted it to be an animated piece, allowing you to go beyond the framework of reality and physics. I have always been interested in abstract forms. When I worked with different VJs, I really didn’t want the body to be shown in the visuals. I didn’t want a visual reference to corporeality. Instead, I wanted to transport this corporeality, this impressionistic quality in an abstract way. For me, this is how I see poetry, too – I don’t want to speak directly. Cinematic VR was not my cup of tea. I was attracted to forms that can be made unreal, fascinated by shape shifting. I have always been interested in using very abstract means to present ambiguity and emotional states. For example, animations in which figurative forms dance together in a frame to music, synchronised or unsynchronised. This composition and dynamic, the way the objects approach each other, the way they interact with each other, the way they open way to “speaking indirectly” – this is simply closer to poetry. And the six degrees of freedom gave me exactly those possibilities (apart from the interaction space itself, of course). 

AP: What meaning, or function, does ASMR have in your work? I am asking this question, because “Nightsss” VR is often described as a piece that combines animation with ASMR. What does that involve?

WL: Usually, when someone thinks of ASMR, they think of sounds, of different textures, of the tactile sensation that sound brings. ASMR as something that characterises “Nightsss” did not come up until I got down to preparing a description for the finished VR experience.  Still, “Nightsss” VR is ASMR-like on many levels. When we talk about ASMR, we mean constructing the domain of impressions for the different senses, by getting close to the body and establishing intimacy with the body. The fact that sound can transport these tactile sensations is a feature of sensory substitution. For me, movement is also an ASMR element. I once wanted to explore what the movement of abstract forms does to us: how it engages our empathy, or what emotions and states are evoked by abstract forms dancing around us like another body. The ASMR quality of movement is created by the pace, dynamics, relations between them, tensions connected with e.g. something sliding over something else, the dynamics of changing textures. All this evokes a physical reaction of our body. That is why “Nightsss” VR is an ASMR experience for me, but this notion comes here as if post factum, only when I look at what we’ve managed to do and what means we’ve used.

AP: I also wondered about the linguistic dimension of VR “Nightsss/Noccc.” You haven’t shown your spoken word poem in translation on international stages (although there are some great translations of the text). Similarly, your VR experience is presented around the world in Polish (you can also listen to the English translation at the end). Why?

WL: This is a Polish text, while the domain of impressions (perception and how we interpret performative activities) is already beyond the linguistic barriers. Mind you, the language (and its sound) is one of the elements that appear in space and that you interpret. I’ve asked myself this question before: “what do you do with a poem when you have sound and image on stage at the same time? Would projecting a translation of the text redirect the audience’s attention, thus competing with the non-verbal, affective dimension of the poem performed by the body?” I think I actually started “doing” VR before I got to know this medium in practice. 15 years of experience collaborating with other artists, including dancers, VJs and musicians – it was learning how to create immersive experiences and work in an interdisciplinary team. Also, I learnt how to reach out into different spaces with my poem. One of those first moments was what you said about letters on paper. I also tried to construct space using typography. Later, of course, it was my body that became the medium of recording and presentation. When I couldn’t create some actions and artefacts on the computer (graphic, film, programming), I created them with my body and my actions in space.

Weronika Lewandowska with plan.kton

AP: And have you ever considered VR experiments with typography? Bringing the experience behind the spoken word scene into the VR space, in such a way that the letters are also something you can interact with?

WL: No. At the moment I am more drawn to situations in which you are immersed in something that’s in between these letters. When I was working on the script for “Nightsss”, I thought about how I would like the person experiencing it to actually write the poem themselves, so that when they immerse themselves in the VR, they feel that it’s their experience and that they are as if writing from it. I interpreted the space of my experience in such a way that “Nightsss” came into being, and now someone else immerses themselves in all these things and brings out their own potential version of this text. They find themselves in the same process and emotion that I was in. My intention was to bring people closer to this beautiful experience that I had and that was captured and stored in this poem.

AP: You told me that you keep returning to your memory, that it somehow repeated in you, too. How did you imagine the audience of your text? Did you want them to interact with this VR experience once, or did you hope for multiple “readings”?

WL: I knew there were too many elements in our experience and that you get to feel some kind of inability to read: it’s just that your perception is not able to grasp everything at once. This kind of overstimulation, perceptual chaos, was very deliberate. In general, when experiencing, we don’t break things down into elements to understand what’s happening. When you experience the VR environment of “Nightsss”, it consists of different layers. You grasp whatever you manage to grasp and in this way you construct your memory. Then, you can immerse in it once more to discover some more things. First, it is simply being that you have, just you being there. That is why I want that first entrance experience to be as strong as possible. So, referring to your question, I would like it to be experienced repeatedly. Just like when you go through a poem, you read it many times and you can have lots of different interpretations, depending on how you feel and what you direct your attention to.

AP: But you can always have a poem on paper, even a visual one. In the case of “Nightsss” VR, on the other hand, you just have to experience it again, you have no way to “take” it with you.

WL: But each successive experience of “Nightsss” means also entering into the memory of that first experience.

AP: And what do you mean when you say you appreciate the power of the first experience? Have you also been involved in designing the so-called “onboarding”?

WL: I don’t really like the word “onboarding,” I would rather call it… a form of invitation or ambient perception tuning. It is about how to properly prepare the place, how to synchronise the space that surrounds you with the space you are about to experience, so that there is no such gap. That is why some of the “Nightsss” performances were accompanied by artistic installations, whose task was to tune the senses of the audience.

AP: Why is this important?

WL: I know that people who experience VR for the first time are often intimidated by the technology and the fact that they could be observed by someone. I think it should be a space for intimate experience (just like VR is for one person). I wanted everyone who enters this space to have a sense of safety, which would translate into a relaxed and open body. You can get to another level of experiencing, if you don’t have to struggle at the beginning with the fact that someone is watching you, thinking about what is outside all the time. I wrote a manual for those who want to immerse themselves in “Nightsss” VR. I included in it information that they are not going to be photographed and they are safe, but also about what activities they can expect and how they can move in virtual reality. 

AP: You care for the comfort of your audience, you want to put them at ease. 

WL: I wanted an intimate experience. And that may not be simple love at first sight. Here, you need time to get to know the technology, time to build trust in the environment in which you are going to experience. When you start to trust the situation, then you are able to open up more to the experience of virtual reality

AP: Your first VR project was a real success. Do you have advice for authors, especially spoken word artists, who want to experiment with VR?

WL: First of all, I would recommend experiencing different things, orienting yourself to multisensory reception and space, to what the relationship between body and space is and what gets recorded and stored in your memory; this sort of working with the multisensory memory of experiencing. Once a person begins to immerse themselves in their memories, they can see that they are constructed in such a way as to create an environment. The more senses are involved, the stronger the memory is. It is worth thinking about how different our perceptions are and about when entering the intimate zone, the mental comfort zone, turns into abuse. It is very important to see the possibilities and dangers that arise from the means you use. This is thinking all the time about where the recipient is, where their sensitivity is. It is good to observe how others work, so that when it comes to directing, you know what is possible. “Nightsss” VR had, above all, a great team of sensitive creators.

AP: When you talk about the whole process of creation, you emphasise the fact that you have to have a vision, that you need to know what you want to achieve and know the ways, the means, needed to complete this task (or know how to ask for them). And to find people who will do it so that it is in line with your vision. Do I understand correctly that you don’t need to be a coder at all to write poetry in VR? That you rather need to be… a poet?

WL: No, you don’t have to be a programmer. Still, without vnLab, I would have thought that I was not the kind of person who could do something like that. There was a lot of trust there from the beginning, faith in us that we could do it and, as I mentioned, there was a well-tuned team. But first of all, you have to have a well-constructed vision, or know what kind of experience you don’t want to create. To respect the other person’s psyche.

AP: In your opinion, is such a “poetic VR experience,” as “Nightsss” VR is often referred to, “VR literature”? Is there any point in using such a category at all?

WL: “I believe that “VR thinking” has always been included in my work, because I have taken space and audience perception into account from the very beginning (but I only realised this after some time!).” VR is an experiential situation, so if your intention in making literary works is to create experience and space, then you can naturally make a literary experience in VR. It’s just that I now think that the poem “Nightsss” is not a VR piece, it was just written as a spatial experience, so it somehow settled easily in those VR techniques. It is a cool, maybe even appropriate, way of constructing texts for VR: starting from concrete moments, elements, experiences. I wonder, though, what would happen if at this point I redirected my attention to writing something for VR completely from scratch. You know what I mean, in “Nightsss” VR, we have a text that has already been there, and the question is whether I, having already had the experience of constructing this environment, with the knowledge of all these means, would be able to enter a different way of writing and create something equally strong, which could already really be called VR literature. For now, it’s just looking for means and techniques, converting one action into another, translating one medium into another.

AP: Wasn’t the difficult access to VR, the sort of hermetic nature of the medium, a kind of artistic barrier for you? VR doesn’t seem to be a popular technology among poetry and literature enthusiasts. You know, I can understand, of course, that the slam space is also a closed circle, that you don’t have a wide audience there either, but in the case of VR it often happens that even when you really want to experience something, it just doesn’t work out because of various, sometimes very basic, problems with the technology. And this is discouraging and tiring.

WL: But on the other hand, there’s something cool about it all: the fact that it is a unique situation (once you manage to experience VR), and the fact that you have the opportunity, as the author, to meet people who have never experienced VR before. This is why it is so important what you offer to them for this first contact with VR. If I were to say what VR experience is good for this, I think it would be our VR, because it just makes sure to guide the person immersed in it in a subtle way, with attention to their psyche.

AP: And it’s not too long.

WL: Yes, it’s not too long, which is why it leaves you a little unsatisfied. You can of course immerse in it a few times, if you feel the need to do so. VR is not a mass medium, but the same is true about poetry. I like working with this sense of “specialness”, the fact that there’s something magical about it, that you have to make an effort to find a VR event and get there, that you have to really want to break through your fear of the new technology. I think there will be more and more VR experiences, because people are curious; the technology will become more popular, and VR experiences will enter other spaces.

AP: I wonder if we’ll actually be talking about literature, or at least poetry, in VR in the future. Thank you, both for taking the time to have this conversation and for stimulating all the senses of my reading-loving body. The experience of your poem in VR will certainly stay long on my list of memories to return to.

The “Nightsss” VR project was realized in Visual Narrative Lab (vnLab) and financed under the Ministry of Science and Higher Education programme within the framework of the “Regional Initiative of Excellence” for the years 2019-2022, project number 023/RID/2018/19. 

The Polish version of this interview will soon be published in “Techsty” magazine.

This interview arose as a part of the research conducted within Bristol+ Bath Creative  R+D Amplified Publishing Pathfinder. Realisation of Agnieszka’s research on VR and AR as literary platform was possible thanks to the funding from the Polish National Agency for Academic Exchange in the Bekker programme (grant agreement No PPN/BEK/2019/1/00264/U/00001).

Have you ever felt your skin prick up as you walked into a foreign environment? Throughout my life there have been times where I’ve entered an unknown venue, be it when exploring the world as a solo traveller, or starting a new job. The mixture of excitement and anxiety that I’ve associated with an entry into the unknown has been a full body experience, reflected in the pores of my skin and in the heat pulsing through my limbs.

I reflected on this phenomenon after my somewhat recent experience of Tender Claws’ The Under Presents, which is described by its makers as ‘an intriguing multiplayer experience set between two worlds’. Others might describe it as a clever and mysterious VR game that affords the user six degrees of freedom.

Image from: https://tenderclaws.com/theunderpresents

First activating the free demo version of this experience in the Oculus Quest, I found myself in a dark and waterlogged wasteland, with little idea of what to expect. I edged towards the ruins of a nightclub from which muffled music exuded. After several moments of uncertainty, a mysterious guide appeared and shepherded me into the building. I was followed into a room full of doors by masked figures in black capes. After some moments, I realised that their strange actions were actually a replay of my own awkward explorations of the space moments earlier. Past and present collided, giving me an uncanny insight into my own behaviour. As with the ‘real world’ experience described above, I felt acutely aware of my own body in this unknown virtual environment.

After much experimentation, I managed to access the depths of the nightclub at the heart of The Under Presents, where a number of other black hooded spectators milled about, enjoying the on-stage entertainment. Are these real people like me, I wondered? It certainly seemed a possibility. Like me, the other guests clumsily engaged in creating and eating food while exploring the environment. One hooded figure handed me a keyboard guitar before scurrying towards an exit.

Image from: https://tenderclaws.com/theunderpresents

Unfortunately for me, I came to this VR experience somewhat late. It’s notable that earlier iterations of The Under Presents saw live actors teleported into the multiplayer experience to interact with users in what could be described as virtual live immersive theatre. A quick scan of the reviews of the project on the Oculus website reveals the novelty of this spontaneous and communal experience. A video on Tender Claw’s website outlines the innovative interface that allows actors to enter into the VR experience, choosing their appearance and powers that they can utilise to interact with players, so as to enhance the narrative experience. 

The affective nature of video games featuring navigable 3D environments has been explored by a number of scholars (Crick, 2011; Klevjer, 2012; Scriven, 2018). This comes after significant work interrogating the way that cinema ‘moves’ and ‘touches’ us on a bodily level (Marks, 1996; Shaviro, 1994; Sobchack, 2004; Williams, 1999). Klevjer describes how video games extend our bodies into the screen space:

When we play, because the avatar extends the body rather than pure agency or subjectivity, screen space becomes a world that we are subjected to, a place we inhabit and where we struggle for survival. We learn to intuitively judge, like we do in the real world, the opportunities and dangers of the environment. Klevjer, 2012, p. 13

This quote resonates with my experience of The Under Presents, where I slowly learned the rules of the virtual world, thus enabling me to progress through the experience. In doing so, I had to embrace ‘unknowing’, meaning I had to come to terms with my vulnerability as a virtual body in a foreign environment. I was never quite clear on where I was located or where I was headed, but in retrospect, this was half the fun. 

In recent years, emerging scholarship has probed the specificity of narrative VR experiences in terms of spectator positioning, presence and/or embodiment (for example, Dooley, 2021; Nicolae, 2018; Vosmeer & Schouten, 2017). The bulk of this work is concerned with cinematic virtual reality rather than more interactive VR experiences with narratives that are emergent or open-ended. I would suggest that The Under Presents offers a case study of a narrative experience that successfully draws upon immersive theatre and video game conventions, as well as cinematic tropes, giving rise to a unique embodied experience that is about ‘story making’ rather than ‘story telling’. The computer-generated story world provides a playground for multiplayer interaction, but the mystery of a stranded ship (the ‘time boat’) sits at the heart of the narrative experience. The limits of the game seem linked to the user’s tolerance for learning the rules of the world; in other words, there are seemingly endless threads to explore, if one takes the time to uncover them. 

My enjoyment of The Under Presents makes me think that the notion of embracing ‘unknowing’ is an apt one for VR creators as well as audiences. In the past, I’ve written about the need for creators to guide the viewer through a narrative VR experience, creating attention cues and allowing time for exploration and acclimatisation. As described above, the makers of The Under Presents adopted these techniques to some degree. However, I’d argue that the experience’s frequent lack of signposting and foreshadowing fosters a sense of uncertainty and intrigue that increases rather than decreases engagement. Maybe this is what great VR storytelling is all about – creating an affective experience that allows the user to ‘live’ the story while navigating unfamiliar territory. 

Postscript: The Under Presents is once again featuring live actors in the environment in the month of April 2022, so this is a good time to visit or revisit the experience!

References

Crick, T. 2011. ‘The game body: Toward a phenomenology of contemporary video gaming’. Games and Culture, 6(3), 259-269.

Dooley, K. 2021. Cinematic Virtual Reality: A Critical Study of 21st Century Approaches and Practices. Springer Nature.

Klevjer, R. 2012. ‘Enter the avatar: The phenomenology of prosthetic telepresence in computer games’. In The philosophy of computer games (pp. 17-38). Springer, Dordrecht.

Marks, L. U. 1996. The skin of the film: experimental cinema and intercultural experience. University of Rochester.

Nicolae, D. F. 2018. ‘Spectator Perspectives in Virtual Reality Cinematography. The Witness, the Hero and the Impersonator’. Ekphrasis. Images, Cinema, Theory, Media, 20(2), 168-180.

Scriven, P. 2018. ‘The phenomenology of the “other” in computer game worlds’. Games and Culture, 13(2), 193-210.

Shaviro, S. 1994. The cinematic body. U of Minnesota Press.

Sobchack, V. 2004. Carnal thoughts. University of California Press.

Vosmeer, M., & Schouten, B. 2017, June. ‘Project Orpheus: a research study into 360 cinematic VR’. In Proceedings of the 2017 ACM International Conference on Interactive Experiences for TV and Online Video (pp. 85-90).

Williams, L. 1999. Hard Core: Power, Pleasure, and the” frenzy of the Visible”. Univ of California Press.

The Writing Platform is looking to commission articles of approx 750 to 2,000 words on creating stories with augmented and virtual reality technologies. This might include the use of AR and VR in fiction and non-fiction literature, journalism, theatre, movies, and games as well as articles that explore AR and VR as tools to promote existing works or as alternatives for live social events (e.g. face-to-face meetings with authors). 

We are looking for papers that examine forms such as: AR stories in which you become one of the protagonists (e.g. Wonderscope), VR experiences in which spoken and written word plays an important role (e.g.The Chalk Room) or VR poetry and VR experiences that blend spoken poetry with dance (e.g. VR Nightsss), AR books and comics (e.g. Modern Polaxis), VR theatre and opera (e.g. The Under Presents: The Tempest), performances that blend VR with interaction between the performer and spectator (e.g. Draw Me Close), holographic theatre (e.g. Chronicle Of Light Year) and live augmented reality glasses performances (e.g. Verrat der Bilder), as well as AR and VR games (e.g. Pokémon Go! or Virtual Virtual Reality). We are also interested in articles on VR experiences that can be used for therapeutic purposes (e.g The Wayback), VR and AR documentaries (e.g. The Waiting Room), VR and AR literary adaptations (e.g. Metamorphosis VR) or VR literary archives (e.g. Digital Fiction Curios), and many more. 

We are interested in the ways in which technologies can produce new forms of storytelling and are also keen to receive articles focusing on expanding and diversifying audiences for immersive storytelling experiences as well as on the ethics of using new technologies and platforms in storytelling. 

The deadline for submission of proposals and ideas for articles has been extended to 1st November 2021.  Articles will be published on The Writing Platform website late 2021/early 2022. Once your proposal is accepted we will negotiate a deadline for the full submission with you.

Your proposed article might fit into one of the following categories: 

Resource: for example, a how-to guide for practitioners about creating stories with any AR/VR tool

Research: for example, an overview of a collaborative research project on AR and VR in storytelling or an examination of the impact of using VR and AR on enhancing audience immersion 

Experience: for example, an account of a VR and AR experience that you have experienced or developed

News: for example, highlighting a new project or opportunity in the VR/AR field that our readers might not have heard of before

Projects: a case study of an especially innovative or inspiring project with impactful outcomes 

We have a small commissioning fund for freelancers (£100 per article).

To propose an article, email Agnieszka Przybyszewska (a.przybyszewska@bathspa.ac.uk) with a 100 word overview of your idea and tell us which category or categories it would best fit into. 

Timothy West’s satirical radio play “This gun in my right hand is loaded” is a wonderful demonstration of what happens when writers who are used to one particular medium (in this case the screen) adapt their idea for another (in this case radio) and fail to account for the affordances and limitations of the form.

At one point early on, the protagonist says (adopt posh 1960s BBC radio voice) “Whiskey eh? That’s a strange drink for an attractive, auburn-haired girl of 29”, hilariously exposing one of the singularities of radio, which might be more subtly manoeuvred by a writer with the right expertise. 

Every medium requires a different approach and set of skills in order to really make it sing and virtual reality is no different. But VR is still evolving its form and even its terminology. What one person means by virtual reality may not correspond to what another does, so let’s start by trying to pin down what we mean by VR. 

For many, VR is any computer-simulated environment that you can access via a VR headset. However, purists say that 360 video, though it conforms to this definition, is not ‘true’ VR. If you try to move forward inside a 360 video, the world’s edges move with you. It’s a bit like having a fishbowl on your head (stay with me).You can look at the fish to the left or the right, up and down, or behind you, but if you try to get a closer look and take a step forward, the whole bowl comes with you – a very disorientating feeling when you first experience it.

 VR that uses a gaming engine like Unity or Unreal, whether that world is created using CGI, photogrammetry, volumetric capture or a combination of all three, does not have this ‘depth’ problem and is generally more interactive. You can choose whether to approach certain things or move away from them just as you can in the real world.

Writing for so-called ‘true’ VR is different from writing for 360 video because the former generally entails branching narratives and interactions and is a more complex process. Writing for 360 video tends to involve a straightforward linear narrative, but whether you are writing for one or the other there is something very important to bear in mind with both – the non-traditional point of view of the user. 

Again, terminology is tricksy here. ‘User’ is a gaming term and for less gamesy experiences, many still use the word ‘audience’ (originally from latin meaning listening or hearing). Others talk about ‘participants’ or ‘viewers’ or even ‘viewsers’ – a useful hybrid coined by media theorist Dan Harries.

When writing for VR, it’s important to realise that the viewser’s POV is self-directed, omnidirectional and present. The viewser is not being told where to look and what to concentrate on by the framing of a shot. There are no shots or cuts and they can (and will) look in any direction around them. This needs to be accounted for by the writer.

It’s wonderful that in VR the viewser has such freedom to choose where to focus her attention. Gone are the slow from-the-legs-up lingering shots on women’s bodies, for example. If you want to, you can turn your back on the leading lady and just take in the sky or the floor. But it’s very likely that some of your viewsers will completely miss something important that is unfolding in ‘front’ of them and check out the skirting boards at precisely the wrong time. If something in the story is crucial and should not be missed, it needs to be signposted with sound design or some other sleight of hand, for example with a “Coooey! Over here!”, just before it happens.

By saying that the viewser’s POV is ‘present’, I’m referring to the immersive nature of VR. Writers who are used to writing for film or TV will need to take into consideration the fact that viewsers are not outside of a frame looking in. They are inhabiting the world – not necessarily acting or participating in it (though they may be), but always taking up space in that world and experiencing themselves as part of it, which entails a completely different mindset from traditional writing for frames. There is no hard and fast rule, but the viewser should probably be acknowledged in some way and it should make sense for her to be there. 

Story Studio, Oculus’ explorative VR studio, pinpointed this need to have the viewser’s presence acknowledged with what they called The ‘Swayze’ Effect (after Patrick Swayze’s character’s feeling of dislocation in the film Ghost). Viewsers are not just observers; they are experiencers, embodied in a world and they should be accounted for and considered as such throughout the writing process.

VR is a rapidly evolving art form that is still finding its feet so it’s pretty hard to predict exactly what kind of writing it might entail in a year or even a few months’ time. Until fairly recently, there were no consumer headsets, let alone hand tracking, haptics or wireless rigs. Things are shifting quickly, but we can hazard a guess that VR will become increasingly social and increasingly interactive with writers needing to be able to come up with complex branching narratives, which means a lot of writing!

With the pandemic leading to increased headset sales and with the market for VR growing, now may be the time to try out some VR for yourself, see what works and what doesn’t and to have a go thinking outside the frame. 

Screenshots is a regular feature by Simon Groth, highlighting a project, app, or other resource of interest.

A Place Called Ormalcy
by Mez Breeze

Meet Mr Ormal, a happy-go-lucky law-abiding citizen of Ormalcy, a Utopian world full of contented creatures and happy citizens. Happy, happy citizens.

Right from the beginning, something is off in A Place Called Ormalcy. Its nonsense  language, garish colours, and warped illustrations might come across as camp if not for the clear sinister undertone that becomes more overt as the story progresses. Each chapter is presented in its own VR environmentand the technology adds to the unsettling nature of the piece. These three-dimensional spaces, suspended in a void and frozen in time, enable the reader to zoom, rotate, and deconstruct. You’re left with the feeling you can access parts of a picture book that should be hidden from view.

Told in a storybook style over seven short chapters, A Place Called Ormalcy is a clever allegory using a child-like sensibility to evoke a chilling tale of authoritarianism and conformity.

http://mezbreezedesign.com/vr-literature/a-place-called-ormalcy/

Screenshots is a regular feature by Simon Groth, highlighting a project, app, or other resource of interest.

Our Cupidity Coda

by Mez Breeze

To read through the text of this VR poem by Mez Breeze takes only minutes, but it would be a mistake to think of this work as slight or even brief. Our Cupidity Coda is deceptively simple, using the VR environment as an extension of a text that already carries heavy emotional resonance charting the course of a relationship from spellbound beginning to bittersweet end. The imagery experienced early in the piece gives away to arresting majesty and even moments of fear.

Press Image for “Our Cupidity Coda”: VR Literature

Created in and intended to be experienced as VR, this piece avoids the pitfalls of its technology. It emphasises emotional and intellectual immersion over the pure sensory experience and rewards multiple viewings. It was recently shortlisted for the QUT Digital Literature Award.

http://mezbreezedesign.com/vr-literature/our-cupidity-coda/

Way back in the wilds of the year 2008, artist-extraordinaire James Morgan and I engaged in an animated discussion about Augmented and Virtual Reality. At that time James and I were collaborators-in-crime in the Third Faction Collective, a group of digital artists intent on constructing game interventions in Massively Multiplayer Online Spaces. During this discussion, I pitched to James an idea to establish an online space devoted to all things Synthetic Reality based (my umbrella term for Virtual Reality, Augmented Reality, and Mixed Reality). This space, called Augmentology 101, intrigued James to the point where a decision was made to sponsor it through the Ars Virtua Foundation and CADRE Laboratory for New Media. What followed was an amazing exploration into the creative potentials of Synthetic Reality – what’s now known as XR (Extended Reality) – and how it might manifest within the realm of electronic literature.  

It’s now been 10 years since the initialisation of the Augmentology 101 project. During this decade, there’s been a major upswing in VR and AR production and development, with impactful XR content such as Firebird – La Péri (a 2016 English/Chinese/French multilingual VR Experience) and Queerskins VR (2018) being standout examples.

Screenshot from the 2016 Multilingual Virtual Reality Project “Firebird – La Peri”

My own attempts at merging digital literature into developing XR fields have been multiple and varied, originating in delving into VR in the 1990’s when VRML was the shiny new thing. Surprisingly enough, the creative and technical challenges that VR creators faced back then are similar to those we face today: high performance requirements, mainstream adoption hurdles (see: Gartner Hype Cycle), and monetisation dilemmas are all relevant. Likewise, skillsets required by VR content creators in the mid 1990’s again parallel XR creators of today, including developing a deep knowledge of spatial storytelling logistics; emotional intelligence; and the ability to formulate story experiences that take into account various hardware and platform limitations such as field of view constraints, tethered headsets restricting natural movements, and hardware specific limitations like the screen-door effect.

In terms of XR projects I’ve produced in the last decade, a brief selection includes conceiving of and co-developing the 2013 anti-surveillance AR game #PRISOM, and in 2015 mapping out with Andy Campbell the (now unfinished) PC/VR project Square Ebony that was to be filled with: “…movement/imagery like huge ‘Panic Room’ landscaped letters…a force field of green…branches intertwined…tangles being text…[that] revolves around an entity…this entity is slowly reconfiguring itself…at the top of a hill/mountain/plateau surrounded by brackish water” (notes from Square Ebony Project Meeting and Documentation, Breeze and Campbell, March 10th 2015). In 2016 I lectured as part of the “Future Possible: Beyond the Screen” Series which centred on how VR can transform creative practice, and which also included a live VR performance walkthrough using one of my Tilt Brush created works. In 2017 I keynoted at the Electronic Literature Conference with a VR performance presented both live at the Conference and simultaneously in Virtual Reality.

“Layering the New real: Tracking the Self in Disembodied [Un] Virtual Spaces” Keynote

Press Image for “Our Cupidity Coda”: VR Literature

Thorough participation in a high-end VR based experience like Perpetual Nomads hinges entirely on immersion, which is triggered initially through the audience having to don gear that firstly reduces their ability to engage in their actual physical space in standard ways (their vision and hearing being “co-opted” into a VR space). The leap of faith the audience needs to make to establish a valid and willing suspension of disbelief (as Samuel Coleridge so aptly phrased it) is already set in motion by the fact a user is entirely aware from the moment they slip on a VR Headset that their body is in essence hijacked by the experience (haptically, kinetically), as opposed to a more removed projection into a story space via more traditional forms (think book reading, movies, tv). Such body co-opting might lead a user to disengage from the VR experience from the very beginning which will reduce the likelihood of true immersion: alternatively, they may readily fall headlong into the experience with an absolute sense of engagement and wonder (the preferred option as a VR content creator!) if the work has been precisely crafted.

For the most part, XR projects such as those mentioned above currently exist only in the mainstream margins, with a majority of experiences requiring costly high-end VR rigs and expensive desktop computers that demand audiences experience the works in their optimal state. To counteract this selective catering to the exorbitant end of the XR market, in early 2018 I had the idea to create a VR Experience that would reduce the mandatory use of high-end tech. This project would instead cater directly to a range of audiences by crafting a work that could be experienced across a far larger (and much more accessible) range of lower-end tech. This VR Literature work is called A Place Called Ormalcy.

Title Image from the “A Place Called Ormalcy” Press Kit

A Place Called Ormalcy is digital literature designed for, and developed in, Virtual Reality. It was constructed using the Virtual Reality Application MasterpieceVR to craft the 3D models, with each chapter (made up of 3D models, text, and audio components) then combined and hosted via the Sketchfab platform. It’s comprised of a text-based story made up of seven short Chapters which are housed in 3D/Virtual Reality environments. It can be accessed via a wide range (crucial in terms of its social commentary aspect) of mobile devices, desktop PCs and both low-end and high-end Virtual Reality hardware. Audiences using the cheapest type of VR equipment (such as Cardboard headsets) are able to access complete versions of this VR literature experience, as are users of any net connected mobile device with a WebVR-enabled browser.

The story of A Place Called Ormalcy (warning: spoilery parts ahead) unfolds through a series of snapshots of the life of Mr Ormal, a happy-go-lucky law-abiding chap who resides in the aesthetically cartoonish world of Ormalcy. Ormalcy exists in an alternative universe complete with its own idiosyncratic language patterns. The storyworld initially presents as a Utopia full of innocent “claymationesque” contented creatures and happy citizens. As the story plays out, however, it soon becomes apparent that in actuality, this VR Experience allegorically traces the makings of a dystopic society, and how such fascist principles can arise in the most benevolent of places. This VR Literature work has social commentary at its very core, commenting directly on and about the rise of current totalitarian trajectories and the contemporary malaise, confusion and accompanying acclimatization patterns.

“A Place Called Ormalcy” Chapter Progression

A Place Called Ormalcy uses a combination of WebVR, 3D, VR, text and audio assets in ways that mirror a slow dystopian creep. In the desktop and mobile versions, each chapter becomes progressively visually cloistered, with dark fog and grainy distortions increasing to finally create a type of gun-barrelled claustrophobic effect. This combines with a gradual leaching of the intense colours found in the free-flowing organic imagery of the initial Chapters which results in a startlingly stripped back, fuzzy palette and model constructions: vibrancy gradually bleaches out to stark black, white and greys. Correspondingly, the 3D tableaus and audio tracks likewise alter from an initial complexity – Mr Ormal begins his story journey waving directly to the audience in “Chapter Wonne” in a bright and blooming space – which incrementally shifts towards the dramatically minimal in the final “Chapter Severn” where Mr Ormal transforms into (…spoiler alert here…) something vastly other.

In the VR version of A Place Called Ormalcy, additional effects mark the dystopic “boiling frog” dilemma that Mr Ormal faces. Each VR tableau subtly increases in size and scale as the Chapters progress, with the audience finding themselves in the climatic Chapter in a looming monochromatic set surrounded by huge windowless block-shaped buildings devoid of detail – except multiple, and menacing, “88” shaped logos (and the awfully transfigured Mr Ormal). In the VR version, the text becomes increasingly difficult to navigate, with the audience having to teleport, twist and turn in the VR Environment to read each annotation, echoing the “fake news” proclamations of our contemporary Western world where it is becoming increasingly difficult to access truth over relentless propaganda.

Although A Place Called Ormalcy may seemingly convey a message of hopelessness or helplessness, the ending does contain clues that all is not lost in this particular dystopian scenario – the final soundtrack offers hope, with protestors chanting and proclaiming resistance as key. Just as VR Literature can work to extend the creation of accessible electronic literature beyond the text-centric to truly encapsulate the haptic and the spatially-oriented, A Place Called Ormalcy illustrates how XR projects can act as relevant social commentary at a time when it is sorely needed. I look forward to continuing to promote, create, and experiment with stretching the limits of VR and AR while producing XR projects that are openly accessible, as well as socially relevant.

Last year, I almost made an empathy game.

It never quite progressed far enough to merit a title, but it did have a main character, some painstakingly-drawn pixel background art, and a mini-game where you could choose what to watch on TV from a list of fictional programmes with names like “Nature Squash” and “Changing Feet”. Here’s a screenshot of it, which – by the way – features my first ever serious attempt to draw a human face.

I never finished this unnamed game, chiefly due to massive technical incompetence and a serious case of scope creep. After a couple of spirited attempts at pruning the mess it had become, I filed it away as a lesson learned and went back to writing about sexually-frustrated mermaids and haunted theme parks. A few months later I was trying to explain the failed project to a friend and struggling somewhat for a definition. After several of my stumbling attempts to outline its basic premise, their expression cleared and they nodded enthusiastically. “Oh,” they said, “I get it. It’s an empathy game.”

Empathy Games

Until that point, I had only been vaguely aware of the label and had assumed it applied solely to a very specific bunch of video games which eschewed the normal notions of winning and losing in favour of generating empathy for the protagonist. Depression Quest was, in my mind, the archetypal example – a text-based adventure game in which the options that might most help your character were frequently crossed out or inaccessible, just like many potential self-care options for people in a state of depression.

As it turned out, however, the term had adopted a much wider meaning while my back was turned. Games with any kind of serious disposition are now routinely slapped with the empathy game label – something that makes as much sense to me as describing any serious work or literature or cinema as an “empathy book” or “empathy film”. Perhaps it’s due to the relatively nascent state of storytelling in gaming that we feel the need to section off anything slightly serious and label it with one of its intended effects. Given that “games” encompasses everything from Candy Crush to The Last Of Us, it makes sense to perhaps segment the serious from the not-so-serious in some fashion. Even so, the label of empathy game often seems a little… off. Surely any game generates empathy to some degree? Most readers will, for example, have witnessed a player wince as though in pain when their character is hurt.

Whether the classification of empathy game is here to stay or whether, as storytelling in games develops, it will soon be folded into the wider taxonomy of genre remains to be seen… but while the category exists, it seems worthwhile to take a quick survey of the kind of things that are being put into it. To that end, here are four empathy games that have made a distinct and reaching impact.

That Dragon Cancer

This point-and-click style game was created by two parents whose child Joel succumbed to cancer during its development. It chronicles his short life from a multitude of perspectives – there are scenes where you search for Joel in an eerily empty stretch of parkland, scenes where you try vainly to soothe his pain as he undergoes treatment, and scenes where you play as the character in a story being read to Joel as he lies sick in hospital. Although the game can be completed, it cannot be “won” or “lost” in any traditional sense – no matter what you do Joel cannot be saved. Instead, the game attempts only to give the player a brief window into the experience of having a child with a terminal illness.

This War Of Mine

This War Of Mine is frequently cited as a fine example of the empathy game genre – and the mechanics of play (direct a small group of civilians as they attempt to survive in a war zone by means of scavenging, improvisation and careful resource management) undoubtedly generate a great deal of empathy not just for the characters involved, but for any civilian caught in the midst of a conflict. It does, however, manage also to cater extremely well to players who just want some hours of entertainment or a challenge to beat.

Papers, Please

Billed as “A Dystopian Document Thriller”, Papers, Please puts you in the shoes of an immigration officer in the fictional nation of Aristotska. Your job is to process a seemingly-endless line of immigrants and returning citizens, checking their passports and other documents against an unwieldy list of rules and regulations before stamping them with either a green ACCEPTED or a red DENIED. Making a decision, however, is not always a straightforward process. Do you bend the rules to avoid splitting apart families or turning aside desperate immigrants? And, if you do, how will you keep your own family alive and well when your salary is docked in response to your “mistakes”? Papers, Please provides no easy answers to these questions, and it’s up to the player to balance their conscience against the risks involved.

Spent

Browser-based game Spent is shorter than the other titles listed here, and is additionally free to play online. It is, essentially, a poverty simulator. After selecting a few basic details to define your virtual life, you are tasked with surviving an entire month on a budget of just one thousand dollars. After dropping a minimum of seven-hundred-and-forty dollars of that on rent, it’s a significant (but not impossible) challenge. Along the way, you’ll have to make decisions about the way you work, the way you treat your children, and the medical treatments you seek. The graphical interface is simple enough, consisting entirely of buttons, sliders, text and icons, but the descriptions themselves are generally enough to paint a grim picture. Supported by the Urban Ministries of Durham, the game is designed to impress upon players the very human cost of living in poverty.

Games That Inspire Change

The gushing reviews of some of these titles often seem to suggest that people are surprised that games can generate real feeling, convey serious points, or even affect social change. But why should this be news to us? Stories have long since been used to great effect to sway feeling and change behaviour – and games are a superb method of immersive storytelling. There’s a wealth of evidence that giving a player agency over the outcome of a story can increase their investment in it and that interactive or gamified experiences are more absorbing and compelling than their linear or passive counterparts.

360 Syria is a project that quantifiably demonstrates this. The VR experience (it functions, for the most part, like a self-guided virtual tour) features photospheres of Syria taken in the aftermath of barrel bombings. By putting on a headset, participants can experience sights and sounds as though actually present on location. Amnesty International – who used the VR experience to solicit donations – reported an almost ten percent increase in the number of direct debits over traditional fundraising methods.

Games, clearly, can not only generate empathy but also spur people to act on those empathetic feelings. Indeed, they’re very, very good at doing so (did I mention that Spent drew in almost twenty-five-thousand dollars in donations?) Perhaps, as the form develops its storytelling abilities, we’ll fold away the term empathy game (in the same way that we might fold away the term “nutrition food”) and start instead dividing games only along genre lines. When that happens, no doubt, we’ll have generally accepted that games are something akin to any other form of art and that it’s not unusual for them to have a real and meaningful impact.

SOURCES

http://www.tandfonline.com/doi/abs/10.1080/00913367.2001.10673646

http://www.digra.org/wp-content/uploads/digital-library/11307.06025.pdf

http://mprcenter.org/review/gregory-video-game-engagement/

http://digitalcommons.lsu.edu/gradschool_theses/1858/

http://www.thirdsector.co.uk/third-sector-awards-2016-digital-innovation-year-winner-amnesty-international-uk-360-syria/digital/article/1406932