Designing a VR Experience in a Covid-19 World

In September 2019, five months before the pandemic, I moved to Toronto to begin a PhD at York University. I had been to the city before, and as I was a new international student  I thought I would make new social connections in school. However, in February 2020 the world transformed into its virtual ‘metaverse’ form, and I realised I would not be able to make friends and deepen any new relationships through Zoom, Teams or Slack. The loneliness hit me hard and even though leaving Canada was an option I was worried that I would not be able to return to finish my studies, so I stayed.

Looking back at the two years through a linear order of events, it seemed to me as though the time had been a vacuum. I could not put events into a linear narrative and I could not remember many things that happened in those two years. As a community as well as individually, we went through a repeated experience of fear, uncertainty, and freedom limitation. Negative emotions, such as boredom and anxiety, influence our long-term memory. Moreover, numerous studies have reported that ‘human cognitive processes are affected by emotions, including attention, learning and memory, and reasoning’ (Chai M. Tyng et al., 2017, p. 1454).

So, I wondered whether one of the post-pandemic consequences could be trauma (and possibly a collective trauma). Research on trauma shows that “the social environment does not have a direct and static impact but is mediated by emotional experience, the way it is lived through, interpreted, and processed on the basis of social, personal, and situational resources (today often termed as potential for resilience)” (Busch & McNamara, p.324). Each person experiences the hardships of life differently and some are more resilient than others. I was stuck, anxious and could not focus so I decided to do the one thing that satisfies me: making things. One of these ‘things’ was an interactive experience dealing with my own pandemic trauma. I decided to experiment on myself and find healing methods through interactive VR. If we take into account that the mind finds tools and technologies in the world in order to expand cognitive space constantly. And, if thinking is feeling and feeling is thinking, our emotions co-expand into this space. If we take creation into the equation and if “making is thinking is feeling” (Gauntlett, 2018), I am persuaded that emerging media are the best form for interactive collaboration between humans and machines.

The Making

The experience was created through the AI Storytelling Project as part of the Immersive Storytelling Lab (ISLab) at York University in Toronto. During my job there as an XR creator, I started to explore how to use interactive and immersive media; such as mixed, virtual and augmented reality (XR) and co-creation with the machines (Loveless, 2020; Wolozin, Uricchio and Cizek, 2020; Guzman & Lewis, 2020) for post-pandemic experiences. Specifically I focused on Natural language processing (NLP), a subdivision of artificial intelligence often used for different aspects of human-machine communication (e.g. speech recognition, text generation, speech-to-text and text-to-speech transformation, etc). Thanks to the cooperation between ISLaband the NLPsoftware-based storytelling platform Charisma.AI, I was given access to its Beta version, and I started exploring different options for an immersive and conversational experience.

My methodology during the creation of the virtual reality piece Home Is the World VR took the form of creation-as-research, where “creation is required in order for research to emerge” (Chapman & Sawchuk, 2012, p. 19). Via this VR piece, I investigated the relationship between technology, creation and the human condition, or as Guzman and Lewis explain, “how people understand AI in relation to themselves and themselves in relation to AI” (2020, p. 77).

I followed in the footsteps of the rich tradition of human-machine interaction, elaborated by Sherry Turkle when the personal computer was being adopted into everyday spaces. She described it as a “metaphysical machine,” a concept which led to the study of AI as a challenge to existing conceptualizations of the nature of humans” (Guzman & Lewis, 2020, p. 80). Emerging technologies may eventually push past the boundaries of human communication, meaning that human-AI communication may change how we communicate with each other as humans and with other entities.

Before I start describing the process of designing Home Is the World VR, I want to give a short synopsis to introduce the technology and the main idea of the piece. The VR experience is aimed at headsets with a passthrough option (i.e. Oculus Quest, HP Omnicept Reverb 2 etc.) –  a feature that allows users to step outside their view in VR to see a real-time view of their surroundings. Passthrough uses the sensors on the headset to approximate what one would see if they were able to look directly through the front of their headset. Hence, the interface is created by an ephemeral space between virtual reality and the user’s physical environment where they converse with the AI character and are asked to answer its questions and to follow a series of sensory-led tasks to induce pleasant memories before the Covid-19 pandemic. Such tasks include smelling coffee beans, conjuring the smell of an early summer morning, the sound of walking on snow, the taste of a delicious dessert, the touch of a plant etc. Then, the Charisma.AI software records a user’s spoken memories of the pandemic and generates a personal pandemic story for each user, individually. The twist: it re-tells the experience somewhat differently. It re-contextualizes the user’s story in a world full of positive news.

Project workflow

 Through Charisma.AI’s dialogue engine, and Google Cloud NLP services, players and audiences can meet with virtual characters, converse with them, and change the story. The AI character is created through the Charisma.AI platform and Google Cloud Speech-to-Text service. Charisma.AI uses the language of storytelling, with built-in features such as emotion, memory, scenes and subplots (Charisma, 2017 – 2021). However, it is not GPT3 based and works with a machine learning dialogue engine, which I found exciting when designing a semi-scripted interactive piece.

My research and ideation of the project started with the following questions. What if we could go back in time and re-shape our memories through an interactive AI storytelling experience? What happens when a human and AI re-create memories together? What if the experience played with the idea of the flexibility of memory and its ability of re-constitution? The connecting points in these questions were the ability of language to be performative, making sense of the world by creating stories, putting events and experiences into linear narratives, and the flexibility of human mind. And it seemed as though Charisma.AI, as a storytelling platform using such NLP modes as intents and sentiment, could not only be used as a tool to make stories but also as the content itself.

The saying ‘think before you speak’ has meaning in neuroscience – the formation of words acts as a delaying function, giving the brain time to deal with information input and retrieval. Traumatic experiences leave such an emotional mark on our brains that, when recalling certain events, we tend to be overcome by emotions and are not able to express what is happening to us through language. It is not by accident that trauma is often dealt with by speech therapy and storytelling. Remembering traumatic events through language and storytelling helps us to deal with  them.

Interestingly, there is a connection to sociolinguistics here, specifically to  J.L. Austin’s theory of performative speech acts. In his book How To Do Things with Words, Austin establishes performative utterances, which not only describe a given reality, but also change the social reality they are describing (1962). The theory was later elaborated by Judith Butler in her book Gender Trouble (first published in 1990), where she introduces gender as a discursive and performative practice.

According to van der Kolk, “Traumatic experiences are exceptional because these intensely emotional events are not encoded into the ongoing narrative states” (2014). The traumatic experience is recorded as separate and dissociated from other life events, and, thus, it takes on a timeless and alien quality. In healing trauma, language is crucial.

John J. Ratey claims that when a subject tries to recall a traumatic experience they are overcome with emotion and are not able to express it in words. They are ‘dumb struck’ for a variety of reasons, one of them being that an important part of our brain responsible for emotion amygdala ‘overreacts’ while another part responsible for language and speech, the so called “Broca’s area shuts down” (2001, p. 210).

Further, Ratey claims that “the formation and recall of a memory is dependent on the environment, mood and gestalt at the time the memory is formed or retrieved” (2001, p. 208). Each memory is created from a vast interconnected network of pieces in our brain such as language, emotions, beliefs. And our daily experiences alter these connections and, therefore, we remember things differently in different phases of our lives.

Apart from neuroscience and linguistics, the theoretical approach framing the project is affect theory, which helps to disclose the ways technology intersects with our limited proximal senses, rhythm and sense of motion and embodiment. When affect is processed through cognition – once the signal from the amygdala (limbic system) reaches the prefrontal cortex in our brains – it becomes an emotion. It occurs through cognitive action and relations between agents (humans, non-humans, things, environment).

These cognitive actions are inherently performative, such as language, bodily and facial gestures, or tone of voice. Sara Ahmed (2014) calls these relations “contact”. She argues that already when we feel that something is good or bad, it involves “reading the contact we have with objects in a certain way” (Ahmed, 2014, p. 6). Contact involves a process of reading, attribution of significance, it “involves also the histories that come before the subject” (Ahmed, 2014, p. 6). Emotions are, thus, culturally and linguistically constructed performative actualizations of affect. Ahmed posits emotion as cultural construct through language (2014) as when we name and perform our emotions they become solidified and transmitted to others.

This process of emotion transmission can be also called “affect contagion” (a perfect example is sentiment contagion in social media through text). Plus, according to Brian Massumi, emotions are culturally and linguistically situated, “emotions are embedded in the arbitrariness of language and gestural code (including face) involving cognition, and through which we assign these qualities, and which carry meanings in order to be communicated” (1995, p. 89).

Stage Two: The Design

Scripting in CharismaAI

The main aim of the piece is to trigger positive emotions via sensual memory associations and language. When reframing the memories through emotions, I applied methods in trauma research, using questions and sensory led tasks that connect past, present and future through emotion and affect (Damasio, 2005; Van der Kolk, 2014; Busch & McNamara, 2020; Bloomaert et al., 2007).

Upon entering the virtual world, users encounter an omnipresent AI character, which converses with them and gives them a series of sensory-led tasks to induce their memories and emotions associated with the pre-pandemic world and apply them in the ‘now’. First, the user is asked by the AI character to remember a situation during the Covid-19 pandemic. Second, they are prompted to answer questions such as: what sign are you? Where did you spend your pandemic time? Were there other people? In what month did the significant event happen? After the AI character collects all necessary information, the software generates a positive event, which happened in the same month in 2020 or 2021. These generated pieces of text come from a corpus of news articles about positive events during the pandemic. Here are some examples:

“In March 2020, we learned that a group of dogs trained to protect rhinos from poachers have saved 45 rhinos in South Africa. The dogs, including beagles and bloodhounds, among other breeds, were trained from birth to track down poachers alongside humans in Greater Kruger National Park.”

“June 2020: The Supreme Court rules that no one can be fired for being gay or transgender, and Beyonce releases Black Parade. Yaas, Queen B!”

An example from my Miro board of the corpus of  positive news events in 2020

Apart from the language, different senses, such as smell, touch, sound, vision and taste, are directly connected to our emotions (e.g. the sense of smell has direct relation to our limbic system, which is why it triggers memories and emotions without any cognitive interference). Therefore, in the second part of the experience, the user is asked to follow sensory-led tasks to deepen the positive emotions. The next image presents different sensory tasks which were brainstormed during a workshop with several participants.

Things to feel with

Blueprinting in UE4 with CharismaAI plugin

In the finale, the AI character summarises the collected information about the user. However, it replaces the negative event with a positive one. And it asks: do you feel better?

Prototyping

Stage Three: User Testing

User testing is crucial when making interactive and immersive experiences (e.g. for improving the user design). Moreover, it is vital to check that the VR experience doesn’t cause any physical or mental issues in the participants. When I tested the prototype in our lab, some participants felt as though it was a ‘confessional’ experience. They saw it as a safe space where they could talk to a scripted AI character who tried to make them feel better about the pandemic by taking them to other emotional landscapes.

On the other hand, when I did an embodied proof of concept exercise with participants at a real-life conference this year, some told me they did not want to go back to the pandemic years. Does this prove or denounce the healing effect of the experience, and the traumatic element of the pandemic?

Busch and McNamara claim that:

“Avoidance of painful intrusions as a measure of self-protection is only one of the reasons that makes it difficult to share a traumatic experience with others. There are many other reasons such as: speaking about what happened can be subject to interdiction, social taboo, or shame; a common ground of experience or knowledge is missing; one does not want to burden others with one’s own pain; the violation and the suffering have not yet been socially acknowledged” (2020, p. 329).

Whereas one user may go through an intimate immersive and interactive experience with an AI character, which not only lends them their ear but also gives a social acknowledgment of the pandemic as traumatic. For another user, it may be a reiteration of the painful memories which they are not ready to share.

Conclusion

A new research question needs to be proposed: When does a VR experience from the first-person perspective heal and when does it re-traumatize? I have experienced far too many re-traumatizing VR pieces in the past (e.g., being sexually, physically, and verbally assaulted in VR can re-traumatize a user if they experienced a similar situation in real life, and if they have no agency or control over the situation in VR). My VR piece aims to stand in opposition to such XR works. It re-iterates the negative memories into positive pandemic stories. And I hope that it can transform the negative emotions into positive ones through language, conversation, and sensory triggers. The iterative process of creation-as-research doesn’t end with a final build of an experience. My work on this project is to be continued.

References

Ahmed, S. (2014). The Cultural Politics of Emotion. Edinburgh University Press.

Austin, J.L. (1975). How to Do Things with Words. Clarendon Press.

Blommaert, J., M. Bock, and K. McCormick. (2007). ‘Narrative inequality in the TRC hearings’
in C. Anthonissen and J. Blommaert (eds): Discourse and Human Rights Violations, pp. 33–63.
John Benjamins Publishing.

Busch, B., & McNamara, T. (2020). Language and Trauma: An Introduction. Applied Linguistics, 41(3), 323–333. https://doi.org/10.1093/applin/amaa002

Butler, J. (2006). Gender Trouble. Routledge Classics. New York: Routledge. 

Chapman, O. B., & Sawchuk, K. (2012). Research-Creation: Intervention, Analysis and “Family Resemblances.” Canadian Journal of Communication, 37(1), Article 1.

Charisma — Storytelling powered by artificial intelligence (no date). Available at: https://charisma.ai/ (Accessed: 15 December 2021).

Damasio, A. (2005). Descartes’ Error: Emotion, Reason, and the Human Brain. Penguin.

Gauntlett, D. (2018). Making is Connecting (Second). Polity Press.

Guzman, A. L., & Lewis, S. C. (2020). Artificial intelligence and communication: A Human–Machine Communication research agenda. New Media & Society, 22(1), 70–86.

Loveless, N. (2019). How to Make Art at the End of the World. Duke University Press.

Massumi, B. (1995). The Autonomy of Affect. Cultural Critique, 31, 83–109.

Milgram, P. and Kishino, F. (1994). ‘A Taxonomy of Mixed Reality Visual Displays’, IEICE Trans. Information Systems, E77-D, no. 12, pp. 1321–1329.

OpenAI (no date) OpenAI. Available at: https://openai.com/ (Accessed: 15 December 2021).

Ratey, John J. (2001). A User’s Guide to the Brain. New York: Random House.

Tyng, C.M. et al. (2017). ‘The Influences of Emotion on Learning and Memory’, Frontiers in Psychology, 8, p. 1454. doi:10.3389/fpsyg.2017.01454.

Van der Kolk, Bessel (2014). The Body Keeps the Score. Viking.

Wolozin S., Uricchio W., Cizek K. (2020). Collective Wisdom. Massachusetts: MIT Press.