What3words is a tool that allows you to communicate your location – wherever you are on earth – using three words. It is backed by a geolocation system that has divided the world into 3m x 3m squares and given each one a unique three-word address. 

The idea behind what3words is that it is much easier to share your location using three words compared to other geolocation systems, such as GPS, which involve listing lots of numbers separated by dots and commas. 

This means that, in an emergency, you could tell responders your location more quickly and precisely. Other suggested use cases include pointing delivery drivers to the correct entrances for large buildings, or helping people create designated meeting places at music festivals.     

Most people’s experience of what3words, however, will be looking up the 3-word address for where they live and being amused by the odd combination of words that they’re shown. 

This amusement reveals something important: that the words used in what3words addresses aren’t empty signifiers. They have meaning, and when combined and used to refer to places, our response to those addresses reveals something about how we see the world. 

Pin the Tale takes this observation one step further. It is an online game that encourages players to write stories about places in the UK using their what3words addresses, pinning them to a digital map. It involves deliberately playing with the associations that what3words addresses have and thinking about their relationships with the places they refer to. 

Stories on Pin the Tale must include all three words in the what3words address of the place being written about. They can be up to 300 words in length, and writers must include a photograph taken at the location, alongside a short hint that describes the place in plain English. 

Once stories have been added to the platform, other users can read them and attempt to identify their exact locations. After they put a correct what3words address in a story’s answer box, the story is added to their collection.       

As the game’s creator, I’ve been able to observe the unique ways this system affects the processes of creative writing and storytelling, the kinds of stories told and how they connect to the places in question. 

In this article, I’m going to talk about some trends I’ve noticed in how people write stories with Pin the Tale, and what we can learn from these observations about the affordances of what3words as a storytelling tool.

From description to juxtaposition 

The first thing to note is that the words in each what3words address have no deliberate link to the place they are attached to, instead being determined arbitrarily by an algorithm. 

This lack of context makes the words feel ‘random’, and part of the amusement we often get from reading 3-word addresses is the obscurity of the word combinations themselves, and the fact that these weird groups of words have been linked to places we’re familiar with. 

What’s happening here is juxtaposition: a process of putting elements side-by-side, where you are invited to consider the relationship between them. In this case, juxtaposing the individual words with each other, and the combination of words with the place. 

Occasionally, you can find places where the words seem particularly apt (for example, a bench in Canterbury where one word in its what3words address is ‘bench’) and 3-word addresses that fit well together in a sentence or seem to suggest a narrative. But most of the time the words have no obvious connection to each other or the place to which they’re attached. 

What does this mean when you attempt to write site-specific stories with them? 

Well, creative writing already makes ample use of techniques where the words being used do not literally describe the thing you’re writing about. 

Imagery – the use of similes and metaphors to help paint a picture in the reader’s mind – is a prime example.  

Idioms are another example: sayings or turns-of-phrase used to express common sentiments, that often cannot be easily understood without being familiar with the cultural context in which they’re expressed. 

Given the lack of literal connections between 3-word addresses and their locations, it is perhaps no surprise that many contributors to Pin the Tale are drawn to styles of writing that are more figurative. 

Pin the Tale story written using the words ‘valve’, ‘minute’ and ‘gown’.

The story above uses the words ‘valve’ and ‘minute’ to craft a fantasy about its location in Bath, inspired by the true history of the architect John Wood, who designed the road layout to look like a key from above. The third word, ‘gown’, is used metaphorically to describe how the tree’s leaves appear as if they are cloaking a secret. 

Poetic forms are also commonly employed by writers on Pin the Tale. 

Pin the Tale story written using the words ‘leans’, ‘wing’ and ‘organ’.

In the example above, two of the address words – ‘leans’ and ‘wing’ – are metaphors that conjure an image of a person’s position in relation to the city. The rest of the poem is crafted around the core theme of a spiritual resonance that can exist between a single human and a place at a particular time. 

What we can say, then, is that writing with what3words often inspires a turn from the literal to the figurative; from description to juxtaposition. Contributors to Pin the Tale employ a style of writing that is typically less factual and on-the-nose, but rather evokes the relationships between things, and particularly how they pertain to the places being written about.                  

Shifts in perspective 

Many people approach the task of writing with what3words by thinking about a place that is meaningful to them in some way. Then they look at the 3-word address for their chosen spot and think “How on earth am I going to fit these words in?” The words simply don’t align with their understanding of what is significant about the place, even if they try thinking more figuratively. 

Sometimes, the words are just too oddly specific to make them work in the context of your narrative. It’s going to be hard to write a story that includes words like ‘crouton’, ‘peroxide’, ‘gearbox’ and ‘megawatt’ unless the tale you’re telling happens to relate to those specific things. 

Another common culprit is verbs. Verbs used in what3words addresses can be conjugated in all sorts of ways, and it is surprising how challenging it can be to include a verb that is in a different person or tense to the one you intended to write in. 

How do you overcome these constraints? 

The solution that many Pin the Tale players have adopted is to write their story from a different perspective. 

Pin the Tale story written using the words ‘petal’, ‘ready’ and ‘short’.

In the above example, the word ‘petal’ led the author to write a story about a friend visiting their canal boat from the perspective of a strawberry plant. The details contained within the narrative have a different quality when described from this point of view. The weather, and the sense of expectation around the fruit, become driving forces in the story. And in my reading, they interestingly convey the complexities of timing in adult friendships.                   

Pin the Tale story written using the words ‘nips’, ‘smarting’ and ‘bashful’.

In ‘Just follow your nose!’, with one of the three words being ‘nips’ – a verb in the third person – it becomes difficult to talk about the stinkpipe from the first-person perspective of a human observer saying that they ‘nip past it’. By telling the story from the perspective of the object, the human becomes the third person described by the verb. And it gives the author the opportunity to personify the stinkpipe in an entertaining yet informative way. 

Players often tell me that it is more difficult to come to Pin the Tale with a particular story in mind that you want to write. When you do approach the task this way, it can become a tactical process of finding ways to shoehorn your given words into a narrative you have already mapped out. Or alternatively, if your chosen place is covered by more than one what3words square, finding an address with words that are easier to fit into your story. 

By letting your story be shaped by the given words, you go from thinking about what a place means to you, or what a place is commonly known for, to a sense of what it could mean: to a different person or entity, in a different context. The words change your perspective. 

Jumping-off points 

What we can begin to see from these observations is that writing with what3words has a peculiar effect of revealing relations between things, perhaps more than the things themselves. If you’re going to make sense of the significance that three arbitrary words might have for a place, then you’re going to have to find some kind of connection between them, whether that is based on material reality or imagination. 

What’s exciting about this process is that different people will perceive completely different connections between the same sets of words in the same places. 

I recently ran a poetry workshop using Pin the Tale, where one of our activities was to write a collaborative poem about a chosen place using its what3words address. It was fascinating how the different attendees interpreted the task. One person wrote about their memory of meeting a friend in the place. Another person imagined the place as the epicentre of a future political uprising. Other attendees wrote about a wide variety of things that you can physically find there. 

It’s for this exact reason that Pin the Tale allows multiple stories to be pinned to the same what3words square. Even within such a small area, with the same combination of words, the range of potential associations is vast. 

One of the most common pieces of feedback I get from players is that the game’s constraints are a source of inspiration. Having to include the three words in your story means that you aren’t starting from a blank sheet. Instead, there is already a jumping-off point; something for writers to connect to and branch out from in their own unique ways. 

Extend this writing process over an entire city, or a country, and you can imagine the rich tapestry of stories that can be woven. Click on the Find Stories button when you first load up Pin the Tale – when the map shows the whole of the UK – and you will see what I mean. The platform’s stories are highly varied. 

Both in the writing of stories and in how they are engaged with by readers, the platform provides geolocated jumping-off points through which narrative connections can be established. The interface extends your scope from the individual event, the individual thing, person or character, to their significance as part of the wider storied tapestry that we understand as a place. 

A tool for connection 

What3words is a technology with a particular set of characteristics. Firstly, it is a geolocation system that points to 3m x 3m spaces on the earth’s surface. Secondly, this geolocation system uses sets of three words: real words that have existing meanings to people. Thirdly, these word combinations are arbitrary: the words within each 3-word address do not have any intentional link with each other or the location with which they’re associated. 

When adopted for storytelling in Pin the Tale, this system has particular affordances. As writers attempt to bring the geolocated word combinations into a meaningful structure – a narrative – what3words addresses become jumping-off points; inspirational constraints used to reveal and make connections between a wide range of site-specific things. Juxtaposition and playing with perspective are two examples of relational methods that writers use to create unique pieces of storytelling out of the address words. 

Writing, and storytelling in particular, are often said to forge connections between disparate people and contexts. This is their superpower. In its use of the what3words system, Pin the Tale’s interface is designed to centre place as a hotbed of such connections. To show how places allow you to go beyond yourself; to think about all the other lives that have been, are being and will be lived in that locality. 

From generative writing tools to augmented publishing processes, artificial intelligence is rapidly changing and challenging the landscape of creative writing and publishing. To respond, MyWorld and Bath Spa University’s Centre for Cultural and Creative Industries (CCCI) and Narrative and Emerging Technologies (NET) Lab have developed a series of free webinars, Writing with Technologies, to offers an in-depth look at AI’s emerging influence across writing and publishing in multiple fields. 

Our third webinar of the series brought Imwen Eke, Joseph Wilks and Richard A Coles together to explore the cutting-edge intersection of artificial intelligence and the gaming industry to discuss how AI is transforming both game development and player experiences with a focus on accessibility and inclusion. 

Speaker Bios

Imwen Eke

Imwen (She/Her) is a play alchemist, creative technologist, facilitator, educator, and TEDx speaker, specialising in play as a powerful tool for exploration and transformation. 

With nearly two decades in immersive entertainment and digital transformation, her expertise spans interactive design, game mechanics, workshop facilitation, and creative technology, making complex transitions accessible through playful methodologies. She focuses on play in adulthood to address social, political, and environmental challenges. 

She is the Founder of New Party Rules Labs, Chair of MyWorld’s Advisory Board (£30M creative technology initiative), and co-organizer of BXRN (Black in Xtended Reality Network). In 2024, she spoke at the UN Geneva Permanent Forum and G20 Rio, advocating for the transformative power of play in governance and society. 

Previously, she worked with SHUNT, Punchdrunk, taught at ISCOM Paris, University of the Underground, and Bristol University. Rooted in London and Bristol, Imwen continues expanding approaches to cultural play and technology. 

Joseph Wilk 

Jospeh is an artist and programmer who uses the digital to explore disability and disability to explore the digital. He deconstructs, misuses and repurposes software and hardware to challenge notions of ownership, narrative and visibility. 

Richard A Cole

Richard Lecturer in Digital Classics at the University of Bristol and co-Director of the Bristol Digital Game Lab. His research focuses on how the history and culture of antiquity intersects with new media, in particular video games, virtual reality, and artificial intelligence. From 2020-2023, he worked on the multidisciplinary Virtual Reality Oracle project, which created a ground-breaking VR experience of ancient divination that is improving educational outcomes in schools. He has published leading research on Classics in video games, while his latest work looks at how questions around the authorship of the Iliad and Odyssey can help us better understand AI generated texts today. Through the Game Lab, he is leading the research component of a collaborative R&D project with industry partner Meaning Machine on generative AI systems for video game dialogue. The aim? To understand how players respond to conversing with AI-powered characters. 

For the last few years I have been experimenting with telling complex, challenging and nuanced historical stories using creative technology. In September 2022, I received a fellowship from Bath Spa University’s Narrative and Emerging Technologies (NET) Lab to explore ways in which to use immersive and emerging technologies to tell stories within some 19th Century family letters and diaries. I have a background in documentary filmmaking and in writing for theatre. More recently I have been working on large R&D projects aiming to promote the use of immersive technology in the creative industries, so the fellowship seemed like the perfect opportunity to try and marry the three areas together. 

Photograph of some of the nineteenth century letters and diary. Credit: Rachel Pownall

Initially, I spent time exploring different types of immersive narrative experiences. I looked at everything from one of the highly commercial immersive Van Gogh exhibitions to VR experiences like Randall Okita’s The Book of Distance to the immersive audio piece, Intravene, by Darkfield. Alongside this I spent time getting to grips with the source material. As well as examining the family documents in detail, I had to do a lot of detective work to try to fill in the narrative gaps; delving into archives, consulting experts, accessing online census and church records then re-examining and interpreting family stories in the light of new information thrown up by this research. 

While there were several fascinating narratives contained within the family letters and documents, I focused mainly on the story of my great, great grandmother Charlotte Coostriah Denman who was half British and half Indian. She was brought from Maharashtra, India to North Wales in 1833 at the age of three by her father William Denman who was an Ensign in the East India Company’s army. She was told that her mother, who died giving birth to Charlotte, was Spanish and she grew up knowing nothing of her Indian heritage. 

As a white British woman, albeit a direct descendent of Charlotte Coostriah, I am very aware of the cultural sensitivities around telling this story. During my fellowship, I was lucky enough to have Dr Priya Atwal from the University of Oxford, as consultant. Her research specialism is cultural politics of empire, particularly across Britain and South Asia in the 19th and early 20th Century. She was able to advise on the historical and cultural ramifications of the East India Company’s colonisation of India and acted as script consultant as I pieced together and started to tell Charlotte’s story and that of her mother. In addition, in order to understand the cultural norms in the UK at the time, particularly relating to women’s experiences, I consulted with Dr Jackie Collier from Bath Spa University. Her research specialism is the social political and cultural history of the long eighteenth century in Britain with a specific interest in gender. 

My research unearthed a huge amount of information including some revelations. With Dr Atwal’s help, I discovered that a mistranslation of some blurry 19th Century writing meant that Charlotte’s mother was not called Lasiml Coostriah as the family had always believed, but Laksmi Coostriah, a name that gives a much stronger indication of her cultural and religious background. I also uncovered parallel stories relating to other women whose lives contrasted with those of Charlotte and Laksmi but were also bound by the strict class, religious and gender conventions of the time. For example, Charlotte’s Aunt Elizabeth was forced to renounce her fiancé, Capt John Molyneux, because his wealthy and powerful father threatened to disinherit him if they went ahead with the wedding. She never married and went on to play a big part in bringing up the orphaned Charlotte.  

I found Twine really helpful in structuring the different narrative strands and offering alternative ways through that focused on different areas.

Screenshot showing the complex narrative structure of one overall storyline in Twine

I had a small amount of money as part of my fellowship to pay for a Proof of Concept (POC) prototype and chose to create one using Augmented Reality (AR) with smartphones because I wanted the story to be as accessible to as many people as possible. I commissioned the AR & VR studio Zubr to create an Instagram filter that would tell a simplified version of William Denman’s journey to India and his subsequent return to Britain with the infant Charlotte.

Screenshot showing simplified narrative structure of prototype in Twine

The ultimate aim was to create a museum-based exhibit where the public would move from element to element triggering sections of the story using their smartphone and acting as detective to uncover the hidden truths behind them. The POC prototype was a first step to test if this was possible.  

For the prototype, we used a mixture of audio that included quotes from some of the letters, music and sound effects, and visuals like the boat, The Lord Lowther, that William sailed on to India and a 19th Century sketch of a Spanish lady in traditional dress (purchased from archive companies). 

Ship Lord Lowther engraved by E. Duncan England. Credit: Peter Horee/ Alamy Stock Photo

Costume of a Spanish lady, 19th century. Credit: Florilegius/ Alamy stock photo                               

Artwork by Bhavana Ram Mohan

We also used elements from some beautiful mood boards that I commissioned from artist-research and illustrator, Bhavana Ram Mohan, whose work explores decolonial curatorial and heritage practices.

The story elements were triggered by holding a smartphone over an object, in this case one of Elizabeth’s diaries. While it was an interesting process to distill the story into five simplified elements, ultimately the prototype did not work as a POC. The audio elements had to be less than two lines of script, and it was just not possible to do justice to so complex and nuanced story in such bitesize pieces. The story demanded a more narrative led approach that can only really be achieved, I now believe, by using an audio first approach. 

Photograph of Elizabeth Denman’s diary from 1833. Credit: Rachel Pownall

Since attempting this prototype, I have looked at other ways to tell the story. This includes plans to create an interactive AI driven chatbot using RocketMaker’s PORTRAIT technology, which would enable participants to converse with a virtual avatar of Charlotte about her life. However, our application to the MyWorld More than AI Sandbox fund was unsuccessful.  

Subsequently, I have written a couple of monologues from the perspective of Charlotte and her Aunt Elizabeth and I am currently looking at how to create an audio led immersive, interactive VR experience to tell their stories. I have put together a diverse team of creatives to achieve this but gaining funding is the most difficult issue with trying to create such an experience. In the meantime, I am looking at low-cost DIY ways of telling the stories. Recently I have started to learn to code in Unity and I am intrigued about the possibility of including elements of gameplay in the telling of the stories. This is one more avenue to investigate as I continue to look for interesting and engaging ways to explore these narratives using different types of creative technology. 

Ephemeral City (Boxed Edition)

We were in lockdown when the final piece of the Ephemeral City puzzle fell into place. It was late 2021 and my book Ex Libris had been out for more than a year. Though it had been released at the height of the first pandemic wave, the idea of a novel made from recombinant chapters was intriguing enough to find a decent audience without the aid of launches or festivals or any of the usual publishing palaver. I had already dedicated a good eighteen months purely to Ex Libris as a publishing project and here, stuck in my house in late 2021, I was finally contemplating what might come next. 

Okay. I say I was busy doing the publishing and promotion for Ex Libris but this is an excuse. In truth, I had been at a creative impasse for some time: filled with ideas but lacking the concentration and energy to see them through, to face the blank screen and blinking cursor.  

Do I blame the pandemic for that? Sure. Why not?  

Weird times, man. 

So rather than force a new idea to fruition, I turned to my archive and a concept that had languished for the better part of fifteen years: the portrait of city over time, told through the stories of people from the margins of history, the people who never expected to be remembered. 

I had a complete manuscript. Several, actually.  

The original idea was to create a collection of stories linked by setting and themes. The individual stories had been successful at the time, published in a few of Australia’s prestigious literary journals, but the collection as a whole gained no traction. I was encouraged at the time to turn the stories into a novel, a reasonable suggestion but one that took years to realise and ultimately led nowhere. After multiple attempts at making it work, I shelved the idea though I never abandoned it. Now, here at this low ebb, the project again beckoned. 

I re-read the various versions of the text and immediately began to imagine the things—big and small—that I wanted to change. However, something was missing.  

Each of the eight stories is set in Brisbane at a specific time, liberally spread between 1931 and 2011. When first writing, I had done a lot of background research on each period seeking to capture the flavour of the time without wanting to draw direct attention to it. In the process, I had amassed quite the collection of digitised ephemera: photographs, print ads, a newsletter, a lottery ticket. Now, reading between the stories and the ephemera—jumping between text and images and people, places, and times—reminded me of less of a short story anthology and more of personal collection of keepsakes, a container—maybe something like a cigar box—filled with treasures. 

That was the moment.  

Indeed, something had been missing and in my reading and re-reading of the stories and research, I stumbled on the ideal way to present the stories. Ephemeral City was created there and then, but it would take another two years for the project to be completed. 

Ephemeral City (Obsolete Edition)

I can explain the floppy disk.  

See, due to a tangled series of false starts and delays, my publisher and I began accepting pre-orders for Ephemeral City before the shipment of book boxes—the actual containers these stories would sit inside—had arrived from the manufacturer. It was a gamble that led to a stressful few weeks, wondering if we would indeed be able to ship the product when promised. We were surrounded by stacks of content, but there was no way to compile them until the shipment arrived. See, Ephemeral City is published in a boxed edition as well as a regular paperback edition (for the normies).  

Each copy of the boxed edition consists of nine booklets and nine pieces of reproduction ephemera. The lottery ticket goes inside the Christmas card which goes into an envelope. The small string of random words needs to be folded and carefully placed at the bottom of the pile. Three separate print runs were involved from two printing companies. And that all had to be organised before we could finalise the design and manufacture of the box. 

I had initially wanted to use an actual cigar box for the container; the reading experience would commence with removing a slipcase. Then, like the ephemera, the box itself would form part of the fictional world of the stories. Remove the slipcase and you enter the story world. I still like the idea, but cost quickly rendered it impractical and I had to disappoint a very excited cigar box manufacturer based in Texas. I have the sample they sent me and it’s cool, but I suspect the solution we arrived at is better. Certainly, it’s more practical. 

All of which is to say yes, a book in a box is a complex thing to put together. I mean that should come as no surprise to anyone, but no amount of forewarning actually prepares you for the experience. There were an alarming number of variables for which we had to take an educated guess and hope everything worked out okay. To keep the workload manageable for my long-suffering designers, I suggested the paperback and the boxed edition’s booklets should be identical in size, meaning the same page layout could apply to both editions, a very early decision that determined the size of the package. There were decisions on choice of paper, typefaces, each with their own knock-on effects.  

And then there was the question of what depth to set the box. How thick would this pile of booklets and ephemera turn out to be? We had no way of knowing for sure without actually making all the things first. What we could be sure of was that nothing would be worse than a box that can’t close properly or one that looks half-full.  

My publisher, Sue Wright of Tiny Owl Workshop, is possibly the only person on this planet capable of helping me shepherd a project like this. Our mantra throughout was that we would give the book whatever time was needed to get it right. That’s pretty much how it played out. There’s not much I would change if given the opportunity even though every stage of the process took at least twice as long as initially anticipated. When we made the agreement to work together in early 2022, I thought we could probably knock the completed book out by the end of that year.  

Sigh. 

The thing is, as you approach the finishing line, frustrations over delays begin to bite that much harder. There’s a point at which you just want this book out of your head and into someone else’s. Lots of someone else’s, if we’re going to be honest. 

And yet, by early this year the text had been finalised, the designs for the ephemera and the cover were done, and delivery of the book’s internal elements was imminent. The bound edition paperback was done. The promotional material was prepared, including a series of videos blogs visiting locations from the book and revealing the real–life stories behind the book, a separate project that could be subject of its own essay. So, I took the best educated guesstimate I could at the size of the final package, approved the box design, and set the manufacturing in motion.  

Now what? What does one do in that final interregnum?  

I noticed the zip file I’d sent the printer containing the nine booklets for the boxed edition came to a very satisfying 1.1MB. 

Hmm. You know what format that file would neatly fit onto? And for a collection built around historical artefacts, why not play around with some dead media while I’m at it? 

That’s how I found myself scouring eBay for a brand-new floppy disk drive and a box of old-new-stock diskettes, fashioning a never-initially planned-for obsolete edition of the book. 

Yes, it’s real. Yes, it works. And yes, real people have parted with some of their hard-earned for one.  

There are still a few available. And, should you acquire one, what you do with it is up to you. If need be, I can hook you up with a drive via eBay. 

* 

Compared to my previous work, Ephemeral City leans much more heavily to the physical and the organic. I have always maintained a strict separation between the technologies used in the making of literature and the technology through which it is read. The fluid and flexible nature of the digital tools we use every day can be applied intelligently and creatively to the craft of narrative and the presentation of text, even when the output is purely ink and paper, still for me the gold standard of reading experiences. 

And as our interconnected electronic world become increasingly undermined, artificialised, scammy, and enshittified, as archivists and thinkers around the world contemplate the possibility that we have already entered what future generations will consider a dark age, personally I find it more rewarding to create a tactile reading experience that rewards a longer engagement through unconventional means than shovelling yet more instant consumption into an ephemeral attention economy.  

I don’t know. Maybe that’s just me. 

Ephemeral City is available now online. 

Immersive project inspires a new generation of readers and new ways of reading

One to One Development Trust is an award winning arts organisation based in Wakefield, UK, led by Judi Alston and Andy Campbell. One to One’s mission is to use digital technologies to engage communities in projects that push the boundaries of creativity. 

As an organisation, we have a long history of creating imaginative projects using immersive technologies. In 2019, we heard that visual artist Luke Jerram’s Museum of the Moon was coming to Wakefield, where One to One has been based for many years. The Council were looking for innovative creative work for their event Festival of the Moon and we had a wave of ideas on how we could use Virtual Reality to create something unique, fun, and spectacular.  

A lot of our work celebrates the relationship between past, present, and future, exploring a sense of place and identity. The Festival of the Moon marked the 50th anniversary of the moon landing, so what better place to start than looking at the local archives of what was happening in Wakefield in 1969. 

A photograph of the mobile library van that toured the borough of Castleford in the District in the 1960’s really resonated with us, with its metallic tin can-like design, it echoed that of a space capsule. 

Photo courtesy of Twixt, Aire and Calder

We invited two senior managers from Wakefield Libraries to our studio to discuss our ideas. We suggested that we turn the historic library van into a moon-orbiting space shuttle filled with book recommendations, a gallery of photos of the area from 1969, and some short films we would make about people’s relationships with the Moon. It would ambitiously take the form of a Virtual Reality experience, include a trip to the Moon’s surface to ‘plant the flag for Wakefield’ and involve members from 12 local libraries. 

The pitch prompted a momentary silence, followed by some excited laughter. The relief was tangible as Claire Pickering, Senior Librarian, exclaimed ‘Yes, I love it.’ 

PHASE 1

We had just over three weeks to build and test the experience before its launch at the Festival of the Moon event. We accessed NASA’s free archive of imagery to help with visuals and created a series of experimental ‘sandbox’ environments in Unity, where books floated around inside an amusingly van-shaped space shuttle.

Photo by One to One Development Trust

Readers from 12 libraries took part. They gave us their recommendations for a reading list themed around ‘what book would you take to the moon?’. These chosen books – rendered in 3D with their original front covers and readers’ group reviews on the back – drift around in zero gravity; using VR hand controllers, readers/players can ‘grab hold’ of the books, flip them over to read the short review, and then toss them back into the shuttle to float. 

To give a sense of reward and progression, we decided to link the number of books/book reviews discovered to the opening of an ‘air lock’ at the bottom of the ship that offers a trip down to the Moon’s surface. With the project being developed for live events with potentially large audiences, we needed a mechanic that would time-limit the experience so that lots of people could take part on our relatively small number of VR headsets. So, we turned this element of the experience into a mission to ‘plant the Fleur de Lys flag’ (Wakefield’s iconic civic symbol we took from the gates of the Town Hall) before your oxygen runs out. After that, it would be ‘game over’.

To enrich the experience and make it attractive to a wide local audience, we created an ‘onboard gallery’ featuring historic local photographs and the short films we produced about different people’s relationships to the Moon. The films include stories from a Pagan, an astronomer, a spoken word artist, and a refugee.

Photo by One to One Development Trust

We completed work on the project just in time for the event. We embarked on an intensive social media campaign focusing on the short films and their subjects and inviting new book suggestions for inclusion in the shuttle. We also featured screenshots as we developed Zero Gravity Lunar Library to document its progress and – as book reviews began to be submitted – highlighted different books of choice. We completed work on the project just in time for the event.

Photo by One to One Development Trust

Throughout the Festival of the Moon, hundreds of people took part in the Zero Gravity Lunar Library at different events. For many, it was their first time experiencing Virtual Reality and the response was staggering. We were amazed at how long people were prepared to queue to have a go, and were particularly encouraged by the younger users who, on average, spent more time trying to catch all of the books in order to read the reviews.

We used a mix of Virtual Reality headsets, large touch screens, and digital projection to run all the components of the project, ensuring that it could be seen and heard from any position in Wakefield’s historic Old Market Hall where the Festival of the Moon was held.

Photo by One to One Development Trust

Several people described the experience as ‘therapeutic’, saying that ‘being in space’, ‘floating around’ and ‘looking back at planet Earth from the Moon’ felt peaceful and relaxing. It provided a great framework to talk to audiences about reading, books, and technology – evoking excited responses from participants because they wanted to share their thoughts and reactions. 

One of our main objectives was to surprise audiences with high production standards. Although the project’s core concept is entertaining, we wanted people to put on the headset and really feel like they were floating around inside a virtual library orbiting the Moon, playing on a sense of epic scale and freedom to explore. We developed realistic high-end graphics to add to the immersion, and this paid off in multiple ways. “This is fantastic, when’s it out for Xbox?” (or other gaming platforms) was a common response with some participants shocked that a small creative team had produced the project. 

The positive feedback from the project and our own research and evaluation pinpointed that this was not the end of Zero Gravity Lunar Library.

Photo by One to One Development Trust

PHASE 2

Five months after we had submitted our internal project evaluation to the funders, Covid hit. The response had been overwhelmingly positive, the key take-away from the evaluation being ‘we need to develop this project further and do more with it.’ However, lockdown and a shift in priorities to respond to the needs of our local community prevailed, and the Zero Gravity Lunar Library was temporarily filed away.

Then, in 2022, Wakefield Council announced a call-out for projects to be included in Word Fest – Wakefield’s month-long literature festival running throughout May. It felt like a great opportunity to revisit Zero Gravity Lunar Library and build on user feedback, particularly the interest we’d had from children and young people  in reading book reviews and wanting to contribute to their own. We were very pleased to be awarded funding to take the project into Phase 2. 

There were key things we wanted to achieve in this phase: the first was to engage with young people aged 7+ in library settings who could help us road-test a new modified version of the project while also compiling and co-producing a new book list for young readers. 

Secondly, we wanted to pilot the experience with a school aimed at key Stage 2 learning (7 to 11 years old). This would inform how we might develop an Education Resource Pack to sit alongside the project to explore its potential as a learning resource. 

Photo by One to One Development Trust

With a younger audience in mind, we decided to change the look and feel of the experience, brightening up the graphics and making the interactive elements deeper and more intuitive; we took out the film and photography exhibition and swapped it for some fun and quirky gravity experiments. We also embedded a live webcam feed from our gaming laptops into the space shuttle’s cockpit/dashboard, so that the children and young people could ‘see themselves’ (and anyone standing near them) wearing the VR headset as they explored the environment. This proved to be immensely fun and popular.

Photo by One to One Development Trust

We ran six sessions in libraries across the Wakefield district. Around 100 children and young people took part. The response – again – was extremely positive, with librarians enthusiastically commenting that it seemed to resonate even with ‘reluctant readers’ (often noted as teenage boys, particularly from disadvantaged backgrounds). Young people embraced the idea behind this version as an engagement resource to encourage reading.  ‘I loved the VR because it made me look more into books,’ said one participant at South Elmsall Library, ‘I would recommend it for other young people because you can experience so much and learn new things.’ 

One of the factors that the children commented on was the quality of the graphics and overall production. We felt that this aspect of the project was an important part of ‘hooking in’ our young participants, most of whom regularly played video games and were used to a high level of visual quality and rich interaction; it was unlikely they’d fully engage in something that felt ‘below-par’ to what they were used to playing/experiencing at home. Additionally, as it had with the adult audiences at Festival of the Moon, it drew surprise at having been developed locally and by a small team. Several young people asked if they could come and do work placements with us, as the project felt to bring immersive production a step closer to within their reach as future creatives. 

We used the experience to facilitate meaningful conversations with our cohorts about reading, games, books, and ideas for how they could integrate. After having explored the ship, discovered new books, and planted their flag on the Moon, many participants were keen and excited to find their own ‘recommended book’ within the virtual library, to write or speak a review of it to be included in the project. 

Photo by One to One Development Trust

Being part of the actual development process – and evolution of the concept – behind Zero Gravity Lunar Library held very strong appeal for many of our participants. Their parents were often similarly enthusiastic, with one parent at Pontefract library saying ‘I think this is a wonderful, interactive way of learning for children who learn better through experience rather than sitting at a desk. It’s so important we have options like this for our children.’ 

Running the project in libraries had its challenges however. Setting up tethered VR headsets/gaming laptops in varied public spaces sometimes proved awkward; a few libraries wanted the event to be pre-booked only, whilst in other libraries we ran (extremely busy) drop-in sessions. But in all instances, library staff made an effort to understand the concept, caught on to the excitement around it, and supported it by helping the young people find their book, or listened to their experiences of the project. A librarian at Wakefield Central Library said, ‘The use of new technology like this shows people that Libraries are moving with the times. They’re not the dark, dusty, and dated places that some people believe them to be. Participants on the day really enjoyed the opportunity they were offered and gave feedback like it was “excellent”, “the best day of my life”, “fantastic” and ”totally immersive”.’

Photo by One to One Development Trust

This collaborative approach with Wakefield Libraries has enabled Zero Gravity Lunar Library to better reach its potential as a powerful promotional tool for reading and literacy. In Phase 2, we continued using the project to research the themes and gather evaluation from participants. This is hugely useful in our practice as creative producers and informs how we move forward. 

We were delighted to pilot Zero Gravity Lunar Library with twenty six Year 6 pupils at Mackie Hill Junior and Infant School in Wakefield, with the support of the Head Teacher, the class teacher, and a teaching assistant. The experience of delivering the project in a school setting was useful, as it helped inform our ideas for an Education Resource Pack. It also gave us the opportunity to talk to pupils and staff about literacy in schools and how they are currently using (or not using) technology in the classroom.

Photo by One to One Development Trust

We also engaged families who are home-educating and ran a specific session for homeschoolers. These families appreciated the recommended reading lists offered by the project. 

Zero Gravity Lunar Library has proven its worth through extensive road-testing and research. We are now ready to seek out opportunities for Phase 3 of the project to explore the potential of disseminating the VR experience – and a new forthcoming web-browser version – to schools alongside an Education Resource Pack as a conduit for class-based learning across 3 main areas of national curriculum: English (including Creative Writing), Maths, and Science. 

Zero Gravity Lunar Library is an important project for us as creatives. It has given us the artistic freedom to explore new ideas, push boundaries and use technology in a fun and innovative way. Without arts development funding this project would not have been possible. It has also given us the opportunity to create a project which we hope will go into further development phases, inform debates and discussion on non-traditional ways to read, the future of digital technology in education, and explore new ways to encourage literacy. 

For more information about the project and to access the downloadable and browser-based versions when they are released later this year, please visit zerogravitylunarlibrary.co.uk

Photo by One to One Development Trust

We are two academics who met in Cape Town when  our children became friends. Conversations around our shared interests in history, technology and Cape Town’s District Six got us speculating about social justice pedagogies in our respective disciplines. Our preliminary discussions revealed  many parallels in our approaches but also intriguing  distinctions, enough to seed a commitment to collaborate. At the time, Siddique was teaching geomatics students who were studying towards qualifications in land surveying and Geographic Information Systems (GIS) at the Cape Peninsula University of Technology (CPUT). David, an archivist, was organising immersive professional experiences for University of Michigan (U-M) graduate students with NGOs in Cape Town through his school’s Global Information Engagement Program (GIEP). In this article we discuss our three-plus years of collaboration to enhance social memory through joining traditional primary sources with emerging information technology tools in District Six, with emphasis on our latest effort to integrate Augmented Reality (AR) with GIS, archives, and storytelling.

District Six in Cape Town is perhaps the most well-known site of apartheid forced removals (some 60,000 Black individuals between 1968 and 1983). It was a cosmopolitan, multi-racial and multi-denominational urban community viewed by the government as a slum that violated the edicts of apartheid’s ‘separate development’ segregation ideology. It was also favourably located close to the city centre, nestled in between Table Mountain and the sea. Upon removal of the residents, almost every building (houses, community halls, shops, public amenities, health facilities, cinemas, nightclubs, hotels, factories, businesses) was demolished by bulldozers, except for a few mosques, churches and schools. Today, the site sits largely unreconstructed and its legacy and future remains deeply contested. The last four months of 2021 saw District Six mired in controversies over redevelopment planning, city evictions of the homeless, restitution and resettlement for ex-residents, and increasingly realised fears that many returning claimants will pass away before securing one of the District’s newly constructed homes (Bantom, 2021; Charles, 2021; Lepule, 2021; Thebus, 2021).  

Our collaboration started in 2018 and builds on the surveying and mapping work carried out by Siddique on the site since 2006. As a much-needed decolonising pedagogy in engineering education, Siddique attempted to conscientise students to the socio-political-historical aspects of the site that their university was situated on. The university, which was originally called Cape Technikon, was built on a large portion of the demolished District Six, and was a Whites-only institution during apartheid. In Cape Technikon’s curriculum, District Six’s troubled history was silenced and buried under the technical language of geomatics. Fieldwork tasks were ahistorical – the rubble under their feet was nothing more than land that needed to be accurately measured and represented on a map. This needed to be changed, and Siddique’s students started to dig up the past, linking mapped locations to what (and who) used to exist there before. 

Figure 1: District Six aerial photographs. Top left – 1953; Top right – 1968; Bottom left – 1983; Bottom right – 2019.

Figure 1 shows four GIS maps of the same geographical space. The red line is the outline of District Six, the area that was declared Whites-only in 1968. One can observe the drastic changes on the landscape. In 1953, District Six was fully developed, with narrow streets and dense concentration of buildings. By 1968, the first signs of demolition were observable. In 1983, District Six resembled a war zone, with most buildings destroyed, and in 2019, much of the district was open space, with the CPUT campus a dominant feature.

Our first efforts in 2018 were consciously designed as an interdisciplinary effort to join geomatics, archiving, and storytelling. We sought to virtually reconstruct aspects of District Six in terms of the lost built environment and stories of still living ex-residents and their experiences before, during, and after demolition. We  brought together a wide range of stakeholders, including ex-residents, St. Mark’s Church (one of the few structures that survived demolition), the District Six Museum, GIEP, and the departments of Town & Regional Planning and Civil Engineering & Surveying at CPUT. St. Mark’s then current rector wanted to foreground the church’s history and reached out to Siddique. At an early project planning meeting at St. Mark’s, we pitched the idea of using its baptismal records and integrating them with Siddique’s mapping efforts and the memories and archives of selected ex-resident parishioners.The baptismal records date from the late 1800s and are a potent symbol and source of District Six’s heritage, documenting individuals, families, social networks, livelihoods and how they manifested across space and time. 

We created a prototype demonstrating that it’s possible to accurately plot the addresses extracted from the baptism records and connect them to the life stories of ex-residents. The prototype selected and mapped approximately 2,000 baptisms between 1950-1958. This was a detailed undertaking that involved onerous human and computing efforts of extracting, checking, and scripting to clean the data. CPUT students created a GIS map that geolocated addresses and integrated it with maps of other sites in District Six. This work represents the first comprehensive digital historical spatial record of District Six that contains discrete and granular locations of a diverse range of important sites (such as businesses, public services, streets, and the homes of baptised babies). Figure 2 shows some of the sites, overlaid on the 2019 map.

Figure 2: Geo-location of specific sites in District Six on the contemporary landscape.

A collaborative exercise with the GIEP team confronted the vagaries of deciphering handwritten baptism entries, confirming and normalising street names and addresses and locations. We worked with St. Mark’s to identify ex-residents who were willing to share their District Six experiences through recorded oral histories supplemented by their own personal archives. Six ex-resident St. Mark’s parishioners agreed to participate. Interviews covered a range of community members. Each participant was photographed with their baptismal record, and copies made of any records they chose to bring in to aid their storytelling, such as childhood photographs and/or mementos. All generated materials from this iteration – including the digitised baptism registries, the oral history methodology, permissions, and oral history recordings – was deposited with the District Six Museum under a deed of gift. (Huang, Leal and Rubin, 2018).

In 2019, we built upon this work with a second GIEP and CPUT cohort. The District Six Museum hosted this effort. In this iteration, we sought to strengthen our oral histories, identify and plot key landmarks on the GIS map, and integrate all of this as a publicly accessible interactive multimedia website. We re-interviewed five of the six 2018 storytellers and added two new ones. We mined the District Six archives to locate archival images and supplementary primary sources of twenty-nine landmarks: eight social centres, a hospital, nine places of worship, seven schools, and 4 cinemas/bioscopes. Finally, we integrated all of this information onto the museum’s website as an online experience that enables visitors to overlay historical District Six with the contemporary landscape, learn storytellers’ biographies through geolocated photographs, documents, and audio, and identify, locate, and learn about historically significant landmarks (Cox et al, 2019). This effort is now hosted by the District Six Museum’s website as the ‘St Mark’s Memory Mapping Project’ (Figure 3). Siddique’s follow-up work, a 21 minute video provides vivid evidence of the fruits of joining geomatics and archiving to tell the story of erased pasts (Motala, 2019). 

Figure 3: Snapshot of the St. Mark’s Memory Mapping Project website.

Over the past two years, Siddique developed a ‘Haunted Walks of District Six’ walking tour, using analog printouts of the GIS map and historical images of the tour’s stopping points. To date, this analog version of Haunted Walks has been delivered to some 200 individuals – academics, students, artists, filmmakers, and ex-residents. The goal is to provide walkers with a deeply affective and transformative experience by integrating the primary sources described above with real time traversing of the contemporary landscape. This approach deploys the ‘counter-surveying’ methodology, which combines land surveying and mapping to find and mark the locations of sites on the physical landscape that no longer exist (Motala and Bozalek 2021). This method can identify the exact locations of demolished homes in the present day for the benefit of ex-residents, as well as to locate other sites to reinforce and graphically illustrate the gravity of what was lost during demolition. 

We now believe that the current analog version of the walking tour needs to be reimagined, strengthened, and experientially enriched with the thoughtful incorporation of digitised and digital content via AR technologies. As an interactive technology that runs in real time and registers and co-locates contemporary and virtual historical objects with one another through handheld devices such as mobile phones and tablets, AR is a logical next step in our collaboration. We are now recasting the Haunted Walks tour as an AR enhanced experience. Users will be directed to the precise locations of important demolished sites in District Six and be shown what used to exist at that location through stories, superimposed archival images, documents, videos and sound clips. AR will enable walking tour participants to immerse themselves simultaneously in both the contemporary and historically erased landscape, essentially co-geolocating themselves across time. 

To continue this work, in 2021 we received a collaborative faculty seed grant from  U-M’s African Studies Program’s African Heritage and Humanities Initiative, running through 2022. This work is directed towards a decolonial praxis based on connecting developments related to our scholarship in social justice geomatics (Zembylas et al 2021) and archival social justice (Wallace et al 2020). It will explore new heritage-related theoretical and praxis directions for engineering, archiving, information technology (AR and GIS), and their intersections in social justice memory work. Work is proceeding along three axes, 1. Theoretical (joining counter-surveying and social justice archiving to probe the intersections of ephemerality, affect, and non-institutionalized memory making); 2. Methodological (development of counter-surveying and memory making and transmission praxis) 3. Socio-technical (incorporation of AR to transform the Haunted Walks experience). We will reinvent the embodied experience of the Haunted Walk, with prospect for application to other contexts of forced removal, recovered and contested memory, and ongoing justice-seeking and activist efforts (acknowledgment, memorialisation, restitution, and reparations). 

However, we also take seriously the advice of Tuck who warns against such work as being solely aimed at the documentation of damage. Whilst damage-centred narratives are often collected and deployed benevolently, the resulting work provides a one-dimensional view of Blackness and disenfranchised communities (Tuck & Yang, 2014). We seek to meaningfully contribute both to more fine-grained understandings of apartheid planning practices and the experiences of those subjected to them. Instead of solely focusing on the documentation of damage, we are equally ‘concerned with understanding complexity, contradiction, and the self-determination of lived lives’ (Tuck, 2009, p. 416). We also want to attract a younger generation of technologically adept South Africans, which is especially critical in light of the nation’s ongoing efforts to decolonise its curricula and grapple with its apartheid past and its afterlives. Through this community-based work, we seek to leverage creative technologies, stimulate affective responses in walker-users, and assist students and researchers in reimagining the historical remnants of the ex-apartheid city and its continued effects on the contemporary urban landscape.

To date we have laboured to reimagine and reconstruct the walking tour as an AR app by plotting out specific notable and not well known points of interest. We are  collaboratively working with the District Six Museum’s archives to identify photographs that can be precisely located and superimposed onto the contemporary landscape. We have ideated and envisioned the user experience via a 2.5km walking loop and how that will manifest as virtual content delivery and interactivity via historical narratives and stories and locational positioning in the real world. We have identified necessary pragmatics of pre-tour set-up (route and tour points, disclaimer and liability waiver, operationalising the tour on users’ devices, interface design, location awareness, and toggling between the historical and contemporary landscape), user orientation to and experience at tour points (navigation, multimedia content display of text, audio, visual and moving image, overlaying historical photos on the contemporary landscape), and ending (feedback, sharing on social media, contact information). 

The biggest early challenge has been identifying and evaluating the tradeoffs on alternative AR platforms, given due consideration of the constraints facing prospective users in District Six: device requirements and operating system; navigation; data usage, and; feasibility to realise the completeness of our vision. We want a user to be able to use an Android smartphone or tablet to download our app and be empowered to take a self-guided tour. Project staff have explored AR possibilities mindful of these needs. We have evaluated the status and capabilities of available AR tools / applications along the following parameters: functionality; usability; learnability; incorporating multimedia content and geolocational data; content management; open source vs proprietary; costing; editability, and; prospects for persistence (longevity). After carefully reviewing nine alternative possibilities, we are using Unity3D as our AR authoring environment. It will allow us to integrate our GIS maps, develop our own user interface, activate the user device’s GPS for navigation and camera for transposing historical photographs onto the contemporary landscape.

Parallel to the AR effort, project staff are modifying the current analog version of the tour by developing a curated digital repository of existing and new archival and other supporting primary source materials. This will strengthen and expand walking tour content. This will be supplemented by the development of new and expanded tour narratives / storytelling. For example, we envision that one of the points that will be visited through the app will be Horstley Street (Figure 4), which was an important street in the history of District Six. The buildings of Horstley Street were completely razed, but the original cobblestone street remains.

Figure 4: Horstley Street before demolition (left, District Six Museum archive) and today (right).

Ultimately we value the expansion into AR not as technology for its own sake but rather as a tool to create powerful affective embodied experiences for walkers-users. It is one thing to see the remnants of District Six as pins located on digitised historical and contemporary aerial photographs, as was done with our previous work. We believe the embodied and embedded process of walking across the contemporary landscape to visually ‘see’ what used to be there provides a more tangible, meaningful, and moving experience that will create a lasting impact. We anticipate launching the app in 2022.

Bibliography

Bantom, K (2021), ‘D6 ‘caretakers’ named,’ news24, 14 September [online]. Available at: https://www.news24.com/news24/SouthAfrica/Local/Peoples-Post/d6-caretakers-named-20210913 (Accessed 13 December 2021) 

Charles, M (2021), ‘City of Cape Town evicts homeless in District Six during massive operation,’ news24, 20 September [online]. Available at: https://www.news24.com/news24/southafrica/news/watch-city-of-cape-town-evicts-homeless-in-district-six-during-massive-operation-20210920 (Accessed 13 December 2021) 

Cox, V, Dall, V, Qui, V and Yelk, J (2019) ‘St. Mark’s memory capture and interactive mapping.’ (Global Information Engagement Program, School of Information, University of Michigan)

Huang, J, Leal, M and Rubin, M (2018) ‘Final report: St. Mark’s community heritage and baptismal record project.’ (Global Information Engagement Program, School of Information, University of Michigan)

Lepule, T (2021) ‘Claimants dying off before ever returning to District Six,’ Weekend Argus, 28 November [online]. Available at: https://www.iol.co.za/weekend-argus/news/claimants-dying-off-before-ever-returning-to-district-six-b8f3ed06-866c-4185-82eb-a9ac13c67d0c (Accessed 13 December 2021) 

Motala, S (2019) “District Six and CPUT: a carto-story.” Available at: https://www.youtube.com/watch?v=iEzydbcVWV4 (Accessed 13 December 2021) 

Motala, S and Bozalek, V (2021) ‘Haunted walks of District Six: propositions for counter-surveying,’ Qualitative Inquiry [online]. Available at: https://doi.org/10.1177/10778004211042349 (Accessed 13 December 2021)

Thebus, S (2021) ‘Plan to resettle District Six claimants plods along,’ Cape Argus, 16 September [online]. Available at: https://www.iol.co.za/capeargus/news/plan-to-resettle-district-six-claimants-plods-along-113c2d18-f4b8-48f4-a6fe-eefa89235742 (Accessed 13 December 2021)

 Tuck, E (2009) ‘Suspending damage: A letter to communities’. Harvard Educational Review, 79(3), 409–427. Available at:  https://doi.org/10.17763/haer.79.3.n0016675661t3n15 (Accessed 14 December 2021)

Tuck, E and Yang, KW (2014). ‘R-words: refusing research’ in Paris D and Winn, MT (eds.) Humanizing Research: Decolonizing Qualitative Inquiry with Youth and Communities. Thousand Oaks, California: SAGE Publications, 223–248. Available at:  https://doi.org/10.4135/9781544329611.n12 (Accessed 14 December 2021)

University of Michigan (undated) ‘Engaged learning,’ [online]. Available at https://engaged.umich.edu/engaged-learning/ (Accessed 13 December  2021)

Wallace, D, Duff, W, Saucier, R, Flinn A (eds.) (2020)  Archives, recordkeeping & social justice. London: Routledge

Zembylas, M, Bozalek, V, and Motala, S (2021) A pedagogy of hauntology: decolonising the curriculum with GIS. In Bozalek, V, Zembylas, M, Motala, S, and Hölscher, D (eds) Higher education hauntologies: living with ghosts for a justice-to-come Oxon: Routledge, 11-28). DOI: 10.4324/9781003058366 

In September 2019, five months before the pandemic, I moved to Toronto to begin a PhD at York University. I had been to the city before, and as I was a new international student  I thought I would make new social connections in school. However, in February 2020 the world transformed into its virtual ‘metaverse’ form, and I realised I would not be able to make friends and deepen any new relationships through Zoom, Teams or Slack. The loneliness hit me hard and even though leaving Canada was an option I was worried that I would not be able to return to finish my studies, so I stayed.

Looking back at the two years through a linear order of events, it seemed to me as though the time had been a vacuum. I could not put events into a linear narrative and I could not remember many things that happened in those two years. As a community as well as individually, we went through a repeated experience of fear, uncertainty, and freedom limitation. Negative emotions, such as boredom and anxiety, influence our long-term memory. Moreover, numerous studies have reported that ‘human cognitive processes are affected by emotions, including attention, learning and memory, and reasoning’ (Chai M. Tyng et al., 2017, p. 1454).

So, I wondered whether one of the post-pandemic consequences could be trauma (and possibly a collective trauma). Research on trauma shows that “the social environment does not have a direct and static impact but is mediated by emotional experience, the way it is lived through, interpreted, and processed on the basis of social, personal, and situational resources (today often termed as potential for resilience)” (Busch & McNamara, p.324). Each person experiences the hardships of life differently and some are more resilient than others. I was stuck, anxious and could not focus so I decided to do the one thing that satisfies me: making things. One of these ‘things’ was an interactive experience dealing with my own pandemic trauma. I decided to experiment on myself and find healing methods through interactive VR. If we take into account that the mind finds tools and technologies in the world in order to expand cognitive space constantly. And, if thinking is feeling and feeling is thinking, our emotions co-expand into this space. If we take creation into the equation and if “making is thinking is feeling” (Gauntlett, 2018), I am persuaded that emerging media are the best form for interactive collaboration between humans and machines.

The Making

The experience was created through the AI Storytelling Project as part of the Immersive Storytelling Lab (ISLab) at York University in Toronto. During my job there as an XR creator, I started to explore how to use interactive and immersive media; such as mixed, virtual and augmented reality (XR) and co-creation with the machines (Loveless, 2020; Wolozin, Uricchio and Cizek, 2020; Guzman & Lewis, 2020) for post-pandemic experiences. Specifically I focused on Natural language processing (NLP), a subdivision of artificial intelligence often used for different aspects of human-machine communication (e.g. speech recognition, text generation, speech-to-text and text-to-speech transformation, etc). Thanks to the cooperation between ISLaband the NLPsoftware-based storytelling platform Charisma.AI, I was given access to its Beta version, and I started exploring different options for an immersive and conversational experience.

My methodology during the creation of the virtual reality piece Home Is the World VR took the form of creation-as-research, where “creation is required in order for research to emerge” (Chapman & Sawchuk, 2012, p. 19). Via this VR piece, I investigated the relationship between technology, creation and the human condition, or as Guzman and Lewis explain, “how people understand AI in relation to themselves and themselves in relation to AI” (2020, p. 77).

I followed in the footsteps of the rich tradition of human-machine interaction, elaborated by Sherry Turkle when the personal computer was being adopted into everyday spaces. She described it as a “metaphysical machine,” a concept which led to the study of AI as a challenge to existing conceptualizations of the nature of humans” (Guzman & Lewis, 2020, p. 80). Emerging technologies may eventually push past the boundaries of human communication, meaning that human-AI communication may change how we communicate with each other as humans and with other entities.

Before I start describing the process of designing Home Is the World VR, I want to give a short synopsis to introduce the technology and the main idea of the piece. The VR experience is aimed at headsets with a passthrough option (i.e. Oculus Quest, HP Omnicept Reverb 2 etc.) –  a feature that allows users to step outside their view in VR to see a real-time view of their surroundings. Passthrough uses the sensors on the headset to approximate what one would see if they were able to look directly through the front of their headset. Hence, the interface is created by an ephemeral space between virtual reality and the user’s physical environment where they converse with the AI character and are asked to answer its questions and to follow a series of sensory-led tasks to induce pleasant memories before the Covid-19 pandemic. Such tasks include smelling coffee beans, conjuring the smell of an early summer morning, the sound of walking on snow, the taste of a delicious dessert, the touch of a plant etc. Then, the Charisma.AI software records a user’s spoken memories of the pandemic and generates a personal pandemic story for each user, individually. The twist: it re-tells the experience somewhat differently. It re-contextualizes the user’s story in a world full of positive news.

Project workflow

 Through Charisma.AI’s dialogue engine, and Google Cloud NLP services, players and audiences can meet with virtual characters, converse with them, and change the story. The AI character is created through the Charisma.AI platform and Google Cloud Speech-to-Text service. Charisma.AI uses the language of storytelling, with built-in features such as emotion, memory, scenes and subplots (Charisma, 2017 – 2021). However, it is not GPT3 based and works with a machine learning dialogue engine, which I found exciting when designing a semi-scripted interactive piece.

My research and ideation of the project started with the following questions. What if we could go back in time and re-shape our memories through an interactive AI storytelling experience? What happens when a human and AI re-create memories together? What if the experience played with the idea of the flexibility of memory and its ability of re-constitution? The connecting points in these questions were the ability of language to be performative, making sense of the world by creating stories, putting events and experiences into linear narratives, and the flexibility of human mind. And it seemed as though Charisma.AI, as a storytelling platform using such NLP modes as intents and sentiment, could not only be used as a tool to make stories but also as the content itself.

The saying ‘think before you speak’ has meaning in neuroscience – the formation of words acts as a delaying function, giving the brain time to deal with information input and retrieval. Traumatic experiences leave such an emotional mark on our brains that, when recalling certain events, we tend to be overcome by emotions and are not able to express what is happening to us through language. It is not by accident that trauma is often dealt with by speech therapy and storytelling. Remembering traumatic events through language and storytelling helps us to deal with  them.

Interestingly, there is a connection to sociolinguistics here, specifically to  J.L. Austin’s theory of performative speech acts. In his book How To Do Things with Words, Austin establishes performative utterances, which not only describe a given reality, but also change the social reality they are describing (1962). The theory was later elaborated by Judith Butler in her book Gender Trouble (first published in 1990), where she introduces gender as a discursive and performative practice.

According to van der Kolk, “Traumatic experiences are exceptional because these intensely emotional events are not encoded into the ongoing narrative states” (2014). The traumatic experience is recorded as separate and dissociated from other life events, and, thus, it takes on a timeless and alien quality. In healing trauma, language is crucial.

John J. Ratey claims that when a subject tries to recall a traumatic experience they are overcome with emotion and are not able to express it in words. They are ‘dumb struck’ for a variety of reasons, one of them being that an important part of our brain responsible for emotion amygdala ‘overreacts’ while another part responsible for language and speech, the so called “Broca’s area shuts down” (2001, p. 210).

Further, Ratey claims that “the formation and recall of a memory is dependent on the environment, mood and gestalt at the time the memory is formed or retrieved” (2001, p. 208). Each memory is created from a vast interconnected network of pieces in our brain such as language, emotions, beliefs. And our daily experiences alter these connections and, therefore, we remember things differently in different phases of our lives.

Apart from neuroscience and linguistics, the theoretical approach framing the project is affect theory, which helps to disclose the ways technology intersects with our limited proximal senses, rhythm and sense of motion and embodiment. When affect is processed through cognition – once the signal from the amygdala (limbic system) reaches the prefrontal cortex in our brains – it becomes an emotion. It occurs through cognitive action and relations between agents (humans, non-humans, things, environment).

These cognitive actions are inherently performative, such as language, bodily and facial gestures, or tone of voice. Sara Ahmed (2014) calls these relations “contact”. She argues that already when we feel that something is good or bad, it involves “reading the contact we have with objects in a certain way” (Ahmed, 2014, p. 6). Contact involves a process of reading, attribution of significance, it “involves also the histories that come before the subject” (Ahmed, 2014, p. 6). Emotions are, thus, culturally and linguistically constructed performative actualizations of affect. Ahmed posits emotion as cultural construct through language (2014) as when we name and perform our emotions they become solidified and transmitted to others.

This process of emotion transmission can be also called “affect contagion” (a perfect example is sentiment contagion in social media through text). Plus, according to Brian Massumi, emotions are culturally and linguistically situated, “emotions are embedded in the arbitrariness of language and gestural code (including face) involving cognition, and through which we assign these qualities, and which carry meanings in order to be communicated” (1995, p. 89).

Stage Two: The Design

Scripting in CharismaAI

The main aim of the piece is to trigger positive emotions via sensual memory associations and language. When reframing the memories through emotions, I applied methods in trauma research, using questions and sensory led tasks that connect past, present and future through emotion and affect (Damasio, 2005; Van der Kolk, 2014; Busch & McNamara, 2020; Bloomaert et al., 2007).

Upon entering the virtual world, users encounter an omnipresent AI character, which converses with them and gives them a series of sensory-led tasks to induce their memories and emotions associated with the pre-pandemic world and apply them in the ‘now’. First, the user is asked by the AI character to remember a situation during the Covid-19 pandemic. Second, they are prompted to answer questions such as: what sign are you? Where did you spend your pandemic time? Were there other people? In what month did the significant event happen? After the AI character collects all necessary information, the software generates a positive event, which happened in the same month in 2020 or 2021. These generated pieces of text come from a corpus of news articles about positive events during the pandemic. Here are some examples:

“In March 2020, we learned that a group of dogs trained to protect rhinos from poachers have saved 45 rhinos in South Africa. The dogs, including beagles and bloodhounds, among other breeds, were trained from birth to track down poachers alongside humans in Greater Kruger National Park.”

“June 2020: The Supreme Court rules that no one can be fired for being gay or transgender, and Beyonce releases Black Parade. Yaas, Queen B!”

An example from my Miro board of the corpus of  positive news events in 2020

Apart from the language, different senses, such as smell, touch, sound, vision and taste, are directly connected to our emotions (e.g. the sense of smell has direct relation to our limbic system, which is why it triggers memories and emotions without any cognitive interference). Therefore, in the second part of the experience, the user is asked to follow sensory-led tasks to deepen the positive emotions. The next image presents different sensory tasks which were brainstormed during a workshop with several participants.

Things to feel with

Blueprinting in UE4 with CharismaAI plugin

In the finale, the AI character summarises the collected information about the user. However, it replaces the negative event with a positive one. And it asks: do you feel better?

Prototyping

Stage Three: User Testing

User testing is crucial when making interactive and immersive experiences (e.g. for improving the user design). Moreover, it is vital to check that the VR experience doesn’t cause any physical or mental issues in the participants. When I tested the prototype in our lab, some participants felt as though it was a ‘confessional’ experience. They saw it as a safe space where they could talk to a scripted AI character who tried to make them feel better about the pandemic by taking them to other emotional landscapes.

On the other hand, when I did an embodied proof of concept exercise with participants at a real-life conference this year, some told me they did not want to go back to the pandemic years. Does this prove or denounce the healing effect of the experience, and the traumatic element of the pandemic?

Busch and McNamara claim that:

“Avoidance of painful intrusions as a measure of self-protection is only one of the reasons that makes it difficult to share a traumatic experience with others. There are many other reasons such as: speaking about what happened can be subject to interdiction, social taboo, or shame; a common ground of experience or knowledge is missing; one does not want to burden others with one’s own pain; the violation and the suffering have not yet been socially acknowledged” (2020, p. 329).

Whereas one user may go through an intimate immersive and interactive experience with an AI character, which not only lends them their ear but also gives a social acknowledgment of the pandemic as traumatic. For another user, it may be a reiteration of the painful memories which they are not ready to share.

Conclusion

A new research question needs to be proposed: When does a VR experience from the first-person perspective heal and when does it re-traumatize? I have experienced far too many re-traumatizing VR pieces in the past (e.g., being sexually, physically, and verbally assaulted in VR can re-traumatize a user if they experienced a similar situation in real life, and if they have no agency or control over the situation in VR). My VR piece aims to stand in opposition to such XR works. It re-iterates the negative memories into positive pandemic stories. And I hope that it can transform the negative emotions into positive ones through language, conversation, and sensory triggers. The iterative process of creation-as-research doesn’t end with a final build of an experience. My work on this project is to be continued.

References

Ahmed, S. (2014). The Cultural Politics of Emotion. Edinburgh University Press.

Austin, J.L. (1975). How to Do Things with Words. Clarendon Press.

Blommaert, J., M. Bock, and K. McCormick. (2007). ‘Narrative inequality in the TRC hearings’
in C. Anthonissen and J. Blommaert (eds): Discourse and Human Rights Violations, pp. 33–63.
John Benjamins Publishing.

Busch, B., & McNamara, T. (2020). Language and Trauma: An Introduction. Applied Linguistics, 41(3), 323–333. https://doi.org/10.1093/applin/amaa002

Butler, J. (2006). Gender Trouble. Routledge Classics. New York: Routledge. 

Chapman, O. B., & Sawchuk, K. (2012). Research-Creation: Intervention, Analysis and “Family Resemblances.” Canadian Journal of Communication, 37(1), Article 1.

Charisma — Storytelling powered by artificial intelligence (no date). Available at: https://charisma.ai/ (Accessed: 15 December 2021).

Damasio, A. (2005). Descartes’ Error: Emotion, Reason, and the Human Brain. Penguin.

Gauntlett, D. (2018). Making is Connecting (Second). Polity Press.

Guzman, A. L., & Lewis, S. C. (2020). Artificial intelligence and communication: A Human–Machine Communication research agenda. New Media & Society, 22(1), 70–86.

Loveless, N. (2019). How to Make Art at the End of the World. Duke University Press.

Massumi, B. (1995). The Autonomy of Affect. Cultural Critique, 31, 83–109.

Milgram, P. and Kishino, F. (1994). ‘A Taxonomy of Mixed Reality Visual Displays’, IEICE Trans. Information Systems, E77-D, no. 12, pp. 1321–1329.

OpenAI (no date) OpenAI. Available at: https://openai.com/ (Accessed: 15 December 2021).

Ratey, John J. (2001). A User’s Guide to the Brain. New York: Random House.

Tyng, C.M. et al. (2017). ‘The Influences of Emotion on Learning and Memory’, Frontiers in Psychology, 8, p. 1454. doi:10.3389/fpsyg.2017.01454.

Van der Kolk, Bessel (2014). The Body Keeps the Score. Viking.

Wolozin S., Uricchio W., Cizek K. (2020). Collective Wisdom. Massachusetts: MIT Press.

In 2021, I attended the 10 Days on the Island festival in lutruwita/Tasmania as part of a research project exploring the social impact of the creative arts in Regional Australia. On my second day at the festival, I went to a small workshop Reaching Global Audiences with Local Storytelling led by Catherine Pettman from Rummin Productions and Rebecca Thompson. After an hour, it was clear that these two filmmakers were creating, producing and sharing some of the most interesting and community-led stories I had seen in a very long time. Their approach to respecting the communities and individuals they are collaborating with is centred on meeting participants where they are: emotionally, creatively and physically. This was challenged in 2020, and their extraordinary film There is No ‘I’ in Island demonstrates their innovative solution to enforced isolation. This short documentary series weaves the fears, dreams, reflections, and songs of the island community of lutruwita/Tasmania into a fantastical, animated landscape. Every voice heard in the series was self-recorded in May 2020, during Tasmania’s COVID-19 pandemic lockdown, and reflects in a personal way on the experience. Catherine Pettman reflects below on the process of collaborative storytelling and its potential to create change.

Tell us a bit about who you are and how you came to be working in film and documentary filmmaking?

In the southeast of lutruwita/Tasmania there’s a unique meeting of two saltwater bays, divided by a narrow isthmus with high coastal mountains that dip their wooded toes into the icy Southern waters. It was a remote place to grow up and our family activities were completely dictated by the climate and what needed to be caught or grown for the dinner table. I played in the ocean, and on the ocean. I walked through the temperate rainforest to find the waterfalls. Mostly, I rode for hours on my single speed bike, travelling here and there, exploring every nook and cranny. They were fearless years full of freedom and adventure. teralina/Eaglehawk Neck was a magical environment, which hosted a complement of curious visitors and unique local characters full of news of their adventures out there in the wilds. Listening to these storytellers, particularly my parents, was a wonderful way to spend the time. Perhaps this is where my love of story began.

I left home in my late teens. Not an unusual occurrence as we tend to leave the Island early on. It’s a migration really. For me, it was to gain higher education in the Dramatic Arts and beyond, and traveling and exploring other cultures. Journeying through the Northern Territory and then living and working with the Yolngu community in NE Arnhem Land had a huge impact on my life, and for which I will always be grateful. Filmmaking started on the other side of the camera as through my performance studies I saw myself as an actor and theatre maker. Eventually, my heart brought me home to Tasmania where I began crewing on commercial productions, gaining experience through a variety of roles, and eventually began to produce my own content for the screen. I’ve particularly loved documenting stories of exemplary people sharing their life’s passion. Oftentimes, there are themes that relate to the wicked issues that we face as a society. Funnily enough, there’s a collective interest in making and sharing these kinds of stories that build community capacity, and quite a few of my shorter documentaries have resonated in places far removed from our little island at the bottom of the world. 

How would you describe your approach to finding the stories you want to tell?

I have a strong suspicion that stories find me. At least it often feels that way. I hear or read about something intriguing, or a colleague shares an experience or an idea, and, suddenly, we’ve been talking for an hour together, building on what’s fascinating or why it’s compelling and how badly we need to capture it, and share it with others. If activities within the story are time critical then there’s extra pressure to try and solve the puzzle of how to develop the idea, how to finance the production, and how to find a pathway to an audience. That was certainly the case with There Is No ‘I’ In Island. It was developed in response to our COVID-19 lockdown and the reality of not having capacity to film live interviews. Hence, the idea to ask participants to self-record their responses was born. 

All of a sudden, we found ourselves pouring over hours of intimate stories, which we collated into story threads for our five episodes. It was a humbling experience, opening each recording full of stories gifted to us during such a poignant moment in time. There are hours of stories I’d love to share but there’s not enough time to make them all into films. Although this creates tension, I also recognise it’s a privileged position to be in. Overall, the stories that rise to the surface are more often than not those that have a strong community of collaborators and supporters all the way along. There’s an authenticity within the story, the process and the community of storytellers, which eventually translates to an authentic film that hopefully connects with the intended audience. Authenticity is probably the most fundamental aspect in choosing the story I want to tell. Truth mirrored back through story has the power to challenge our values and beliefs and, just possibly, transform the way we see the world – perhaps even offering up a renewed perspective on how we can be better citizens within it. We can but try.

What innovations in the ways that community stories can be made and shared are most exciting for you?

The There Is No ‘I’ In Island series was a break away from the conventional, and I suppose we considered it innovative as it was new ground for us as creatives. Myself and co-creator Rebecca Thomson prompted experiential responses from the community by asking specific questions around a particular subject, and participants self-recorded their answers rather than having a filmmaker present to guide the narrative – quite an untraditional method. By doing this, the authority transferred across to the individual and already we can see the impact of community ownership of their ‘voices’ in the way There Is No ‘I’ In Island  has been embraced and joyously shared through physical space and online. The production most likely felt innovative as it occurred during a very uncertain time during lockdown when we were all unsure how production could continue. It became this very nimble and explorative way that we could keep producing content and share human experiences and stories from our own community, with community participation being at the core of the process. 

Everything about the project made perfect sense and the community responded in kind with openness and enthusiasm. We could sense that the community were feeling ‘seen’ and ‘heard’, which gave their stories even more weight as it was clear how valued they felt being involved. It was hugely exciting to develop from the ground up using this pool of natural, charming, exotic, and relatable characters who were completely anonymous to us, yet who also became extraordinarily familiar throughout our production process. 

As There Is No ‘I’ In Island was conceived during lockdown in May 2020, it was also designed so that we could pair five visual artists each with an experienced animator.

It was hugely satisfying to match these creative teams with an episode that best suited their tone and style, creating a rich and valuable opportunity to build their skills, push themselves creatively and to form new professional collaborations. This network of participants and creatives has resulted in a multi-faceted cross-section of community participants, artistic collaborators, supporters and viewers.  

Do you think stories can create change?

One of the most beautiful things about stories is they have the power to open us up to new perspectives. The question I tend to ponder is how exactly is the story creating change in a person? Is it conscious? Or are these archetypal themes resonating with our subconscious selves, shifting our deepest values and beliefs in a new direction? What happens when new concepts and ideas settle in and relax our learned perspectives and prejudices? Do stories somehow deliver intangible meanings that nurture us and provide sustenance for personal growth? Stories deliver such rich meaning to our lives, they connect us to higher ideals whilst, at the same time, they also connect us to our own hearts. 

Stories have the capacity to make us care deeply about the world and the wicked issues that we face in society. Stories help us understand how to tackle the challenges around us, they inspire us, and remind us we are not alone. Stories provide a pathway to express our emotions and our dreams, which is the perfect combination when you think about it. When stories create change in people, they become empowered to make change. A story that communicates a call to action is probably the most powerful tool in making change. We need to get active to see the changes we want beyond talking, listening and sharing as these activities aren’t enough by themselves. But they are a fantastic start to the conversation towards meaningful action.

Which are your favourite or most impactful projects and why?

Doing it Scared was one of the most beautiful projects I’ve produced, it was a deeply meaningful story featuring rock climbing legend Paul Pritchard and his attempt to finally conquer the ‘Totem Pole’, a “fearsome sea-stack”, which 18 years prior had almost cost Paul his life. Paul was one of the world’s leading climbers and mountaineers of the 1980s and 90s, renowned for his hard and extremely bold first ascents. In 1998, Paul was abseiling in to climb the Totem Pole in Southeast Tasmania when he dislodged a rock, which hit him on the head, causing a severe head injury that he was lucky to survive. The aftermath of the accident left him with hemiplegia, which means he has little feeling or movement in the right side of his body. Despite this disability, Paul continues to live a life filled with adventure. So much so that when Paul decided to return to the Totem pole to finish the climb and asked us to film this extraordinary attempt, we leapt at the chance to support his ambition and help document the final chapter of this remarkable story. 

The resulting edge-of-your-seat film is just 12 minutes in length. It has been seen all over the world in every major outdoor adventure festival, and still continues to screen in cities throughout Europe, China and the US. Paul has been invited to speak at many of these festivals and uses the film within his own presentations to large NFP and Governmental organisations, and within smaller classroom settings, sharing his story and demonstrating his physical and spiritual experience of life through leading practical activities with a goal of breaking down prejudice with empathy, education and inspiration. 

You can find There is No ‘I’ in Island here:

www.rummin.com

https://www.facebook.com/rumminproductions

https://www.facebook.com/TasmanianVoices

Think of a song or piece of music that is important to you. What happens when you hear it? Does it take you somewhere else and bring you back? Are you saddened by it? Does it transport you away from the everyday? What is it about this song that is important? 

This last question lies at the heart of a project called Mapping the Music of Migration (MaMuMi), a two-year pan-European collaboration focused on music and the migration experience. Across Europe, migrants have been vilified and we wanted to combat this issue by finding a shared language, i.e. music.

Mobility and movement have long been central aspects of human life. People traverse national borders for complex and varied reasons, out of choice or necessity. War, poverty, economic downturns, love and retirement can all prompt displacement. There is no universal migration story, but the word ‘migrant’ narrows the narrative lens; it cuts out the lives prior to and after those journeys. It forces them into one, repeated across media platforms.

In the UK and abroad, we see tabloid media images of migrants in camps. We have seen footage of dead toddlers on Greek beaches. We hear stories about boats capsizing and people drowning in the Mediterranean. We know the numbers: 41 here, 43 there, more than 17,000 since 2015. UK politicians openly equate ‘illegal immigrants’ with terrorists. Those migrating to the UK face a hostile environment, their very status ‘as’ migrants denigrated. The constant reduction of their humanity to migrants results in racism, xenophobia, and segregation.

How do we change this narrative? How can we open up spaces where commonalities can be explored? We did this by using everyday technologies, in the form of mobile phones, to interview and record  participants talking about pieces of music. Using something so integral to people’s daily lives in this way engineered spaces that allowed people to speak about anything that the piece of music triggered in them. Putting them on a mobile app made them visible and audible to others.

At MaMuMi, we wanted to offer such spaces where we might share and discover those common stories. We knew from our previous research with NGOs across Europe that no one was using the method of talking about music to open up such a space. Trivial though that may seem, it is in fact hugely valuable. This is because two people may enjoy the same song, but have differing memories attached. This point of connection facilitates open discussion on grounds that are non-political / neutral. This is where we began to develop our app, which made these stories accessible to a wider audience.

The project’s seven partners are universities and NGOs working with migrants in Bulgaria, Cyprus, Greece, Italy, Norway, Spain and the UK. Between them, the project team have collected thirty anonymous stories recorded in refugee camps, offices, and over Zoom, in which migrants discuss a piece of music or song that they deeply associate with their journey. This track is then edited and interweaved with the music itself, into what the project call a ‘Song Story.’ These are then uploaded to the MaMuMi app, where they are represented by a pin placed on a map that spans the globe. It might be Iraq, Gambia or Italy. 

Selecting a pin will play a song-story, where users will begin tracing the speaker from origin to their journey into and/or across Europe, until at last they come to where they currently reside.  Their journeys see them from West Africa, overland to Libya, to Cyprus by boat. They move from Syria to a holding camp in Kos, they relocate from the North of England to Spain to retire and travel from Austria to Spain, for work, and from Greece to Cyprus for the same reason. 

A myriad of narratives are represented here, some of which may relate a song with certain memories, others which remind the speaker of traditions lost. One participant who grew up in Morocco chose Bonnie Tyler’s Holding Out for a Hero, for example, which he listened to on repeat before taking and passing his UK driving test. Another participant, now working in Bulgaria, chose his own song, sung in his native Breton tongue. One young Syrian boy from a refugee camp in Kos described a song that exemplified the brutality he had been witness to, saying ‘I have sad and bad memories from this song, because when the youth of our area went out to protest, they thought that the policemen would not attack them, but they hung them and they killed them.’ Poignantly, the song named ‘Ya Hef’ (translation: ‘Shame on You’) is a baleful ballad about the Syrian civil war by Samih Choukeir, a renowned Syrian musician now living in exile in Paris. Likewise, another participant described the meeting of their Senegalese Griot musical heritage with the Norwegian educational system, where they now teach. Their song is about pancakes, a catchy pop tune that they have composed for the children they instruct. They tell us that ‘If you build a country, you have to [focus?] it to the young people. That’s why my focus, my music usually is for the young people, that’s why I make this song, Pannekake’. 

The range of narratives from our participants revealed stories that were both specific and common to many, and hearing them has facilitated conversations across cultures and borders. The pan-European team reported that the project had enhanced understanding of cultural diversity and awareness, and a sustainable set of resources (manual for NGO staff and migration app), which can be used for education and training. The project was also innovative in the sense that it gave the floor to migrants, refugees and asylum seekers (without a musical background) to make their stories visible to the public and their local community and raise multicultural awareness, fulfilling the MaMuMi mission. All of us on the project believed that engineering such a new space was necessary, and we hope it has gone some way to combating the harmful narratives surrounding migrants.

These participants have shared their stories of success, loss, frustration, memory and hope with us. We are not therapists, we cannot offer therapeutic spaces, but, using technology, we have opened up a place where stories can be told, heard and shared. And as these song stories get mapped onto our app, we offer alternative narratives of migration to those that dominate our news media. 

More information and workshop methods can be found on our website: www.mamumi.eu

It’s August 2020 in Melbourne, Australia – roughly a month into what would become one of the world’s longest COVID-19 inspired lockdowns, at 111 days. I’m out for my daily run along the Merri Creek, a serpentine trail winding through the city’s north. Normally quiet at this time of the week, it’s bustling with people, with locals generously interpreting the government’s directive to only spend one hour outside the home each day. We’re here to breathe in the crisp winter air, but also for the proximal human contact, scrunching eyes as we pass each other in hope of forging a connection from behind our masks. My phone buzzes, signalling an incoming SMS. I don’t recognise the number.

I press my thumb down on the link, ‘Where are you today?’. It directs me to a website where I’m then prompted to switch on my phone’s location services.

‘You are 1375km away from Farhad Rahmati, who recorded this 14 hours ago’, it reads, followed by a ‘play’ icon. I press on the button, and the audio I’m listening to – a news podcast detailing the latest on the pandemic – fades down through my earbuds into the gentle sound of birdsong. It continues for 10 minutes – a crow chimes in intermittently, the steady buzz of crickets gradually increases in volume, and some other nondescript sounds can be heard far off in the distance. Possibly cars, or maybe a train. The line between the recording and my surroundings is hard to detect at times, as Merri Creek bird populations – crows and magpies – weave through my audio bubble. Finally, there are footsteps, seemingly drawing closer, then an abrupt silence.

This is one of thirty audio pieces recorded by men seeking asylum, detained indefinitely by Australia after attempting to arrive by boat sometime after 2013. The issue of refugees has played a key role in Australian political life over the past two decades, with both the ruling Liberal-National Coalition and Australian Labor Party advancing a punitive agenda.

Prior to 2001, asylum seekers who arrived by boat in Australia would be detained in facilities on the Australian mainland while their claims for refugee status were processed. If found to be a genuine refugee they would then be released into the community on a protection visa; if unsuccessful, they would be returned to their country of origin.

This all changed following a notorious standoff in August 2001 between the Australian government and Norwegian cargo ship the MV Tampa, which had come to the rescue of more than 400 asylum seekers stranded on a fishing vessel in waters between Indonesia and Australia. This saga led to the ‘Pacific Solution’, whereby the Australian government established offshore detention camps in Nauru and Papua New Guinea and implemented a policy of boat turnbacks.

Coming just after the September 11 attacks in the US, the policy proved popular, with the Howard government achieving an unlikely win in the federal election later that year. The Labor government dismantled the Pacific Solution in 2008, however facing growing numbers of boat arrivals and an increasingly hostile political climate it reopened detention camps in Nauru and Papua New Guinea in 2012, going even further than the previous policy in declaring that no asylum seeker arriving in Australia by boat would ever be settled in the country. The approach has continued under successive coalition governments, with former Prime Minister Malcolm Turnbull winning praise from US President Donald Trump for the cruelty inherent to Australia’s policy of deterrence.

I received a new recording from one of six asylum seekers each day throughout August last year. The SMS would come at random times, with a short description giving some insight into their surroundings. The project was devised by the Manus Recording Project Collective – a group made up of asylum seekers and Australian journalists and audio producers Michael Green, André Dao and John Tjhia. You could sign up by texting ‘Hello’ to a mobile number – originally supplied to me via email by my producer at the community radio station I’m affiliated with, Triple R – after which time the messages would begin.

‘Where are you today?’ follows a similar project of the Manus Recording Project Collective’s  from 2018. Titled ‘How are you today?’, it involved 84 recordings documenting the experiences of men imprisoned at the Manus Island detention facility in Papua New Guinea, which were then presented as part of an installation at the Ian Potter Museum of Art in Melbourne.

One of those involved in that project, Behrouz Boochani, has since been widely celebrated for his writing and advocacy. His book No Friend But the Mountains, written via individual text messages sent from Manus Island to his translators and collaborators on the outside, won numerous high-profile awards and is being adapted for both film and stage. Another, Abdul Aziz Muhamat, was recognised with the Martin Ennals Award for Human Rights Defenders in 2019. This came following his central role in the multi-award-winning 2017 podcast The Messenger – a story documenting his time interred on Manus Island, pieced together by more than 3500 WhatsApp voice messages sent between Aziz and Michael Green. Both men are now free, having been granted asylum by New Zealand and Switzerland respectively.

Digital media and platforms like WhatsApp have proved an effective mechanism for piercing the secrecy surrounding the Australian government’s treatment of asylum seekers. Journalists have for the most part been barred from gaining access to detention facilities on Manus Island and Nauru, and the Australian government has resisted calls for more transparency over its military-led maritime immigration and surveillance program, Operation Sovereign Borders. The extraordinary stories smuggled out of Manus prison, likened to Gramsci’s prison notebooks, have given profound insights into the reality of life for those trapped in a system that is designed to be so unbearable as to make people choose abject persecution in their own country over the conditions of Australia’s immigration detention.

In recent times, audio has proved a particularly successful medium for spotlighting the human suffering caused by harsh immigration policies. This American Life’s Pullitzer Prize-winning episode ‘The Out Crowd’ uses sophisticated character-driven storytelling techniques, incorporating atmos, first-person narration and music to explore the emotional toll and trauma experienced by both asylum seekers and migration officers as a result of President Trump’s ‘Remain in Mexico’ policy.

In Australia, recent examples such as the Guardian Australia-affiliated podcasts The Wait and Temporary have provided episodic narrative investigations of the ways in which the government’s policies are impacting those fleeing persecution both within the country and in neighbouring Indonesia. These highly personalised stories reflect the capacity for podcasting to inculcate ‘hyper-intimacy’ (Berry, 2016), where predominantly headphone-based listening practices, combined with an often confessional, self-reflexive storytelling style foster deeply embodied encounters with podcast storytellers (Dowling & Miller, 2019; Lindgren, 2016; MacDougall, 2011).

The specific experience of listening to The Messenger has been referred to as ‘earwitnessing’ (Rae, Russell & Nethery, 2019), with the affective power of voice inviting empathy for Aziz’s plight, and enabling a political engagement with the experience of detention. Much has been made of the role of voice in podcasting, where a personal, conversational style of speaking is thought to make for a more engaging listening experience. What, then, to make of audio recordings documenting imprisonment, such as Rahmati’s, where voice is entirely absent?

Brisbane Immigration Transit Accommodation (BITA) is situated precisely 1375 kilometres from my location in Preston, Melbourne. Rahmati’s short bio, written in small-print below the ‘play’ button on the ‘Where are you today?’ website, provides the crucial context needed to layer meaning onto his gentle recording made during a period of quiet contemplation in the exercise yard at BITA.

“When I record without talking…it talks louder, because I let the audience find themselves and feel themselves in this situation that I have been trapped in,” he said during a radio interview I conducted with he and Green on Triple R.

“And I think that’s very powerful message. To let people feel our frustration and our pain, which has been around us for seven years.”

The act of listening to a snapshot of Rahmati’s soundworld, recorded just a few hours prior, had the effect of eroding the vast gulf of distance and life experience that separated us. It also gave rise to a feeling of shared listening, made all the more resonant with the subtle feedback of birdsong on the Merri Creek.

Five days later, I received a recording that originated from much closer to home.

Yasin, a 24-year-old from Darfur, could be heard intermittently adjusting weights, then exhaling with exertion, as he went about his exercises, set to the backdrop of country music. Like Rahmati, he was brought to Australia under the government’s short-lived ‘Medevac’ legislation that gave medical professionals the ability to order the transferral of asylum seekers from offshore camps to the Australian mainland for medical treatment.

The Mantra Bell City Hotel is one of a number of Alternative Places of Detention (APOD) used to hold asylum seekers transferred under Medevac. It’s located just a few blocks from my house, on a busy road that cuts across Melbourne’s north, leading out to the airport. A growing collective of pro-refugee activists assembled there in the latter part of 2020 in protest against the 60 asylum seekers held inside, often risking fines for breaking laws restricting outdoor gatherings introduced to curb the spread of COVID-19. In December the detainees were moved to a CBD hotel, then a month later the government announced some would be released on bridging visas. Still though, the government’s no-resettlement policy on boat arrivals stands, so there’s little certainty as to where these men will end up.

In one sense, my closeness to the saga unfolding at the Mantra should have made me care more about the plight of those men. But why should this be so? Proximity, we’re told, is one of the primary news values, but audio has a way of removing the perception, and significance, of distance. Listening to Yasin and Rahmati’s audio recordings had the effect of placing me, momentarily at least, by their side – offering the opportunity to ‘find myself’ in their plight, as Rahmati put so well. And if we’re to truly empathise with those detained indefinitely and without charge in Australia’s name, then we need to overcome the idea that a mere physical closeness to something – or its visibility – is a marker of its importance.

Reflecting on the project, Michael Green points to the lingering effects of listening to these audio pieces.

“I think almost one of my favourite parts of the of the recording is the moment when it finishes, it just cuts out after 10 minutes,” he said

“But actually, it carries on in my mind. I continue to listen along or imagine what Farhad had continued to do in that space afterwards. I think that’s kind of a powerful part of it the way it takes you with them.”

For a time late last year, Rahmati’s whereabouts were unknown. Refugee advocacy groups held concerns for his welfare after it emerged he had been removed from BITA following an interview on Australia’s national broadcaster, ABC, where he spoke about the plight of refugees held in detention. This came after a similar turn of events months earlier, when he was moved from a hotel facility to BITA after talking to media. It turned out he’d been sent to the Villawood Immigration Detention Centre in Sydney, without explanation. Then, on February 17 this year he announced on Twitter that he had been granted asylum in the US, as part of a refugee swap deal negotiated with the Obama Administration. After eight years imprisoned by Australia, he would be free.

The pandemic has prompted us to grapple with the significance of distance and human connection. Podcast listening increased as people looked for ways to connect with the outside, with numerous COVID-19 specific ventures providing important news and insights into the health crisis, and many others offering a valuable source of companionship.

My experience listening to ‘Where are you today?’ reveals how audio has a unique power to forge an imagined relationship with others, not only through emotional, personalised storytelling, as has become ubiquitous, but also through the simple act of sharing a segment of one’s soundworld, allowing you to temporarily co-habit with those whose lives have been thrown into turmoil through the imposition of border politics.

After months of relative normalcy, where Australia’s privilege of isolation and closed-border policy served to keep COVID-19 at bay, Melburnians are once again in lockdown. The virus initially entered via a hotel in Adelaide, used to quarantine those relatively few overseas arrivals. We spluttered along for a while with low case numbers, trying to keep up with changing restrictions, before an outbreak in Sydney eventually drew us into its web. We’re now up to two-hundred-and-something-days in lockdown, with a tentative exit date of October 26. Some days are quiet along Merri Creek, others more busy. People tend to come out with the sun, gathering in larger numbers than are strictly allowed in the hope, or assumption, they’re not being policed. Still, the birds sing, as they ever did.

References

Berry, R. (2016). Part of the establishment: Reflecting on 10 years of podcasting as an audio medium. Convergence: The International Journal of Research into New Media Technologies, 22(6), 661-671. 

Dowling, D. O., & Miller, K. J. (2019). Immersive audio storytelling: Podcasting and serial documentary in the digital publishing industry. Journal of Radio & Audio Media, 26(1), 167-184. 

Lindgren, M. (2016). Personal narrative journalism and podcasting. Radio Journal: International Studies in Broadcast & Audio Media, 14, 23-41. 

MacDougall, R. C. (2011). Podcasting and Political Life. American Behavioral Scientist, 55(6), 714-732. 

Rae, M., Russell, E. K., & Nethery, A. (2019). Earwitnessing detention: Carceral secrecy, affecting voices, and political listening in The Messenger podcast. International Journal of Communication 13, 1036–1055.