Children’s literature has always been a genre curious to experiment and play with media. Just think of pop-up-books and how the Alice in Wonderland adaptation Alice for the iPad (2010) by Atomic Antelope was one of the first apps to explore the features of the iPad. Since the introduction of the iPad in 2010, several attempts have been made to explore meaningful alliances between the print book and the digital device using augmented reality technology; creating hybrid experiences combining the traditional medium for children’s literature and its newest carrier. Such experiences depend on the user installing an app on her digital device and pointing its camera at the pages of a book. The device reads or decodes the data on the paper page and activates and displays content on the screen. The user’s physical, multi-sensory and cognitive engagement in such experiences is also one of complex character as she is managing two technologies at the same time. 

On the one hand, the book or codex technology which include turning the pages and navigating in specific ways, and on the other hand, the digital device and its specific interface navigation. In this way, and as is the defining nature of augmented reality, the book that exists in the user’s real-world environment is enhanced by computer-generated sensory information thus playing with the user’s perception. Off-hand, the effort to create this union seems slightly paradoxical if we consider the fact that the iPad was conceived and designed specifically as the unification of the book and the computer. So, what can actually come of this persistent Sisyphean task of making the print book and the tablet computer work together? 

This article will dig into this question by taking you through some remarkable international examples of literary AR book projects for children, their development over the past decade and the experiences they can produce. It will then shift from an international to a more local, Danish perspective and explore the potential of AR books for children for supporting reading motivation. In Denmark, a new partnership saw the light of day in 2021 in the company Smart Books. The company consists of the popular YouTuber and influencer Rasmus Kolbe, best known under his old boy scout name Lakserytteren (directly translated: the salmon rider) and Søren Jønsson, who is a successful and experienced producer of games for children. Smart Books deliver an augmented reality ‘smart book’ concept, where the reader chooses the path through the paper book’s narrative, interacting with both a book and digital content on a smartphone or tablet, and in this way gains an interactive reading experience. While this strategy is new in a Danish context, the venture also stands on the shoulders of a line of previous AR book projects.

Literary “AR + book” projects for children

When looking over the last decade’s projects that combine AR and paper books aimed at children, it is clear that this media interplay has gained a stronger footing in non-fiction and educational publications than in more literary, narrative projects. Generally, in these latter projects there has been a development from early projects that mainly ‘digitize’ the content of paper books, such as 3D animation of characters in picture books (e.g., Resin’s Two Left Feet, 2013), without asking for the user’s engagement to any significant degree, to newer projects that play with the potential of the augmented digital environment more fully and call for the user’s engagement via interactive game elements (such as Books & Magic’s The Little Mermaid, 2016). However, in some of the latter projects the print book’s materiality and role is in turn neglected. The crux of the matter seems to be to find a balance between the media where one is not a gimmicky appendix to the other. 

Resin’s Two Left Feet

Books & Magic’s The Little Mermaid

If we look to projects that can be characterised as literary in the sense that storytelling and an aesthetic experience are at the forefront of the works, projects produced  by the now hibernating American multi-platform storytelling company Moonbot Studios stand out. These works include The Fantastic Flying Books of Mr. Morris Lessmore and The Numberlys. Both works are aimed at children, The Numberlys at young children (3-7 years old) and Lessmore at slightly older children, yet a target audience is not mentioned anywhere. They both exist in several media at the same time constituting cluster works (Mygind 2017): as picturebooks and AR apps that can be used in conjunction with the picturebooks, as standalone interactive apps and animation films. In the changeable, fleeting world of apps, these works are already old (the books, the apps and films came out in the period 2011-2014) and are not available for purchase anymore, but this does not mean that they are not worth mentioning here. On the contrary, these works draw closer to a balance and a mutual relationship between the print and digital media than many other works. 

Moonbot Studios’ The Fantastic Flying Books of Mr. Morris Lessmore, teaser for all versions

Moonbot Studios Imag-n-o-tron: Numberlys edition

The picturebook The Numberlys takes full advantage of the book medium by playing with the reader’s ingrained expectation to the book by using the oblong format and mixing the reading directions of the book. The book is bound in the short, left side, which would normally mean that the book is read horizontally, but already from the title page the expectation is denied as this page must be read vertically and the book must, therefore, be turned. The title is one large image that spreads from top to bottom, with the five little main characters marching across the page at the bottom. This vertical reading direction enhances the impression of the vast, oppressive world of all-pervading numbers that the characters inhabit. This is a characteristic of the picturebook that it takes advantage of the book medium, its materiality and reading conventions and plays with them to convey meaning. In relation to the hybrid AR book experience, it is noteworthy to point out how the interactivity between the reader and the medium becomes part of the way meaning is conveyed in this universe and therefore not something that is reserved for the digital component.  

In relation to the rest of the cluster work that The Numberlys compose through its many independent media versions, the AR app IMAG-N-O-TRON: Numberlys Edition is the only one that cannot stand alone. Since the app is dependent on the picturebook to activate its content, it can be characterised as an intracompositional transmedia phenomenon (Dena 2009), meaning that the ‘AR app + book’ composition is transmedial in itself, and the app is not a self-contained narrative unit. When opening the AR app, the reader is transformed into and staged as a player, collector and detective, using the digital device as a magnifying glass through which to explore and investigate the picturebook. The AR app encourages the player to scan the book for objects, which,  when located on the paper page via the camera, will turn into animated objects on the screen and be stored in the apps interface. When the objects from the book have been collected, the player can build new, fun constructions in the digital space and practice constructing letters and numbers. In this way, the app encourages the player to perform creative, educating tasks that mimic what the fictional characters do in the picturebook, thereby extending what we might call the core values or message from the picturebook to the digital environment via the AR technology.  

In relation to the aforementioned balance between the media in AR book compositions, it is noteworthy how, on the one hand, The Numberlys picturebook composes a self-contained narrative entity, while, on the other hand, the ‘AR app + book’ composition actually works independently as well. Of course, the reader/player will gain a deeper, more informed experience if she reads the story in picturebook, but it is not a prerequisite to engage with the part of the Numberlys universe that is available in the AR book composition as they offer two distinct kinds of engagement: engagement in a narrative and engagement in playful activities.

Disrupting reading and media cultural hierarchies

Moonbot Studios’ few productions were, and still are, innovative and remarkable examples of AR + book compositions and, on the whole, of experiments with multi-platform, transmedia storytelling, however, the venture did not continue and did not set a precedent for subsequent international AR books for children. If we look to the recent Danish Smart Book concept, this endeavour is targeted at older children, specifically children from 9-13 years old, and here we find yet another approach to the composition of the AR book. 

The Smart Book concept consists of a series of paperback books with individual accompanying AR apps. Currently, three books have been published in Danish and two in English are forthcoming. The series takes place in a fantasy universe of wizards and magic where the reader assumes the role as the protagonist “you”, the First Student of the Firemaster. Just like in the so-called gamebook series for children from the 80-90s Choose You Own Adventure, you navigate the book by reading short numbered chapters that present you with a choice and, depending on the choice, directs you to a new chapter. Some chapters also present the reader with challenges and puzzles, often small maths related puzzles, that must be solved to move on. These obstacles are presented in the book via simple illustrations and text but must be met and solved in the AR app that, when accomplished, will direct the reader to a new chapter in the book. The content of the puzzles is most often not related to the narrative and therefore not narratively motivated. Instead, they offer different kinds of cognitive engagement.       

An interesting aspect of this AR book composition is that the traditional way of reading a book is turned into something else via the non-linear and unknowable reading path. We might say that the book and the AR app create both a material and fictional space in which the reader moves back and forth, yet with a feeling of moving forward without knowing when the journey will end. Normally the reader of a book can see, feel and count the number of pages read and the number of pages remaining in the book. This conventional way of navigating a narrative in a book is suspended and disrupted both by integrating the AR app and through the non-linear structure resulting, paradoxically, in both a higher degree of agency assigned to the reader and a higher degree of obscurity or mystery for the reader.

In their public presentations of the Smart Book concept, the two owners of the company, Søren Jønsson and Rasmus Kolbe/Lakserytteren, associate their project and their motivation behind it with the widespread concerns about the decrease in reading among children. A Danish study of children’s reading habits conclude that children’s joy of reading literature decreases with age and that the drop happens between 5th and 6th grade (Hansen et al. 2017). This situation is the point of departure for the Smart Book project, and the owners’ stated mission is to reignite children and young people’s joy for the written word, and showing them the way to the AR books especially through Lakserytteren’s media channels, such as YouTube, TikTok and Instagram.

The Smart Book series adapts the known “choose your own adventure” form to the AR book, hereby disturbing existing notions of what it means to read literature and proposes a new way of reading. These hybrid works of fiction combine print and digital media in a mutually dependent fashion that challenges prevalent public debates about onscreen reading. In these debates a notion of a media hierarchy is formed. A hierarchy that contrasts print media as the authentic, educational way of reading with digital reading (both visual and audio) as the inferior way of reading. Transmedia AR book projects have an intrinsic potential of being able to break down this hierarchy and offer new multi-sensory, transmedia experiences that support different reading styles. 

References

Dena, C. 2009. Transmedia Practice. Theorising the Practice of Expressing a Fictional World across Distinct Media and Environments. PhD dissertation.

Hansen, S. R. et al (2017): Børns læsning 2017: En kvantitativ undersøgelse af børns læse- og medievaner i fritiden, Læremiddel.dk: Tænketanken Fremtidens Biblioteker.

Mygind, S., 2017. “A Chinese cluster: Danish-born digital comic as source for transmedia design and innovation” in Ensslin, A. et al (eds.): Small Screen Fictions. Vashon Island, Washington: Paradoxa

In introducing my new novel, author Ryan O’Neill puts it most succinctly:

This is an introduction to a novel you will never read.

He adds hastily that he is referring not to the book in your hands, the one he hopes you’re about to begin, but the novel that inspired his words, the novel he read.

The novel in question is Ex Libris and regardless of which copy you read it contains twelve chapters that can be shuffled into any order. The number of variations possible with such a structure is a little over 479 million. It has been published in both standard paperback and ebook editions, each copy a newly shuffled order of chapters unique to that copy alone. The manuscript that Ryan read in order to create his introduction is different to the finished copy now in his possession, which is in turn different from every other copy ever made.

I have written about Ex Libris previously where I noted that this kind of storytelling has its precedents, the most significant of which all hail from the 1960s. Nanni Balestrini’s Tristano was conceived and written using early computer programming to randomise its content between copies, though it wasn’t published as intended until print technology had caught up in 2007. Other similar books were housed in a box, either as loose leaves (Composition No. 1 by Marc Saporta) or as chapter booklets (The Unfortunates by B. S. Johnson). Of these, Johnson’s novel provided the most direct influence on the structure of Ex Libris: the fluid pieces of the story are defined not arbitrarily by the size of the page, but by the narrative itself. The story is broken into discrete, meaningful components that combine to form a larger picture.

What Ryan alludes to in his opening statement is that any work structured in this way presents a challenge to critical reading. How can readers universalise their experience if the texts they read are never consistent? You may disagree with someone else’s reading of a text, but you do so on the fundamental understanding that both of you have at least read the same words in the same order. John Bryant’s scholarship on textual fluidity through editions, translations, and adaptations demonstrates that texts are never as concrete as we might assume. But variation between editions is a long way from a narrative that changes by design between individual copies. Although it is possible to arrange Ex Libris in approximate chronological order (some events in the story clearly happen before others), each of the novel’s fluid chapters is a vignette, dependent on the others for context, but not for prior knowledge. I have used the analogy of a jigsaw puzzle to explain this to readers: smaller narratives link together to form a larger picture. The order in which the pieces are placed changes the individual’s progress but doesn’t change the ultimate picture.

The coding to compile finished print-ready files is done in Automator, the computer equivalent of a Rube Goldberg machine.

It can be difficult to get past the structure itself and the mathematics behind it as many contemporary and more recent reviews of recombinant works demonstrate. Umberto Eco in his introduction to Tristano, focuses almost exclusively on the novel’s number of permutations with only a cursory nod to the story. This might be understandable for a novel that, though beautiful, has a deliberately tenuous grip on character, plot, and setting. But the same approach is repeated in reviews of Saporta, Johnson, and other similar works. It is as though the flashy acrobatics of the novel’s physical construction obscure what the writers are doing within. And the critics’ resulting performative bewilderment or pithy dismissal of a wacky experiment seem to me like missed opportunities.

When the assumed shared experience of an audience is modified or removed altogether, how does their engagement with a narrative change? Some clues may be found in my own experience on both sides of the reader/writer divide. How I initially read and thought about a fluid novel like The Unfortunates, for example, is very different to how I have come to think about Ex Libris and that change in point of view has been illuminating.

My experience with The Unfortunates suggests that a first reading looms large in one’s perception of story. While reading, I had to keep reminding myself that the clever positioning of two adjacent scenes was attributable not only to the author’s craft but also to sheer happenstance. We’re trained to read stories as linear and it’s a hard habit to break. When I return to The Unfortunates today, no matter how many times I reshuffle its contents, the story is always coloured by that first reading and how the chapters initially unfolded. That first reading has become my definitive version of the novel from which all others deviate.

Readers of Ex Libris may have a similar experience, perhaps moreso given their copy cannot be physically reconstructed. Information that colours the perception of the characters and their actions may come earlier or later and its impact will undoubtedly shift. Readers who see more of a particular character earlier, for example, may centre the story around them in a way others won’t. Several of the fluid chapters also contain crucial pieces of information that change a character’s image or motivation and cast events elsewhere in the story in a different light. Reviewing the chapter order for each copy, I frequently pay attention to where these chapters fall, wondering how their precise location changes the tenor of the story.

I say I wonder because, primarily, I must rely on guesswork. My perception of the novel is not of a puzzle but of narrative pieces in constant motion, a true fluid state. As I worked on it, Ex Libris formed a kind of web, a set of interlocking shorter narratives that fed into a larger complex. For me there can never be a definitive version of the story, only discrete narrative chunks that cross-reference, echo, or contrast, but never line up precisely.

Unlike The Unfortunates which can be endlessly reshuffled, Ex Libris is presented to the reader as a single, fixed manifestation of the narrative. But it’s also a window, a viewport through which you might catch a glimpse of what I see. Without the capacity to physically manipulate pages, the reader must instead imagine that fluid state and the differences in emphasis that come with changes in how the story unfolds. With Ex Libris, like with all fluid texts, a critical reading should regard not only the text as it’s presented, but also with the text in every conceivable other version. The success or otherwise of any one version of the narrative is merely a subset of nearly half a billion possible narratives in the aggregate. Though difficult to fully conceive, this is something I suspect many readers instinctively know. A common reaction from those who have finished the novel is to seek out other readers to compare notes.

But what readers who squint to catch glimpses of the author’s view may not realise is that they have experienced the story in a way I cannot. I can cast an eye over any number of versions of my story, but I can never see the flow of a linear narrative, only a single path running through that fluid web of chapters. For better or for worse I can never have the experience I had reading The Unfortunates.

I suspect that’s why the story that emerged turned out far more self-reflexive than I had originally intended. Maybe it was inevitable that a narrative featuring a band of literary misfits reconstructing a library from fragments in a dystopian world would eventually turn in on itself, a comment on how fiction can become a vehicle for revealing how we construct our own truths. In the same way the story’s characters can never truly reach the author, so too a reader’s and writer’s experiences always remain tantalisingly out of reach for each other.

Ex Libris is out now.

Reference

Bryant, J., 2005. The Fluid Text. Ann Arbor: University of Michigan Press.

When writers discuss plot and pacing in narrative craft, especially in creative writing classes, we often talk about the curve of stories, the rise and fall in tension that characterises the most common story structures. Now usually, at least in my experience, that curve is not something a writer actively thinks about while composing a work. It’s more instinctive. Manipulating pace is one of the writer’s primary tricks in taking a simple sequence of events and turning them into narrative. But what in retrospect looks deliberate and disciplined, is in the act of writing more like manipulating the feel of the story as you go.

When it came to my current publishing project, all that instinct counted for nothing. An experiment in recombinant narrative structure requires careful consideration and active manipulation of the curve.

Ex Libris is a novel containing twelve chapters that can be shuffled in any order, yet always presents as a cohesive narrative arc. It is being published in a print run that randomises the chapters between each copy. With close to half a billion possible combinations, each copy will contain a unique version of the text, yet all will tell the same story.

The title for ‘Ex Libris’ comes from the nineteenth century fad for bookplates.

The two books that, more than any others, inspired the structure of Ex Libris are The Unfortunates by B. S. Johnson and Tristano by Nanni Balestrini. Curiously, both were written in the 1960s, though Tristano wouldn’t find its true form until 2007.

The Unfortunates is a beautiful but restless story about grief and the intrusion of memories that overlay the banality of daily life. The novel was structured with a fixed opening and closing and with freely fluid chapters between. The first edition and its more recent reproduction was published as chapter-length booklets contained in a box, which the reader was free to arrange in whatever order they desired.

Balestrini envisaged Tristano as a standard bound work with content that was randomised between copies. Sound familiar? The author was unable to realise the work as intended until forty years after its initial publication and with the advent of digital-based print technology. As the title suggests, Tristano builds its text using Tristan and Isolde as scaffold, which frees Balestrini to desiccate the narrative into the smallest of fragments, hints of meaning that only ever briefly come into focus.

Both works experiment boldly, not just with structure, but also with the language itself. The result is intoxicating: as a reader you feel like you’re having fun, even as you stumble around the text, constantly trying to find your footing. Tristano is one of the best examples of what I call ‘narrative drift’, the sense that, as a reader, you must let go of any sense of structure or meaning and allow the pages to take you wherever they lead. The Unfortunates is more focused, a narrative that initially drifts, but tightens as more of its pieces fall into place.

When I began writing what would become Ex Libris, I didn’t have a particular structure or publishing method in mind. What I wanted to do was write a work with fluid text without sacrificing a reader’s sense of plot or narrative arc.

I started with much more complicated mechanics and elaborate concoctions of fixed and fluid chapters. I ground my way through three drafts of the story, never completely satisfied, trying to find some magic key that would unlock how the story should work.

Eventually, I abandoned these versions of the story altogether. After a break from the manuscript, I returned and found myself back at first principles. Finally, I contemplated the curve.

I created a storyboard of sorts in Scriviner—movable lists in dot points—obstinately refusing to write anything resembling finished prose until a supporting structure had been mapped in sufficient detail. Slowly, a new structure began to take shape. The story begins in media res, at the beginning of the climax. Then it backtracks. It fills in details and circumstances that led directly to the opening scene. Then it jumps to the rest of the climax and conclusion. This means Ex Libris, like Johnson’s The Unfortunates, opens and closes with fixed chapters that frame the narrative. I had hoped not to invite such direct comparisons with Johnson, since clearly I would come off a distant second best. But the structure he pioneered, with its parallels to classic storytelling technique, is compelling in its simplicity.

Beyond the framing device, the fluid or recombinant chapters in Ex Libris primarily concern themselves with exploring character and world. These chapters exist in a weird state of semi-independence. A fluid chapter is episodic, with its own miniature arc. It cannot rely on prior knowledge. That doesn’t make it a short story. Although it shares traits with the short story form, a fluid chapter’s raison d’etre is to contribute to a greater whole. Detached from their surroundings and the framing of the novel, these little stories might struggle to pass a ‘so what?’ test.

Story and the structure developed in tandem. Part dystopia, part satire, with doses of paranoia and farce, and a self-reflexive bent, the novel is set in a hyper-networked surveillance state that has abandoned and almost forgotten the book. It focuses on a small band of subversives who collect the fragments and scraps of stories left behind. Calling themselves the ‘free readers’, they are attempting to rebuild a grand library they know must have once existed. A fragmented book about fragmented books, Ex Libris both feeds off and contributes to its own structure, a virtuous cycle of knowing winks.

I was very conscious of the reader’s experience.

I was very conscious of the reader’s experience, signposting and orienting the text at every opportunity to counter and minimise the sense of narrative drift. I maintained strict upper and lower word limits for each chapter. Too long indicated waffle that needed to be broken up. Too short pointed to a lack of substance. Often throughout the long planning stage of the project, I would stare at a dot-point breakdown for a chapter and think ‘but where’s the story?’.

I also avoided working on chapters in any particular order. Instead, I jumped around. From its initial use as a storyboard, Scrivener became a kind of reference tool as I wrote, a way to maintain a wide-angle view of the story, while moving the chapters around. The texts themselves were composed in separate documents, organised by character name and working title. Early printouts were separated into chapters, each one held together with a bulldog clip, so that I could shuffle and reshuffle while reading.

When I finally created the first complete manuscript, I used a random number generator and manually combined the chapters into a single file. I’ve never considered putting together a preferred or canonical order. The thought of it seems a bit…wrong to me. The chronology of the story can be reconstructed in part—some events clearly happen before others—but a grand overarching chronology would be impossible to determine. That’s not how this story works.

At the end of an exhaustive process, I wasn’t sure if I’d succeeded. It wasn’t until the first feedback from beta readers (each of them with their own unique random shuffle) that I suspected maybe this was working as intended. A good indication was that some of these early readers did their own reshuffling to see if I had cheated.

The coding to compile finished print-ready files is done in Automator, the computer equivalent of a Rube Goldberg machine.

The long process of conceiving, planning, and writing Ex Libris has led me to a different way of thinking about raising tension in a narrative arc. The behaviour of the characters introduced in the opening sequences is gradually becomes clearer as their background is revealed. It doesn’t matter in what order those revelations happen.

The best analogy I’ve found is that it’s like a jigsaw puzzle. The order in which you place the pieces doesn’t change the final picture, but it does change how you experience the journey towards it. Adjacent chapters might flow or they might juxtapose. A character might disappear from the story for a while. A particular piece of key knowledge might be revealed earlier or later. The story has a different rhythm between copies. If the traditional narrative arc is the linear curve, this is more two-dimensional.

So does it work? That remains my burning question as I finalise editing and prepare to publish. It’s impossible to speak for every possible combination. There are 479,001,600 of them so I can’t check. It’s something every individual reader will have to determine on their own based on the version of the text they receive. I’ve always hoped that the story might be good enough to transcend its construction. I imagine a reader happening across a copy of Ex Libris, with no prior knowledge of its creation, who will read from cover to cover and enjoy it.

Is that even possible? I guess we’ll see.

The crowdfunding campaign to publish Ex Libris is live until 25 November 2019.

Screenshots is a regular feature by Simon Groth, highlighting a project, app, or other resource of interest.

Core Values
by Benjamin Laird

Shortlisted for the QUT Digital Literature Award, Core Values is a response to the iconic Australian poem My Country, by Dorothea Mackellar. Updating the original text, it uses technology to not only animate language but transform the experience of the poem itself. The formality of the original poem is replicated, but also cut apart and interspersed with dehumanising jargon, map coordinates, GIS data, and technobabble made to scroll endlessly within a three-dimensional box, lined by historical maps of the nation.

It’s a poem and a representation of an Australia in which you are quite literally trapped, a prison. The poem’s ‘stereoscopic mode’ for viewing in a simple VR device only accentuates the feeling of being closed in, a confronting and powerful match between text and technology.

https://poetry.codetext.net/core-values/

Way back in the wilds of the year 2008, artist-extraordinaire James Morgan and I engaged in an animated discussion about Augmented and Virtual Reality. At that time James and I were collaborators-in-crime in the Third Faction Collective, a group of digital artists intent on constructing game interventions in Massively Multiplayer Online Spaces. During this discussion, I pitched to James an idea to establish an online space devoted to all things Synthetic Reality based (my umbrella term for Virtual Reality, Augmented Reality, and Mixed Reality). This space, called Augmentology 101, intrigued James to the point where a decision was made to sponsor it through the Ars Virtua Foundation and CADRE Laboratory for New Media. What followed was an amazing exploration into the creative potentials of Synthetic Reality – what’s now known as XR (Extended Reality) – and how it might manifest within the realm of electronic literature.  

It’s now been 10 years since the initialisation of the Augmentology 101 project. During this decade, there’s been a major upswing in VR and AR production and development, with impactful XR content such as Firebird – La Péri (a 2016 English/Chinese/French multilingual VR Experience) and Queerskins VR (2018) being standout examples.

Screenshot from the 2016 Multilingual Virtual Reality Project “Firebird – La Peri”

My own attempts at merging digital literature into developing XR fields have been multiple and varied, originating in delving into VR in the 1990’s when VRML was the shiny new thing. Surprisingly enough, the creative and technical challenges that VR creators faced back then are similar to those we face today: high performance requirements, mainstream adoption hurdles (see: Gartner Hype Cycle), and monetisation dilemmas are all relevant. Likewise, skillsets required by VR content creators in the mid 1990’s again parallel XR creators of today, including developing a deep knowledge of spatial storytelling logistics; emotional intelligence; and the ability to formulate story experiences that take into account various hardware and platform limitations such as field of view constraints, tethered headsets restricting natural movements, and hardware specific limitations like the screen-door effect.

In terms of XR projects I’ve produced in the last decade, a brief selection includes conceiving of and co-developing the 2013 anti-surveillance AR game #PRISOM, and in 2015 mapping out with Andy Campbell the (now unfinished) PC/VR project Square Ebony that was to be filled with: “…movement/imagery like huge ‘Panic Room’ landscaped letters…a force field of green…branches intertwined…tangles being text…[that] revolves around an entity…this entity is slowly reconfiguring itself…at the top of a hill/mountain/plateau surrounded by brackish water” (notes from Square Ebony Project Meeting and Documentation, Breeze and Campbell, March 10th 2015). In 2016 I lectured as part of the “Future Possible: Beyond the Screen” Series which centred on how VR can transform creative practice, and which also included a live VR performance walkthrough using one of my Tilt Brush created works. In 2017 I keynoted at the Electronic Literature Conference with a VR performance presented both live at the Conference and simultaneously in Virtual Reality.

“Layering the New real: Tracking the Self in Disembodied [Un] Virtual Spaces” Keynote

Press Image for “Our Cupidity Coda”: VR Literature

Thorough participation in a high-end VR based experience like Perpetual Nomads hinges entirely on immersion, which is triggered initially through the audience having to don gear that firstly reduces their ability to engage in their actual physical space in standard ways (their vision and hearing being “co-opted” into a VR space). The leap of faith the audience needs to make to establish a valid and willing suspension of disbelief (as Samuel Coleridge so aptly phrased it) is already set in motion by the fact a user is entirely aware from the moment they slip on a VR Headset that their body is in essence hijacked by the experience (haptically, kinetically), as opposed to a more removed projection into a story space via more traditional forms (think book reading, movies, tv). Such body co-opting might lead a user to disengage from the VR experience from the very beginning which will reduce the likelihood of true immersion: alternatively, they may readily fall headlong into the experience with an absolute sense of engagement and wonder (the preferred option as a VR content creator!) if the work has been precisely crafted.

For the most part, XR projects such as those mentioned above currently exist only in the mainstream margins, with a majority of experiences requiring costly high-end VR rigs and expensive desktop computers that demand audiences experience the works in their optimal state. To counteract this selective catering to the exorbitant end of the XR market, in early 2018 I had the idea to create a VR Experience that would reduce the mandatory use of high-end tech. This project would instead cater directly to a range of audiences by crafting a work that could be experienced across a far larger (and much more accessible) range of lower-end tech. This VR Literature work is called A Place Called Ormalcy.

Title Image from the “A Place Called Ormalcy” Press Kit

A Place Called Ormalcy is digital literature designed for, and developed in, Virtual Reality. It was constructed using the Virtual Reality Application MasterpieceVR to craft the 3D models, with each chapter (made up of 3D models, text, and audio components) then combined and hosted via the Sketchfab platform. It’s comprised of a text-based story made up of seven short Chapters which are housed in 3D/Virtual Reality environments. It can be accessed via a wide range (crucial in terms of its social commentary aspect) of mobile devices, desktop PCs and both low-end and high-end Virtual Reality hardware. Audiences using the cheapest type of VR equipment (such as Cardboard headsets) are able to access complete versions of this VR literature experience, as are users of any net connected mobile device with a WebVR-enabled browser.

The story of A Place Called Ormalcy (warning: spoilery parts ahead) unfolds through a series of snapshots of the life of Mr Ormal, a happy-go-lucky law-abiding chap who resides in the aesthetically cartoonish world of Ormalcy. Ormalcy exists in an alternative universe complete with its own idiosyncratic language patterns. The storyworld initially presents as a Utopia full of innocent “claymationesque” contented creatures and happy citizens. As the story plays out, however, it soon becomes apparent that in actuality, this VR Experience allegorically traces the makings of a dystopic society, and how such fascist principles can arise in the most benevolent of places. This VR Literature work has social commentary at its very core, commenting directly on and about the rise of current totalitarian trajectories and the contemporary malaise, confusion and accompanying acclimatization patterns.

“A Place Called Ormalcy” Chapter Progression

A Place Called Ormalcy uses a combination of WebVR, 3D, VR, text and audio assets in ways that mirror a slow dystopian creep. In the desktop and mobile versions, each chapter becomes progressively visually cloistered, with dark fog and grainy distortions increasing to finally create a type of gun-barrelled claustrophobic effect. This combines with a gradual leaching of the intense colours found in the free-flowing organic imagery of the initial Chapters which results in a startlingly stripped back, fuzzy palette and model constructions: vibrancy gradually bleaches out to stark black, white and greys. Correspondingly, the 3D tableaus and audio tracks likewise alter from an initial complexity – Mr Ormal begins his story journey waving directly to the audience in “Chapter Wonne” in a bright and blooming space – which incrementally shifts towards the dramatically minimal in the final “Chapter Severn” where Mr Ormal transforms into (…spoiler alert here…) something vastly other.

In the VR version of A Place Called Ormalcy, additional effects mark the dystopic “boiling frog” dilemma that Mr Ormal faces. Each VR tableau subtly increases in size and scale as the Chapters progress, with the audience finding themselves in the climatic Chapter in a looming monochromatic set surrounded by huge windowless block-shaped buildings devoid of detail – except multiple, and menacing, “88” shaped logos (and the awfully transfigured Mr Ormal). In the VR version, the text becomes increasingly difficult to navigate, with the audience having to teleport, twist and turn in the VR Environment to read each annotation, echoing the “fake news” proclamations of our contemporary Western world where it is becoming increasingly difficult to access truth over relentless propaganda.

Although A Place Called Ormalcy may seemingly convey a message of hopelessness or helplessness, the ending does contain clues that all is not lost in this particular dystopian scenario – the final soundtrack offers hope, with protestors chanting and proclaiming resistance as key. Just as VR Literature can work to extend the creation of accessible electronic literature beyond the text-centric to truly encapsulate the haptic and the spatially-oriented, A Place Called Ormalcy illustrates how XR projects can act as relevant social commentary at a time when it is sorely needed. I look forward to continuing to promote, create, and experiment with stretching the limits of VR and AR while producing XR projects that are openly accessible, as well as socially relevant.

Screenshots is a regular feature by Simon Groth, highlighting a project, app, or other resource of interest.

The Cartographer’s Confession
By James Attlee

The Cartographer’s Confession is the story of Thomas Andersen, who, as a child, migrates to London with his mother during the second world war and the fallout from that event in the decades that follow. Presented as a series of source documents—tapes, letters, and photographs—collected by the present-day researcher and screenwriter, Catriona Schilling, the app reveals its story through layers of fiction and non-fiction, timeframes, and locations.

Commissioned by the Ambient Literature project, The Cartographer’s Confession is designed to be experienced on location, appropriately using a map as its primary navigation. It’s not hard to imagine the power of walking through the streets of London as the story unfolds on in your ears and your phone. Though it does offer a chronological ‘armchair mode’, the app’s dreamy soundscapes and contrast of present and past lose some of their impact 16,000km away.

But even with a diminished experience, Attlee’s writing is concise and emotive, the performances are solid, and the app’s design, especially its sound, shows beautiful attention to detail. The soundtrack by The Night Sky is also very cool, if sometimes distracting. Given the quality of its writing and production values, it’s easy to see how The Cartographer’s Confession won over the judges of the 2017 New Media Writing Prize.

The Cartographer’s Confession is available to download from the App Store and Google Play.

Many writers have explored the use of new forms of software to supplement their writing practice, such as the use of a preferred interface to increase focus, or an application that boosts productivity. Equally notable are the advancements and innovations taking place in emergent technology such as: VR, AI, facial recognition, algorithms, and creative code, that have had a transformative effect on all forms of creative thinking, extending to the literature world.

Emergent technology encompasses the increased accessibility for people to explore technologically-focused ideas, as well the ability to transform novel approaches to computing, such as a text editor, into interactive and engaging systems that emphasise creative experiences for the user. My project, the Forgetful Typewriter, is situated within the remit of emergent technology projects, such as Ambient Literature. The projects of which are currently questioning and opening avenues of discussion about what these innovations mean for literature, from reading and writing to publishing.

The Forgetful Typewriter, is a project that was initially developed at Goldsmiths, University of London. It is a text editor programmed in such a way for words to fall, fade and reappear, reflecting the very process of writing process. These elements, in the text editor, occur both randomly (or serendipitously),  and simultaneously make use of data and parsing tools.

Emergent Technology and the Future of Literature

The connections between experimental literature and technology are deep rooted. A project that I recently became aware of is David Bowie’s verbasizer software, which explores techniques of cut up poetry in a digital context. The software helped enhance Bowie’s exploration of randomness that influenced much of his work and is a thought-provoking example of how artists from all kinds of backgrounds could consider a software component to their ideas. Writer and researcher, Oscar Schwartz, also has a fascination for computer-generated poetry. His research focuses on using algorithms that generate poetry with results that are sophisticated enough to provoke questions of what it means to be human. Moreover, the AHRC funded project, Ambient Literature investigates the interactive potential of digital text with a focus on location.

Another significant development in emergent technology is how these software ideas are becoming more accessible due to the maker movement. This encompasses components like Open Sourcing software, with online communities who have the ability to access and contribute to projects remotely from anywhere in the world. However the maker movement is equally considerable offline, when makerspaces and projects such DiDIY project (‘digital do it yourself’) provide platforms for collaboration and opportunities for members of the community, of all levels of technical ability, to come together and work on technology project, typically with an emphasis on creativity and contributing to improving society.

Creative Code & User Feedback (interaction with software)

I was introduced to code whilst completing my final major project in design at Goldsmiths, University of London. I was inspired by ideas of everyday technology’s use of impacting forgetfulness, such as memory outsourcing; which I also experimented with through prototyping forms of writing software that dynamically remove words from the typed text.

Discovering the processing environment, I developed these ideas as software sketches. The subject of the project has moved from themes of memory outsourcing, into rich discoveries and textures of results that occur from working in and with creative code.

Designing instructions and exercises, geared towards creative writing, I have continued to conduct research, involving writers from traditional and experimental backgrounds to participate in testing the software. These participants have included author Daniel Bürgin, and musician and lyricist Harry Burgess. Based on the feedback, there are two main themes for how the software could be beneficial and interesting for writers to use. The added sense of pressure comes from the unexpected results of the texts destruction as well as the word fragments and constellations these interactions create.

The process of words falling away as the text is being written, was described by Bürgin (2016) as being ‘competitive’:  “One competes with the words falling away and ripping apart the text… almost as if running out of time” . The software also heightens the immersive experience of writing. As a traditional writer of books and essays, Bürgin sees potential in adopting the software as one of the rituals that help overcome writer’s block.

Burgess who writes lyrics and shorter texts, has a completely different style in which ideas move around and are refined from intense writing sessions that push exciting, edgy language. He describes the process of lines fading as “racing against the interaction” , the result of which increases the expression. There is evidence that the constraint of not being able to revisit or re-edit the text is creatively conducive.  

Other features of use of the software are the resulting fragments and constellations of words, some of which Bürgin found to be creatively interesting: “I thought it got more interesting the more words that were lost. I thought that space was helping creativity more because fragmentation brings it down to the bare bones”. The fragmented version of the text that is returned by some of the interactions rearranges and condenses the writing can aid rethinking work and progress into new ideas.

Future aims for the project and Conclusion

The programme is not only useful for creative writing but can also support other areas too: linguistics or education. For example, other interactions, such as using tools to parse the text and apply the same kind of feedback based on a word type, could have exciting applications in the field of linguistics. Therapeutic writing is another subject that has been strongly recommended for further exploration through workshops.

It would also be fascinating to pursue ideas for how writers could play a part in programming their own experiences. For example, what if the project was approached in a similar way to the developments happening 3D printing and open sourcing, available for dismantling, taking apart and sharing the results? This could also function as an alternative introduction for non technical people to get to grips with the fundamentals of code/programming.

With the developments and progress of the maker movement  (emergent technology and open source communities) there is a compelling level of evidence that writers should be empowered to explore a creative and software-based component to their ideas, this is the main of objective of the Forgetful Typewriter.

Word of Colour Productions interviewed the self confessed tech geek, writer and editor Jacob Sam-La Rose on the impact of digital platforms and trends on his writing for The Writing Platform.

In 2012, Jacob’s poetry collection ‘Breaking Silence’ (Bloodaxe) was shortlisted for both the Forward Felix Dennis Award and the Aldeburgh Fenton Award. A techie and writer since the 1990s, Jacob has developed websites for literature development agencies, including Spread the Word, Apples & Snakes and Black Inc, and continues to advocate for the positive impact of new technology on literary and cross-art practice.

This interview was filmed by Words of Colour Productions in partnership with The Writing Platform. It will be the first of four profiles that Words of Colour Productions will produce for the new portal.

Interview by Joy Francis
Filmed by Nathan Richards
Supported by The Writing Platform

Welcome to The Writing Platform! Our aim is to provide neutral information and informed opinion on digital transformations in writing, reading, and publishing. In the build-up to the launch of this site, we’ve been surveying writers on their digital needs; a complex, fascinating picture of writers today has emerged. For more on the results of the first phase of the survey, go here.

We writers live in interesting times. Great change is taking place throughout the interlinked industries we rely upon for our livelihoods – publishing and bookselling. Reading and writing themselves are changing; new devices and new platforms proliferate. Phones are as powerful as computers; being online means you can publish yourself freely, no matter how big or small your audience. While the ‘end of the book’ has long been predicted, pundits are now predicting the death of the e-reader as tablets come down in price. Bookshops are vanishing from the high street, libraries struggle to redefine themselves while fending off cuts in funding, and a battle worthy of Star Wars rages over our heads between the three major tech corporations whose rapid infiltration of the world of books threatens to overwhelm even the largest of publishers.

On the other hand, books sales continue to boom. We are living through a Golden Age of reading; the ‘heavy reader’, that figure so beloved of all writers (and described for us here by Kassia Kroszer), has greater access to a larger variety of books, at lower prices, than ever before. Passionate readers around the world make use of both local and online book clubs; writing and reading continue to be activities central to the way we define ourselves as people. Good books still find their way to readers.

As well as that, opportunities for writers who are interested in moving beyond the book are also proliferating. Away from the world of traditional writing and publishing, new hybrid forms of literature have been emerging over the past decade, and with them, new business models are appearing.

And writers continue to write. But whether we are well established in our careers, or at the very beginning, or somewhere in-between, we are all part of an industry that is in extreme flux, an industry that will, no doubt, continue to shift and change for the foreseeable future. And this ever-changing landscape is difficult to navigate. Established writers have a tradition of out-sourcing their knowledge of the publishing industry to their agents; the recommended trajectory for most writers remains as follows: write that book, get that agent, let the agent worry about the rest. Our survey has thrown up a number of interesting trends: when asked ‘Where do you find out about developments and new opportunities in writing and publishing?’, writers listed websites (85%), other writers (63%) and live events (36%), with less than ten percent mentioning publishers (9.8%). The rise of self-publishing has disrupted the writer-agent-publisher trajectory; the one key thing that the successful self-publisher possesses – and that the successful traditionally published writer often does not – is an insider’s knowledge of how to publish a book.

All writers need to be bettered informed. We need to have access to clear, neutral, information about digital transformation and how it affects us; we need access to informed opinion and debate. The internet is full of information, of course, and a new future-of-publishing event or conference takes place every couple of minutes somewhere in the world, or so it seems. But very little of this information is aimed directly at writers. And that’s where The Writing Platform comes in; a website for writers, created by people who are dedicated to sharing knowledge and information. Please feel free to get in touch with us with your questions, comments, and ideas. We are commissioning content: tell us what you need to know.