Children’s literature has always been a genre curious to experiment and play with media. Just think of pop-up-books and how the Alice in Wonderland adaptation Alice for the iPad (2010) by Atomic Antelope was one of the first apps to explore the features of the iPad. Since the introduction of the iPad in 2010, several attempts have been made to explore meaningful alliances between the print book and the digital device using augmented reality technology; creating hybrid experiences combining the traditional medium for children’s literature and its newest carrier. Such experiences depend on the user installing an app on her digital device and pointing its camera at the pages of a book. The device reads or decodes the data on the paper page and activates and displays content on the screen. The user’s physical, multi-sensory and cognitive engagement in such experiences is also one of complex character as she is managing two technologies at the same time. 

On the one hand, the book or codex technology which include turning the pages and navigating in specific ways, and on the other hand, the digital device and its specific interface navigation. In this way, and as is the defining nature of augmented reality, the book that exists in the user’s real-world environment is enhanced by computer-generated sensory information thus playing with the user’s perception. Off-hand, the effort to create this union seems slightly paradoxical if we consider the fact that the iPad was conceived and designed specifically as the unification of the book and the computer. So, what can actually come of this persistent Sisyphean task of making the print book and the tablet computer work together? 

This article will dig into this question by taking you through some remarkable international examples of literary AR book projects for children, their development over the past decade and the experiences they can produce. It will then shift from an international to a more local, Danish perspective and explore the potential of AR books for children for supporting reading motivation. In Denmark, a new partnership saw the light of day in 2021 in the company Smart Books. The company consists of the popular YouTuber and influencer Rasmus Kolbe, best known under his old boy scout name Lakserytteren (directly translated: the salmon rider) and Søren Jønsson, who is a successful and experienced producer of games for children. Smart Books deliver an augmented reality ‘smart book’ concept, where the reader chooses the path through the paper book’s narrative, interacting with both a book and digital content on a smartphone or tablet, and in this way gains an interactive reading experience. While this strategy is new in a Danish context, the venture also stands on the shoulders of a line of previous AR book projects.

Literary “AR + book” projects for children

When looking over the last decade’s projects that combine AR and paper books aimed at children, it is clear that this media interplay has gained a stronger footing in non-fiction and educational publications than in more literary, narrative projects. Generally, in these latter projects there has been a development from early projects that mainly ‘digitize’ the content of paper books, such as 3D animation of characters in picture books (e.g., Resin’s Two Left Feet, 2013), without asking for the user’s engagement to any significant degree, to newer projects that play with the potential of the augmented digital environment more fully and call for the user’s engagement via interactive game elements (such as Books & Magic’s The Little Mermaid, 2016). However, in some of the latter projects the print book’s materiality and role is in turn neglected. The crux of the matter seems to be to find a balance between the media where one is not a gimmicky appendix to the other. 

Resin’s Two Left Feet

Books & Magic’s The Little Mermaid

If we look to projects that can be characterised as literary in the sense that storytelling and an aesthetic experience are at the forefront of the works, projects produced  by the now hibernating American multi-platform storytelling company Moonbot Studios stand out. These works include The Fantastic Flying Books of Mr. Morris Lessmore and The Numberlys. Both works are aimed at children, The Numberlys at young children (3-7 years old) and Lessmore at slightly older children, yet a target audience is not mentioned anywhere. They both exist in several media at the same time constituting cluster works (Mygind 2017): as picturebooks and AR apps that can be used in conjunction with the picturebooks, as standalone interactive apps and animation films. In the changeable, fleeting world of apps, these works are already old (the books, the apps and films came out in the period 2011-2014) and are not available for purchase anymore, but this does not mean that they are not worth mentioning here. On the contrary, these works draw closer to a balance and a mutual relationship between the print and digital media than many other works. 

Moonbot Studios’ The Fantastic Flying Books of Mr. Morris Lessmore, teaser for all versions

Moonbot Studios Imag-n-o-tron: Numberlys edition

The picturebook The Numberlys takes full advantage of the book medium by playing with the reader’s ingrained expectation to the book by using the oblong format and mixing the reading directions of the book. The book is bound in the short, left side, which would normally mean that the book is read horizontally, but already from the title page the expectation is denied as this page must be read vertically and the book must, therefore, be turned. The title is one large image that spreads from top to bottom, with the five little main characters marching across the page at the bottom. This vertical reading direction enhances the impression of the vast, oppressive world of all-pervading numbers that the characters inhabit. This is a characteristic of the picturebook that it takes advantage of the book medium, its materiality and reading conventions and plays with them to convey meaning. In relation to the hybrid AR book experience, it is noteworthy to point out how the interactivity between the reader and the medium becomes part of the way meaning is conveyed in this universe and therefore not something that is reserved for the digital component.  

In relation to the rest of the cluster work that The Numberlys compose through its many independent media versions, the AR app IMAG-N-O-TRON: Numberlys Edition is the only one that cannot stand alone. Since the app is dependent on the picturebook to activate its content, it can be characterised as an intracompositional transmedia phenomenon (Dena 2009), meaning that the ‘AR app + book’ composition is transmedial in itself, and the app is not a self-contained narrative unit. When opening the AR app, the reader is transformed into and staged as a player, collector and detective, using the digital device as a magnifying glass through which to explore and investigate the picturebook. The AR app encourages the player to scan the book for objects, which,  when located on the paper page via the camera, will turn into animated objects on the screen and be stored in the apps interface. When the objects from the book have been collected, the player can build new, fun constructions in the digital space and practice constructing letters and numbers. In this way, the app encourages the player to perform creative, educating tasks that mimic what the fictional characters do in the picturebook, thereby extending what we might call the core values or message from the picturebook to the digital environment via the AR technology.  

In relation to the aforementioned balance between the media in AR book compositions, it is noteworthy how, on the one hand, The Numberlys picturebook composes a self-contained narrative entity, while, on the other hand, the ‘AR app + book’ composition actually works independently as well. Of course, the reader/player will gain a deeper, more informed experience if she reads the story in picturebook, but it is not a prerequisite to engage with the part of the Numberlys universe that is available in the AR book composition as they offer two distinct kinds of engagement: engagement in a narrative and engagement in playful activities.

Disrupting reading and media cultural hierarchies

Moonbot Studios’ few productions were, and still are, innovative and remarkable examples of AR + book compositions and, on the whole, of experiments with multi-platform, transmedia storytelling, however, the venture did not continue and did not set a precedent for subsequent international AR books for children. If we look to the recent Danish Smart Book concept, this endeavour is targeted at older children, specifically children from 9-13 years old, and here we find yet another approach to the composition of the AR book. 

The Smart Book concept consists of a series of paperback books with individual accompanying AR apps. Currently, three books have been published in Danish and two in English are forthcoming. The series takes place in a fantasy universe of wizards and magic where the reader assumes the role as the protagonist “you”, the First Student of the Firemaster. Just like in the so-called gamebook series for children from the 80-90s Choose You Own Adventure, you navigate the book by reading short numbered chapters that present you with a choice and, depending on the choice, directs you to a new chapter. Some chapters also present the reader with challenges and puzzles, often small maths related puzzles, that must be solved to move on. These obstacles are presented in the book via simple illustrations and text but must be met and solved in the AR app that, when accomplished, will direct the reader to a new chapter in the book. The content of the puzzles is most often not related to the narrative and therefore not narratively motivated. Instead, they offer different kinds of cognitive engagement.       

An interesting aspect of this AR book composition is that the traditional way of reading a book is turned into something else via the non-linear and unknowable reading path. We might say that the book and the AR app create both a material and fictional space in which the reader moves back and forth, yet with a feeling of moving forward without knowing when the journey will end. Normally the reader of a book can see, feel and count the number of pages read and the number of pages remaining in the book. This conventional way of navigating a narrative in a book is suspended and disrupted both by integrating the AR app and through the non-linear structure resulting, paradoxically, in both a higher degree of agency assigned to the reader and a higher degree of obscurity or mystery for the reader.

In their public presentations of the Smart Book concept, the two owners of the company, Søren Jønsson and Rasmus Kolbe/Lakserytteren, associate their project and their motivation behind it with the widespread concerns about the decrease in reading among children. A Danish study of children’s reading habits conclude that children’s joy of reading literature decreases with age and that the drop happens between 5th and 6th grade (Hansen et al. 2017). This situation is the point of departure for the Smart Book project, and the owners’ stated mission is to reignite children and young people’s joy for the written word, and showing them the way to the AR books especially through Lakserytteren’s media channels, such as YouTube, TikTok and Instagram.

The Smart Book series adapts the known “choose your own adventure” form to the AR book, hereby disturbing existing notions of what it means to read literature and proposes a new way of reading. These hybrid works of fiction combine print and digital media in a mutually dependent fashion that challenges prevalent public debates about onscreen reading. In these debates a notion of a media hierarchy is formed. A hierarchy that contrasts print media as the authentic, educational way of reading with digital reading (both visual and audio) as the inferior way of reading. Transmedia AR book projects have an intrinsic potential of being able to break down this hierarchy and offer new multi-sensory, transmedia experiences that support different reading styles. 

References

Dena, C. 2009. Transmedia Practice. Theorising the Practice of Expressing a Fictional World across Distinct Media and Environments. PhD dissertation.

Hansen, S. R. et al (2017): Børns læsning 2017: En kvantitativ undersøgelse af børns læse- og medievaner i fritiden, Læremiddel.dk: Tænketanken Fremtidens Biblioteker.

Mygind, S., 2017. “A Chinese cluster: Danish-born digital comic as source for transmedia design and innovation” in Ensslin, A. et al (eds.): Small Screen Fictions. Vashon Island, Washington: Paradoxa

“Alexa, open The Messlins” says one of my kids out loud to the smart speaker setup in our living room. After eight months of research, design and production, I was about to witness my children interacting with my voice-enabled audio story for the very first time.

With a background in performance and media production, I have always been driven to find new ways to connect with an audience, whether taking the stage to do sketch comedy, giving live readings of my self-published children’s book, or producing audio-visual content for broadcast, film, and web. In 2019, I began to explore the ways in which storytelling can transform into storyliving; an undeniable shift that we have been seeing more of in the last decade. This exploration eventually led me to Ryerson University’s Media Production MA program. 

I completed my master’s degree in 2020 with a research and design focus on interactive audio narratives using a smart speaker. Interactive stories told through a smart speaker are a relatively new way to engage an audience. As part of my research, I developed a prototype for an interactive audio narrative for children titled “The Messlins,” using Amazon’s voice-enabled device, the Echo. The story used a branching narrative design. Branching narrative is a form of storytelling that allows audiences to decide the path and outcome of the story (gamasutra.com). A child can decide how the story unfolds by voicing their choice when prompted by the characters heard in the story. Later, I will describe further why the Amazon Echo was chosen as the device for the design and user-testing of the prototype.

As a parent, my world presently revolves around all things that educate and entertain my twin 6-year olds, so it made the most sense to write a story for an audience for that age group.  Through my research, I learned that children between the ages of 4 to 7 have developed the skills to use their imagination and play make-believe (Calvert and Wilson 291; Vygotsky 1967). Many of us can recall a time in our own childhoods when a parent reads a bedtime story. Even though my mother or father had the responsibility of reading the words on the page, I was always encouraged to ask questions, add sound effects or speak “directly” with the characters. The beautiful thing about an audio narrative is that it can open a world of possibility that allows the listener to paint their own imagery of the storyworld and characters in their minds. 

Since the prototype was designed to be an interactive experience, it was important to understand what motivates a young child to interact with the story. One consideration was a child’s ability to communicate verbally. Storytelling with elements of play can help a child with their communication skills (Phillips 4; Ryokai and Cassel 2). Throughout the story of “The Messlins”, children are meant to interact with the characters as well as move around their physical space as if they are in the story. The design of the characters was another important factor when considering a child’s motivation to interact with the story. The main characters are siblings who would be described as tweens, as it has been found that young children respond better to older characters that they can look up to (Miller 2014).  As a side note, the inspiration for the main characters with whom the user interacts came from the adventurous spirit of both my son and daughter, so I decided to create a young male and female character with equal status. I also wanted to pay a small tribute to my heritage as a Filipino-German-Canadian, so I created another main character with a German name while weaving in Filipino references throughout the script. 

My years of experience as a video producer taught me to always start with a script outline, a “blueprint” that ensures flow and connectivity between the beginning, middle and end. An interactive story with multiple plots and endings adds an extra layer of complexity. Before writing the audio script, I first mapped out the main plot points of the story which evolved into a branching map, which looks very much like a tree, hence the name. Unlike a linear plotline, stories that employ a branching narrative require a lot of work and pre-planning to ensure that every choice available makes sense for the story progression (Crawford 117). I personally like to get literally hands-on for a project like this, so to begin I created the branching map by hand with lots of different coloured markers and a large piece of paper; this is where being a parent with an abundance of craft material comes in handy! I then created a digital version of the branching map using a mapping software called Lucid Chart.

There is one differentiating factor that sets interactive content apart from a linear audio story; the ability to make choices. In a branching narrative, the listener can influence the direction of the story. A narrative designer must consider how a user will interact with the system and factor in as many outcomes as possible (Crawford 31). The element of play involved in an interactive experience also changes the role of the child. When a child simply listens to a linear story, this is considered a passive activity. However, in an interactive story a child is an active participant who can be viewed as either a player, user, learner, or a combination of the three (Markopoulos et al 28). Designing for an active participant requires empathising with them, in this case the child, and seeing the story through their eyes. In the case of my prototype, it would be hearing it through their ears. 

Once an idea is sparked, the hardest part of the creative design process for me is often simply starting. I was carrying out two roles in parallel; the writer of a fairytale story and the VUI designer of an interactive experience. The writer in me needed to think about how to tell an engaging story that hit all the major beats. If the experience wasn’t entertaining, I would lose the interest of the listener (Buurman 1). While my inner-VUI designer had to consider the conversation between the user and the device from beginning to end (Pearl 8).  For no particular reason, other than it being one of the most popular brands of smart speakers that I also happen to own, I chose the Amazon Echo, which uses the virtual assistant AI technology called Alexa. 

Voice-enabled apps that are built for the Echo are called “Skills.”  Early in the design process, I knew that I wanted a narrator to introduce the Skill before the story began. I also wanted it to be a voice that was different from the voice assistant (in this case, Alexa) so that the user would know that the experience had officially begun. This meant that the narration portion would have to be scripted and recorded. 

Once I had established how the Skill would launch, mapping out the branches was a relatively fluid process. I would complete one whole branch from start to finish before going onto another branch. Some branching narrative designers believe that it is best to follow through to the end with one branch otherwise one may run the risk of creating several half-complete branches (LudoNarraCon 2020). In addition, this approach insured that every path was given the same importance in the design process. To avoid the map from branching out exponentially, I used a technique referred to as foldback, which involves linking certain points of different branches together (Crawford 121). Once completed, the first version of the branching map resulted in a total of 31 scenes which included the opening narration and credits. 

With the map created, I was then able to write the script, a process which was once again twofold; the beginning and end of each scene were driven by principles of VUI design while the middle section was all about the narrative. The front and backend of each scene were treated as conversational markers (Pearl 40). This means that a scene opens with positive feedback to the user indicating that the choice they had made in the previous scene was heard and accepted. It also means that the same scene would eventually end with one of the characters prompting the user to make a choice for the next scene. I opted for the characters to suggest how to respond as it has been found that providing examples is a better user experience (Pearl 21). 

The original branching map included two distinct choices for the user at every major plot point. My original thought was that it was best to keep the answer short and simple as to make it easier for the child to repeat. I later discovered that short commands like “Study” can be problematic for a voice-driven application and it is easier for the system to recognise phrases such as “Go to the Study” (Pearl 131). Of course, we can’t forget about the middle section of each scene; this was my moment to shine as a storyteller and comedian. I was making a prototype for kids after all and humour can make an experience more enjoyable (Hall and Maeda 95) but it is important to note that it is different from adult humour (Miller 2014). Fortunately, I have a strong background in sketch comedy which makes me keenly aware of comedic timing and the art of being silly. I added little moments in the script that would hopefully elicit a giggle from the listener, whether it was for the child or their parent. 

When the university closed its doors due to the Covid-19 pandemic, I had to be proactive to ensure that the production process would continue as smoothly as possible. Unfortunately, the social distancing rule that was put in place in mid-March 2020 made in-person recording sessions impossible and I had to pivot my approach. I planned a virtual sound production that required building a temporary sound booth in my basement, delivering equipment to the homes of my voice talent, directing them over a video conference call, and uploading the audio files onto a shared drive which my sound engineer could access.

While I was able to find alternate methods to keep production going, it did not come without its challenges and setbacks. The sound booth that I was able to build at home was the perfect size but I was unable to cover it with blankets from top to bottom, which would have been ideal in order to cut out any external sounds. This meant that there were moments when noise in the home like the furnace going on, would get picked up by the microphone. Directing the voice talent remotely was also challenging. I had to listen to my voice actors through my computer speakers. There was no way of really knowing how the recordings sounded until I received the files. Fortunately, my voice actors also live together which made it easier to match their ambient sound. The recordings that my husband and I did at home also matched. Adjusting the reverb and adding ambient sounds created the illusion that the characters were in the same space. This is where having a sound engineer who can make this happen is incredibly important.

The audio clips were made up of various sound elements which included voice recordings, original and stock music, and sound effects. The voice recordings were provided by myself, my husband, and two friends. Since this was an interactive audio story for children, I wanted all the characters including the mischievous creatures, to be lively and likeable. The high energy delivery of the character voices was intended to keep a young listener engaged. As sound can affect us psychologically and emotionally (Treasure 2020), I also wanted to avoid any of the characters from sounding too menacing or scary for a child. I had a similar objective for the music selection. All the stock music in the prototype are instrumental pieces that were part of a family-friendly music library. The main theme song of the story includes original lyrics that I wrote with three characteristics in mind; simple, fun, and catchy. 

The sound effects evolved throughout production. I had originally planned for sound effects that would help support each scene such as ambient sounds of a room (eg. the sound of feet walking down a long hallway) and the characters’ movements and interactions with objects (eg. a character pulls a cork out of a bottle). Further research on VUI design prompted me to consider how sound effects could be used to provide the user a context on how to interact with the story (Pearl 62). For instance, a twinkling sound effect was later added to cue the user whenever it was time to make a choice in the branching narrative. This type of sound effect is considered an earcon because it is a piece of information that lets the user know that they are expected to respond to the system (material.io 2020). Another addition into the audio mix were various hero sounds, which indicated whenever the user had selected a choice that would advance the narrative closer to success. This could also be considered a form of feedback, which has been found to be effective in a VUI designed for children (Cieślak 2020).

Despite all the unexpected ways in which I had to shift production, overall I was pleased with the end result.  Once the produced audio clips were ready, I incorporated them into the conversational map that I created with Voiceflow. I was then able to test out the prototype with my children using our home smart speaker and observe how they interacted with the story. To witness their intuitive responses to the characters and get up to take part in some of the activities brought a huge smile to my face. As a creator, there is no better feeling than to see the content elicit a reaction from the audience, and on a new level, interact with the content. Observing my children playing with the interactive audio narrative allowed me to confirm what worked and didn’t work. The children’s interactions and reactions to the story confirmed that receiving feedback from the characters affirmed their role as the choice-makers. Music played its most important role as a cue for commands. The children knew that they could respond once the music had ended. Their attentiveness throughout the story was due in large part to their required involvement to determine the story progression (Miller 2014; Sperring and Strandvall 233).

The prototype test also revealed what needed to be improved. Using vocal commands to communicate with a smart speaker is intuitive but the moment the voice assistant cannot pick up a command, which happens more often with children, is when it becomes frustrating for a user. I also found it interesting that my children were more invested in chasing after the Messlins than to do the side activities. For them, that was the main objective of the game, and anything beyond that was an obstacle. In terms of the character design, I spent considerably more time on the development of the prince and princess characters because I had assumed a child would feel more connected to them because of their age and shared goal of finding the Messlins. So, it came as a surprise that my children were more engaged whenever a Messlin was in the scene. Although the Messlins had considerably fewer lines compared to the other characters, they were the most memorable characters because they were, as one of my kids described them, “silly and funny.”It is an exciting time for voice technology, and with huge brands like Disney and Hasbro starting to develop their own content for voice-enable devices it’s evident that we are just witnessing the beginning of a new way to connect with audiences. It is especially important these days, with everyone at home and relying more on screen time. Providing children with a new way to interact with content while also encouraging them to imagine the scene in their own minds is fulfilling to me as both a creator and a parent. In the coming months, I look forward to testing out the prototype with other users through the Ryerson University Transmedia Zone Incubator program of which I am currently a member. Further testing is an integral part of the process and I hope to gain valuable new insight on the user-experience.  Much like a branching narrative, where this creative process leads me to next is still unknown, but I’m looking forward to the adventure nonetheless. 

References

Buurman, H. A. Virtual Storytelling: Emotions for the narrator. MS thesis. University of Twente, 2007.

Calvert, Sandra L., and Barbara J. Wilson. The Handbook of Children, Media, and Development. Wiley-Blackwell, GB, 2008.

Cieślak, Katarzyna. “How to design a Voice-First game for kids? – Voice Tech Global Medium.” Medium, 24 Jan. 2020, medium.com/voice-tech-global/how-to-design-a-voice first-game-for-kids-79ec345d0c35.

Crawford, Chris. Chris Crawford on interactive storytelling. New Riders, 2012.

Hall, Erika, and John Maeda. “Conversational design.” (2018).

LudoNarraCon. “Developing Branching Narratives.” YouTube, uploaded by Fellow Traveller, 5 May 2020, https://www.youtube.com/watch?v=mgD81pQlu1o.

Material.io. “Applying sound to UI.” 20 March 2020, https://material.io/design/sound/applying-sound-to-ui.html 

Markopoulos, Panos, et al. Evaluating children’s interactive products: principles and practices for interaction designers. Elsevier, 2008.

Miller, Carolyn H. Digital Storytelling: A Creator’s Guide to Interactive Entertainment. Focal Press/Elsevier, US. 2008; 2014;.

Pearl, Cathy. Designing voice user interfaces: principles of conversational experiences. ” O’Reilly Media, Inc.”, 2016.

Phillips, Louise. “Storytelling: The seeds of children’s creativity.” Australasian Journal of Early Childhood 25.3 (2000): 1-5.

Ryokai, Kimiko, and Justine Cassell. “Computer support for children’s collaborative fantasy play and storytelling.” CSCL. 1999.

Sperring, Susanne, and Tommy Strandvall. “Viewers’ Experiences of a TV Quiz show with Integrated Interactivity.” International Journal of Human-Computer Interaction: Social Interactive Television, vol. 24, no. 2, 2008, pp. 214-235.

“The evolution of video games as a storytelling medium, and the role of narrative in modern games.” 6 Apr. 2020, www.gamasutra.com/blogs/ChrisStone/20190107/333798/The_evolution_of_video_games_as_a_storytelling_medium_and_the_role_of_narrative_in_modern_games.php.

Treasure, Julian. “The 4 ways sound affects us.” 13 Mar. 2020,  www.ted.com/talks/julian_treasure_the_4_ways_sound_affects_us?language=en

Vygotsky, Lev S. “Play and its role in the mental development of the child.” Soviet psychology 5.3 (1967): 6-18.