It can be scary to look back sometimes, almost like a fear of heights. It’s six years ago (six years!) that I first started thinking about the project that was just released in the iOS App Store. A lot has happened in those years. All of it helped my meditative reading app ‘Lotus’ take its ultimate shape. But it still makes me a little dizzy when I think about all the time that passed…

I was frustrated, all those years ago. I didn’t like the e-book of my last novel; it was a boring carbon copy of the paperback with some typesetting errors to boot. As a video games player (and sometimes writer), I knew that the phone on which I was trying to read my e-book could do so much more. Mix in a touch of naiveté, and my frustration turned into hope: maybe I could come up with a better way to present my books on a touchscreen device?

My quest began: first there was fundraising (I managed to get two subsidies from Dutch cultural funds). In the first subsidy application (and the only one I got without a hitch), I presented three goals: 1. re-imagine an existing serial story I wrote (based on my years as a games journalist traveling to games trade shows) in a new, digital-native form, 2. investigate ways to present text differently, experiment with the user interface and graphical elements, and 3. maybe use some sound.

To explore and realise these goals, I wanted to work together with experts in various fields (code, design, audio) and use my learnings to shape the story itself, like an iterative process, ultimately making everything come together in one logical whole.

That text was actually the toughest. As I built my team, prototyped, tested, iterated, the story was the one part I kept pushing back. I found that managing a complex project with many involved parties was a poor bed-fellow with the old writing profession. I never seemed to have the time (or patience) to sit down and write. And when I did get around to it from time to time, progress was tedious. I felt the pressure of my teammates (who were all so talented and productive), the subsidy funds, my own self-worth, and instead of pushing me forward, it held me back.

It was also easy to get distracted. Some of the work we were doing turned out to be super interesting, like:

– breaking text up into neat portions instead of filling the screen;
– using background colour and audio as a sort of ‘sense scape’;
– formatting the text in such a way the aforementioned sense scape could unfold in sync with the text;
– a zooming-out ‘outline view’.

The first of those seemed to strick a nerve with nearly everyone who I presented our prototype to. People said they wanted to read everything this way. It was very tempting to think: let’s build a business out of this. And to think: screw that novella!

So I decided to launch Immer, a start-up looking to bring the ideas from Lotus to a broader set of books, before Lotus was even finished. For a while there, it didn’t look like it would ever surface at all.

But when we decided that Lotus would work fine as a pilot showcasing some of our ideas, I realised I would have to bite the bullet and write my long-postponed book after all. It was ‘almost’ done, after all, and not only would it show people how this new way of reading would work, it would also show potential investors in the business that we were able to deliver the goods. Which I was, right?

Late in 2016, the Dutch Foundation for Literature offered me a short residency in the translator’s house in Amsterdam, and I used it to work on Lotus almost non-stop. I think I really was almost done. But almost directly after that, I found out I had kidney cancer… They cut the whole thing out and I’m fine now. But it delayed the writing by almost a whole year.

Early last year I could no longer avoid it. I had to finish Lotus. It wasn’t pretty. Over the years I’d kept adding new elements to the story to make it interesting for myself to write, and the whole thing just got too complex. It had no focus. I wouldn’t have been able to tell you what it was about.

It was painful to continue, costing a huge amount of energy. It felt like something I should give up on. Nobody was waiting for this and I didn’t even like it that much myself. Still, I kept going. I’d never gotten the whole story on paper from A to Z, and now, with all my might, I finally did it. Well, I finished something.

We organised a big beta test in the summer of 2018. People liked the app, but the response to the story was lukewarm. Just as I’d feared, although a part of you always hopes that people will somehow suddenly love something you struggled with.

I put the text aside and focused on something different first: another pilot with an existing book by Dutch author Arnon Grunberg. It was a success, the app worked great and people seemed to love it. That got me hyped. I started warming up to Lotus, too. It helped immensely that there was an actual manuscript to bounce off of this time. I wasn’t adding ideas to ideas, I was adding ideas to actual work.

When I finally sat down to rewrite it in March, I did something I’d never done before: print the manuscript, lay it down besides my laptop and type the whole thing again. I was very critical of my own work. Threw out at least half of it. Added a little bit of just the right thing. For the first time since the instigation of the project six years ago, the story part of Lotus started to feel like a good idea.

I’d finally found my focus. Suddenly, I could see how the entire six-year-journey had distilled itself in the narrative: the traveling stories where I’d started, but also the experience trying to develop and sell the reading app, and to top it all off the cancer too. All of it was in there in the end.

There was one other problematic element: sound. From the start, testers seemed to be doubtful about the idea of listening to audio on their headphones while reading. It was either too minimalistic (not interesting) or too full (distracting from the text). On our side, we never seemed able to find the right shape. But my enthusiasm about the text finally coming together seemed infectious and the audio folks at SonicPicnic dove into the project with renewed energy. They ultimately created a really great dynamic soundscape, built in layers that swell up and ebb away, really adding something to the story.

In the end, we even were able to add an English translation by Jenny Watson over the summer. I’m really proud of being able to release a bilingual story app, innovative reading software with a text that took a lot to get right but is finally ready. I can’t wait to see what the world thinks!

Lotus can be downloaded for iOS now: https://apps.apple.com/app/read-lotus/id1441103711

Despite the rise of digital devices and associated media, paper does not seem to be disappearing anytime soon. In fact, most print-based documents, including books, tend to be used alongside digital devices, in educational (i.e. textbooks and laptops) and leisure (i.e. guidebooks and smartphones) contexts for example. The Next Generation Paper Project (NGP) is a £1.17 million Digital Economy programme project administered by EPRSC.  This project aims to revolutionise paper and the way books are written and read by converging these two mediums, developing new opportunities for writers and other content creators alike to collaborate and further embellish and/or enhance the meaning of text.

An example of the a-book reading paradigm

To do this, NGP is developing new approaches to interactive paper.  Despite its ordinary appearance the paper is connected to additional multimedia content (i.e. audio, video, imagery, websites), generating hybrid books referred to as ‘a-books’. The term ‘a-books’ was originally introduced by Mackay et al. (2002) to describe a laboratory note-book prototype but is employed broadly here to describe the kinds of books created by NGP. By facilitating the embedding of additional complementary digital that can to be consumed simultaneously with print (e.g. video can be played while reading a passage) , a-books can be seen to support novel forms of multimedia reading experiences co-created by writers, readers and other contributors such as graphic designers, publishers and advertisers. Co-creation is used here to refer to ‘joint creation and evolution of value with stakeholder individuals, intensified and enacted by platforms of engagement’ (Ramaswamy and Ozcan, 2014). This point will be returned as the types of interactive paper technologies being developed by NGP will now be introduced.

These include the use of visual recognition technology—referred to as 2G paper—or embedded touch and light sensitive sensors in the paper—named 3G paper. 2G paper does not require specialised equipment or printed alterations to the page; 2G books appear as normal paper books, but supplementary digital content can be viewed by taking a picture of its pages using a smartphone with the specialised app installed, called Cornwall a-book.

Through 2G paper specifically, any publication can have digital media linked to its pages without any changes to its appearance or interaction. The a-book also utilises a range of digital media (i.e. websites, video, audio, imagery). This is unlike previous augmented books that have tended to focus on using audio and/or video alone. For example, Listen Reader is an augmented book that plays additional related audio when a page is turned (Back et al., 2001) and Interactive Newsprint is a newspaper with touch sensitive buttons that play associated audio when pressed (Frohlich et al., 2017). Moreover, the Cornwall a-book app—that pairs with and allows multimedia content from 2G and 3G paper prototypes to be played on a smartphone—has a digital recording feature enabling users to capture personal video, audio, imagery or found weblinks, and add them to the app or certain book pages. When digital media is recorded and linked to a page, it becomes available through the app when a specific paper book page is selected by taking a picture or searching for it—page numbers can be spoken, typed manually or selected by pressing the present display image on the animated slideshow too.

An example of 2G paper interaction design

Content shown in green indicates the digital media present for that page and the ‘+’ symbol in the top right hand corner of the Cornwall a-book app allows the selection and recording of personal digital media. Pages can also be digitally bookmarked as well as physically. The bookmark icon can be pressed when a page’s digital content is displayed on the app to bookmark the page, and a shortcut to the page will appear under the bookmark icon on the app homepage. This merges physical book interaction associations, such as annotating and bookmarking with corresponding digital functions, converging print and digital features into a platform for enhanced reading engagement and co-creation. This ‘zero delta’ approach allows the technology to be available for immediate public consumption once final developments are complete.

The 3G paper books contain wiring in the spine and touch sensitive printed buttons in the cover depicting different media content (e.g. audio, video, websites); when pressed, these buttons cause related content digital content for the spread open to be played on a paired smartphone with the custom Cornwall a-book app installed.

This approach is not yet ready for public distribution due to the novelty and complexity of the technology and its construction charges. However, its development is pushing and exploring the boundaries of current augmented paper technologies and, with this, new opportunities for supporting intuitive multimedia reading processes in the future.

The Cornwall a-book app’s homepage with features annotated

This project is initially exploring its usage within travel and tourism by prototyping a travel guide to Cornwall published by Bradt Travel Guides, one of the world’s leading publishers of travel guides for tourists. It is expected that this technology will also be easily transferable to other areas such as Education, Health Care and Transport to name a few—basically any sector that uses paper alongside digital media. Indeed, NGP is comprised of a large interdisciplinary research team with expertise in user research, design, hardware and software technology, and business enterprise, that each lead bespoke work packages. Progress so far, has included the completion of an ethnography study with twenty-two travellers that investigated their media practices when preparing, having and reflecting on holidays, and the creation of a fully working 2G a-book and 3G a-book section. Resulting ethnography findings directed the a-book’s design, using a ‘research through design’ approach. The 2G a-book is now undergoing evaluation with prospective users through a series of reading evaluation studies employing a ‘technology elicitation’ method—participants are encouraged to play with and explore the prototype both alone and while talking with NGP researchers to facilitate deep discussions about its possible future applications in a travelling context, such as how it might change current practices and provide a platform for new interactions to emerge.

Importantly, NGP remains unique to previous augmented book concepts because it maintains the original format of the paper book. Many prior augmented paper prototypes either require additional markings to be added and/or need specialist equipment for operation. Examples include the use of QR codes commonly seen in magazines and advertising posters, and other visual markers such as barcodes. For instance, Books with Voices is an augmented book with barcodes linked to video clips that requires a PDA (personal digital assistant) device to scan the codes and view the footage (Klemmer et al., 2003).

As previously mentioned, the introduction of digital text and media seems to have created a dichotomy of reading experiences between digital and printed texts where these two mediums are for the most part consumed separately. Relatedly, previous research investigating expert reader habits describes online platforms as mostly facilitating discontinuous fragmented reading, whereas printed media seems to support continuous immersive reading, and, at times, imaginary if fictional text is present (Hillesund, 2010). Considering this, a-books could be said to allow for semi-immersive imaginary reading, in the context of travel guides at least, as its digital dimension expectedly both breaks and enhances in-depth reading and imaginings of place. The digital content requires some activation, temporarily removing the reader from the activity, but it’s playback can also accentuate the text’s meaning, making the overall process a semi-immersive imaginary reading experience. For example, a more holistic atmosphere of an area being discussed might be facilitated by playing a soundscape while reading the text, or additional weblinks might be activated to explore content in further detail. Correspondingly, the example image of the a-book reading paradigmFig. 1 shows a video interview on the smartphone with a member of the National Lobster Hatchery described in the text, talking about how lobsters are raised. Prior research exploring augmented paper has also suggested that readers can and do practice ‘synchronized reading-and-listening’ while reading (Frohlich et al., 2017).

Additionally, personal recordings of experiences abroad, such as through video, image or audio, might be captured, added to pages using the app during a holiday and played back to facilitate reminiscing when the user has returned. Useful web content, including links to immersive images, video and sound of a locality, can also be digitally connected to relevant book pages by readers and played back to enable personalised semi-immersive reading. This supports the co-creation of alternative reading interactions as it allows users, employing presented print and digital media as a starting point, to choose what multimedia content they want to facilitate their reading and envisioning of places visited or to be experienced. Lastly, as the main book text provides in-depth elaborations of entertainment options and facilities at different sites, the travel book remains the primary platform for engagement and reading and can be being read and interacted with as an ordinary book if so desired.

The Cornwall a-book app display when a page is selected

Kirsty Fergusson, the official travel writer for the Cornwall ‘Slow Travel Series’ travel guide published by Bradt Travel Guides, was commissioned by NGP to collect the additional digital audio/visual media while collating research to update the written content for the latest edition. For example, some of the items recorded include audio of Kirsty’s voice either discussing her experiences of scenic areas and places of interest in-situ, or interviewing residents and local business owners, as well as imagery of them and/or their location (e.g. farm, café, garden). Although multimedia content delivered by Kirsty required some editing (e.g. lighting adjustments applied, noise disturbances removed) and preparation (e.g. conversion into compatible a-book file formats) for inclusion in the a-book, care was taken by the responsible design researcher, Dr Emily Corrigan-Kavanagh, (notably playing the role of the designer in this instance) to preserve its authenticity. For example, imagery that contained overexposed aspects or dull colours, usually caused by weather conditions, was only slightly altered, to preserve and reflect the authenticity of their origin, namely content captured by a travel writer, busy and on the move, experiencing Cornwall through an array of different circumstances and seasonal changes. The media was therefore prepared and presented in the a-book by the designer with the intention of offering an honest depiction of the writer’s experiences, both visually and aurally. New classifications of digital media, such as audiophotos, audiophoto narratives and photo narratives, co-created by the writer and designer during this process, can be used to support semi-immersive imaginary reading experiences in a-books.

Audiophotos were first introduced by audiophotography and refer to images that are recorded with sound (Frohlich, 2004, p. 3). In the a-book, they consist of still shots of significant people, objects and/or places with accompanying audio of corresponding people being interviewed, or soundscapes or solo narrations of featured photos. These were carefully assembled by the designer, which included making judgements on what images and audio submitted by the writer to combine. Audiophoto narratives are similar to audiophotos, but they include panning camera shots of static images as well as related sound overlay. In these instances, as well as deciding what content to use and what order they should appear, the designer needed to compose the composition of each image initially appearing in view so that the areas gradually revealed, as the camera panned out, would accentuate the meaning of the audio content. An exemplary audiophoto narrative included an oyster farmer talking about his career path with accompanying photos of him and his father working on the farm to be viewed alongside a written section on ‘Oysters and Oyster Catchers’. Lastly, photo narratives simply refer to image slideshows, where more than one image is activated when the associated page has its picture taken using the Cornwall a-book app.  To create these, the designer needed to judge what imagery best related to each other and compose these as a sequence to better portray visual aspects of descriptions in the text. The affordances of the a-book, such as the facilitation of semi-immersive imaginary reading experiences that can be co-created by designers, readers and writers among others collectively, presents exciting developments for the future of writing and reading more generally. The written word can now be linked to an array of related multimedia content without compromising the intuitive and fluid interaction with a physical book, while adding another layer of authenticity through its digital embellishment. Moreover, designers can creatively interpret this digital media to create photo narratives, audiophotos and audiophoto narratives, while readers can supplement this with their own gathered and recorded digital links (i.e. audio, video, imagery, websites).

An example of 3G paper interaction design

Recent developments of NGP make the success and accessibility of Next Generation Paper already apparent. A prototype of the 2G travel guide has been released February of this year for public purchase in participating stores and a partial version of the accompanying Cornwall a-book app (does not have personalisation functionality) is also  free to download from Google Play Store, allowing access to the additional complementary multimedia content. It appears that we are already living in the age of Next Generation Paper.

References

Back, M., Cohen, J. and Gold, R. (2001) ‘Listen reader: an electronically augmented paper-based book’, in SIGCHI’01. Seattle, Washington: ACM, pp. 23–29.

Frohlich, D. et al. (2017) ‘Designing interactive newsprint’, International Journal of Human Computer Studies, 104(2017), pp. 36–49.

Frohlich, D. (2004) Audiophotography: Bringing photos to life with sounds. 3rd edn. New York: Springer Science & Business Media.

Hillesund, T. (2010) ‘Digital Reading Spaces: How Expert Readers Handle Books, the Web and Electronic Paper’, First Monday Peer-Reviewed Journal of the Internet, 15(4–5). Available at: http://firstmonday.org/htbin/cgiwrap/bin/ojs/index.php/fm/rt/printerFriendly/2762/2504.

Mackay, W. E. et al. (2002) ‘The missing link: augmenting biology laboratory notebooks’, in UIST’02. Paris: ACM, pp. 41–50.

Ramaswamy, V. and Ozcan, K. (2014) The co-creation paradigm. Stanford: Stanford University Press.

Acknowledgements:

Special thanks are given to travel writer Kirsty Fergusson for collecting additional digital material when writing the Cornwall Guide, and colleagues at Bradt Travel Guides for their contribution to this work, including Adrian Phillips, Rachel Fielding, Anna Moores, Carys Homer and Ian Spick. Team members and colleagues on NGP, Prof David Frohlich, Prof Caroline Scarles, Dr George Revill, Dr Jan van Duppen, Dr Haiyue Yuan, Prof Miroslaw Bober, Dr Radu Sporea, Dr Brice Le Borgne, Ms Megan Beynon and Prof Alan Brown are also sincerely thanked for their hard work and contribution to the project so far.