Next Generation Paper

Despite the rise of digital devices and associated media, paper does not seem to be disappearing anytime soon. In fact, most print-based documents, including books, tend to be used alongside digital devices, in educational (i.e. textbooks and laptops) and leisure (i.e. guidebooks and smartphones) contexts for example. The Next Generation Paper Project (NGP) is a £1.17 million Digital Economy programme project administered by EPRSC.  This project aims to revolutionise paper and the way books are written and read by converging these two mediums, developing new opportunities for writers and other content creators alike to collaborate and further embellish and/or enhance the meaning of text.

An example of the a-book reading paradigm

To do this, NGP is developing new approaches to interactive paper.  Despite its ordinary appearance the paper is connected to additional multimedia content (i.e. audio, video, imagery, websites), generating hybrid books referred to as ‘a-books’. The term ‘a-books’ was originally introduced by Mackay et al. (2002) to describe a laboratory note-book prototype but is employed broadly here to describe the kinds of books created by NGP. By facilitating the embedding of additional complementary digital that can to be consumed simultaneously with print (e.g. video can be played while reading a passage) , a-books can be seen to support novel forms of multimedia reading experiences co-created by writers, readers and other contributors such as graphic designers, publishers and advertisers. Co-creation is used here to refer to ‘joint creation and evolution of value with stakeholder individuals, intensified and enacted by platforms of engagement’ (Ramaswamy and Ozcan, 2014). This point will be returned as the types of interactive paper technologies being developed by NGP will now be introduced.

These include the use of visual recognition technology—referred to as 2G paper—or embedded touch and light sensitive sensors in the paper—named 3G paper. 2G paper does not require specialised equipment or printed alterations to the page; 2G books appear as normal paper books, but supplementary digital content can be viewed by taking a picture of its pages using a smartphone with the specialised app installed, called Cornwall a-book.

Through 2G paper specifically, any publication can have digital media linked to its pages without any changes to its appearance or interaction. The a-book also utilises a range of digital media (i.e. websites, video, audio, imagery). This is unlike previous augmented books that have tended to focus on using audio and/or video alone. For example, Listen Reader is an augmented book that plays additional related audio when a page is turned (Back et al., 2001) and Interactive Newsprint is a newspaper with touch sensitive buttons that play associated audio when pressed (Frohlich et al., 2017). Moreover, the Cornwall a-book app—that pairs with and allows multimedia content from 2G and 3G paper prototypes to be played on a smartphone—has a digital recording feature enabling users to capture personal video, audio, imagery or found weblinks, and add them to the app or certain book pages. When digital media is recorded and linked to a page, it becomes available through the app when a specific paper book page is selected by taking a picture or searching for it—page numbers can be spoken, typed manually or selected by pressing the present display image on the animated slideshow too.

An example of 2G paper interaction design

Content shown in green indicates the digital media present for that page and the ‘+’ symbol in the top right hand corner of the Cornwall a-book app allows the selection and recording of personal digital media. Pages can also be digitally bookmarked as well as physically. The bookmark icon can be pressed when a page’s digital content is displayed on the app to bookmark the page, and a shortcut to the page will appear under the bookmark icon on the app homepage. This merges physical book interaction associations, such as annotating and bookmarking with corresponding digital functions, converging print and digital features into a platform for enhanced reading engagement and co-creation. This ‘zero delta’ approach allows the technology to be available for immediate public consumption once final developments are complete.

The 3G paper books contain wiring in the spine and touch sensitive printed buttons in the cover depicting different media content (e.g. audio, video, websites); when pressed, these buttons cause related content digital content for the spread open to be played on a paired smartphone with the custom Cornwall a-book app installed.

This approach is not yet ready for public distribution due to the novelty and complexity of the technology and its construction charges. However, its development is pushing and exploring the boundaries of current augmented paper technologies and, with this, new opportunities for supporting intuitive multimedia reading processes in the future.

The Cornwall a-book app’s homepage with features annotated

This project is initially exploring its usage within travel and tourism by prototyping a travel guide to Cornwall published by Bradt Travel Guides, one of the world’s leading publishers of travel guides for tourists. It is expected that this technology will also be easily transferable to other areas such as Education, Health Care and Transport to name a few—basically any sector that uses paper alongside digital media. Indeed, NGP is comprised of a large interdisciplinary research team with expertise in user research, design, hardware and software technology, and business enterprise, that each lead bespoke work packages. Progress so far, has included the completion of an ethnography study with twenty-two travellers that investigated their media practices when preparing, having and reflecting on holidays, and the creation of a fully working 2G a-book and 3G a-book section. Resulting ethnography findings directed the a-book’s design, using a ‘research through design’ approach. The 2G a-book is now undergoing evaluation with prospective users through a series of reading evaluation studies employing a ‘technology elicitation’ method—participants are encouraged to play with and explore the prototype both alone and while talking with NGP researchers to facilitate deep discussions about its possible future applications in a travelling context, such as how it might change current practices and provide a platform for new interactions to emerge.

Importantly, NGP remains unique to previous augmented book concepts because it maintains the original format of the paper book. Many prior augmented paper prototypes either require additional markings to be added and/or need specialist equipment for operation. Examples include the use of QR codes commonly seen in magazines and advertising posters, and other visual markers such as barcodes. For instance, Books with Voices is an augmented book with barcodes linked to video clips that requires a PDA (personal digital assistant) device to scan the codes and view the footage (Klemmer et al., 2003).

As previously mentioned, the introduction of digital text and media seems to have created a dichotomy of reading experiences between digital and printed texts where these two mediums are for the most part consumed separately. Relatedly, previous research investigating expert reader habits describes online platforms as mostly facilitating discontinuous fragmented reading, whereas printed media seems to support continuous immersive reading, and, at times, imaginary if fictional text is present (Hillesund, 2010). Considering this, a-books could be said to allow for semi-immersive imaginary reading, in the context of travel guides at least, as its digital dimension expectedly both breaks and enhances in-depth reading and imaginings of place. The digital content requires some activation, temporarily removing the reader from the activity, but it’s playback can also accentuate the text’s meaning, making the overall process a semi-immersive imaginary reading experience. For example, a more holistic atmosphere of an area being discussed might be facilitated by playing a soundscape while reading the text, or additional weblinks might be activated to explore content in further detail. Correspondingly, the example image of the a-book reading paradigmFig. 1 shows a video interview on the smartphone with a member of the National Lobster Hatchery described in the text, talking about how lobsters are raised. Prior research exploring augmented paper has also suggested that readers can and do practice ‘synchronized reading-and-listening’ while reading (Frohlich et al., 2017).

Additionally, personal recordings of experiences abroad, such as through video, image or audio, might be captured, added to pages using the app during a holiday and played back to facilitate reminiscing when the user has returned. Useful web content, including links to immersive images, video and sound of a locality, can also be digitally connected to relevant book pages by readers and played back to enable personalised semi-immersive reading. This supports the co-creation of alternative reading interactions as it allows users, employing presented print and digital media as a starting point, to choose what multimedia content they want to facilitate their reading and envisioning of places visited or to be experienced. Lastly, as the main book text provides in-depth elaborations of entertainment options and facilities at different sites, the travel book remains the primary platform for engagement and reading and can be being read and interacted with as an ordinary book if so desired.

The Cornwall a-book app display when a page is selected

Kirsty Fergusson, the official travel writer for the Cornwall ‘Slow Travel Series’ travel guide published by Bradt Travel Guides, was commissioned by NGP to collect the additional digital audio/visual media while collating research to update the written content for the latest edition. For example, some of the items recorded include audio of Kirsty’s voice either discussing her experiences of scenic areas and places of interest in-situ, or interviewing residents and local business owners, as well as imagery of them and/or their location (e.g. farm, café, garden). Although multimedia content delivered by Kirsty required some editing (e.g. lighting adjustments applied, noise disturbances removed) and preparation (e.g. conversion into compatible a-book file formats) for inclusion in the a-book, care was taken by the responsible design researcher, Dr Emily Corrigan-Kavanagh, (notably playing the role of the designer in this instance) to preserve its authenticity. For example, imagery that contained overexposed aspects or dull colours, usually caused by weather conditions, was only slightly altered, to preserve and reflect the authenticity of their origin, namely content captured by a travel writer, busy and on the move, experiencing Cornwall through an array of different circumstances and seasonal changes. The media was therefore prepared and presented in the a-book by the designer with the intention of offering an honest depiction of the writer’s experiences, both visually and aurally. New classifications of digital media, such as audiophotos, audiophoto narratives and photo narratives, co-created by the writer and designer during this process, can be used to support semi-immersive imaginary reading experiences in a-books.

Audiophotos were first introduced by audiophotography and refer to images that are recorded with sound (Frohlich, 2004, p. 3). In the a-book, they consist of still shots of significant people, objects and/or places with accompanying audio of corresponding people being interviewed, or soundscapes or solo narrations of featured photos. These were carefully assembled by the designer, which included making judgements on what images and audio submitted by the writer to combine. Audiophoto narratives are similar to audiophotos, but they include panning camera shots of static images as well as related sound overlay. In these instances, as well as deciding what content to use and what order they should appear, the designer needed to compose the composition of each image initially appearing in view so that the areas gradually revealed, as the camera panned out, would accentuate the meaning of the audio content. An exemplary audiophoto narrative included an oyster farmer talking about his career path with accompanying photos of him and his father working on the farm to be viewed alongside a written section on ‘Oysters and Oyster Catchers’. Lastly, photo narratives simply refer to image slideshows, where more than one image is activated when the associated page has its picture taken using the Cornwall a-book app.  To create these, the designer needed to judge what imagery best related to each other and compose these as a sequence to better portray visual aspects of descriptions in the text. The affordances of the a-book, such as the facilitation of semi-immersive imaginary reading experiences that can be co-created by designers, readers and writers among others collectively, presents exciting developments for the future of writing and reading more generally. The written word can now be linked to an array of related multimedia content without compromising the intuitive and fluid interaction with a physical book, while adding another layer of authenticity through its digital embellishment. Moreover, designers can creatively interpret this digital media to create photo narratives, audiophotos and audiophoto narratives, while readers can supplement this with their own gathered and recorded digital links (i.e. audio, video, imagery, websites).

An example of 3G paper interaction design

Recent developments of NGP make the success and accessibility of Next Generation Paper already apparent. A prototype of the 2G travel guide has been released February of this year for public purchase in participating stores and a partial version of the accompanying Cornwall a-book app (does not have personalisation functionality) is also  free to download from Google Play Store, allowing access to the additional complementary multimedia content. It appears that we are already living in the age of Next Generation Paper.

References

Back, M., Cohen, J. and Gold, R. (2001) ‘Listen reader: an electronically augmented paper-based book’, in SIGCHI’01. Seattle, Washington: ACM, pp. 23–29.

Frohlich, D. et al. (2017) ‘Designing interactive newsprint’, International Journal of Human Computer Studies, 104(2017), pp. 36–49.

Frohlich, D. (2004) Audiophotography: Bringing photos to life with sounds. 3rd edn. New York: Springer Science & Business Media.

Hillesund, T. (2010) ‘Digital Reading Spaces: How Expert Readers Handle Books, the Web and Electronic Paper’, First Monday Peer-Reviewed Journal of the Internet, 15(4–5). Available at: http://firstmonday.org/htbin/cgiwrap/bin/ojs/index.php/fm/rt/printerFriendly/2762/2504.

Mackay, W. E. et al. (2002) ‘The missing link: augmenting biology laboratory notebooks’, in UIST’02. Paris: ACM, pp. 41–50.

Ramaswamy, V. and Ozcan, K. (2014) The co-creation paradigm. Stanford: Stanford University Press.

Acknowledgements:

Special thanks are given to travel writer Kirsty Fergusson for collecting additional digital material when writing the Cornwall Guide, and colleagues at Bradt Travel Guides for their contribution to this work, including Adrian Phillips, Rachel Fielding, Anna Moores, Carys Homer and Ian Spick. Team members and colleagues on NGP, Prof David Frohlich, Prof Caroline Scarles, Dr George Revill, Dr Jan van Duppen, Dr Haiyue Yuan, Prof Miroslaw Bober, Dr Radu Sporea, Dr Brice Le Borgne, Ms Megan Beynon and Prof Alan Brown are also sincerely thanked for their hard work and contribution to the project so far.