XR Babel Library
In today's digital age, the internet provides us with access to an enormous collection of digital and scientific cultural heritage, available through platforms such as Europeana and Gallica. However, it is not always clear how to leverage old and new technologies to enhance the experience of this heritage. Moreover, for the general public, discovering content that is relevant or meaningful to their interests is not always a straightforward task.
Objective:
To create innovative interfaces that facilitate exploration and engagement with digital cultural and scientific heritage. By combining old cultural heritage technologies such as books with newer ones, such as semantic web, generative and NLP AI, and XR technologies, we are working towards creating more meaningful and engaging experiences for users.
Method:
I presented our team with different designs of new interfaces as a starting point for several collaborative design sessions. These interfaces integrate Holographic technologies, Computer vision algorithms, single-board computers, IR motion controllers, generative and NLP AI algorithms, semantic web, and game engine technologies into the design of a book.
I developed prototypes with increasing complexity and presented them to the team for collaborative revision. Through an iterative process, we refined the design to achieve a more functional and engaging interface.
Tools:
Raspberry Pi
Unity3D
OpenCV
TensorFlow
SPARQL
Leap Motion
Results:
While the project was terminated before completion and widespread dissemination of the research outcomes, the resulting prototypes from each iteration are presented below.
Iteration 1: Magical Book
Our initial augmented book interface design consisted of a Raspberry Pi with a camera positioned above a book page. The camera was connected to a holographic display that I built using a Pepper's Ghost pyramid on an LCD screen. I implemented a computer vision algorithm to detect the silhouette of a finger over keywords on the page. When the finger was over a keyword, a video played on the holographic display.
Iteration 2: Book that Builds Itself
The first prototype I developed gave us insight into the user experience of digitally augmenting a book. I proposed several AI-based content generation/augmentation ideas, such as using NLP to analyze sentence semantics and generate/download images of keywords, rewriting text in a different author's style, using a user's finger doodle to create photorealistic images via an algorithm, applying a style transfer technique for artistic enhancement, and using a selfie to create a 3D model for the story's characters.
I upgraded the device with a touchscreen LCD, a Leap Motion, and a more advanced holographic display (Looking Glass). Users could interact with the holographic book with the leap motion and doodle, and access the book's UI on the LCD screen. The holographic display showcased related 3D models and videos. Pages from the "Encyclopédie ou Dictionnaire raisonné des sciences, des arts et des métiers" from the Bibliothèque nationale de France are downloaded at runtime. This iteration led the team to focus on using the book as a gateway to other books on the BNF.
Iteration 3: Library that Builds Itself
I created an interface that downloads and spawns cultural and scientific heritage based on metadata curated in a CMS. This led to the idea of a self-building book collection inspired by the Library of Babel. In the first iteration of the holographic library, 9 books were spawned from metadata curated by the literature team at the CRILCQ.
Iteration 4: Semantic VR Babel Library
I used SPARQL queries to Gaellica and Europeana to develop a self-building library that continuously selects related books and digital cultural heritage. Users can explore new clusters of media by traversing the semantic graph. Initially, I created a Python console script to demonstrate the creation and navigation of such a semantic graph, which I later turned into a more user-friendly GUI application.
As a creative proposal for the library design, I suggested a fractal experience to the 3D artist team at the NAD center. The fractal design would feature pages clustering to form books, which in turn would cluster into hexagonal columns of related books. However, the project was terminated due to the COVID-19 pandemic before the 3D artists could fully implement the concept in an interactive VR experience using TouchDesigner.
