Participatory Opera

The demand for public participation in storytelling is increasing across various mediums such as theater, fanfiction, and choose-your-own-adventure movies and video games. The use of natural movements and sounds in virtual environments can enhance the feeling of immersion and make the interaction more intuitive. Allowing the audience to impact the story through these embodied interactions can create thematically compelling narratives that engage the public in conversations around issues such as tolerance, gender identity, bullying, disability, and mental health.

Objective:

To develop a one-act interactive opera that includes an online interactive environment that engages the audience in the collaborative creation of content and social interactions before and after the performance.

Method:

I collaborated with a diverse team of experts in gamification, arts and technology, virtual communities, 3D art, literature, dance, social sciences, music, and game development, as well as a choreographer, a writer, and a designer.
I created an algorithm to detect audience singing, which was used to interact with virtual flora and fauna, show support for the main character at the opera's climax, and affect the story's ending.
I developed an augmented reality and an augmented virtuality opera that seamlessly integrated the libretto, soundtrack, stage, characters, animations, and audience interaction.
I also implemented a gamified online multiplayer environment where users could chat, direct the choreography of virtual characters, collaborate on a community song, and synchronize their breath with others.
I presented workshops, participated in live performances, and contributed to the authoring of research articles and book chapters.

Tools:

Unity3D

Google Cardboard

Blender

iClone

Results:

XR Opera

In the initial exploratory stage, I created an augmented reality (AR) app that let the audience choose and embody characters on stage. To increase engagement, the audience could sing along to support the main character. If the audience sang loudly enough, the virtual wind would support the character's wings during the climax of the opera.

Imposing the user's face under the character's hat. — AR test featuring the audience's face imposed under an opera singer's hat.

Imposing virtual elements over stage — AR test showcasing virtual elements superimposed on a live scene background.

Virtual Opera elements — Virtual elements displayed during the opera's climax.

To expand the opera's reach, I developed a VR version using the audience microphone to bring live elements into the virtual opera. To enable low-power mobile device use, I pre-recorded a 360 video of the opera and added interactive elements with a server authoritative networking system. This resulted in an immersive and accessible 360 VR collaborative experience.

screenshot of 360° immersive video of the opera with interactive elements — Screenshot of 360° immersive video of the opera featuring interactive elements in Unity3D.

multiplayer demo of 360 vr collaborative opera — Multiplayer demo test of collaborative 360 VR opera.

Co-Creation Environment

I developed a multiplayer online co-creation environment to engage the opera audience in collaborative content creation and social interactions before and after performances. Participants could chat and meet at different stations offering dance, literature, and music-based interactions. For instance, the music station featured flora that extracted notes from user-generated sound and played back a piano melody. By colliding with different floras, players could orchestrate co-created songs. This interactive experience encouraged collaboration and creativity among the audience.

Tomat Navigator in hands of a visually impaired participant

Research Dissemination

Edwards, G., Kiss, J., Morales, E., McLaren, C., Lacasse, S., Bourbeau, M. L. , Falcon, J. N. , & Guimond, J. P. (2019). Designing a Participatory and Interactive Opera. In (Ed.), Interactive Multimedia - Multimedia Production and Digital Storytelling. IntechOpen. https://doi.org/10.5772/intechopen.82811

Edwards, G., Kiss, J., & Nino, J. (2019) “Tales From the Humanitat” A Multiplayer Online Co-Creation Environment. In 14th International Symposium on Computer Music Multidisciplinary Research (p. 977). https://hal.archives-ouvertes.fr/hal-02382500/file/Proceedings_CMMR2019.pdf#page=983

Edwards, G., Kiss, J., Morales, E., (2017) “Pinnacle: An Interactive Opera Dealing With Acceptance of Differences”, live opera presentation at the Symposia For Qualitative Health Research Conference, Quebec City, Quebec, Canada. https://journals.sagepub.com/doi/full/10.1177/1609406917748704#_i41

Edwards, G., Kiss, J., & Nino, J., (2017) “Pinnacle- an interactive opera”, live opera presentation at the 11th Conference of the European Society for Literature, Science and the Arts. University of Basel, Switzerland.