Real-Time MIR Algorithms for Music-Reactive Game World Generation

Abstract

In music education, an important learning objective is to acquire a sense of rhythm and melody, e.g., by tapping or singing along to a musical performance. In this context, music games are becoming increasingly popular due to their accessibility and interactivity. However, existing games are often tailored to a specific musical repertoire. In this paper, we will explore the potential of music information retrieval (MIR) algorithms for generating dynamic game worlds that are reactive to musical input in real-time, e.g., to music from a stereo system or a live performance. Such games may be used to explore musical performances in an interactive and playful way. As a case study, we therefore developed a jump-andrun game prototype that challenges players to tap along the beat of the music in their environment. Its music-reactive game world relies on past and predicted beat positions estimated by a real-time beat tracking algorithm.