Sympathetic Orchestra: A Responsive Virtual Orchestra for Embodied Interpretive Practice in Conducting
‍Bob Tianqi Wei, Shm Almeda, Shayne Shen, Ethan Tam, and Dor Abrahamson.
CHI 2026, Accepted

ABSTRACT
Learning to conduct effectively depends on a tight loop between perception and action, in which students’ hand gestures are continuously shaped by audible and situational feedback. However, students often rehearse with static recordings that cannot response to their physical actions or interpretive intentions. This breaks the feedback needed to develop tacit musical knowledge, such as how timing, cueing, balance, and phrasing are communicated through embodied motion. We present Sympathetic Orchestra, a real time interactive system that maps conducting gestures to dynamically responsive orchestral playback and a stage oriented visual layout, making the consequences of actions immediately perceivable. The system reveals higher engagement and distinct interaction patterns that show how responsive feedback supports the development of tacit musical understanding through embodied practice.
‍

Demonstration of Sympathetic Orchestra: An Interactive Conducting Education System for Responsive, Tacit Skill Development
Bob Tianqi Wei, Shm Almeda, Ethan Tam, and Dor Abrahamson.
UIST 2024
https://doi.org/10.1145/3672539.3686783
PDF
‍
ABSTRACT
Students learning musical conducting often practice along to static recordings, which do not provide real-time feedback similar to that of a live orchestra during rehearsals. Novice conductors need better solutions for practicing with feedback that mimics the experience of conducting a live orchestra. We can leverage emergent multimodal and spatial interaction technologies to support a “virtual orchestra” practice experience that allows students to develop tacit, live-practice knowledge. Through formative interviews with conducting experts and students, we designed and developed a dynamic, multimodal interaction system that targets key goals held by students developing their orchestral conducting skills, and that traditional practicing methods lack support for. Sympathetic Orchestra is an interactive virtual orchestra system that uses Google AI edge-powered Computer Vision hand and face tracking on webcam data to responsively interact with dynamic audio music playback and develop tacit practicing experiences.