Long-Term, Store-Front Robotics: Interactive Music for Robotic Arm, Caxixi and Frame Drums

The integration of interactive music systems and robotic musicianship into commercial retail environment is a largely under-explored area [1]. More commonly, such systems have been confined to laboratory settings or specific art installations and performances, where the controlled conditions differ vastly from the unpredictability and diversity of the real world [2]. Robotic musicians however have extensive potential in the real-world to engage with a range of audiences.

This project centers around a three-week-long interactive in-store installation featuring a UR3 industrial cobot (collaborative robot) arm, custom-built frame drums, and an adaptive music generation system. The store front was in the city center of Sydney, Australia at AESOP, Pitt St. The installation was designed to interact with the ambient soundscape of the store, creating a dynamic and engaging musical experience for patrons. This paper demonstrates the technical feasibility of such an integration and also explores the artistic and commercial implications of embedding interactive robotic musicianship within a retail environment.

This project’s core contributions include the development of a robotic arm integrated with actuated frame drums to forge a distinctive musical instrument, deploying interactive music to elevate the commercial store’s ambiance, and showcasing the robotic system’s resilience through its uninterrupted operation for a three-week span. Crucially, the project navigated the retail environment’s unique challenges by prioritizing system reliability to avoid any disruptions or negative impacts on the customer experience.

We also delve into the critical outcomes and reflections arising from the deployment of our interactive music system in a commercial context. Central to our discourse are the pivotal themes of system reliability, the nuanced dynamics of human-robot interaction within the retail sphere, and the implications of robotic agency as perceived by store staff and patrons. We finally explore the strategic design choices that underpinned the system’s integration into the retail environment, ensuring its alignment with the store’s brand identity while enhancing the customer experience.

For this system, we chose to focus on the frame drums, selected for their versatility and adaptability to diverse musical styles, which aligns with the objective of creating an interactive music system that resonates with a broad audience. Their wide range of sizes and tunings, coupled with a potential uniform hitting mechanism, enabled the integration of these instruments into the robotic setup without necessitating significant alterations for each drum size.

The physical design of the frame drum actuation system incorporates motors equipped with dual beaters, enabling hits on both the rim and the interior surface of the drum (see Figure 3). This design choice expands the tonal possibilities while also clearing displaying movement to the audience, through longer than required beater arms. Cabling for the actuation mechanism is routed through the beater holder, to maintain a clean appearance. The drums are positioned at a slant, primarily for aesthetic reasons, to draw attention to the central robotic arm and create a visually engaging setup that complements the store’s ambiance.

The actuation mechanism is powered by six Dynamixel XC430 W240 T motors, controlled through a U2D2 motor controller. This setup facilitates precise control over the drum strikes, with each motor’s range of motion pre-configured and stored within the motor’s firmware. The entire system is managed via a Linux-based control platform, utilizing Python for motor control and system integration. This approach allows for real-time adjustments and synchronization with the central robotic arm, ensuring a cohesive performance.

The control software is designed to interface directly with the U2D2 motor controller, enabling dynamic control of the drum actuation system. Python serves as the backbone for the software architecture, providing a flexible and powerful tool for programming the musical sequences and integrating the frame drums with the broader system. This software layer is critical for achieving the desired musical output and ensuring that the frame drums contribute effectively to the interactive music experience.

The integration of frame drums into the robotic music system was designed around best practices for robotic musicianship, focusing on displaying movement to humans, even if at the cost of mechanical efficiency. This design choice reflects the overarching goal of the project: to create an engaging and immersive musical experience that leverages the unique capabilities of robotic systems while respecting the aesthetic and acoustic properties of traditional instruments.

For imitating a percussionist playing the caxixi, we utilised an off-the-shelf UR3 robot arm which is designed to be safe to operate around humans due to its auto-shutoff capability which utilises internal force sensing to detect unexpected external forces, such as colliding with an object. The UR3 is a highly capable robot with six degrees of freedom (DOF) which means that it can reproduce any arbitrary motion, within its workspace and motor limits, that a human can perform. Robotic arms have recently been used in multiple musical works, such as guitar performance and to convey dancing [21].

The caxixi was attached to the end effector of the robot using a 3D-printed mount, shown in Fig. 4(a). To execute motions on the arm we utilised Robotic Operating System (ROS) [22] which has ready-to-use robot specific drivers. To simulate the robot and environment we used Pybullet [23] with OpenAI Gym and IKFast plugin [24], shown in Fig. 4(b). This was used for ensuring the commanded trajectories were executable and also for avoiding environment collisions.

To program the playing of the caxixi we utilised a recording of a human percussionist using a VICON motion capture system. We track the trajectories of the both the percussionist’s back palm and the caxixi relative to their shoulder. The marker setup is shown in Fig. 5. The VICON captures 3D position and orientation, which we refer to as a pose, at 300 Hz.

To reproduce this trajectory on the robot arm we utilise Gaussian process (GP) regression, a non-parametric probabilistic interpolation method [25]. First a GP is fitted to the trajectory and then using GP regression a smoothed out trajectory of poses at any arbitrary time sequence can be generated. The amount of smoothing can be controlled by adjusting the length-scale parameter of the GP kernel which is useful for ensuring the trajectory is executable on the robot arm which has physical joint velocity limits. More details can be found in [26].

We then took three snippets from this recording with different rhythmic signatures and then translated this motion to the robot arm. For mapping this trajectory to the robot arm we calculate the inverse kinematics solutions for each pose. Since a 6-DOF robot arm can produce multiple solutions for the same pose, for example elbow-up or elbow-down, we bias these solutions to be visually close to the natural configuration of the percussionist’s arm. The robot arm also has an option of moving to a random pose to emulate a lively feel. For ensuring executability, we experimentally chose a GP kernel length-scale which ensured the arm did not violate its joint limits or cause an unintended safety stop.

For the music analysis and generation component of the installation, Pure Data (Pd) was selected for its ability to run on linux. The decision to utilize Pure Data (over pure Python) was also motivated by its capacity to provide a straightforward interface for store staff, ensuring that the system could be managed and troubleshooted with minimal technical expertise. This aspect was crucial for maintaining the system’s operational integrity throughout its deployment in a commercial setting.

The Pure Data patch designed for this project was intentionally kept simple, focusing on stability and ease of use. The primary goal of the patch was to analyze the in-store music in real time and output two key pieces of information: the beats per minute (BPM) and the density of the music. This simplicity in design was essential for ensuring that the system could remain functional over extended periods without requiring complex interventions and avoiding (or at least lowering) the chance of crashing.

The analysis process begins with a volume threshold mechanism. If the in-store music’s volume falls below this threshold, the system remains inactive, ensuring that the installation does not generate output in the absence of sufficient audio input. For detecting musical onsets, the patch employs the Bonk object, a widely used tool within Pure Data for onset detection. The number of detected onsets over time is then used to calculate the density of the music, providing a measure of its rhythmic complexity.

To determine the BPM of the in-store music, the patch utilizes the Beat  object from the ELSE library by Alexandre Torres Porres. The patch monitors the stability of the detected BPM, and in cases of significant variation that suggests instability, it temporarily ceases to generate musical output. This safeguard ensures that the installation’s responses are musically coherent and aligned with the ambient music.

After analyzing and detecting the BPM and density, a Python script creates drum rhythms. The generative systems uses a stochastic model that operates on the incoming level of density, combined with a random value for syncopation. Utilizing these parameters, the system allocates drum strikes across a two-bar grid, determining positions for drum hits. Elevated levels of syncopation facilitate an increased frequency of placements off the beat, whereas density levels dictate the quantity of notes generated. Subsequently, each note is given a probability of being played during each cycle. This secondary layer of probability ensures a continuous variability in the rhythm.

The design of the Pure Data patch was designed for flexibility to accommodate the diverse range of music played in the store, which included genres as varied as classical, jazz, pop, rock, and soundscapes. By focusing on the fundamental aspects of music analysis—tempo and density—the system was equipped to adapt to this wide array of musical styles. This adaptability was key to achieving the project’s objective of enhancing the retail environment with an interactive musical experience that could dynamically respond to the store’s soundtrack, regardless of its genre.

The integration of the interactive music system is orchestrated through a central Linux laptop, which serves as the center for all operations. This centralized approach ensures cohesive control and synchronization over the system components. Audio input is analyzed by a Pure Data patch, which creates the density and BPM that are then streamed to a Python script. The script interprets the BPM and density data to generate a rhythmic grid for the frame drums, using a rule-based system to create varying patterns. This rhythmic grid, structured as a 2-bar pattern repeats until there is a significant change in music density or when it has played for 32 bars, and dictates the activation signals for each frame drum hit and the robotic arm movement (See figure 6). The robot arm is sent either an instruction to play one of the three caxixi rhythms, or to move to a random pose in-time with the beat. The random pose ensures that the robot is relatively active, while not constantly playing the caxixi and distracting from the retail experience.

One of the critical challenges in system integration is ensuring the synchronization of musical output with the detected beats. This requires considering any digital latency and accounting for the physical time required to actuate the frame drums and robotic arm. To manage this, the system implements a delay strategy for playtime, by the distance between each beats minus 120 ms, accounting for the movement time and system latency. This formula ensures that the actuation of the drums and robotic arm is precisely timed to match the rhythm of the music being analyzed.

To enhance the system’s usability, especially for staff, auto-run scripts are configured on the desktop of the central Linux laptop. These scripts enable easy system start-up and restart, ensuring that the interactive music installation can be maintained with minimal effort and technical knowledge.

In conjunction with the installation, a special three hour live performance was organized towards the project’s culmination (Fig. 7). The primary objective of this event was to explore the acoustic dynamics of the space further and to highlight the installation’s presence and capabilities. This initiative served not only as a demonstration of the system’s versatility but also as an opportunity to engage the audience directly with the interactive elements of the robotic musicianship in a more pronounced and focused setting. We invited musicians to improvise on violin, alto saxophone and djembe.

From a technical standpoint, modifications to the setup for the live performance were minimal. The key change involved the integration of a Rode Bluetooth clip-on microphone, which was employed to capture the audio from live musicians. This adjustment facilitated a direct and seamless connection between the musicians’ performances and the installation, allowing the system to process and respond to live musical input in real time. The live performance was an exploratory venture into the soundscapes that could be generated within the given space, involving all combinations of the three instruments. This exploration was aimed at showcasing the dynamic interaction between human musicians and the robotic installation, drawing attention to the robot’s role as an autonomous performer.

This section reflects on the key outcomes and insights derived from the deployment of the interactive music system in a commercial storefront, emphasizing the significance of reliability, the perception of robotic agency, the dynamics of store operations, and the strategic design choices that influenced the system’s integration and reception.

This project successfully demonstrated the feasibility and potential of integrating interactive robotic musicianship into a commercial retail environment. By deploying a UR3 arm alongside custom-built frame drums within a storefront, we showcased a novel application of automation in artistic expression but also enriched the ambient retail experience for customers. By focusing on enhancing the interaction between the system and the store’s ambient music, rather than attempting to alter the store’s musical landscape, we have demonstrated a novel approach to enriching the customer experience without compromising the store’s brand identity.

Our exploration into the realms of anthropomorphism, human-robot interaction, and musical adaptability underscores the nuanced relationship between technology, music, and retail spaces. The spontaneous anthropomorphization of the robotic elements by store staff, attributing agency and character to the system, highlights the potential of such installations to become integrated, engaging parts of social and commercial settings. This project has also addressed the technical challenge of designing a system capable of responding to a diverse array of musical genres, emphasizing interaction and adaptability as central to the success of the installation. The system’s ability to operate continuously for three weeks without significant interruptions highlights its reliability and suitability for long-term installations in public spaces.

Moving forward, this project serves as a foundation for future research and development in the intersection of music, robotics, and retail. It invites further exploration into how these systems can be optimized to meet diverse user needs, adapt to changing environments, and contribute to the creation of innovative retail experiences. By continuing to focus on the seamless integration of technology with human-centric design principles, we can unlock new possibilities for enhancing public spaces with interactive, engaging, and meaningful installations.

This research is partially supported by the Industrial Transformation Training Centre (ITTC) for Collaborative Robotics in Advanced Manufacturing (also known as the Australian Cobotics Centre) funded by ARC (Project ID: IC200100001)