The SYMBODY project is an endeavour aimed at exploring the intricate connections between human body movement and sound synthesis, to unravel the multifaceted aspects of human movement. It aspires to extend its influence beyond the boundaries of art, venturing into significant realms of scientific inquiry and societal progress.

The project aims to reveal the dynamic interplay between body movements and sound by crafting a bespoke, customizable toolbox. This toolbox aims to capture and visualize these relationships in real-time, generating spatialized audio and graphics. Through the utilization of machine learning, we seek to contribute to the understanding of human-machine interaction and advance the field of movement analysis and prediction.

With a focus on modelling human movement and enriching its comprehensive representation, SYMBODY aims to extend predictive models for professional movement. The project envisions contributions that could hold promising implications for disciplines such as kinesiology and biomechanics, potentially refining techniques utilized in diverse fields including dance, sports science, rehabilitation, and ergonomics.

Moreover, within the domain of explainable AI, SYMBODY strives to shed light on the intricate connections between body movements and sound. Through innovative tools, our project not only aims to expand comprehension of human-machine interaction but also provides avenues for artistic exploration, as well as future applications in assistive technology, entertainment, and therapy.

The project aims to develop a comprehensive toolbox comprising a streamlined methodology for dataset creation, machine learning tools to explore the relationship between body movement and sound, data visualization tools for enhanced comprehension, and a live application capable of representing movement-sound correlations in real-time.

By fostering connections between art, science, and society, SYMBODY emerges as a pioneering experimental project with the potential to deepen our understanding of human movement, enrich scientific dialogue, and contribute to societal well-being. SYMBODY represents an exciting collaboration between artist Natan Sinigaglia, In4Art, HekaLab (PiNA), and TMC.

SYMBODY: reflection and thoughts on project by Natan Sinigaglia

Main learnings:

“Choosing different input data for the training process can drastically change the performance of the model. This variability underscores the critical importance of carefully selecting and experimenting with diverse datasets to optimize the model’s effectiveness. We recognize that exploring different input data will be an integral part of our research, driving our understanding of how various factors influence model outcomes.”

“To address this, we decided to prioritize the development of a robust yet flexible pipeline. This pipeline is designed to be adaptable, capable of handling all sorts of sequential data with ease. Its flexibility ensures that it can accommodate a wide range of input types and structures, making it highly versatile for our research needs. By being agnostic in its design, the pipeline does not favor any specific type of data, allowing us to experiment freely and adjust our approach based on the unique characteristics of each dataset we encounter.In this way, our pipeline serves as a foundational tool, providing the necessary infrastructure to support extensive experimentation and iterative refinement. This approach not only enhances our ability to develop high-performing models but also empowers us to uncover new insights and innovations. As we progress, this adaptability will be crucial in navigating the complexities of sequential data, ultimately leading to more robust and insightful research outcomes.”

Collaboration with AIR partners:

“There were several calls between my team, a few partners, and our mentors. During these discussions, all the experts were highly positive and provided us with constructive feedback, invaluable insights, and suggestions on possible directions for development. Their enthusiasm and support were immensely encouraging and helped us refine our approach.”

“Our first visualization prototypes captivated the AI experts. They recognized the value and potential of our data visualization workflow, which further validated our efforts and approach. The experts were particularly impressed with how our visualizations could effectively communicate complex data in a more accessible and intuitive manner. They saw great promise in the ability of these visualizations to enhance understanding and drive more informed decision-making processes.”

“This positive reception from seasoned professionals in the field not only boosted our confidence but also underscored the significance of our work. Their feedback has been instrumental in guiding our next steps, ensuring that we continue to innovate and improve. We are excited to incorporate their suggestions and further develop our prototypes, with the ultimate goal of creating a robust, impactful data visualization tool.”

This project is funded by the European Union from call CNECT/2022/3482066 – Art and the digital: Unleashing creativity for European industry, regions, and society under grant agreement LC-01984767