The SYMBODY project is an endeavour aimed at exploring the intricate connections between human body movement and sound synthesis, to unravel the multifaceted aspects of human movement. It aspires to extend its influence beyond the boundaries of art, venturing into significant realms of scientific inquiry and societal progress.

The project aims to reveal the dynamic interplay between body movements and sound by crafting a bespoke, customizable toolbox. This toolbox aims to capture and visualize these relationships in real-time, generating spatialized audio and graphics. Through the utilization of machine learning, we seek to contribute to the understanding of human-machine interaction and advance the field of movement analysis and prediction.

With a focus on modelling human movement and enriching its comprehensive representation, SYMBODY aims to extend predictive models for professional movement. The project envisions contributions that could hold promising implications for disciplines such as kinesiology and biomechanics, potentially refining techniques utilized in diverse fields including dance, sports science, rehabilitation, and ergonomics.

Moreover, within the domain of explainable AI, SYMBODY strives to shed light on the intricate connections between body movements and sound. Through innovative tools, our project not only aims to expand comprehension of human-machine interaction but also provides avenues for artistic exploration, as well as future applications in assistive technology, entertainment, and therapy.

The project aims to develop a comprehensive toolbox comprising a streamlined methodology for dataset creation, machine learning tools to explore the relationship between body movement and sound, data visualization tools for enhanced comprehension, and a live application capable of representing movement-sound correlations in real-time.

By fostering connections between art, science, and society, SYMBODY emerges as a pioneering experimental project with the potential to deepen our understanding of human movement, enrich scientific dialogue, and contribute to societal well-being. SYMBODY represents an exciting collaboration between artist Natan Sinigaglia, In4Art, HekaLab (PiNA), and TMC.

SYMBODY: reflection and thoughts on project by Natan Sinigaglia

Main learnings:

“Choosing different input data for the training process can drastically change the performance of the model. This variability underscores the critical importance of carefully selecting and experimenting with diverse datasets to optimize the model’s effectiveness. We recognize that exploring different input data will be an integral part of our research, driving our understanding of how various factors influence model outcomes.”

“To address this, we decided to prioritize the development of a robust yet flexible pipeline. This pipeline is designed to be adaptable, capable of handling all sorts of sequential data with ease. Its flexibility ensures that it can accommodate a wide range of input types and structures, making it highly versatile for our research needs. By being agnostic in its design, the pipeline does not favor any specific type of data, allowing us to experiment freely and adjust our approach based on the unique characteristics of each dataset we encounter.In this way, our pipeline serves as a foundational tool, providing the necessary infrastructure to support extensive experimentation and iterative refinement. This approach not only enhances our ability to develop high-performing models but also empowers us to uncover new insights and innovations. As we progress, this adaptability will be crucial in navigating the complexities of sequential data, ultimately leading to more robust and insightful research outcomes.”

Collaboration with AIR partners:

“There were several calls between my team, a few partners, and our mentors. During these discussions, all the experts were highly positive and provided us with constructive feedback, invaluable insights, and suggestions on possible directions for development. Their enthusiasm and support were immensely encouraging and helped us refine our approach.”

“Our first visualization prototypes captivated the AI experts. They recognized the value and potential of our data visualization workflow, which further validated our efforts and approach. The experts were particularly impressed with how our visualizations could effectively communicate complex data in a more accessible and intuitive manner. They saw great promise in the ability of these visualizations to enhance understanding and drive more informed decision-making processes.”

“This positive reception from seasoned professionals in the field not only boosted our confidence but also underscored the significance of our work. Their feedback has been instrumental in guiding our next steps, ensuring that we continue to innovate and improve. We are excited to incorporate their suggestions and further develop our prototypes, with the ultimate goal of creating a robust, impactful data visualization tool.”

KNOWLEDGE TRANSFER

Natan Sinigaglia engaged in three different formats of knowledge sharing events, each designed to reach specific audiences and facilitate different types of exchange . He participated in closed, expert AI sessions with scientists from Institute of Artificial Intelligence of Serbia , Universitat de Barcelona  and  HLRS,  where technical discussions focused on the development and refinement of auto-encoder architectures for motion capture. These intimate sessions proved invaluable for addressing complex technical challenges and exploring innovative approaches to movement analysis.

On a public level, he presented his work through a trailer video at the innovation market of the TEKNOWLOGY event, an annual innovation festival where knowledge and innovation lead to societal impact. At TEKNOWLOGY, visitors experienced the technical innovations of the future firsthand, which provided the perfect space to discuss the developments of auto-encoder AI and its practical applications. The festival setting allowed for engaging demonstrations that made complex technical concepts accessible to a broader audience, while gathering valuable insights about public perceptions of AI-driven movement analysis.

Lastly, there was a special dedicated knowledge-sharing event called “Motion Prompting,” which brought together diverse perspectives from artists, technologists, researchers, and industry leaders. This event explored the frontier where AI meets human movement, addressing crucial questions about how artificial intelligence can interpret and respond to physical expression. Through the presentation of  the SYMBODY project participants examined the current capabilities and future potential of motion prompting technology. The event sparked discussions about moving beyond text-based AI interactions, investigating how neural networks perceive the human body, and exploring methods for AI to analyze movement quality.

These knowledge transfer sessions revealed growing interest in body-based AI interfaces and highlighted the importance of considering diverse movement patterns in technical and societal development. The multi-layered approach to knowledge sharing not only advanced the technical aspects of the project but also fostered valuable connections between the artistic, technical, and research communities, opening the discourse on its potential to transform human-AI interaction in the future.

“The combination of expert sessions, public demonstrations, and specialized events created a comprehensive knowledge transfer strategy that served multiple purposes: advancing technical development, increasing public understanding, and facilitating cross-disciplinary exchange. This approach has established a strong foundation for the continued development of motion prompting technology while ensuring that diverse perspectives inform its evolution.” – In4Art

This project is funded by the European Union from call CNECT/2022/3482066 – Art and the digital: Unleashing creativity for European industry, regions, and society under grant agreement LC-01984767