EU Horizon 2020 FET PROACTIVE project (2019-2022)

Impact on research, innovation, industry and policy

EnTimeMent aims at a radical change in scientific research and enabling technologies for human movement qualitative analysis, entrainment and prediction, based on a novel neuro-cognitive approach of the multiple, mutually interactive time scales characterizing human behaviour.

Research and Development Funding

The EnTimeMent results, particularly the advances in theoretical frameworks and computational models demonstrated by 50+ scientific papers and surveys available free access, and the EnTimeMent Stack Framework Architecture to support industry standard platforms in creative industry, can be used by national and EU research funding agencies to increase their funding and support in the field of expressive gesture and emotion recording technologies. Our attempt at integration extends to a range of renowned platforms, such as Unity for game development, Blender for 3D modelling, Adobe Premiere for video editing, and Autodesk Maya for digital animation. If from one hand, these explorations represent initial efforts, we believe they can be foundational contributions to the future of motion capture technology, in a direction of innovation, advancement, and refinement of existing hardware and software solutions. Specifically, we encourage funding agencies to support time-aware multimodal markerless expressive motion tracking, with particular focus to individual as well as group movement analysis, automated analysis of motion signature, and cross-modal interactive sonification of movement, as they have been observed as being crucially important in human social interactions.

Standards and Guidelines

EnTimeMent results, in particular the emerging taxonomy of time-aware expressive and emotion features from human movement, can be used for the establishment of standards and guidelines for the ethical and responsible use of expressive gesture and emotion recording technologies. This could include considerations such as privacy, security, informed consent, and the protection of sensitive information.

Integration in Education

EnTimeMent results can be used to emphasize the integration of the developed technologies into educational curricula, and to stimulate the creation of novel trans-disciplinary curricula. For instance, the novel “Affective Computing, Arts, and Cultural Welfare (ACW)” international track of the Master’s degree in “Digital Humanities – Interactive systems and digital media” at the Polytechnic School of University of Genova is inspired also by EnTimeMent results. This would involve promoting the use of EnTimeMent results in fields like performing arts, cultural wellness, psychology, human-computer interaction, and therapy. Integrating these technologies into trans-disciplinary educational programs would facilitate skill development, research, and applications across various disciplines opening novel directions and opportunities in job creation.

Industry Collaboration and Support

Our results, for instance the propagation of emotional qualities across multiple temporal scales during solo, dyad, or group synchronization situations in humans encourage collaboration between the developers of expressive gesture and multi-scale emotion recording technologies, with relevant industries, such as entertainment, gaming, healthcare, sport, culture, and education. This collaboration could foster the creation of new applications and products that enhance user experiences and benefit different sectors. As one concrete example, the BeatHealth startup we created recently in EuroMov (2021) and which develops rhythm-based digital solutions for health and wellness, focuses on music-movement synchronization as a key factor in predicting and rehabilitating rhythm-based pathologies such as Parkinson Disease. The results obtained in EnTimeMent helped us to realize the very multi-scale nature of synchronization, not only at behavioural level but also across physiology and cognition. This realization has consequences for the personalized efficacy of these solutions, which is currently being investigated.

Accessibility and Inclusivity

This policy direction resulting from our results may focus on ensuring the accessibility and inclusivity of new expressive gesture and emotion recording technologies. This would involve addressing affordability, compatibility with different devices, and the development of user-friendly interfaces that accommodate individuals with diverse abilities and needs.

Ethical Considerations

The consortium has explored hardware and software solutions to collect human movement data that may have a clear path towards applications in health and disease (e.g., diagnosis, behavioral prediction, intention decoding, goals understanding, etc.). Given the fast-pace of current advancements both in the direction of AI-based methods to extract these information without the participant being aware and the availability of miniaturized distributed sensing technologies it is now more than ever necessary to start a serious discussion on how to embed privacy by design in any of these software/hardware solutions. Human movement data collection in public spaces is already a reality and, given the potential unleashed by current AI-solutions, it is no longer sufficient to request consent to the user. Rather, the policy maker has to impose technical and organizational solutions to all technological players such that data is safe from the very first acquisition step.

In addition to this, and in consideration of the previous points, guidelines for the ethical use of innovative expressive gesture hardware and software must address potential biases. These policy directions would provide a framework for the responsible development, deployment, and use of hardware and software for recording expressive gestures and emotional qualities in movements, fostering innovation while safeguarding individual rights and societal well-being.