The mother’s arm is moving upwards: how are we able to see in her gesture the tender sweetness leading her to caress the cheek of her sleeping child or the inner violence preparing to hit the soldier’s cheek? How can we foresee in the heavy murderer’s hand movement a slight faltering revealing his fragile uncertainty in finishing his task, how can we measure the hesitation of the woman’s arms closing to protect her crying, in resonant dialogue with the bodies moving around her?
Understanding, measuring and predicting the qualities of movement imply a dynamic cognitive relation with a complex non-linearly stratified temporal dimension. Movements are hierarchically nested: a gesture sequence has a hierarchical layered structure: from high level layers down to more and more local components (as a language: from syntax to phonemes) where every layer influences and is influenced by every other (bottom-up/top-down). Every layer is characterized by a different temporal dimension: a proper rhythm from macro to micro temporal scales of action. This organization does not only apply to action execution, but also to action observation and is at the basis of the unique human ability to understand and predict conspecific gestural qualities. Human skill in understanding and predicting gestural qualities, and attempting to influence one another’s actions, depends on the capacity to create intercrossing relations between these different temporal and spatial layers through feedforward/feedback connections and bidirectional causalities, with the body as a time keeper, coordinating different internal, mental and physiological clocks. In 1973, Johansson showed that the human visual system can perceive the movement of a human body from a limited number of moving points. This landmark study grounded the scientific bases of current motion capture technologies. Recent studies proved that the information contained in such a limited number of moving points does not concern only the activity performed, but can also provide hints about more complex cognitive and affective phenomena: for example, Pollick (2001) showed that participants can infer emotional categories from point-light representations of everyday actions). Studies using naturalistic images and videos have established how fluent we are in body language (de Gelder, 2016).  Very few studies consider the temporal dynamics of the stimulus, and how affective qualities may be perceived faster than other qualities (Meeren et al 2016), be interlinked and change over time. In other words, time is a crucial variable for these processes. Such time intervals are the time intervals of human perception and prediction, i.e., this is a human time, which integrates time at the neural level up to time at the level of narrative structures and content organization. Current technologies either do not deal with such a human time or they do in a quite empirical way: motion capture technologies are most often limited to computation of kinematic measures whose time frame is usually too short for an effective perception and prediction of complex phenomena. While a lot of effort is being spent improving such technologies in the direction of more accurate and more portable systems (e.g., wearable and wireless), such developments are incremental with respect to a conceptual and technological paradigm that remains unchanged. Furthermore, most systems for gesture recognition or for analysis of emotional content from movement data streams adopt time processing windows whose duration is fixed and is usually empirically determined. EnTimeMent proposes a radical change of paradigm and technology in human movement analysis, where the time frame for analysis is grounded on novel neuroscientific, biomechanical, psychological, and computational evidence, and dynamically adapted to the human time governing the phenomena under investigation. This is obtained through innovative scientifically-grounded and time-adaptive technologies that can operate at multiple time scales in a multi-layered approach, transforming the current generation of motion capture and movement analysis systems and endowing them with a completely novel functionality to achieve a novel generation of time-aware multisensory motion perception and prediction systems. The goal of EnTimeMent is to develop the scientific knowledge and the enabling technologies for such a transformation. EnTimeMent will enable novel forms of human-machine interaction (human-machine affective synchronization) and human-to-human entrainment experiences, primarily concerned with the non-verbal, embodied and immersive, active and affective dimensions of qualitative gesture. For instance, novel machine learning techniques inspired by deep networks (Bengio, 2015) might afford the detection of patterns in motion to represent motion at gradually increasing levels of abstraction using Empirical Mode Decomposition (Huang et al., 1998) to extract local characteristic and time scale of the data automatically. This will effectively reduce the originally high-dimensional and redundant raw sensor observations to something that is more manageable for inference of, for example, emotions. The experimental platform will integrate signals from sensors capturing movement in relation to respiration, heart rate, muscle activity, sound, and brain activity across multiple time scales. By these means, the main technological breakthrough of EnTimeMent will be promoting novel perspectives on understanding, measuring and predicting the qualities of movement (at individual and group level) in motion capture, multisensory interfaces, wearables, affective and IoT technologies.

EnTimeMent project objectives...

EnTimeMent project objectives description...

EnTimeMent measurable objectives can be summarised as follows:

EnTimeMent project scientific objectives...

EnTimeMent project technological objectives...

EnTimeMent project community-building objectives...

EnTimeMent project scenarios...

Scenario 1: Healing with multiple times

Research questions addressed: How can multi-time EnTimeMent technologies improve everyday life communicative and prosocial skills in severely disabled persons with low or no verbal capabilities?

Narrative description: Giorgio is a 4 year-old child with a severe case of dystonic cerebral palsy. As a consequence, he is not able to control his movements, feed himself or effectively communicate with others, which threatens learning and participation. His mom Francesca struggles with her child’s condition which she hasn’t still fully accepted. Therapists know the situation is extremely complex, with the family living far from the hospital, in an area where social assistance and services are less effective.

In cases as severe as Giorgio’s, in which a complex rehabilitation project takes place as the person is still growing up, the healing process becomes a life project, with multiple aims and goals unfolding in multiple temporal dimensions. The rehabilitation team has leveraged a “home-care-unit” that affords the family to live autonomously while having Giorgio monitored by the EnTimeMent technology , and allows to read Giorgio’s body communicative signals at multiple temporal scales, in order to:

at a lower temporal scale, enhance movement control by interpreting bodily sensorimotor and muscular signals to regulate and better tune care and assistance; Improve emotion regulation by decoding affective bodily signals and providing most appropriate responses and care.

at a middle temporal scale, foster attention and motivation by interpreting bodily signals of vigilance, focus and interest; Improve control of bodily movements while learning daily-life processes such as drinking; Support emotion regulation by interpreting residual bodily intentional communication

at a higher temporal, foster planning and control of complex activities such as play and social interaction by decoding participative signals; Enhance metacognitive processes by reading resilience signals.

Scenario 2: Chronic Musculoskeletal Pain Management with multiple times

Research question addressed: Can the modeling and enhancement of diverse temporal-scales of movement  in chronic musculoskeletal pain facilitate self-management and functional activity?

Narrative description: Since developing chronic back pain, Paul’s attention is constantly on pain and on fearing increased pain or damage. He no longer understands what his body can and cannot do, or learn from experiencing movement. When he tackles physical tasks, his concern is about pain so he rushes them, feels no satisfaction at completing them and often feels defeated by increased pain. His central nervous system is ‘stuck’ on pain. The newly released EnTimeMent motion capture system captures his movements at different temporal-scales through the day and sonifies it. Each temporal-scale of movement carries particular meaning. Low-level temporal dynamics of Paul’s taking a book from a shelf tells him how far he stretches and how much more he could move, constraining anxiety. Longer temporal aspects of his movement, outside his awareness, represent task achievement (e.g. washing his car) and remind him to take regular breaks before pain builds up. Even longer temporal dynamics, over weeks and months, show him progress in his capabilities, while increased fluidity of his movement reflects reduced anxiety and increased confidence. This progress sustains him during setbacks and higher pain days, as it reminds him how to build activity again, and his central nervous system is processing normal messages about movement. We will investigate new computational models that capture these different temporal dynamics of movement and the multiple temporal phenomena characterising them and their interactions (kinematics, muscle activity, respirations, brain activation) and transform them into sound enriched by metaphorical musical structures that through embodiment enhances the awareness of movement capability, and provides a path for taking back control from pain. Such computational models will also aim to model even lower temporal dynamics, capturing protective and avoidant movement responses (habits outside awareness) at physiological and neural levels that maintain pain. We will aim at developing new metrics of progress (e.g. individual motor signature as in Słowiński et al. 2016) based on these different temporal-scales.  Such computational models can inform the design of rehabilitation for chronic pain self-management supported by the novel multi-temporal multi-modal EnTimeMent motion capture technology. We will also explore how it can be used to facilitate communication between the person with chronic pain and her home-carer and healthcare staff to facilitate learning of management skills.

Scenario 3:  Dancing with multiples times

Research questions addressed: How are individual and group motor signatures of dancers characterized at multiple time scales during improvisation among them and with non-anthropomorphic partners?

Narrative description: A group of dancers engaged in a performance with non-anthropomorphic dancing partners perform joint improvisation sequences, playing with social roles (leader, follower, improvising), emotional gesture qualities (hesitant, aggressive etc.) and physical/ghost presence of their dancing partners. Dancers perform in an architectural space, which can be interactively manipulated by sound (interactive movement sonification, to enhance the communication at multiple temporal scales) or light (e.g., in the light or in the dark), and which can even be structurally transformed by means of interactive mobile scenery (e.g. a theatre stage technologically equipped simulating a future living architecture). Using the EnTimeMent motion analysis system, biological and social signals are recorded at multiples time scales, from milliseconds (e.g., EMG) to seconds (e.g., MOCAP) to minutes (e.g., breathing), allowing the computation of Individual Motor Signatures of each dancer (IMS), Group Motor Signature (GMS), together with their dynamics at their respective temporal scales, as well as entrainment, saliency, and prediction of qualities. Applying the new Curiosity-based EnTimeMent Learning Algorithm (CELA), relevant individual and social performance qualities are extracted simultaneously. Concrete instances of this group interaction scenario will be developed, from dance to sports, fitness, wellness, and entertainment group activities.

EnTimeMent project objectives...

EnTimeMent project objectives description...

EnTimeMent measurable objectives can be summarised as follows:

EnTimeMent project scientific objectives...

EnTimeMent project technological objectives...

EnTimeMent project community-building objectives...

EnTimeMent project scenarios...

EnTimeMent aims at a radical change in scientific research and enabling technologies for human movement qualitative analysis, entrainment and prediction, based on a novel neuro-cognitive approach of the multiple, mutually interactive time scales characterizing human behaviour.

Our strategy facilitates the development of computational models for automated detection, measurement, and prediction of movement qualities. These models, based on behavioural signals, work on multi-layered parallel processes at nonlinearly stratified temporal dimensions, offering new prospects for human movement analysis.EnTimeMent's innovative and time-responsive technologies operate on multiple time scales, introducing a new dimension to motion capture and movement analysis systems. Our goal is to create time-aware multisensory motion perception and prediction systems.

Our proposed models and technologies undergo systematic testing and refinement through a range of controlled and ecological experiments. This includes action prediction in a laboratory setting and in the context of dyadic and small group interactions. EnTimeMent's research scenarios span several sectors such as healthcare, performing arts (e.g., dance), sports, and entertainment group activities. We also engage with living architectures where applicable.

We're dedicated to promoting community-building and exploitation through practical initiatives. This involves creating a network of users and stakeholders, innovation hubs, and SME incubators to provide a basis for project continuity and potential market expansion post-project.

Project Objectives

Project Objectives Description

EnTimeMent measurable objectives can be summarised as follows:

Scientific

Technological 

Community-building

Scenarios