Symbolic Human Motion Programming and Style Disentanglement (Jie Chen et al.)

We study the problem of automatic motion generation of 3D trajectories that animate the human body. It has great potential in various research and application arenas such as film production, CG (Computer Graphics) character animation, performing arts, motion science, and mixed reality applications. The Generated motion should not only obey physical and kinematic laws, more importantly, the aesthetic style and messages conveyed by the motion should be controllable, and consistent with the music and other artistic context. These make it an extremely challenging problem.

Existing motion generation research over-simplifies the task as an autoregressive process where future poses are predicted based on information of a small local window to the past. Probabilistic models tend to generate unsatisfactory and repeated motion. Although reinforcement learning can avoid explicit supervision on the generation target, the agent’s action space is still expensive to model and almost impossible to interpret. As such, we propose to understand and manipulate motion through the symbolic language programming using Labanonation. By studying the connection between symbolic notations with artistic context, we aim to investigate the deep logic of motion and choreography. In specific, we will investigate a hierarchical symbolic motion representation framework with tokenized Motif codes at the highest abstraction level. A note-based language model will be designed for high-level motion concept programming. Based on the Motif notes, Labanotation will be conditionally programmed under a reinforcement learning framework to choreograph the exact use of the body parts, time, direction, and energy. The figure bellow shows a preliminary framework we are investigating.

In addition, we will create a large motion capture dance dataset professionally performed and densely labelled with Labanotation, together with rich context information. Based on the dataset, we will investigate a novel style disentanglement framework which enables genre-specific motion generation and style translation between traditional Chinese dance and classical western ballet.

We have the best motion capture studio in Hong Kong, and we are in close collaboration with Hong Kong Academy of Performing Arts, and the Hong Kong Dance Company.


References:


For further information on this research topic, please contact Dr. Jie Chen.