Aligning Human Motion Generation
with Human Perceptions

Human motion generation is a pivotal task in computer vision, computer graphics, and artificial intelligence, aiming to produce natural and realistic human pose sequences [53]. This field has seen substantial advancements with the rise of deep generative models [22, 33, 9, 16]. Previous works have explored text-conditioned motion generation that transform narrative descriptions into coherent pose sequences [38, 37, 25, 31, 28], audio-conditioned methods that synchronize movements with rhythmic cues [17, 35, 40], and scene-conditioned generation that integrates environmental contexts to produce contextually appropriate motions [5, 42, 1]. Despite significant progress, current mainstream data-driven kinematic motion generation methods sometimes produce unnatural motions that are jittery, distorted, or violate physiological and physical constraints. These issues could be attributed to the inherent uncertainty of the task, limitations of supervision signals, and dataset noises. Furthermore, evaluating generated human motions presents additional challenges. Conventional metrics such as error and FID fall short in capturing the nuanced details essential for producing lifelike and visually appealing movements [40, 41]. These measures can overlook critical aspects like the fluidity and biomechanical plausibility that are fundamental to human perceptual judgments. Given these challenges, it is imperative to develop metrics that are more closely aligned with human perception to more accurately evaluate and enhance the motion generation results.

Pioneer work [51] collect human perceptual similarity dataset and propose to utilize distance in deep features as perceptual metrics. Some works [30, 2, 50, 14, 6] in language models to explore aligning model performance with human intent by first training a reward model, then performing reinforcement learning with the reward model. Recent works  [49, 43, 24] also explore utilizing human feedback to improve visual generation results. For example, ImageReward [44] propose a reward feedback learning method (ReFL) to to align text-to-image generative models with human judgements. In human motion generation, however, few studies have explored modeling human feedbacks, even though the generated motion quality is highly relevant to human perceptions. One recent work, MoBERT [41], constructs a dataset of human ratings for generated motions. Our work differs from MoBERT in that we collect real human data on a scale tens of times larger (52.6K vs 1.4K) and use comparisons instead of ratings, which is more robust. We design the critic model to learn ratings from these comparisons automatically. Additionally, our approach could not only evaluate motion quality but also effectively improve motion generation results.

We first collect generated human motion sequence pairs for subsequent perceptual evaluation. We utilize state-of-the-art diffusion-based motion generation method MDM [38] and FLAME [21] to generate human motion sequences parameterized by SMPL [27]. For MDM [38], we utilize the action-to-motion model trained on HumanAct12 [13] and UESTC [19] respectively. For FLAME [21], we utilize the text-to-motion model trained on HumanML3D [12]. For each group of $4$ motion sequences to be annotated, we use the same condition (text prompt or action labels) while sampling different random noises. This makes the motions similar in content while still having distinguishable differences, thereby making it easier to annotate the choices.

Human perceptual evaluation is the core component of MotionPercept, therefore we implement a rigorous pipeline to ensure annotation quality. We first introduce the question design of the perceptual evaluation, then describe the protocol for conducting the evaluation. Finally, we present a statistical analysis of the evaluation results.

In total, we collect annotations for 18260 multiple-choice questions covering 73040 unique motions, significantly surpassing previous work [41] (1400 motions). We further investigate the following two questions:

1. 1.

   Based on our experimental setup, can the subjects confidently select the suitable options from the choices provided?

2. 2.

   Is there a significant difference in perceptual preferences among different subjects, or are they well-aligned?

For the first question, we calculate the proportion of cases where a choice could not be made (including “all good” and “all bad”), and find a total of 418 such groups (2.29%). The result indicates that most of the time subjects can make a definite judgment, demonstrating the validity of our protocol design.

For the second question, we conduct a perceptual consensus experiment where all 10 subjects perform perceptual evaluation independently on 312 groups of randomly selected data. We calculate their pairwise and overall consistency in choices. Figures 2(A) and 2(B) show that for most questions (82.37%), all 10 subjects make the unanimous decision. Figure 2(C) reveals that all 10 subjects exhibit high pairwise agreement (90%). These results indicate a high level of consistency in perceptual judgments of human motion among different human subjects. This not only validates the rationality of our perceptual evaluation pipeline but also inspires us to train machine learning models to emulate this consistent judgment capability.

We formulate the problem as follows: given an input human motion sequence $\mathbf{x}$, we assume there is an implicit human perception model $\mathcal{H}$ that rates the motion quality $\mathcal{H}(\mathbf{x})$, where a higher rate indicates better quality. We aim to build a computational critic model $\mathcal{C}$ that best aligns with $\mathcal{H}$. Since $\mathcal{H}$ is not explicitly available, we take a data-driven approach. We obtain the human perceptual evaluation dataset $\mathcal{D}$ containing multiple pairs of samples $(\mathbf{x}^{(i)},\mathbf{x}^{(j)})$. Our training objective is to train the model $\mathcal{C}$ using the dataset $\mathcal{D}$ so that it approximates the human perception model $\mathcal{H}$ as closely as possible. Specifically, we want the model prediction $\mathcal{C}(\mathbf{x}^{(i)})>\mathcal{C}(\mathbf{x}^{(j)})$ if and only if $\mathcal{H}(\mathbf{x}^{(i)})>\mathcal{H}(\mathbf{x}^{(j)})$. Based on the Bradley-Terry model [4, 18], the overall training objective could be written as maximizing the joint probabilities that the model $\mathcal{C}$ makes judgments consistent with $\mathcal{H}$ for each pair of samples in the dataset $\mathcal{D}$:

where $\sigma$ is the sigmoid function.

In practice, we represent human motion by $\mathbf{x}\in\mathbb{R}^{L\times J\times D}$ where $L$ denotes the sequence length, $J$ denotes the number of body joints, and $D$ denotes parameter dimensions. We implement the critic model $\mathcal{C}$ as a neural network that maps the high-dimensional motion parameters to a scalar $s$. We draw pairwise comparison annotations from the collected dataset, where $\mathbf{x}^{(h)}$ is the better instance and $\mathbf{x}^{(l)}$ is the worse. The perceptual alignment loss is thus given by:

Additionally, we explore to utilize the learned human perceptual prior of $\mathcal{C}$ not only for evaluating generated motions, but also improving them. We demonstrate that our motion critic model could be integrated into state-of-the-art diffusion-based motion generation approaches with ease by using MDM [38] as an example. The forward diffusion is modeled as a Markov noising process $\{\mathbf{x}_{t}\}_{t=0}^{T}$ where $\mathbf{x}_{0}$ is drawn from the data distribution, and

where $\alpha_{t}\in(0,1)$ are constant hyper-parameters. When $\alpha_{t}$ is small enough, it’s reasonable to approximate $\mathbf{x}_{T}\sim\mathcal{N}(0,I)$, allow sampling $\mathbf{x}_{T}$ from random noise to begin our denoising process.

Given an MDM model $\mathcal{M}$ with pre-trained parameters $\theta_{0}$, we fine-tune to improve its alignment with a pre-trained critic model $\mathcal{C}$. We develop a lightweight perceptual-aligned fine-tuning approach based on ReFL [44]. Notably, in order to utilize the critic model in a plug-and-play manner, we keep the MDM training step and objective $\mathcal{L}_{\text{MDM}}$ unchanged. Instead, we simply add one optimization step with critic model supervision in each training iteration as shown in Figure 1.

Specifically, we sample a Gaussian noise $\mathbf{x}_{T}$ and inference until $\mathbf{x}_{t}$ where $t\in[T_{1},T_{2}]$ is randomly selected in later denoising steps. Then, a single-step denoising is performed to predict $\mathbf{x}_{0}\textquoteright$ from $\mathbf{x}_{t}$. Based on the predicted motion $\mathbf{x}_{0}\textquoteright$, we compute the critic score $s=\mathcal{C}(\mathbf{x}_{0}\textquoteright)$, which is used to compute the motion critic loss:

where $\phi(s)=-\sigma(\tau-s))$ is a critic-to-loss mapping function, $\tau$ is a constant for shifting the critic value, $\sigma$ is the sigmoid function. We further introduce a Kullback-Leibler (KL) divergence regularization to prevent $\mathcal{M}$ from moving substantially away from the conditional motion generation task:

The overall fine-tuning loss is given by

where $\lambda$ and $\mu$ are constants for loss balancing. The detailed algorithm workflow is shown in Algorithm 1.

We first evaluate whether the proposed critic model could serve as an effective motion quality metric. Specifically, we are interested in the following research questions:

1. 1.

   How does MotionCritic align with human perceptual evaluations?

2. 2.

   Could MotionCritic generalize to different data distributions?

To investigate the first question, we evaluate the performance of our critic model on a held-out test set and compare it with existing motion quality metrics as follows:

• •

  Error-based metrics, including Root Average Error (Root AVE), Root Absolute Error (Root AE), Joint Average Error (Joint AVE), and Joint Absolute Error (Joint AE). These metrics involve directly computing the distance between the generated motion and a pairing GT with the same condition.

• •

  Heuristic metrics, including acceleration [32, 45], Person-Ground Contact [32], Foot-Floor Penetration [32], and Physical Foot Contact (PFC) [40]. These metrics does not compare against GT; instead, they implement intuitive rule-based evluations. For example, PFC models the relationship between center of mass acceleration and foot-ground contact.

• •

  Learning-based metrics. Prior work MoBERT [41] proposes to evaluate motion quality with a motion feature extractor and SVR Regression.

Note that distribution-based metrics (e.g. FID) could not compare quality of individual motion sequences, and the comparison can be found in subsequent experiments. For each metric, we calculate the percentage they align with GT annotations (accuracy) and also their probabilistic distribution distance with GT annotations (log loss). We use the softmax function to convert the scores to probabilities (taking the opposite before softmax for metrics where smaller is better). Table 4 demonstrates that our critic model significantly outperforms previous metrics. These results not only validate the effectiveness of learning from large-scale human perceptual evaluations but also prove that our critic model can serve as a more comprehensive and robust metric for assessing motion quality.

Furthermore, to investigate the second question, we test the critic model on data outside of the training distributions. We collect a standalone test set with a different motion generation algorithm, FLAME [25], and perform perceptual evaluation with a different human subject. Note that this model is trained on a different dataset [12] with the model used to generate critic model training data, which means the action categories have large variations. The results in Table 4 further shows that our critic model could well generalize to the new test set, indicating its efficacy in evaluating different generation algorithms and unseen motion contents.

Additionally, we test the generalization of our critic model on the real GT motion distribution. Figure 4(A) illustrates the critic score distribution of HumanAct12 [13] test set. We group the 1190 GT motions into 5 groups based on their critic scores, evenly distributed from highest to lowest. We compare the average critic score between the groups with distribution-based metric FID and user study. The user study is conducted by comparing motion pairs sampled from each groups and then computing Elo rating [7, 40] for each group. Figure 4(B) clearly indicates that the critic score aligns well with human preferences, while FID does not. Notably, we discover that the outliers with small critic values (group V) are indeed artifacts within the dataset. Please refer to the supplementary materials for video results. The results indicate that our critic model can also generalize to the GT motion manifold, even though the model has never been trained on it. It also highlights the potential of using our critic model as a tool for dataset diagnosis (e.g., discover failure modes).

Furthermore, we investigate whether our critic model can also serve as an effective supervision signal. Specifically, we fine-tune a pre-trained motion generator [38] with the proposed framework, and evaluate on HumanAct12 [13] test set every 200 steps. Additionally, we conduct a standalone user study by comparing motion pairs generated at different fine-tuning steps and compute the Elo Rating [7, 40]. Figure 5 reveals that as fine-tuning progresses, the motion quality consistently improves according to the user study, in line with the training objective of increasing the critic score. We also present a visualization comparison in Figure 6. We discover that as fine-tuning progresses, unreasonable human motions such as jittering, twisting, and floating significantly decrease. Please refer to the supplementary materials for video comparisons. The results also demonstrate that our fine-tuning process requires only hundreds of iterations to take effect, significantly improving the perceptual quality of the model. Compared to the 350K pre-training steps, this accounts for only 0.23% of the training cost. This further demonstrates the advantages of our proposed framework in using a perceptually-aligned critic model to fine-tune the motion generation model, not only improving quality but also being lightweight and efficient.