Multimodal Conditional 3D Face Geometry Generation

Abstract.

We present a new method for multimodal conditional 3D face geometry generation that allows user-friendly control over the output identity and expression via a number of different conditioning signals. Within a single model, we demonstrate 3D faces generated from artistic sketches, 2D face landmarks, Canny edges, FLAME face model parameters, portrait photos, or text prompts. Our approach is based on a diffusion process that generates 3D geometry in a 2D parameterized UV domain. Geometry generation passes each conditioning signal through a set of cross-attention layers (IP-Adapter), one set for each user-defined conditioning signal. The result is an easy-to-use 3D face generation tool that produces high resolution geometry with fine-grain user control.

Figure 1. We propose a novel method for diffusion-based controllable 3D face geometry generation that allows for controlling the results via several conditioning modes: artistic sketches, 2D facial landmarks, Canny edges, FLAME model parameters, portrait photos and text.

1. Introduction

The creation of 3D facial geometry for digital human characters is a modeling task that usually requires tremendous artistic skill. Digital sculpting with 3D modeling tools is a time-consuming and demanding process, especially when the target is as recognizable as a human face. This complexity has prompted research into data-driven sculpting methods (Gruber et al., 2020) and other, more user-friendly, interactive interfaces (Kim et al., 2015).

Several common morphable 3D face models (e.g. FLAME (Li et al., 2017)) simplify the facial modeling task by providing a shape subspace to operate in, as well as simple parameters to control the identity and expression geometry without the need for 3D modeling skills, but they are limited in expressiveness and offer only basic control knobs.

Recent methods can create high quality 3D geometry and textures from text prompts (Zhang et al., 2023a; Wu et al., 2023) via optimization, leveraging large pre-trained text-to-image diffusion models (Rombach et al., 2022). These methods allow layman users to create 3D faces through natural text descriptions. While this is a powerful approach, it can still be difficult to achieve a particular output through text description (Koley et al., 2024). Some concepts like the specific curvature of a face or a unique facial expression are much easier to convey via sketches, edge contours or portrait photos than through text.

In the image domain, approaches like ControlNet (Zhang et al., 2023b) or T2I-Adapter (Mou et al., 2023) have demonstrated controllable image generation beyond text using sketches, images, or edge maps as conditioning signals. These methods provide users with much more fine-grained control over the generation process than text-based methods alone. Ye et al. (2023) propose IP-Adapters to control Stable Diffusion (Rombach et al., 2022) with image prompts by learning new cross-attention layers. However, image-based methods are not easy to extend to 3D facial geometry generation.

We present a flexible new method for 3D facial geometry generation that creates high quality faces from any one of various inputs, including sketches, 2D landmarks, Canny edges, portrait photos, FLAME face model parameters and text. Our approach is to train a conditional diffusion model on a high quality 3D facial dataset constructed from high resolution scans (Chandran et al., 2020) represented in the 2D UV domain. Our model is trained from scratch, without the need for a pre-trained foundation model like Stable Diffusion. To condition our model we train one set of cross-attention layers for each type of conditioning input, following IP-Adapter (2023). First, the diffusion model learns to inject FLAME parameters via the original UNet cross-attention layers. We then freeze the diffusion model while training additional sets of cross-attention layers (e.g. one for artist sketches, 2D landmarks, portrait photos, etc). An interesting side effect of the FLAME-conditioned model is that it allows us to re-interpret FLAME-parameterized faces in a generative sense - providing a space of high resolution stochastic variations on top of the traditional low resolution FLAME model.

Our method allows for fast and user-friendly creation of 3D digital character faces, controlled by the input mode preferred by the user, all within a single model. We demonstrate several applications of our method including sketch-based 3D face modeling, geometry from 2D facial landmarks, Canny edges, or portrait photos, text-to-3D facial geometry, and finally, extending the FLAME model space by allowing stochastic diffusion sampling conditioned on the same semantic parameters.

2. Related Work

In the following, we present relevant related work on 3D face geometry generation with diffusion models, as well as on injecting additional control modes into diffusion models.

2.1. 3D Face Geometry Generation

Recent work uses diffusion models to control the generation of novel 3D face geometry. ShapeFusion (Potamias et al., 2024) generates face geometry by running the diffusion process directly on the mesh input vertices. It allows unconditional and conditional face geometry generation and supports various editing operations on a given mesh, based on selected vertices (anchor points). However, it does not support conditioning signals beyond vertices. 4DFM (Zou et al., 2024) trains an unconditional diffusion model on a set of sparse 3D landmarks for facial expression generation. It can generate dynamic facial expression sequences based on 3D landmarks by retargeting the landmarks to a mesh after the diffusion process. While they support conditioning with different signals such as expression labels and text, they achieve control via classifier-guidance (Dhariwal and Nichol, 2021) which requires training additional classifiers on noisy data.

Other methods focus on 3D face or head avatars, which can generate 3D geometry and texture. Rodin (Wang et al., 2023) can generate triplane-based head geometry with text or image conditioning, but the resulting geometry is extracted with Marching Cubes (Lorensen and Cline, 1987) and thus not in the same topology across generations. DreamFace (Zhang et al., 2023a), Wu et al. (2023) and Bergman et al. (2023) propose pipelines that generate 3D face geometry and textures from text. The geometry is created by optimizing 3D Morphable Model (3DMM) (Blanz and Vetter, 1999) parameters using a Score Distillation Loss (SDS) (Poole et al., 2023). During the optimization, the SDS loss uses the feedback from a pre-trained text-to-image latent diffusion model to update the 3DMM parameters given a geometry render. This setup requires a differentiable renderer and often generates over-smoothed results with little details when compared to diffusion sampling (Zhou et al., 2024). Additionally, they require Stable Diffusion (SD) (Rombach et al., 2022) which was pre-trained on billions of images (Schuhmann et al., 2022) to guide their generations.

In our work, we generate controllable 3D face geometry in a single common topology from several different conditioning modes without reliance on SDS optimization, classifier-guidance or a SD prior.

2.2. Multimodal Conditional Image Generation

Image generation with diffusion models can be controlled with conditioning modes that are different from text such as sketches (Voynov et al., 2023), Canny edges (Canny, 1986), RGB images (Ye et al., 2023), expression parameters (Kirschstein et al., 2024) or face shape (Ding et al., 2023; Gu et al., 2024). To control pre-trained diffusion models with new modes, ControlNet (Zhang et al., 2023b) introduces a trainable copy of the diffusion model’s UNet encoding blocks, which take the new conditioning as input. The output of the copied model is added to the skip-connections of the frozen pre-trained diffusion model. A separate trainable copy of the UNet encoding blocks (361M parameters) is created per conditioning mode. T2I-Adapter (Mou et al., 2023) aligns the internal knowledge of a pre-trained diffusion model with new control modes by proposing a small adapter network that achieves control similar to ControlNet, while requiring less parameters (77M). IP-Adapter (Ye et al., 2023) injects each conditioning via separate cross-attention layers (Vaswani et al., 2017) while requiring even less parameters (22M). It introduces new cross-attention layers whose outputs are added to those of the original UNet. Ye et al. (2023) show that the diffusion model follows the added conditioning signal closely, when it comes through the newly trained cross-attention layers. In general, adapters can add control with new conditioning modes even long after the training of the underlying base diffusion model is concluded. They can also add new conditioning modes for which only limited paired training data is available, because the underlying diffusion model is frozen, avoiding issues such as catastrophic forgetting (Kirkpatrick et al., 2017; Zhang et al., 2023b). While pre-trained text-to-image diffusion models understand the RGB image domain and can generate images conditioned on a large variety of input modalities, they are not trivially extendable to generating 3D face geometry instead. We represent 3D face geometry in the 2D UV domain, which enables us to train a 2D diffusion model that can incorporate new conditioning modes using IP-Adapters. However, even in the 2D UV domain, 3D geometry is still far from what pre-trained text-to-image diffusion models have seen before. Thus, we train our diffusion model from scratch.

3. Multimodal 3D Face Geometry Generation

Figure 2. Our pipeline for diffusion-based controllable 3D face geometry generation which uses a delta UV position map representation $\Delta$$\bf{T}$ to generate results. We can control the results with several conditioning modes (i.e. FLAME parameters $\mathbf{c}_{f}$, sketches $\mathbf{c}_{1}$, Canny edges $\mathbf{c}_{2}$, 2D landmarks $\mathbf{c}_{3}$, portrait photos $\mathbf{c}_{4}$ and text $\mathbf{c}_{5}$).

We propose a novel method for diffusion-based controllable 3D face geometry generation that allows for controlling the results with several conditioning modes (i.e. sketches, 2D landmarks, Canny edges, FLAME parameters, portrait photos and text).

Our method consists of four components: First, we create a dataset of 3D faces, where each face is represented as a UV position map describing the vertex positions (Section 3.1). This data representation can be easily processed with 2D convolutional neural networks. Second, a variational autoencoder (VAE), which compresses our UV position map face data into a latent space representation (Section 3.2). Third, a latent diffusion model (LDM), which learns a non-linear, deep controllable face model in latent space (Section 3.3). Fourth, we learn mode-specific cross-attention layers (IP-Adapters) with the ability to transform and inject conditioning modes into the LDM for controllable 3D face geometry generation (Section 3.4). Each of the components is explained in the following and visualized in Fig. 2.

3.1. Dataset and Geometry Representation

To represent our face geometries, we generate a novel dataset based on the 3D scan data acquired by Chandran et al. (2020), where all faces are stabilized and in topological correspondence. In total, we use 323 identities in our training dataset, where each identity shows 24 different facial expressions (7752 examples). We subtract a template face shape $\bf{T}$ from all faces in the dataset and thus represent each individual face as a delta from the template face. The computed delta representation $\Delta$$\bf{T}$ reduces artifacts in the generated 3D face geometry, when compared to the full face representation (Fig. 3). We transform each delta face into a vertex delta position map in UV space, which is suitable for being processed by neural networks (Feng et al., 2018; Otto et al., 2022). This representation records the x, y, z coordinates of the face geometry within a 3-channel image, similar to traditional color texture maps, but instead of an RGB value at each pixel we store the x, y, z delta values. To improve generalization, we augment our existing geometry training data by synthetic identities which we generate via identity interpolation (50k examples) and by mixing face parts of different identities together (Funkhouser et al., 2004) (150k examples). Adding the augmentations during LDM training improves the generalization to novel identities when conditioning on the FLAME parameters (Table 1). We use the parameter space of a common 3D morphable face model (FLAME (Li et al., 2017)) as a base conditioning because we can fit FLAME to the scan data and to the augmented data and thus generate a large dataset of paired geometry-FLAME parameter data. Additionally, we create paired training data for several conditioning modes that only have limited paired data available. For example, portrait photos are only available for the scans from the dataset of Chandran et al. (2020), but not for the augmentation data. However, it is possible to inject new modalities with limited paired data by training new cross attention layers while keeping the LDM frozen as shown by Ye et al.(2023) (and described in Section 3.4).

Figure 3. On the left is the generated geometry when training on the full vertex map representation, which shows visible artifacts (e.g., on the eyelid). Our proposed training with the delta vertex maps (right) results in removes such artifacts.

3.2. Variational Autoencoder

To reduce the computational requirements for the diffusion model, we downsample our $256^{2}$ UV position map data by a factor of four into a $64^{2}$ latent space using a variational autoencoder (VAE) (Kingma and Welling, 2014) consisting of an encoder $\mathcal{E}$ and a decoder $\mathcal{D}$. We train the VAE from scratch following the autoencoder loss function and architecture as it is presented in related work (Rombach et al., 2022; Esser et al., 2021). Specifically, we use the VQ-GAN (Esser et al., 2021) autoencoder loss:

(1) $$\begin{split}\mathcal{L}_{\text{VAE}}=\mathcal{L}_{\text{rec}}+\mathcal{L}_{\text{GAN}}+\mathcal{L}_{\text{reg}}.\end{split}$$

$\mathcal{L}_{rec}$ consists of a pixel-wise L1 loss and a LPIPS (Zhang et al., 2018) perceptual loss. It compares the input UV position maps to the reconstructions through the VAE. $\mathcal{L}_{GAN}$ evaluates inputs $x$ and reconstructions $\mathcal{D}(\mathcal{E}(x))$ with a patch-based discriminator (Isola et al., 2017) and $\mathcal{L}_{reg}$ employs a codebook loss which serves as a latent space regularizer. Please refer to Esser et al. (2021) for more details.

3.3. Latent Diffusion Model

Next, we train a latent diffusion model (LDM) (Rombach et al., 2022), that learns to generate latent UV position maps $\mathbf{z}=\mathcal{E}(\mathbf{x})$. To train the LDM, a forward diffusion process is defined as a Markov chain, which noises the latents $\mathbf{z}$ following a fixed noise schedule of $T$ uniformly sampled timesteps. At the last time step $T$, the distribution is Gaussian. We can directly sample $\mathbf{z}_{t}$ at an arbitrary timestep $t$ by:

(2) $$\mathbf{z}_{t}(\mathbf{z}_{0},\bm{\epsilon})=\sqrt{\bar{\alpha}_{t}}\mathbf{z}_{0}+\sqrt{1-\bar{\alpha}_{t}}\bm{\epsilon}\hskip 14.22636pt\bm{\epsilon}\sim\mathcal{N}(\mathbf{0},\mathbf{I}),$$

where $1-\bar{\alpha}_{t}$ describes the variance of the noise and $\bar{\alpha}_{t}:=\prod_{s=1}^{t}\alpha_{s}$ according to a fixed noise schedule.

We learn to predict the noise $\bm{\epsilon}$ that was added to a noisy latent image $\mathbf{z}_{t}$ following Ho et al. (2020):

(3) $$\mathcal{L}_{LDM}=\mathop{{}\mathbb{E}_{\mathbf{z}_{0},\mathbf{c}_{f},t,\bm{\epsilon}\sim\mathcal{N}(\mathbf{0},\mathbf{I})}}\left[||\bm{\epsilon}-\bm{\epsilon}_{\theta}(\mathbf{z}_{t},\mathbf{c}_{f},t)||_{2}^{2}\right],$$

where $t$ is the timestep, $\mathbf{c}_{f}=\bm{\rho}_{\phi}(\mathbf{y})$ is a FLAME parameter conditioning and $\bm{\epsilon}_{\theta}(\mathbf{z}_{t},\mathbf{c}_{f},t)$ is the UNet (Ronneberger et al., 2015) neural network with parameters $\theta$. The FLAME conditioning $\mathbf{c}_{f}$ is obtained by fitting FLAME to the face geometry encoded in $\mathbf{z_{t}}$ and mapping it through a MLP $\bm{\rho}_{\phi}(\mathbf{y})$.

During inference (reverse diffusion process) we generate latent 2D UV position maps from the model distribution. We start from $\mathbf{z}_{T}\sim\mathcal{N}(\mathbf{0},\mathbf{I})$ and iteratively compute less noisy latents until we reach a clean latent sample $\mathbf{z}_{0}$. Sampling following DDPM (Ho et al., 2020) or DDIM (Song et al., 2021) computes $\mathbf{z}_{t-1}$ from $\mathbf{z}_{t}$ based on the UNet output.

3.4. Multimodal Conditional Generation

To control the generations with additional conditioning modes (beyond the FLAME parameters $\mathbf{c}_{f}$), we train different sets of cross-attention layers, following IP-Adapter (Ye et al., 2023). The LDM itself is kept frozen. In this way, we can integrate novel conditioning modes post-LDM training even with limited paired mode-geometry data. We train one set for each of the following conditioning modes: sketches $\mathbf{c}_{1}$, Canny edges $\mathbf{c}_{2}$, 2D landmarks $\mathbf{c}_{3}$, and portrait photos $\mathbf{c}_{4}$. The output of the new cross-attention layers is added to the outputs of the existing LDM cross-attention layers, thereby injecting the new conditioning signal into the generation process:

(4) $$\mathbf{Z}=Attention(\mathbf{Q},\mathbf{K},\mathbf{V})+Attention^{\prime}(\mathbf{Q},\mathbf{K^{\prime}}_{m},\mathbf{V^{\prime}}_{m}),$$

where $\mathbf{Q}$ are the intermediate UNet query features, $\mathbf{K}$ and $\mathbf{V}$ are keys and values for our FLAME conditioning $\mathbf{c}_{f}$ and $\mathbf{K^{\prime}}_{m}$,$\mathbf{V^{\prime}}_{m}$ are keys and values for the newly injected modality $\mathbf{c}_{m}$.

(5) $$\mathbf{K} =\mathbf{c}_{f}\cdot\mathbf{W}_{k},\mathbf{V}=\mathbf{c}_{f}\cdot\mathbf{W}_{v},$$

$$\mathbf{K^{\prime}}_{m} =\mathbf{c}_{m}\cdot\mathbf{W^{\prime}}_{k,m},\mathbf{V^{\prime}}_{m}=\mathbf{c}_{m}\cdot\mathbf{W^{\prime}}_{v,m}$$

Here, $\mathbf{W^{\prime}}_{k,m}$ and $\mathbf{W^{\prime}}_{v,m}$ represent the newly added weights that are updated during training.

Prior to passing each of the above-mentioned conditioning modes to the cross-attention layers, we pass each of them through CLIP (Radford et al., 2021) and extract a 768 dimensional global CLIP feature vector, which serves as our conditioning representation. Following IP-Adapter, we train a small projection network consisting of one linear layer and layer normalization, designed to project the CLIP feature vector into several extra context tokens before injecting it into the cross-attention layers. We use 16 tokens for each of our conditionings to allow for meaningful attention computation.

At inference time, we can control the 3D face geometry generation with any of the modes using the respective set of cross-attention layers and classifier-free guidance (Ho and Salimans, 2021). The strength of the conditioning signal can be increased by increasing the hyperparameter $w$:

(6) $$\hat{\bm{\epsilon}}_{\theta}(\mathbf{z}_{t},\mathbf{c}_{f},\mathbf{c}_{m},t)=w\bm{\epsilon}_{\theta}(\mathbf{z}_{t},\mathbf{c}_{f},\mathbf{c}_{m},t)+(1-w)\bm{\epsilon}_{\theta}(\mathbf{z}_{t},t)$$

To condition only on a newly added mode or to generate geometry unconditionally, the FLAME conditioning $\mathbf{c}_{f}$ is set to its null embedding. Additionally, for unconditional generation, the new cross-attention in Eq. 4, which feeds $\mathbf{c}_{m}$ to the diffusion model, is simply not added to the original attention. Our full method pipeline is visualized in Fig. 2.

3.5. Implementation Details

For our dataset, we crop the full head face geometry to allocate more vertices to the face region, representing 50520 face vertices within each $256^{2}$ UV position map. At $256^{2}$ resolution we can represent reasonably high-resolution face geometry while being able to limit our VAE training time to 8 days using our training dataset of 7752 samples ($\sim$1.4 seconds/iteration; batch size 8). We use a learning rate of 4.5e-6 and a codebook size of 8192. Note that for training our VAE, we did not use the data augmentations described in Section 3.1 as the autoencoder was already able to reconstruct test geometries with high accuracy when trained only on the studio dataset. Next, we train our LDM for 4 days ($\sim$1.6 seconds/iteration; batch size 12) with a learning rate of 1e-4 and diffusion timesteps $T=1000$. We utilize geometry data augmentations with corresponding FLAME fits during training (+200k samples) to allow for better generalization across identities during generation. Afterwards, we train each set of cross-attention layers with a learning rate of 1e-4 for 6 days ($\sim$3.3 seconds/iteration; batch size 24) while keeping the LDM frozen. With a probability of 0.05, we randomly set either $\mathbf{c}_{f}$ or $\mathbf{c}_{m}$ or both to their null embeddings during training. This step enables classifier-free guidance at inference. Note that we do not add augmented geometry data to train the new cross-attention layers because we do not have access to paired mode-geometry data for modes such as portrait photos. All experiments were run on a single RTX A6000 GPU. Also note that the VAE and the LDM have to be trained only once and novel conditioning modes can be added by training only the new cross-attention layers. In the next section, we show and evaluate our results either on a separate held-out validation set which contains 18 different identities displaying 24 expression each (432 samples) or on unconstrained in-the-wild test data. Unless mentioned otherwise, we generate every result by running DDPM sampling steps $S=50$ with conditioning strength $w=1$. In total, generating a geometry sample takes $\sim$11 seconds.

4. Results

We now show several results and applications of our new multimodal conditional 3D face geometry generation method. We begin by demonstrating control over the generated facial geometry using FLAME’s identity and expression parameters (Section 4.1). We then demonstrate multimodal conditioned geometry generation by guiding the denoising process using sketches, sparse 2D landmarks, Canny edges and text. We evaluate the effectiveness of these different modalities in guiding the generated geometry both qualitatively and by using quantitative metrics in Section 4.2. In addition to using different conditioning modes, we show how one can also spatially restrict the guidance to a particular face region to perform precise geometry edits in Section 4.3. Finally we generate dynamic facial performances by guiding our model from video inputs and demonstrate that our method can produce facial animations that are stable across time (Section 4.4).

4.1. Identity and Expression Parameter Conditioning

The base conditioning used to train our geometry generator are the identity and expression parameters from the FLAME model (Li et al., 2017). We use 300 identity parameters ($\bm{\beta}$), 100 expression parameters ($\bm{\psi}$) and 3 jaw pose ($\bm{\theta}$) parameters. We combine these FLAME parameters into a 403-dimensional conditioning vector which we pass through a 3-layer MLP with Leaky ReLU activation functions prior to injecting it into the diffusion model via cross-attention.

Recollect that we do not use the geometry from the FLAME model itself to train our diffusion model. Instead we fit the FLAME model to the high quality facial geometry from Chandran et al. (2020) only to obtain identity and expression parameters, and train the diffusion model directly on the geometry captured by Chandran et al (2020). As there is a loss in geometric detail during this FLAME fitting step, our diffusion model is expected to compensate for the information missing in the FLAME parameters, but present in the ground truth facial geometry. In other words, our geometry generator has to re-interpret the FLAME parameters such that it can re-produce the training data and therefore does not memorize the FLAME parameter space exactly.

We visualize geometries generated by our model for unseen FLAME parameters in  Fig. 4. As our underlying mesh topology is different from FLAME and represents $\sim$10-times more vertices, it can express a greater level of detail that is not present in the lower resolution FLAME mesh. This high resolution detail is captured and reproduced by the denoising process. Therefore, by simply varying the noise seed, one can obtain variations of the FLAME-conditioned geometry, each of which contain different mid/high frequency details.

Input	seed $1$	seed $2$	seed $3$

Figure 4. Changing the noise input while conditioning on the FLAME parameters does not affect the identity and expression of the generated geometry, but only the stochastic details that are added on top. Our model can capture richer geometric detail that is not present in the original FLAME mesh, while still respecting FLAME’s identity and expression parameters. Each row shows a different set of FLAME parameters, with the corresponding FLAME mesh visualized in the first column. We generate samples from three different seeds which are shown in the remaining columns.

Disentangling Identity and Expression. Our diffusion model also preserves the disentanglement between facial identity and expression that is present in FLAME. In Fig. 5, we show how a smooth interpolation of FLAME’s identity and expression coefficients results in a smooth, yet nonlinear, interpolation of our generated geometry. We can observe that the facial expression remains fixed when interpolating between identities, and vice versa (please also refer to the supplemental video). To eliminate the randomness in the generation, we used the same initial noise to generate the interpolated geometries along with DDIM sampling (Song et al., 2021).

Extreme Expressions Due to the limited number of extreme expressions in our training datasets, our model can produce suboptimal results when conditioned on FLAME parameters corresponding to extreme expressions (Wide open mouth for example). However, this is purely a data limitation and could be resolved by having a larger and more balanced dataset, by oversampling expressions during training or by weighting the loss towards focusing more on extreme expressions.

Figure 5. Identity and expression disentanglement. Traversing the first dimension in FLAME’s identity space leads to smooth changes in our generated face geometry (rows 1 and 2). Similarly a linear interpolation of FLAME’s expression parameters results in a smooth, yet nonlinear, interpolation of facial expression as seen in rows 3 and 4.

4.2. Multimodal Conditioning

Beyond the underlying FLAME-based control, we introduce additional conditioning modes to control our diffusion model following Section 3.4. We now discuss the results of facial geometry generation by conditioning our diffusion model on sketches, sparse 2D landmarks, Canny edges, portrait RGB photos and text. To create facial geometry from text input, we leverage the fact that CLIP (Radford et al., 2021) embeds both RGB images and text into a shared latent space. Therefore, instead of training separate cross-attention layers for text, we re-use those learned from our portrait images and directly feed them with the CLIP vectors obtained from a text prompt.

Geometries generated for different conditionings from each of these additional modes are shown in Fig. 6. While our generated geometry follows the identity and expression seen in the input conditioning signal, the degree to which the geometry can match the conditioning varies from mode to mode. For example, as portrait photos contain the most amount of information about a face when compared to sketches, Canny edges, landmarks or text, geometries generated by conditioning our model on portrait images tend to more tightly constrain the generated identity and expression (also see Table 2).

Portrait
Output
Sketch
Output
FLAME
Output
Edges
Output
Landmarks
Output
Text
Output

Figure 6. Multimodal conditional generation results on our validation dataset, including conditioning on portrait images, sketches, FLAME parameters, Canny edges, 2D landmarks and text. The FLAME parameter inputs are visualized as meshes. Across all conditioning modes, our model captures the facial identity and expression reasonably well.

V2V error	Mean $\downarrow$	Median $\downarrow$	Std $\downarrow$
No augmentations	4.093	3.692	2.170
With augmentations	3.757	3.352	2.085

Table 1. We compare the 3D face geometry generated by diffusion models that were trained with and without 3D data augmentations. We measure the vertex-to-vertex error (V2V) in mm between FLAME parameter conditioned generations and ground truth geometry on neutral shapes from our validation set. The model trained using data augmentation is able to capture unseen identities better. Results are averaged over three different seeds.

V2V error	Mean $\downarrow$	Median $\downarrow$	Std $\downarrow$
2D landmarks	6.489	5.765	2.314
Canny edges	5.938	5.488	1.897
Sketch	5.630	5.034	2.099
Portrait photo	5.394	4.874	2.025
FLAME parameters	4.772	4.462	1.317

Table 2. We report the vertex-to-vertex error (V2V) in mm on 178 shapes from our validation set for different types of conditioning signals. Conditions that are more descriptive of the end facial geometry like FLAME parameters, and portrait images achieve a lower error than others.

To control the strength $w$ of the guiding condition, we make use of classifier-free guidance (Ho and Salimans, 2021) following Eq. 6. Fig. 7 demonstrates the effect of different guidance strengths for each condition. Increasing the guidance strength increases the effect that the conditioning has on the resulting geometry. For example, in Fig. 7, the expression of the generated geometry of the subject in the first row displays stronger wrinkles, and a closer match to the portrait image when setting $w=3$ compared to setting it to $w=1$. Finally, despite training on purely studio data, we show how our model responds to conditionings derived from in-the-wild data in Fig. 8.

Mode	Input	$w=0$	$w=1$	$w=3$
Portrait
Sketch
Flame
Edges
Landmarks

Figure 7. By varying the guidance strength $w$, we can control the extent to which our conditioning signals affect the generated geometry. Setting $w=0$ results in unconditional generation, while $w>=1$ results in conditional generation.

Figure 8. Generation using conditioning signals obtained from in-the-wild test data (Portrait images top row, Canny edge maps bottom row). Our model produces reasonable facial geometry from in-the-wild conditions despite being trained only on studio data.

4.2.1. Quantitative Evaluation

To evaluate the effectiveness of each of our conditioning modes in guiding the generated geometry towards ground truth scans, we compute the Euclidean error between the generated geometry and the ground truth geometry for each conditioning mode. In Table 2, we report the vertex-to-vertex (V2V) error of 178 shapes from our validation set for each type of conditioning. We observe that conditioning signals that are more descriptive, such as FLAME parameters or portrait photos, obtain a lower error when compared to signals that are less descriptive of the final geometry (2D landmarks, Canny edges and sketches). For the base FLAME parameter conditioning, we also visualize the error maps on the geometry in Fig. 9. We also highlight the importance of training with 3D geometric augmentations in Table 1 where our diffusion model trained with augmentations outperforms one trained without data augmentations, when evaluated on only neutral expressions from our validation set.

Figure 9. Error maps on our validation set. The first row shows the FLAME mesh as generated by the FLAME face model from the input FLAME parameters. The second row shows the generated geometry from our model conditioned on the respective FLAME parameters. The third row visualizes the error from our conditional generations to the original scanned geometry in our validation set. The first four columns are various identities, while the last five columns are different expressions of the same subject.

Finally, as our method involves learning additional modes of conditioning on top of an underlying FLAME conditioned diffusion model, we can also use more than one conditioning signal at inference time to guide the generation according to Eq. 6. In Fig. 10, we show how combining both FLAME parameter and portrait image conditioning lowers the vertex error on a validation sample, as the denoising UNet now has access to more information about the desired identity and expression.

Figure 10. Multimodal conditioning using portrait photo and FLAME parameter conditioning separately and simultaneously. The first row shows the conditioning inputs. The second row shows the generated face geometry. The third row shows the error map when compared with the original geometry from our validation set.

4.3. Geometry Editing

The latent space of our autoencoder preserves the spatial layout of the original UV position map, much like how the latent space of the image autoencoder in text-to-image models (Rombach et al., 2022) preserves the spatial layout of the encoded image. As a consequence, by masking regions in the latent UV position map corresponding to regions we wish to modify, and by denoising the masked regions, one can apply intuitive edits to particular regions of the facial geometry. Please refer to RePaint (Lugmayr et al., 2022) for more details on the masking process. Even when using masks with sharp boundaries, the denoising process can take care of smoothly interpolating at the mask boundaries. This mask-based editing of facial geometries using our model is shown in  Fig. 11. In the top row of  Fig. 11 we mask the nose region of the latent position map, such that it remains fixed throughout the multiple steps of denoising. We then generate multiple geometry samples by varying the initial noise input to the diffusion model. The noise predicted at each denoising step is multiplied with the nose mask before being fed as input to the denoising UNet for the next time step. This denoising procedure leads to generations where the generated samples all share the same nose shape, but vastly differing facial identities. In the bottom row of  Fig. 11, we show the result of inverse masking, where the face shape is held fixed while allowing the nose shape to change. Our model produces meaningful results in both cases.

Figure 11. Unconditional face shape editing (inpainting). In the top row, the nose is kept fixed, while we sample the remaining regions unconditionally. In the bottom row, we sample the nose region unconditionally while keeping the other regions fixed.

The editing of facial geometry by masking the latent position map can also be guided further with user conditions. In  Fig. 12, we show an example of an interactive sketching workflow, where an artist can progressively edit a generated geometry by modifying one region at a time. For this application, we start from an initial 3D geometry obtained from our model by conditioning it on a hand-drawn sketch (a). An artist further edits the sketch, such as to open the mouth for instance (b). We automatically mask the latent position map such that only the regions corresponding to the edited sketch lines are allowed to change. The masked regions of the latent position map are populated with noise and denoised through our model while using the edited sketch as the conditioning signal. When used in this fashion, our model can perform local edits on the initial geometry to reflect the changes made by the artist. Additional examples of how subsequently modifying the mouth (c), face shape (d) and nose (e) are translated to changes in the generated 3D face geometry are shown in the remaining columns of Fig. 12. Please see the supplemental video for another illustration of this workflow.

\begin{overpic}[width=86.72267pt,trim=0.0pt 0.0pt 0.0pt 10.0pt,clip]{figures/% sketchedit/inputs_0001.png} \end{overpic}	\begin{overpic}[width=86.72267pt,trim=0.0pt 0.0pt 0.0pt 10.0pt,clip]{figures/% sketchedit/inputs_0002.png} \put(0.0,-9.0){\includegraphics[width=26.01613pt,trim=150.0pt 200.0pt 150.0pt % 100.0pt,clip]{figures/sketchedit/masks_render_0002.png}} \end{overpic}	\begin{overpic}[width=86.72267pt,trim=0.0pt 0.0pt 0.0pt 10.0pt,clip]{figures/% sketchedit/inputs_0003.png} \put(0.0,-9.0){\includegraphics[width=26.01613pt,trim=150.0pt 200.0pt 150.0pt % 100.0pt,clip]{figures/sketchedit/masks_render_0003.png}} \end{overpic}	\begin{overpic}[width=86.72267pt,trim=0.0pt 0.0pt 0.0pt 10.0pt,clip]{figures/% sketchedit/inputs_0004.png} \put(0.0,-9.0){\includegraphics[width=26.01613pt,trim=150.0pt 200.0pt 150.0pt % 100.0pt,clip]{figures/sketchedit/masks_render_0004.png}} \end{overpic}	\begin{overpic}[width=86.72267pt,trim=0.0pt 0.0pt 0.0pt 10.0pt,clip]{figures/% sketchedit/inputs_0005.png} \put(0.0,-9.0){\includegraphics[width=26.01613pt,trim=150.0pt 200.0pt 150.0pt % 100.0pt,clip]{figures/sketchedit/masks_render_0005.png}} \end{overpic}
\begin{overpic}[width=86.72267pt,trim=0.0pt 100.0pt 0.0pt 10.0pt,clip]{figures% /sketchedit/blender_render_0001.png} \put(2.0,5.0){a)} \end{overpic}	\begin{overpic}[width=86.72267pt,trim=0.0pt 100.0pt 0.0pt 10.0pt,clip]{figures% /sketchedit/blender_render_0002.png} \put(2.0,5.0){b)} \end{overpic}	\begin{overpic}[width=86.72267pt,trim=0.0pt 100.0pt 0.0pt 10.0pt,clip]{figures% /sketchedit/blender_render_0003.png} \put(2.0,5.0){c)} \end{overpic}	\begin{overpic}[width=86.72267pt,trim=0.0pt 100.0pt 0.0pt 10.0pt,clip]{figures% /sketchedit/blender_render_0004.png} \put(2.0,5.0){d)} \end{overpic}	\begin{overpic}[width=86.72267pt,trim=0.0pt 100.0pt 0.0pt 10.0pt,clip]{figures% /sketchedit/blender_render_0005.png} \put(2.0,5.0){e)} \end{overpic}

Figure 12. Conditional generation from an input sketch (a), followed by local edits of the mouth (b,c), the face shape (d) and the nose (e). The masks used to constrain the region of modification are shown in the insets.

4.4. Dynamic Generation

Even though our model is only trained with static face shapes, we find that it can generate temporally stable 3D facial geometries when conditioned on per-frame FLAME parameters derived from animation sequences or on CLIP embeddings obtained from individual frames from in-the-wild videos. In Fig. 13, we show the generated 3D face geometry produced by our method when conditioned on various signals derived from videos. To demonstrate the use of sketches as dynamic conditioning, we use a recent face reconstruction technique (Chandran et al., 2023) to track the facial geometry in 3D from an in-the-wild video and then render out 2D sketches using a hand-painted texture map and the tracked geometry. We identify that the only pre-processing required to obtain temporally stable generations from CLIP embeddings is to smooth them with a box filter before using them as the conditioning signal. To further ensure stable generations across time, we use the same noise seed across the video as well as DDIM sampling. We kindly refer to our accompanying video for more dynamic results.

Figure 13. Dynamic geometry generation results given sketch, landmark, FLAME parameters or portrait photos from 4 different input videos as conditionings. Our results change smoothly across time while maintaining a consistent identity rather well. We use the same noise seed across all frames and DDIM sampling to reduce stochasticity of the results.

5. Conclusion

We propose a new framework for 3D facial geometry generation based on latent diffusion models that can be guided using multiple types of conditionings. Our conditional geometry generator operates in a latent geometry space and can produce high resolution UV position maps. It can be seamlessly conditioned on hand-drawn sketches, 2D landmarks, Canny edges, FLAME-parameters, RGB portrait photos and even text; resulting in a comprehensive facial geometry generator that supports many applications like geometry super-resolution, geometry editing, face reconstruction, etc. We train our model from scratch on only static face shapes captured in a studio setting and yet demonstrate that our model can generalize reasonably to in-the-wild conditioning signals, and can also generate facial performances when conditioned on frames from video data. As limitations, we identify that our model can produce geometric artifacts and suboptimal geometry for extreme expressions, especially when controlled using FLAME’s jaw pose parameters. We are hopeful that our work will encourage future research towards building comprehensive facial foundation models that also consider facial geometry.