Neural Volumetric Blendshapes: Computationally Efficient Physics-Based Facial Blendshapes

Algorithms that create personalized anatomical models can essentially be distinguished according to two paradigms: heuristic-based and data-driven. Considering heuristic-based approaches, Anatomy Transfer [2] applies a space warp to a template anatomical structure to fit a target skin surface. The skull and other bones are only deformed by an affine transformation. A similar idea is proposed by Gilles et al. [23]. While they also implement a statistical validation of bone shapes, the statistics are collected from artificially deformed bones. In [26, 30], an inverse physics simulation was used to reconstruct anatomical structures from multiple 3D expression scans. Saito et al. [45] simulate the growth of soft tissue, muscles, and bones. A musculoskeletal biomechanical model is fitted from sparse measurements in [48] but not qualitatively evaluated.

There are only a few data-driven approaches because combined data sets of surface scans and MRI, CT or DXA images are hard to obtain for various reasons (e.g. data privacy or unnecessary radiation exposure). The recent work OSSO [32] predicts full body skeletons from 2000 DXA images that do not carry precise 3D information. Further, bones are positioned within a body by predicting only three anchor points per bone group and not avoiding intersections between skin and skull. A model that prevents skin-skull intersections and also considers muscles is based on fitting encapsulating wraps instead of the anatomy itself [34]. However, no accurate algorithm based on medical imaging but a BMI (Body mass index) regressor [41] is used to position the wraps. A much more accurate, pure face model, was developed by Achenbach et al. [1]. Here, CT scans are combined with optical scans by a multilinear model (MLM) which can map from skulls to faces and vice versa. As before, no self-intersections are prevented and only bones are fitted. Building on the data from [1] and following the idea of a layered body model [34], we create a statistical layered head model including musculature that avoids self-intersections.

A variety of techniques for animating faces have been developed in the past [25, 12, 56, 43]. Data-driven models [36, 26, 37], which have recently been significantly improved by deep learning [57, 21, 13, 20, 51, 4], are certainly dominant. Due to their simplicity and speed, linear blendshapes [36] are still most commonly used in demanding applications and whenever no computationally rich hardware is available. Physics-based models have been developed for a long time [50] and avoid artifacts like implausible contortions and self-intersections, but due to their complexity and computational effort, they are rarely used. Hybrid approaches add surface-based physics to linear blendshapes for more detailed facial expressions [6, 7, 16]. However, by construction they can not model volumetric effects.

The pioneering work of Sifakis et al. [50] is the first fully physics-based facial animation. The simulation is conducted on a personalized tetrahedron mesh, which can only be of a limited resolution due to a necessary dense optimization problem. With Phace [27], this problem was overcome by an improved physics simulation. An art-directed muscle model [17, 5, 18] additionally represents muscles as B-splines and allows control of expressions via trajectories of spline control points. A solely inverse model for determining the physical properties of faces was proposed in [29]. Neural soft-tissue dynamics [46, 14] extend the SMPL (Skinned Multi-Person Linear Model) proposed in [40] with secondary motion. Recently, [53, 55, 16] adapted neural soft-body dynamics to learn the physical properties of a particular person. However, these approaches must be retrained for new objects and are slow in inference. With volumetric blendshapes [26], a hybrid approach has been presented that combines the structure of linear blendshapes with physical and anatomical plausibility. We extend this work to neural volumetric blendshapes to make physical plausibility realtime capable while maintaining the control structure of standard linear blendshapes.

We represent a head $\mathcal{H}=\rho_{\mathcal{H}}\!\left(\mathcal{T}\right)$ with neutral expression through a component-wise transformation (see Section 3.5 for details) of a layered head model (LHM) template

$$\mathcal{T}=\!\left(S_{\mathcal{T}},B_{\mathcal{T}},M_{\mathcal{T}},\hat{S}_{\mathcal{T}},\hat{B}_{\mathcal{T}},\hat{M}_{\mathcal{T}}\right),$$

(1)

that consists of six triangle meshes. $S_{\mathcal{T}}$ describes the skin surface including the eyes, the mouth cavity, and the tongue, $B_{\mathcal{T}}$ the surface of all skull bones and teeth, $M_{\mathcal{T}}$ the surface of all muscles and the cartilages of the ears and nose. $\hat{S}_{\mathcal{T}}$ is the skin layer, i.e. a closed wrap enveloping $S_{\mathcal{T}}$, $\hat{B}_{\mathcal{T}}$ the skull layer that envelopes $B_{\mathcal{T}}$, and $\hat{M}_{\mathcal{T}}$ the muscle layer that envelopes $M_{\mathcal{T}}$. Other anatomical structures are omitted for simplicity. The template structures $S_{\mathcal{T}},B_{\mathcal{T}}$ and $M_{\mathcal{T}}$ were designed by an experienced digital artist. The skin, skull, and muscle layers $\hat{S}_{\mathcal{T}},\hat{B}_{\mathcal{T}}$, and $\hat{M}_{\mathcal{T}}$ have the same triangulation and were generated by shrink-wrapping a sphere as close as possible to the corresponding surfaces without intersections. The complete template is shown in Figure 2.

The skull and muscle layers are further massaged such that the quads of all prisms that can be spanned between corresponding faces of the skin, muscle, and skull layer are as rectangular as possible while preserving the original geometries. To that end, we determine for the skull layer

$\displaystyle\arg\min_{X}$

$\displaystyle w_{\mathrm{rect}}{E}_{\mathrm{rect}}\!\left(X,\hat{S}_{\mathcal{T}}\right)+w_{\mathrm{dist}}E_{\mathrm{dist}}\!\left(X,\hat{B}_{\mathcal{T}}\right),$

(2)

within the efficient projective dynamics [11] optimization framework, initializing with $X=\hat{B}_{\mathcal{T}}$. Here, $E_{d}$ is the two-sided Hausdorff distance to the non-massaged shape and

		$\displaystyle E_{r}\!\left(X,\hat{S}_{\mathcal{T}}\right)=$		(3)
		$\displaystyle\sum_{(x_{i}^{0},x_{i}^{1})\in X}\left(90^{\circ}-\angle\!\left(x_{i}^{0}x_{i}^{1}\hat{s}_{i}^{1}\right)\right)^{2}+\left(90^{\circ}-\angle\!\left(x_{i}^{1}x_{i}^{0}\hat{s}_{i}^{0}\right)\right)^{2}$
		$\displaystyle\quad\quad\quad+\left(90^{\circ}-\angle\!\left(\hat{s}_{i}^{0}\hat{s}_{i}^{1}x_{i}^{1}\right)\right)^{2}+\left(90^{\circ}-\angle\!\left(\hat{s}_{i}^{1}\hat{s}_{i}^{0}x_{i}^{0}\right)\right)^{2},$

induces the rectangular prism shapes. After the optimization we set $\hat{B}_{\mathcal{T}}=X$. The same optimization is run for the muscle layer.

The wrapping layers of the LHM allow for at least two significant advantages. On the one hand, they provide a simplified representation of the skin surface, the musculature, and the skull, which we exploit in Section 3.5 for determining the LHM template transformation $\rho_{\mathcal{H}}$. On the other hand, they can be used for topologically and semantically consistent tetrahedralization of the head volume by splitting the prisms between the layers canonically into tetrahedrons.

More precisely, the construction of the LHM also defines a soft tissue tetrahedron mesh $\mathbb{S}_{\mathcal{T}}$ (i.e. between the skin and the muscle layer) and a muscle tissue tetrahedron mesh $\mathbb{M}_{\mathcal{T}}$ (i.e. between the muscle and the skull layer). The massage of the muscle and skull layers ensures nicely shaped tets with minimal shearing. Further, $\mathbb{S}_{\mathcal{T}}$ can be fine-tuned by removing all vertices and connected tets outside of $S_{\mathcal{T}}$, adding the vertices of $S_{\mathcal{T}}$, and connecting them via Delaunay tetrahedralization. The complexities of all template components are given in Table 1.

Building on the LHM representation, we can now introduce neural volumetric blendshapes. For this, the classical concept of linear blendshapes is reviewed first. Thereupon, we derive volumetric blendshapes and the involved physics-based simulations that are in general not real time capable. Finally, we introduce neural volumetric blendshapes (Section 3.3) as an efficient and fast alternative.

For a specific head $\mathcal{H}=\!\left(S_{\mathcal{H}},B_{\mathcal{H}},M_{\mathcal{H}},\hat{S}_{\mathcal{H}},\hat{B}_{\mathcal{H}},\hat{M}_{\mathcal{H}}\right)$, a linear blendshape model consists of $n$ surface blendshapes

$$\!\left\{{S}_{\mathcal{H}}^{i}\right\}_{i=1}^{n}$$

(4)

and determines an unknown facial expression $S_{\mathcal{H}}^{\mathrm{exp}}$ as the linear interpolation

$$S_{\mathcal{H}}^{\mathrm{exp}}=\sum_{i}w_{\mathrm{exp}}^{i}S_{\mathcal{H}}^{i}.$$

(5)

The $w_{\mathrm{exp}}^{i}$ are the blending weights and determine the share of each blendshape in the expression.

The corresponding set of volumetric blendshapes can be defined as

$$\left\{V^{i}_{\mathcal{H}}\right\}_{i=1}^{n}=\left\{\!\left(\nabla\mathbb{S}_{\mathcal{H}}^{i},\nabla\mathbb{M}_{\mathcal{H}}^{i},B_{\mathcal{H}}^{i}\right)\right\}_{i=1}^{n},$$

(6)

where $\nabla\mathbb{S}_{\mathcal{H}}^{i}$, $\nabla\mathbb{M}_{\mathcal{H}}^{i}$ describe soft and muscle tissue deformations as vectors of $3\times 3$ deformation gradients. $B_{\mathcal{H}}^{i}$ describes the skull after rigid motion of jaw and cranium. An anatomically plausible inverse physics model $\phi^{\dagger}$ (see Section 3.4 for details) relates the linear rig to the volumetric rig as

$$V^{i}_{\mathcal{H}}=\phi^{\dagger}\left(S_{\mathcal{H}}^{i},\mathbb{S}_{\mathcal{H}},\mathbb{M}_{\mathcal{H}},B_{\mathcal{H}}\right).$$

(7)

In words, $\phi^{\dagger}$ determines the biomechanical volumetric deformations of $\mathcal{H}$ that cause the skin surface of a facial expression. Vice versa, a forward physics model $\phi$ acts as a left inverse to $\phi^{\dagger}$ and maps volumetric deformations back to surface deformations as

$$\bar{S}_{\mathcal{H}}^{i}=\phi\!\left(V^{i}_{\mathcal{H}}\right).$$

(8)

In reality, ${S}_{\mathcal{H}}^{i}$ is often not an anatomically legal facial expression with respect to $\phi^{\dagger}$. As a consequence, $\phi^{\dagger}$ is built such that $\bar{S}_{\mathcal{H}}^{i}$ is an anatomically reachable expression which, in Euclidean sense, is close but not necessarily equal to ${S}_{\mathcal{H}}^{i}$.

The major advantage of a volumetric blendshape model is that unknown facial expressions calculated as

$$\bar{S}_{\mathcal{H}}^{\mathrm{exp}}=\phi\!\left(\bigoplus_{i}w_{\mathrm{exp}}^{i}\otimes V^{i}_{\mathcal{H}}\right)$$

(9)

can be shaped anatomically more plausible and thus more realistic, provided a suitable choice of interpolation (and extrapolation) functions $\bigoplus$ and $\otimes$. At this, the essential requirement to $\bigoplus$ and $\otimes$ is the biologically necessary volume preservation of the deformation gradients. The common approach would be to to separately interpolate the stretch and the rotation components of the deformation gradients [26, 49] and the positions of the skull bones. Considering the stretching components, volume-preserving interpolation methods [3, 28] have been proposed. Considering the rotation components, quaternion-interpolation satisfies the volume-preservation by construction. However, to the best of our knowledge, it is yet to be discussed how both components should be extrapolated for facial animations if the blending weights do not form a convex combination. In this work, we therefore use a novel hybrid approach that calculates facial expressions as

$$\bar{S}_{\mathcal{H}}^{\mathrm{exp}}=\phi\left(\phi^{\dagger}\!\left(\sum_{i}w_{\mathrm{exp}}^{i}\bar{S}_{\mathcal{H}}^{i},\mathbb{S}_{\mathcal{H}},\mathbb{M}_{\mathcal{H}},B_{\mathcal{H}}\right)\right).$$

(10)

This way, inter- and extrapolation capabilities of linear surface blendshapes can be used while maintaining the advantages of anatomical plausibility through $\phi^{\dagger}$.

Regardless of the construction of $\bigoplus$ and $\otimes$, the calculation of facial expressions is in general not realtime-capable for high-resolution volumetric blendshapes and sophisticated physics models $\phi$ and $\phi^{\dagger}$. We therefore present the neural volumetric blendshape model that like the linear blendshape model (4) consists of a set of (not necessarily anatomically plausible) expression surfaces $\{{S}_{\mathcal{H}}^{i}\}_{i=1}^{n}$. However, unknown anatomically plausible expressions are formed by a neural network $f$ that is trained such that

$$\bar{S}_{\mathcal{H}}^{\mathrm{exp}}-\sum_{i}w_{\mathrm{exp}}^{i}S_{\mathcal{H}}^{i}\approx f\!\left(\sum_{i}w_{\mathrm{exp}}^{i}S_{\mathcal{H}}^{i}-S_{\mathcal{H}},S_{\mathcal{H}}\right).$$

(11)

The structure of Equation (11) enables us to implement $f$ efficiently as fully connected networks. More precisely, both inputs to $f$, the neutral surface $S_{\mathcal{H}}$ and the vector of the differences to the linear interpolation result $S_{\mathcal{H}}^{\mathrm{exp}}=\sum_{i}w_{\mathrm{exp}}^{i}S_{\mathcal{H}}^{i}$, are tokenized with the help of respective encoders. Subsequently, the tokens are processed by a decoder that outputs the vertex-wise deformations from $S_{\mathcal{H}}^{\mathrm{exp}}$ to the anatomically plausible expression $\bar{S}_{\mathcal{H}}^{\mathrm{exp}}$.

The inputs and outputs of $f$ are justified as discussed next. Considering Equation (10) which is approximated by $f$, it seems reasonable to use the same inputs and only learn the evaluation of $\phi(\phi^{\dagger}(\cdot)$. However, the accompanying tetrahedron meshes are significantly higher-dimensional than the corresponding surface meshes and $f$ would be slow downed severely in the inference speed. We therefore expect $f$ to implicitly learn the linking between linear and volumetric blendshapes as shown in Equation (7) as well as the fitting of the LHM. Since we demonstrate in Section 3.5 how the LHM can be fitted given only the neutral surface and also the linking does not need any further inputs, the neutral surface carries sufficient information to omit the tetrahedron meshes. For the second input, the difference vector, the blending weights or the linear interpolation result could alternatively be inserted. Inputting only the weights, however, would considerably limit the flexibility of $f$ because it does not allow the underlying surface blendshapes to be changed after training. On the other hand, inputting the linear interpolation result exhibits disadvantages at training time. The target deltas are mostly Gaussian distributed and can therefore be learned more easily by a neural network [39]. In the same spirit, the output of $f$ is chosen to be the differences to the anatomically plausible expression.

We evaluated alternatives to fully connected networks such as set transformers [35], convolutional networks on geometry images, graph neural networks [47], or implicit architectures [42], but all have exhibited substantially slower inference speeds while reaching a similar accuracy.

Our design of $f$ is not only fast, i.e., runs only 8 ms per frame on a consumer grade Intel i5 12600K, but it is also straightforward to deploy. A single neutral surface of a head is sufficient on which deformation transfer [9] can be applied. Building on this, an anatomically plausible animation can be performed with $f$ as shown in Figure 3. Further, no more volumetric information has to be processed in complex simulation frameworks. Thereby, and because only simple fully connected layers are used, the mesh is portable and easy to use on many different (computationally weak) devices.

Next, the construction of $\phi$ and $\phi^{\dagger}$ is described and subsequently a pipeline for creating a dataset to learn $f$ is structured.

We realize the anatomically plausible inverse physics $\phi^{\dagger}$ and the left inverse $\phi$ as projective dynamics energies $E_{\phi^{\dagger}}$ and $E_{\phi}$, respectively. Starting with $\phi^{\dagger}$, separate terms for soft tissue, muscle tissue, the skin, the skull, and auxiliary components are applied. Considering the soft tissue $\mathbb{S}_{\mathcal{H}}$, we closely follow the model of [27] and impose

$$E_{{\mathbb{S}_{\mathcal{H}}}}=w_{\mathrm{vol}}\sum_{t\in\mathbb{S}_{\mathcal{H}}}E_{\mathrm{vol}}(t)+w_{\mathrm{str}}\sum_{t\in\mathbb{S}_{\mathcal{H}}}E_{\mathrm{str}}(t),$$

(12)

which for each tetrahedron $t$ penalizes changes of volume

$\displaystyle E_{\mathrm{vol}}(t)=\!\left(\det(\text{F}(t))-1\right)^{2}$

(13)

and strain

$\displaystyle E_{\mathrm{str}}(t)=\min_{R\in SO(3)}\left\|\text{F}(t)-R\right\|^{2}_{F}.$

(14)

$\text{F}(t)$ denotes the deformation gradient of a tetrahedron $t$, $R\in SO(3)$ the optimal rotation, and $\left\|\cdot\right\|^{2}_{F}$ the Frobenius norm.

To reflect the biological structure of the skin, we additionally formulate a dedicated strain energy

$$E_{{S_{\mathcal{H}}}}=\sum_{t\in{S}_{\mathcal{H}}}E_{\mathrm{str}}(t)$$

(15)

on each triangle $t$ of the skin.

For the muscle tets $\mathbb{M}_{\mathcal{H}}$, we follow the arguments of [29] that capturing fiber directions for tetrahedralized muscles is in general too restrictive. Hence, only a volume-preservation term

$$E_{{\mathbb{M}_{\mathcal{H}}}}=w_{\mathrm{vol}}\sum_{t\in\mathbb{M}_{\mathcal{H}}}E_{\mathrm{vol}}(t)$$

(16)

is applied. The weight $w_{\mathrm{vol}}$ is set sufficiently high such that the volume-preservation is almost a hard constraint.

The skull is not tetrahedralized as it is assumed to be non-deformable even though it is rigidly movable. The non-deformability of the skull is represented by

$$E_{B_{\mathcal{H}}}\!=\sum_{t\in B_{\mathcal{H}}}\!E_{\mathrm{str}}\!\left(t\right)+\sum_{x\in B_{\mathcal{H}}}\!E_{\mathrm{curv}}\!\left(x,B_{\mathcal{H}}\right),$$

(17)

i.e. a strain $E_{str}$ on the triangles and mean curvature regularization

$$E_{\mathrm{curv}}\!\left(x,B_{\mathcal{H}}\right)=A_{x}\left\|{\Delta x-R\Delta b_{x}}\right\|^{2}$$

(18)

on the vertices of the skull. The matrix $R\in SO(3)$ denotes the optimal rotation keeping the vertex Laplacian $\Delta x$ as close as possible to its initial value $\Delta b_{x}$. The vertex Laplacian is discretized using the cotangent weights and the Voronoi areas $A_{x}$ [10]. We do not model the non-deformability as a rigidity constrain due to the significantly higher computational burden. Like for the volume constraints, the non-deformability of the skull is weighted to be an almost hard constraint.

To connect the muscle tets as well as the eyes to the skull, connecting tets are introduced. For the muscle tets, each skull vertex connects to the closest three vertices in $\mathbb{M}_{\mathcal{H}}$ to form a connecting tet. For the eyes, connecting tets are formed by connecting each eye vertex to the three closest vertices in $B_{\mathcal{H}}$. On these connecting tets, the energy $E_{con}$ with the same constraints as in Equation (12) is imposed as almost hard constraints. By this design, the jaw and the cranium are moved independent from each other though muscle activations but the eyes remain rigid and move only with the cranium.

Finally, the energy

$$E_{\mathrm{inv}}=\sum_{x\in S_{\mathcal{H}}}E_{\mathrm{tar}}\!\left(x,{S}^{\mathrm{exp}}_{\mathcal{H}}\right)$$

(19)

of soft Dirichlet constraints

$\displaystyle E_{\mathrm{tar}}\!\left(x,{S}^{\mathrm{exp}}_{\mathcal{H}}\right)=\left\|{x-{s_{x}}}\right\|^{2},$

(20)

is added, attracting each vertex $x$ of the skin surface $S_{\mathcal{H}}$ to the corresponding vertex $s_{x}$ from the target expression ${S}^{\mathrm{exp}}_{\mathcal{H}}$. As previously mentioned, the expression ${S}^{\mathrm{exp}}_{\mathcal{H}}$ might be anatomically implausible. As a countermeasure, we can impose a maximum strain by balancing $w_{\mathrm{str}}$ with $w_{\mathrm{tar}}$. Thus, together with the almost hard constraints and by the construction of projective dynamics, $\phi^{\dagger}$ always results in a plausible expression $\bar{S}^{\mathrm{exp}}_{\mathcal{H}}$ close to the target. The weighted sum of the aforementioned energies gives the total energy

	$\displaystyle E_{\phi^{\dagger}}=$	$\displaystyle\>\>w_{\mathrm{{\mathbb{S}_{\mathcal{H}}}}}E_{{\mathbb{S}_{\mathcal{H}}}}+w_{\mathrm{{\mathbb{M}_{\mathcal{H}}}}}E_{{\mathbb{M}_{\mathcal{H}}}}+w_{\mathrm{{B_{\mathcal{H}}}}}E_{{B_{\mathcal{H}}}}$		(21)
		$\displaystyle+w_{\mathrm{{{S}_{\mathcal{H}}}}}E_{{{S}_{\mathcal{H}}}}+w_{\mathrm{con}}E_{con}+w_{\mathrm{inv}}E_{inv}$

of the backward model $\phi^{\dagger}$.

The forward model $\phi$ is considerably simpler in structure and is realized as the energy

	$\displaystyle E_{\phi}=$	$\displaystyle{}\sum_{t\in\mathbb{S}_{\mathcal{H}}}E_{\mathrm{dg}}\!\left(t,\nabla\mathbb{S}_{\mathcal{H}}^{\mathrm{exp}}\right)+\sum_{t\in\mathbb{M}_{\mathcal{H}}}E_{\mathrm{dg}}\!\left(t,\nabla\mathbb{M}_{\mathcal{H}}^{\mathrm{exp}}\right)$		(22)
		$\displaystyle+w_{\mathrm{tar}}\sum_{x\in B_{\mathcal{H}}}E_{\mathrm{tar}}\!\left(x,{B}^{\mathrm{exp}}_{\mathcal{H}}\right)+w_{\mathrm{con}}E_{\mathrm{con}}$

that aims to achieve given deformation gradients $\!\left(\nabla\mathbb{S}_{\mathcal{H}}^{\mathrm{exp}},\nabla\mathbb{M}_{\mathcal{H}}^{\mathrm{exp}}\right)$ while matching the positions of the targeted skull ${B}^{\mathrm{exp}}_{\mathcal{H}}$. Here,

$\displaystyle E_{\mathrm{dg}}(t,\nabla\mathbb{T})=\min_{R\in SO(3)}\left\|{\text{F}(t)-RT_{t}}\right\|^{2}_{F}$

(23)

is similar to Equation (14) and attracts the deformation of each tetrahedron $t$ to a corresponding target deformation gradient $T_{t}\in\nabla\mathbb{T}$.

For both models we resolve self-intersections between colliding lips or teeth in a subsequent projective dynamics update. In the second update, colliding vertices are resolved as in [33]. The distinctive feature here is that no collision-gaps can occur after dissolving the self-intersections.

To approximate Equation (10) with $f$, a defining training dataset is required in first place. By the construction of $f$, this training dataset must consist of instances that relate diverse facial expressions created through linear blendshapes to the corresponding anatomically plausible surfaces. This dataset must also cover a variety of distinct head shapes to train $f$ to be as generally applicable as $\phi$ and $\phi^{\dagger}$.

In the following, we describe a pipeline for creating instances of such dataset, which can be roughly divided into two high-level steps. First, in order to evaluate Equation (10) on a reasonable domain, it is necessary to model a head $\mathcal{H}$ from an extensive head-model and determine the corresponding neutral soft and muscle tissue tetrahedron meshes $\mathbb{S}_{\mathcal{H}}$ and $\mathbb{M}_{\mathcal{H}}$. Second, $S_{\mathcal{H}}$ has to be deformed to an expression ${S}_{\mathcal{H}}^{\mathrm{exp}}$ and mapped to the anatomically plausible $\bar{S}_{\mathcal{H}}^{\mathrm{exp}}$. A more detailed algorithmic description is given in Algorithm 1.

Sampling Head Shapes We start the first part of the pipeline by randomly drawing a neutral skin surface $S_{\mathcal{H}}$ from DECA [20], one of the most comprehensive high-resolution face models currently available. More specifically, we randomly draw an image from the Flickr-Faces-HQ [31] dataset and let DECA determine the corresponding neutral head shape. Further, a precomputed mapping is applied to adapt the DECA topology to our template.

Fitting the LHM Next, the template LHM $\mathcal{T}$ is aligned with the skin surface $S_{\mathcal{H}}$ by finding $\rho_{\mathcal{H}}$ that maps each of the template components individually. For this, we rely on a hybrid approach that is largely data-driven but also based on heuristics that ensure anatomic plausibility and avoids intersections.

As the first of the remaining five template meshes, we fit the skin layer by setting

$$\hat{S}_{\mathcal{H}}=\text{rbf}_{S_{\mathcal{T}}\rightarrow S_{\mathcal{H}}}(\hat{S}_{\mathcal{T}}).$$

(24)

The RBF function is a space warp based on triharmonic radial basis functions [8] that is calculated to displace from the template skin surface $S_{\mathcal{T}}$ to the target $S_{\mathcal{H}}$ and is then applied to the template skin layer. By construction, the skin layer will be warped semantically consistent and stick close to the targeted skin surface.

Next, we fit the skull layer $\hat{B}_{\mathcal{H}}$ by invoking a linear regressor $D$ that predicts the distances from the vertices of $\hat{S}_{\mathcal{H}}$ to the corresponding vertices of $\hat{B}_{\mathcal{H}}$ and subsequently minimizing with projective dynamics

	$\displaystyle\arg\min_{X}$	$\displaystyle\hskip 2.84526ptw_{\mathrm{rect}}{E}_{\mathrm{rect}}\!\left(X,\hat{S}_{\mathcal{T}}\right)$		(25)
		$\displaystyle+w_{\mathrm{dist_{2}}}E_{\mathrm{dist}_{2}}\!\left(X,\hat{S}_{\mathcal{H}},D\!\left(\hat{S}_{\mathcal{H}}\right)\right)$
		$\displaystyle+w_{\mathrm{curv}}E_{\mathrm{curv}}\!\left(X,\hat{B}_{\mathcal{T}}\right).$

Here,

$$E_{\mathrm{dist_{2}}}\!\left(X,\hat{S}_{\mathcal{H}},D\!\left(\hat{S}_{\mathcal{H}}\right)\right)=\sum_{x\in X}\!\left(\sqrt{\left\|{x-{s_{x}}}\right\|^{2}}-d_{x}\right)^{2}$$

(26)

ensures that for each vertex $x\in X$ the predicted distances $d_{x}\in D(\hat{S}_{\mathcal{H}})$ is adhered to. Apart from $E_{\mathrm{dist_{2}}}$, the same regularizing terms as in Equation (LABEL:eq::rect) and 18 are used. The optimization is initialized with $X=\hat{S}_{\mathcal{H}}-D(S_{\mathcal{H}})\cdot n(\hat{S}_{\mathcal{H}})$  where $n(\hat{S}_{\mathcal{H}})$ are area-weighted vertex normals. $D$ is trained on the dataset of [22] (SKULLS) that relates MRI skull measurements to skin surface scans. To ease the learning task, we learn the regressor between PCAs of the skin layers and the skin-to-skull-layer distances. Predicted lengths are set to a minimum value if they fall below a threshold, thus, avoiding skin-skull intersections and numerical issues in downstream physics-based simulations. In Figure 4a, the linear regressor training is visualized.

The muscle layer $\hat{\mathcal{M}}_{\mathcal{H}}$ is fitted by positioning its vertices at the same absolute distances between the corresponding skin and skull layer vertices as in the template, and only passing on a small relative amount $w_{\mathrm{rel}}$ of the distance changes compared to the template. This approach assumes that the muscle mass in the facial area is only moderately affected by body weight and skull size.

The skull mesh is placed by setting

$$B_{\mathcal{H}}=\text{rbf}_{\hat{B}_{\mathcal{T}}\rightarrow\hat{B}_{\mathcal{H}}}\!\left(B_{\mathcal{T}}\right).$$

(27)

The properties of the RBF space warp ensure that the skull mesh remains within the skull layer if the layer is of sufficient resolution. The muscle mesh could be placed in a similar fashion but is not needed in our pipeline any further.

Finally, the soft and muscle tissue tetrahedron meshes $\mathbb{S}_{\mathcal{H}}$ and $\mathbb{M}_{\mathcal{H}}$ can be constructed as described in Section 2. On average, the complete fitting pipeline takes only about 500ms for one instance on an Intel i5 12600K processor. Figure 4b) visualizes the overall fitting process.

Sampling Expressions In the second part of the pipeline, the actual training instance $\!\left({S}_{\mathcal{H}}^{\mathrm{exp}},\bar{S}_{\mathcal{H}}^{\mathrm{exp}}\right)$ is created. Beginning with deformation transfer [9] to transfer ARKit¹¹1https://developer.apple.com/ surface-based blendshapes to $S_{\mathcal{H}}$, we create expression $S_{\mathcal{H}}^{\mathrm{exp}}$ by linearly blending the blendshapes with weights that are obtained from 8 around 10 minutes long dyadic conversations recorded with a custom iOS app.

Training Instance Finally, $\bar{S}_{\mathcal{H}}^{\mathrm{exp}}=\phi\!\left(\phi^{\dagger}\!\left(S_{\mathcal{H}}^{\mathrm{exp}},\mathbb{S}_{\mathcal{H}},\mathbb{M}_{\mathcal{H}},B_{\mathcal{H}}\right)\right)$ can be computed. Computing one training instance takes approximately 40 seconds on a AMD Threadripper Pro 3995wx.

The fitting of the LHM is mainly composed of the data-driven positioning of the skull layer and the subsequent heuristic fitting of the muscle layer. We evaluate the crucial fitting of the skull layer with the SKULLS [22] dataset. Since this dataset consists of 43 instances only, a leave-one-out validation is performed in which the vertex-wise L2 errors are measured. The results are compared to the multilinear model as originally used for SKULLS in [1].

Both models cannot achieve a medical-grade positioning with errors between approximately $2\text{\,}\mathrm{mm}$ and $4\text{\,}\mathrm{mm}$. The MLM achieves a higher precision with a mean error of $1.98\text{\,}\mathrm{mm}$ than our approach that dispositions the skull by $3.83\text{\,}\mathrm{mm}$ on average. However, the MLM cannot prevent collisions that might crash physics-based simulations. Also, our fitting algorithm produces large errors only in regions that are of less importance for facial simulations as can be seen in Figure 5. The errors are predominately distributed in the back area of the skull, since here the rectangular constraints of our fitting procedure can presumably no longer be aligned well to the skin layer. The following section demonstrates in downstream physics-based simulations that the prediction quality in the frontal face region is sufficiently adequate for detailed facial animations. Figure 6 displays fitting examples.

To investigate the quality of $\phi^{\dagger}$ and $\phi$ we perform an experiment on nearly 300 high-resolution optical facial expression scans from the proprietary 3DScanstore²²2https://www.3dscanstore.com dataset. First, we fit the LHM to each of the scans using $\phi^{\dagger}$ and store only the resulting muscle deformation gradients and bone movements. Then, in turn, we apply these to the neutral scans of the corresponding individuals through $\phi$ and simulate the soft tissue as $\phi^{\dagger}$ does. Only if our physic models and the previous LHM fitting operate with reasonable precision, do we expect a low L2-loss between the original expression and the simulated expression as well as visually appealing results.

Across all expression scans, we observe an average L2 reconstruction error of only 1.6 mm. The reproduction quality of the physical models is also reflected in the visual results of Figure 7. Further, the accuracy of the LHM fitting investigated in the previous section is again underlined. Additionally, we inspected expression retargeting by applying the extracted muscle and bone transformations to other identities. Again, visual appealing examples can be found Figure 7. The weights used to realize $\phi,\phi^{\dagger}$, and the LHM fitting in this and the following experiments are stated in Table 2.

To train and evaluate $f$, we assemble a dataset of 50000 training and test instances by using the pipeline from Section 3.5. At this, we sample 10000 different head shapes and compute 5 different expressions per head. For training, the Adam optimizer performs 200k update steps with a learning rate of 0.0001. The learning rate is linearly decreased to 0.00005 over the course of training and a batch size of 128 is used. In total, the training specifications result in an approximate runtime of 8 hours on an NVIDIA A6000. The comparatively short training time can straightforwardly be explained by the less noisy training data as usually encountered for image-based deep learning models and the efficient network design. The identities are used 90% for training and 10% for testing.

First, we evaluate the efficiency of $f$. The initial runtime of $\phi$ and $\phi^{\dagger}$ in Equation (10) with 6 global projective dynamics updates as in [26] is about $950\text{\,}\mathrm{ms}$ on a workstation AMD threadripper pro 3995wx with 128 cores (implemented with ShapeOp [19]). Our implementation of $f$ only takes 8ms on a consumer-grade Intel i5 12600K (implemented with PyTorch³³3https://pytorch.org). Moreover, a forward pass on an NVIDIA RTX3090 is calculated in less than one millisecond. Thus, neural volumetric blendshapes are suitable for realtime applications on weaker hardware setups and also offer advantages when many facial animations are to be executed in parallel on a GPU.

The time advantage alone is not useful if $f$ is not an adequate approximation. We evaluate the approximation quality by measuring the mean L2 error on the test dataset. More precisely, we randomly generate 5 training-test splits of the generated dataset and report the average of the test losses after training on each training set. We achieve an average test error of only $0.13\text{\,}\mathrm{mm}$, meaning that we can successfully achieve the approximation target of Equation (11) and $f$ generalizes well across head shapes and facial expressions. Another open question is the temporal consistency of $f$ which is inherent in physics-based simulations. To this end, we invite the reader to watch the attached video in the supplementary material. Finally, visual results of neural volumetric blendshapes as shown in Figure 8 exhibit the same improvements towards more realistic facial expressions as much slower physics-based simulations before. These include the resolution of self-intersections, the simulation of wrinkles, the adherence to anatomical boundaries, and the realism-enhancing constraining of volume preservation. Close-up images of resolved self-intersections can be seen in Figure 9. The only other approach that to the best of our knowledge can animate detailed faces at this speed, requires only a neutral surface as input, and allows for semantically consistent control, are linear blendshapes paired with a variant of deformation transfer [9, 39, 15]. As can be seen in Figure 8, too, neural volumetric blendshapes produce far more immersive facial animations.