SE-GAN: Skeleton Enhanced GAN-based Model for Brush Handwriting Font Generation

Figure 2 shows our framework with two encoders: the image encoder $E_{i}$ and the skeleton encoder $E_{s}$, which are composed of four residual blocks. The $E_{i}$ is employed to extract character image features from $X_{i}$, including content and style information from source domain whereas the skeleton encoder $E_{s}$ aims to preserve the structure feature from $X_{s}$. To enhance the feature extraction and fusion of two different features, a novel self-attentive refined attention module (SAttRAM) with two variants, SRAM and CRAM, are stacked sequentially to extract the skeleton enhanced image representations. Then, the generator takes the refined image representations which include both content and style information as input to generate the target style image. Following the adversarial training strategy in GANs, we employ two discriminators. The first $D_{i}$ is used to discriminate the target image and generated image. The second $D_{s}$ tries to detect whether the skeleton of generated image is coherent to the skeleton image extracted from the target domain character image. For skeleton extraction, inspired by [19], we adopt an simple but effective skeletonization algorithm to extract the skeleton image by eroding and dilating the binarized character image iteratively.

The previous works [20, 10, 21] demonstrate the effectiveness of class activation map (CAM) in localizing the important regions of input images in both image generation and classification tasks. Inspired by this, CAM can be used to extract style-discriminative attention heatmaps in the font generation task. To obtain style-discriminative features $M(x)$, the decoded feature maps $F(x)\in\mathbb{R}^{C\times W\times H}$ of source font $x$ are first fed into the classifier, a fully connected (FC) layer with weights $\Omega\in\mathbb{R}^{C}$, for domain classification. Then CAM computes the attention heatmaps by linearly weighted summation of all channels:

$\displaystyle M(x)=\sum_{c=k}^{C}\Omega_{k}F(x)_{k}$

(1)

where $M(x)\in\mathbb{R}^{W\times H\times C}$ indicates the attention heatmap at spatial location $H,W$, $\Omega_{k}$ represents the weight for channel $k$ in feature maps, and $F(x)_{k}\in\mathbb{R}^{W\times H}$ represents the feature map of channel $k$ from the last convolutional layer at spatial location $HW$.

However, the CAM lacks the spatial attention and usually leads to over-activation issues for feature capturing [22]. Moreover, it’s difficult to integrate multi-modal features with a CAM module. To refine the style-discriminative feature and integrate multi-modal features, we propose a self-attentive refined attention module (SAttRAM) as shown in Fig 3. We bring self attention as an efficient module to refine the pixel-wise attention heatmaps by capturing spatial dependency. Hence, the refined feature maps can be defined as follows:

$\displaystyle\hat{M}(x)=f(F(x),F(x))g(M(x))+M(x)$

(2)

$\displaystyle f(F(x),F(x))=\sigma(\theta(F(x))^{T})\phi(F(x)))$

(3)

Here $f$ is a pairwise embedding function that computes dot-product pixel affinity as self-refined attention weight normalised by softmax function $\sigma$ in an embedding space. The embedding functions $\theta$, $\phi$ are implemented by individual $1\times 1$ convolution layers, where $\theta(F(x))\in\mathbb{R}^{C1\times WH}$ and $\phi(F(x))\in\mathbb{R}^{C1\times WH}$. The function $g$ reshapes the input feature $F(x)$ to $g(F(x))\in\mathbb{R}^{WH}$, all of which are aggregated with this similarity weights given by function $f(F(x),F(x))\in\mathbb{R}^{C\times WH}$. This self-refined attention weight is normalised by softmax function $\sigma$ and the output is $\hat{M}(x)\in\mathbb{R}^{W\times H\times C}$.

To handle the multimodal input features for image generation, we build two variant attention units on top of the refined attention feature maps, namely the self-refined attention module (SRAM) unit and the cross-refined attention module (CRAM) unit. SRAM takes a group of images features $x_{i}=E_{i}(x)$ as input to obtain attended features, since image features are basic information in image translation. CRAM catches intra-modal interactions between image features $x_{i}=E_{i}(X_{i})$ and skeleton features $x_{s}=E_{s}(X_{s})$, which can further refine the feature maps extracted from character images.

$\displaystyle\hat{M}(x_{i},x_{s}))=f(F(x_{i}),F(x_{s}))g(M(x_{i}))+M(x_{i})$

(4)

In Equation 4, the character encoding feature is embedded into the residual space by function $g$. Whereas $f$ represents the pixel-level feature aggregation between image feature $x_{i}$ and skeleton feature $x_{s}$.

We design two discriminators for learning target font style. Following the traditional setting, the first discriminator $D_{i}$ is employed to calculate how similar the generated image is to the font image of target domain, and the encoder for $D_{i}$ also exploits the refined attention feature maps as mentioned in previous section. Under the assumption that the generated character image should have similar skeleton to its target font image, we also design an extra skeleton discriminator $D_{s}$. Considering that the skeletonization is a non-differential function, we first apply a pre-trained skeleton generator (skeletonGAN) to generate the corresponding skeleton from a given character image. Then the skeleton discriminator $D_{s}$ is applied to distinguish the skeleton of generated image from that of ground truth image. For skeletonGAN, we train a simple CycleGAN model [8] without SAttRAM module, as skeletonization task appears to be much easier than the character generation task. With the help of skeleton discriminator $D_{s}$, the model is encouraged to preserve the content and structure of character during training, meanwhile, $D_{s}$ is also served as a regularizer for SE-GAN to prevent model from overfitting.

The loss function of SE-GAN contains content loss, adversarial loss, cycle-consistency loss and classification loss. The content loss consists $L_{con}$ of two parts: the pixel loss $L_{pix}$ which forces the generated image $G_{F}(X_{i},X_{s})$ to be similar to the target image $Y_{t}$, and the skeleton consistency loss $L_{sc}$ which ensures the skeletons consistency between $SG(G_{F}(X_{i},X_{s}))$ and $SG(Y_{i})$.

$\displaystyle L_{pix}(G_{F})=\mathbb{E}_{X}[||G_{F}(X_{i},X_{s})-Y_{t}||_{1}]$

(5)

$\displaystyle L_{sc}(G_{F},SG)=\mathbb{E}_{X}[||SG(G_{F}(X_{i},X_{s}))-SG(Y_{t})||_{1}]$

(6)

where the $SG$ represents the pre-trained skeletonGAN.

The cycle-consistency loss $L_{cycle}$ is identical to that used in [8], which guarantees that the cycle transformation is able to bring the image back to the original state. In order to distinguish that the image $X_{i}$ belongs to source or target style $y_{cls}$ and promote the style transformation in the refined attention module, we use an auxiliary loss $L_{cls}$ following [10].

Besides, the adversarial loss $L_{adv}$ combines discriminative loss $L_{D_{i}}$ and skeleton-level discriminative loss $L_{D_{s}}$, and these two losses aim to catch different properties of the desired generated image $X_{i}$.

	$\displaystyle L_{D_{i}}(G_{F},D_{i})=$	$\displaystyle\mathbb{E}_{Y}[logD_{i}(G_{F}(X_{i},X_{s}))]$		(7)
		$\displaystyle+\mathbb{E}_{X}[log(1-D_{i}(G_{F}(X_{i},X_{s})))]$

	$\displaystyle L_{D_{s}}(G_{F},D_{s})=$	$\displaystyle\mathbb{E}_{Y}[logD_{s}(Y_{s})]$		(8)
		$\displaystyle+\mathbb{E}_{X}[log(1-D_{s}(SG(G_{F}(X_{i},X_{s}))))]$

where $D_{i}$, $D_{s}$ are pixel-level discriminator and skeleton discriminator respectively.

Finally, we jointly train the generators, discriminators, and classifiers by using the full objective as follows:

$\displaystyle\underset{G_{F},G_{B},RAM_{p}}{min}\underset{D_{1},D_{2}}{max}\lambda_{1}{L}_{adv}$

$\displaystyle+\lambda_{2}{L}_{cycle}+\lambda_{3}L_{cls}+\lambda_{4}L_{con}$

(9)

where $\lambda_{i}$ controls the weights of different losses. Here, we omit the corresponding backward loss functions for simplicity because they are also defined in the same way.

As the deficiency of public brush handwriting font generation dataset, we collect a large-scale image dataset for experiments with six different styles. The statistics of each style is presented in Table 1. The total number of images is 15,799, and the size of each subset ranges from 1,419 to 3,958. During experiments, we split each subset into train/dev/test set by ratio of 8:1:1. We choose the standard print font Liukai¹¹1https://www.foundertype.com/index.php as the source domain, and take each of the six styles as the target domain respectively. We adopt content accuracy [3] and Fréchet Inception Distance (FID) [23] as evaluation metrics. For baselines, we compare our model with four representative font generation models, including zi2zi [1], CycleGAN [14], StarGAN [15], and DGFont [12].