Controllable Text Generation with Focused Variation

As shown in Figure 1, FVN uses four encoders and one decoder during training: the text-to-content encoder $\text{Enc}^{\textit{TC}}(\cdot)$, the text-to-style encoder $\text{Enc}^{\textit{TS}}(\cdot)$, the content encoder $\text{Enc}^{C}(\cdot)$, the style encoder $\text{Enc}^{S}(\cdot)$, and the text decoder $\text{Dec}(\cdot)$.

Text-to-* encoders The text-to-content encoder $\text{Enc}^{\textit{TC}}(\cdot)$ encodes a text $t$ to a dense representation $z^{C}\in\mathcal{R}^{D}$ while the text-to-style encoder $\text{Enc}^{\textit{TS}}(\cdot)$ encodes a text $t$ to a dense representation $z^{S}\in\mathcal{R}^{D}$: $z^{C}=\text{Enc}^{\textit{TC}}(t)$ and $z^{S}=\text{Enc}^{\textit{TS}}(t)$.

In order to learn disjoint latent spaces for the different attributes we want to model, we train two codebooks, one for content $e^{C}\in\mathcal{R}^{K\times D}$ and one for style $e^{S}\in\mathcal{R}^{N\times D}$. They are shown as $[e^{C}_{1},\dots,e^{C}_{K}]$ and $[e^{S}_{1},...,e^{S}_{N}]$ in Figure 1.

These two codebooks are used to memorize the latent vectors for text-to-content variation and text-to-style variation learned during training. Instead of using the $z^{C}$ and $z^{S}$ vectors as inputs to the decoder, we find their nearest latent vectors in the codebooks $e^{C}_{k}$ and $e^{S}_{n}$ and use those nearest latent vectors for decoding the text instead of the original encoded dense representation. Formally, $k=\operatorname*{argmin}_{i}\|z^{C}-e^{C}_{i}\|_{2}$ and $n=\operatorname*{argmin}_{j}\|z^{S}-e^{S}_{j}\|_{2}$.

Like in VQ-VAE, we use the $l$2-norm error to move the latent vectors in the codebooks $e$ towards the same space of the encoder outputs $z$:

	$\displaystyle\mathcal{L}_{\textit{VQ}}^{C}$	$\displaystyle=\|sg(z^{C})-e^{C}_{k}\|^{2}_{2}+\beta^{C}\|z^{C}-sg(e^{C}_{k})\|^{2}_{2},$		(1)
	$\displaystyle\mathcal{L}_{\textit{VQ}}^{S}$	$\displaystyle=\|sg(z^{S})-e^{S}_{n}\|^{2}_{2}+\beta^{S}\|z^{S}-sg(e^{S}_{n})\|^{2}_{2},$		(2)

where $sg(\cdot)$ stands for the stop gradient operator.

Style and content encoders The content encoder encodes a CMR $c$ treating it as a sequence of tokens and producing a matrix $V^{C}\in\mathcal{R}^{L^{\prime}\times D}$, where $L^{\prime}$ is the length of $c$, from which the last element $v^{C}\in\mathcal{R}^{D}$ is returned. The style encoder encodes a style $s$ and obtains a dense representation $v^{S}\in\mathcal{R}^{D}$ selecting the last element of the matrix $V^{S}\in\mathcal{R}^{L^{\prime\prime}\times D}$. Ultimately, $v^{C}=\text{Enc}^{M}(m)$ and $v^{S}=\text{Enc}^{S}(s)$.

Both sets of vectors, $e$ and $v$ are needed as the former learn to memorize the encoded inputs $z$, while the latter learn regularities in the attributes.

The decoder takes the $e^{C}_{k}$, $e^{S}_{n}$, $v^{C}$ and $v^{S}$, which encode content and style, as input and decodes text $t^{\prime}$. We use an LSTM network to model our decoder and provide the initial hidden state $h_{0}$ and initial cell state $c_{0}$. The initial hidden state is the concatenation of $e^{C}_{k}$ and $e^{S}_{n}$, while the initial cell state is the concatenation of $v^{C}$ and $v^{S}$: $c_{0}=v^{C}\circ v^{S}$ and $h_{0}=e^{C}_{k}\circ e^{S}_{n}$.

When we decode the $l$-th word, we encode the previous word $t^{\prime}_{l-1}$ and pay attention to the encoded sequence of content $v^{C}$ and style $v^{S}$ using the last hidden state as a query. Since both content and style are sequences of words, the attention mechanism can help figure out which part of them is important for decoding the current word. We concatenate the embedded previous output word and the attention output as the input for LSTM $x_{l}$. The LSTM updates the hidden state, cell state and produces an output vector $g_{l}\in\mathcal{R}^{2D}$. Since we want to feed the generated text back to text encoders for additional control, we reduce $g_{l}$ to a word embedding dimension vector $o_{l}$ by a linear transformation. Finally, we map $o_{l}$ to the size of the vocabulary and apply softmax to obtain a probability distribution over the vocabulary.

	$\displaystyle x_{l}$	$\displaystyle=\text{Emb}(t^{\prime}_{l-1})\circ\text{Attn}(h_{l-1},V^{C}\circ V^{S}),$		(3)
	$\displaystyle g_{l},(h_{l},c_{l})$	$\displaystyle=\text{LSTM}\Big{(}x_{l},(h_{l-1},c_{l-1})\Big{)},$		(4)
	$\displaystyle o_{l}$	$\displaystyle=W_{\textit{emb}}\cdot g_{l}+b_{\textit{emb}},$		(5)
	$\displaystyle P(t^{\prime}_{l})$	$\displaystyle=\text{softmax}(W_{V}\cdot o_{l}+b_{V}).$		(6)

The loss for text decoding is the sum of cross entropy loss of each word os $\mathcal{L}_{\textit{Dec}}=-\sum_{l}\log P(t^{\prime}_{l})$.

To ensure the generated text $t^{\prime}$ conveys the correct content and style, we feed them to content and style decoders to perform backward prediction tasks that better control the generator. The decoders contain two components: we first reuse the text-to-content and text-to-style encoders to encode the embedded predicted text $o_{L}$ and obtain latent representations $z^{\prime C}$ and $z^{\prime S}$, and then we classify them to predict content $c^{\prime}$ and style $s^{\prime}$, as shown in the right side of Figure 1: $z^{\prime C}=\text{Enc}^{\textit{TC}}(o_{L})$ and $z^{\prime S}=\text{Enc}^{\textit{TS}}(o_{L})$. $\text{Enc}^{\textit{TC}}(\cdot)$ and $\text{Enc}^{\textit{TS}}(\cdot)$ denote the same text-to-content and text-to-style encoders we defined previously. This design is inspired by work on text style transfer dos Santos et al. (2018).

Both $z^{\prime}$ vectors and $e$ vectors are used by two classification heads $F^{C}$ (multi-label) and $F^{S}$ (multi-class) for predicting content and style respectively in order to force those vectors to encode attribute information. We use $g$ to denote the $g$-th element in the set of possible key-value pairs in the CMR and $m(\cdot)$ to represent an indicator function that returns whether the $g$-th element is in the ground-truth CMR.

	$\displaystyle P\Big{(}y^{C}_{z}(g)=m(g)\Big{)}$	$\displaystyle=F^{C}(z^{\prime C}),$		(7)
	$\displaystyle P(y^{S}_{z}=s)$	$\displaystyle=F^{S}(z^{\prime S}),$		(8)
	$\displaystyle P\Big{(}y^{C}_{e}(g)=m(g)\Big{)}$	$\displaystyle=F^{C}(e^{C}_{k}),$		(9)
	$\displaystyle P(y^{S}_{e}=s)$	$\displaystyle=F^{S}(e^{S}_{n}).$		(10)

The loss for training the two prediction heads is:

	$\displaystyle\mathcal{L}_{\textit{CTRL}}$	$\displaystyle=-\sum_{g}\log P\Big{(}y^{C}_{e}(g)=m(g)\Big{)}-\log P(y^{S}_{e}=s)$
		$\displaystyle-\sum_{g}\log P\Big{(}y^{C}_{z}(g)=m(g)\Big{)}-\log P(y^{S}_{z}=s).$		(11)

Finally, we also adopt vector quantization by mapping each generated word’s representation $o_{l}$ to the word embedding $e^{V}\in\mathcal{R}^{|V|\times D}$ to map the output of the decoder and the input of text encoders in the same space. This is needed because the text-to-* encoders expect as input text embedded using word embeddings, but in this case we are providing $o_{L}$ as input, and without this vector quantization loss, $o_{L}$ will not be in the same space of the embeddings. As a result, there is another VQ loss: $\mathcal{L}_{\textit{VQ}}^{V}=\|sg(o_{l})-e_{v}^{V}\|^{2}_{2}+\beta^{V}\|o_{l}-sg(e_{v}^{V})\|^{2}_{2}$.

The total loss minimized during training is the sum of the losses for decoding the text, predicting the content and style, the VQ-loss from two codebooks, and the VQ-loss for word embedding: $\mathcal{L}=\mathcal{L}_{\textit{Dec}}+\mathcal{L}_{\textit{CTRL}}+\mathcal{L}_{\textit{VQ}}^{C}+\mathcal{L}_{\textit{VQ}}^{S}+\mathcal{L}_{\textit{VQ}}^{V}$.

The whole prediction process is depicted in Figure 2. The trained text decoder expect four inputs: $v^{C}$, $v^{S}$, $e^{C}_{k}$, and $e^{S}_{n}$. At prediction time, only content $c$ and style $s$ are given. We can obtain $v^{C}$, $v^{S}$ by providing $c$ and $s$ to their respective encoder, but we also need to obtain $e^{C}_{k}$ and $e^{S}_{n}$ without input text. At the end of the training phase, we map each content $c\in C$ and style $s\in S$ to the indices in the $e^{C}$ and $e^{S}$ codebooks by first obtaining $z^{S}$ and $z^{C}$ vectors from the training data associated with $c$ and $s$, we find the index of the closest codebooks vectors by $\operatorname*{argmin}_{k}\|e^{C}_{k}-z^{C}\|_{2}$ and $\operatorname*{argmin}_{n}\|e^{S}_{n}-z^{S}\|_{2}$ and count how many times each index $k\in K$ was the closes to each $c\in C$ and likewise for indices $n\in N$ for each $s\in S$. By normalizing the counts, we obtain a distribution $P(K|C)$ for content and a distribution $P(N|S)$ for style. The construction of the two distributions is performed only once at the end of the training phase.

To obtain $e^{C}_{k}$ at prediction time, we select the $k$ vector of the codebook by sampling $k\sim P(K|C=c)$ and likewise to obtain $e^{S}_{n}$ with $n\sim P(N|S=s)$. Sampling from those distributions allows to both focus on specific content and style disjointly by conditioning on them, while at the same time allowing variability because of the sampling (we refer to this procedure as focused variation). $v^{C}$, $v^{S}$, $e^{C}_{k}$, and $e^{S}_{n}$ are finally provided as inputs to the decoder to generate the text $t^{\prime}$. Content and style decoders mentioned in the training section are not needed for prediction.

PersonageNLG contains 88,855 training and 1,390 test examples. We reserve 10% of the train set for validation. There are 8 slots in the CMR and 5 kinds of style: agreeable, disagreeable conscientious, unconscientious, and extravert personality traits. The styles are evenly distributed in both train and test sets. All slots’ values are delexicalized. We model the focused variation distribution of the content by jointly modeling the presence of slot names in the CMR, e.g. $P(K|PriceRange\in c\text{ }and\text{ }FoodType\in c)$, because there are no slot values. Style is modeled as a single categorical variable, e.g. $P(N|s=Agreable)$.

E2E contains 42,061 training examples (4,862 CMRs), 4,672 development examples (547 CMRs) and 4,693 test examples (630 CMRs). Like in the PersonageNLG dataset, there are 8 slots in the CMR. Each CMR has up to 5 realisations (references) in natural language. Differently from PersonageNLG, the CMRs in the different splits are disjoint and the texts are lexicalized. Following the challenge guidelines Dušek et al. (2018), we delexicalized only ‘name’ and ‘near’ keeping the remaining slots’ values. Since the E2E dataset does not have style annotations but has lexicalized texts, we model the CMR in the same way we did for PersonageNLG, but we replace the style codebook with a slot-value codebook that help the text decoder generating the slot values in the CMR. We build the focused variation distribution for every slot-value independently over the codebook indices, e.g. $P(N|s=PriceRange[high])P(N|s=FoodType[French])...$. During prediction we sample codes for each slot value in the CMR and use their average to condition text decoding. This is particularly useful when the surface forms in the output text are not the slot values themselves, e.g. when “PriceRange[high]” should be generated as “expensive” rather than “high”.

In both experiments we use $K=512$ and $N=|V|$ as codebooks sizes. More dataset details are shown in Appendix B Table 12 and Table 13, while details about preprocessing, architecture, and training parameters are provided in Appendix A.

We compare our proposed model against the best performing models in both datasets. All of them are sequence-to-sequence based models. For PersonageNLG, TOKEN, CONTEXT (from Oraby et al. (2018)) are variants of the TGEN Novikova et al. (2017) architecture, while token-m* and context-m* are from Harrison et al. (2019) (which adopt OpenNMT-py Klein et al. (2017) as the basic encoder-decoder architecture). token-* baselines use a special style token to provide style information while context-* baselines use 36 human defined pragmatic and aggregation-based features to provide style information ‘-m*’ indicates variants of how the style information is injected into the encoder and the decoder. For the E2E challenge dataset, TGEN Novikova et al. (2017), SLUG Juraska et al. (2018), and Thomson Reuters NLG Davoodi et al. (2018); Smiley et al. (2018) are the best performing models. They have different architectures, re-rankers, beam search and data augmentation strategies. More details are provided in Appendix C.

The results of the baselines Oraby et al. (2018); Harrison et al. (2019) are taken from their original papers, but it’s unclear if they were evaluated using a single or multiple references (for this reason they are marked with $\dagger$), but since these models are not dependent on sampling from a latent space, we would not expect that to change performance.

We also compare to conditional VAEs: CVAE implements the conditional VAE Sohn et al. (2015) framework. Controlled CVAE implements the controlled text generation Hu et al. (2017) framework. The architecture and hyper-parameters of CVAE and controlled CVAE are the same as FVN.

The FVN ablations used in our evaluation are: (1) FVN-ED does not use the codebooks, only uses the content and style encoders and decoders, and is equivalent to an attention-augmented sequence-to-sequence model; (2) FVN-VQ does not use the content and style encoders and decoders, it directly uses the sampled latent vector for text decoding (3.3); (3) FVN-EVQ does not use content and style decoders; (4) FVN is the full network. Refer to Table 11 for architecture details. All VAEs and FVN variants are evaluated using multiple references because the sampling from latent space may lead to generate a valid and fluent text that n-gram overlap metrics would not score high when evaluated against a single reference.

We evaluate the quality and diversity of the generated text on both dataset. PersonageNLG is style-annotated and delexicalized, so we also report style and content correctness for it.

To evaluate quality in the generated text, we use the automatic evaluation from the E2E generation challenge, which reports BLEU (n-gram precision) Papineni et al. (2002), NIST (weighted n-gram precision) Doddington (2002), METEOR (n-grams with synonym recall) Banerjee and Lavie (2005), and ROUGE (n-gram recall) Lin (2004) scores using up to 9-grams. To evaluate content correctness, we report micro precision, recall, and $F_{1}$ score of slot special tokens in the generated text, with respect to the slots in the given CMR $c$. To evaluate diversity, we report the distinct n-grams of ground-truth and baselines’ examples. For style evaluation, we separately train a personality classifier (with GloVe embeddings, 3 bi-directional LSTM layers, 2 feed-forward linear layers) on the PersonageNLG training data. The macro precision, recall, and $F_{1}$ score of the personality classifier on the test set is 0.996. We use this classifier to evaluate the style of the generated text and report our results in Table 4.

In addition to automatic evaluation, we conducted a crowdsourced evaluation to compare our model against the ground truth on the entire test set. We did not compare our model with baselines since a pilot evaluation on a random sample of 100 data points from the test set suggested that baselines did not produce fluent enough text to compare with FVN. We considered the ground truth to be a performance upper bound and compared against it to find how close FVN is to it. Crowdworkers were presented with a personality and two sentences (one is ground truth and the other one was generated by FVN) in random order, and were asked evaluate A) the fluency of the sentences in a scale from 1 to 3 and B) which of the two sentences was most likely to be uttered by a person with a given personality (more details in Appendix D). This evaluation was conducted on the entire test set consisting of 1,390 data points, 278 per personality, and each data point was judged by three different crowdworkers.

We report the result of Question A in Table 8. For each sentence, we averaged the scores across three judges. The overall performance of FVN is very close to the ground truth (2.81 vs. 2.9), which suggests that FVN can generate text of comparable fluency with respect to ground truth texts.

We evaluated Question B using a majority vote of the three crowdworkers. Considering the overall performance, 50.29% of times human evaluators considered FVN generated text equal or better at conveying personality than the ground truth. This suggests that FVN can generate text with comparable conveyance with respect to ground truth.

More details and a full breakdown on the human evaluation are available in Appendix D.

Tables 2, 3 and 4 show the results on text quality, content correctness, and style. As shown in Table 2, FVN significantly outperforms the state-of-the-art methods (context-m), especially on BLEU and NIST, which evaluate the precision of generated text, with the caveat regarding single or multiple references explained above. We believe this is due to the fact that FVN explicitly models CMR and style, while context-m depends on human-engineered features. Comparing FVN with CVAE and controlled CVAE, which are similar methods that also sample from the latent space, FVN performs better on all the metrics. Human evaluation results in Section 4.3 show that FVN is close to the ground truth in fluency and style.

Regarding the content correctness evaluation in Table 3, FVN overall performs much better than other baselines, especially on the recall score. Methods with explicit control decoders (controlled CVAE and FVN) perform better than CVAE and FVN-EVQ, which suggests that the controlling module is useful to enhance the content conveyance. Regarding the style evaluation in Table 4, all methods have good performance. Style is likely easy to convey in the text (the markers are pretty specific) and easy to identify for the separately trained personality classifier. Nevertheless, FVN is the best performing model. The text diversity comparison in Table 7 shows how FVN and its ablations have a diversity of generated texts with respect to the ground truth texts, but so do VAE-based methods. The combination of these findings suggests that FVN can produce text with comparable or better diversity than VAEs and ground truth, while conveying content and style more accurately.

Comparing with the ablations, the full FVN always performs better than FVN-ED and FVN-VQ, especially on the recall of slot tokens. FVN-VQ is able to precisely generate slot tokens from the CMR, but it cannot generate all required slot tokens, while FVN can generate them with high precision and substantially higher recall. An explanation is that the latent vectors in the content codebook only memorize the representations of texts without generalizing properly to new CMRs: since FVN is able to generate text containing most of the required slots, that text is usually longer than FVN-VQ’s, which also explains why FVN performs better than FVN-VQ on METEOR and ROUGE-L that evaluate the recall of n-grams, and suggests that all encoders and codebooks are indeed needed for obtaining high performance.

The comparison between FVN and FVN-EVQ shows how in some cases FVN-EVQ has higher quality, but FVN obtains better scores on correctness and style, suggesting the additional decoder improves conveyance sacrificing some fluency.

In Table 6, we compare our proposed model and variants against the best performing models in the E2E challenge: TGEN Novikova et al. (2017), SLUG Juraska et al. (2018), and Thomson Reuters NLG Davoodi et al. (2018); Smiley et al. (2018). We can see from the results that FVN performs better than all these state-of-the-art models. The reason of the low performance of CVAE-based methods on the E2E dataset is that the CMR are disjoint in the train and test sets (while in PersonageNLG they are overlapping) and CVAEs struggle to handle unseen CMRs. FVNs performs well because it builds focused variations for each attribute independently instead of the entire CMR.

Table 10 shows texts generated by FVN under the same CMR (given 8 attributes, rare in training data) and extravert style. The first three samples have the same CMR latent vector, but different sampled style latent vectors. The remaining three examples have different sampled CMR latent vectors, but the same style latent vector. In the first three examples, the generated texts and the words representing the extravert style are different (“let ’s see what we can”, “I don’t know”, “you know”). In the latter three examples, the words representing style are similar (“you know”), but the aggregation of attributes is different. These examples suggest that the two codebooks learn disjoint information and that the sampling mechanism introduces the desired variation in the generated texts.

Table 10 shows that FVN learns disjoint content and style codebooks and that the vectors in the codebook can be explicitly interpreted by sampling multiple texts and observing the generated patterns. This is useful because, beyond sampling correct style vectors, we can select the realization of a style we prefer (Table 14 shows linguistic patterns associated with the top codes of each style). These patterns are automatically learnt and suggest that there is no need to encode them with manual features. Conditional VAEs do not provide this capability.

Samples obtained providing the same CMR and style to different models and examples of the linguistic patterns learned by FVN’s style codebook are provided in Appendix E. Diverse samples obtained from FVN by sampling different latent codes are shown in Table 15.