Unifying Discourse Resources with Dependency Framework

Borrowing the discourse scheme of PDTB, HIT-CDTB adopts one-predicate two-arguments structure, where a connective serves as the predicate and two text spans as two arguments, as shown in Fig. 1. The connective can be either explicitly identified if it already exists in the original text, or implicitly added by annotators, while the arguments can be phrases, clauses, sentences, or sentence groups. Each connective corresponds to a relation type. In total, there are 4 coarse-grained types (i.e., temporal causal, comparative and extension) and 22 fine-grained types (e.g., temporal is further divided as synchronous and asynchronous). The documents cover multiple domains, such as news, editorials and popular science articles. When labeling each document, the discourse relations are labeled locally by only considering adjacent text spans, so a complete logical structure of the text may not be obtained.

Similar to RST, SU-CDTB represents the inner structure of a text with a hierarchical tree with EDUs as leaves, and uses connectives as intermediate nodes to indicate rhetorical relations. The whole text is first divided into several text spans which are recursively divided until getting EDUs. In SU-CDTB, EDUs are segmented according to the punctuation marks. The nucleus-satellite relation structure in RST is retained through arrows in the trees. Connectives in SU-CDTB are also given some relation attributes. 500 news documents from the Chinese Treebank [Xue et al. (2005] are annotated. A discourse tree is constructed for each paragraph rather than the entire document, with an average of 4.5 EDUs per tree, which are relatively shallow. Besides, the top-down constructed tree cannot cover some particular discourse structures, such as non-adjacent relations. So parsers trained only with this corpus are not very likely to analyze more complex discourse logic.

Sci-CDTB is a small Chinese discourse dependency corpus, where EDUs are directly connected with discourse relations without intermediate levels. Its definition of EDU refers to RST-DT, with some modifications based on the linguistic characteristics of Chinese. The head of each EDU and the relations between them are annotated. Sci-CDTB is a small-scale corpus, only composed of 108 annotated scientific abstracts, so it is hard to support the training of a competitive discourse parser. It is the only Chinese DDS corpus to the best of our knowledge.

Following the work of ?), we implement four Chinese discourse dependency parsers:

• •

  Graph-based Parser adopts Eisner algorithm to predict the most possible dependency tree structure [Li et al. (2014b]. It refers to the graph-based method of syntax parsing, which adopts Eisner algorithm and MST algorithm to predict the most possible dependency tree structure. For simplicity, an averaged perceptron is implemented to train weights.

• •

  Vanilla Transition-based Parser adopts the transition method of dependency parsing proposed by ?), employing the action set of arc-standard system [Nivre et al. (2004]. An SVM classifier is trained to predict transition action for a given configuration.

• •

  Two-stage Transition-based Parser [Wang et al. (2017] first adopts the transition-based method to construct a naked dependency tree, and then uses another SVM to predict relations, which can take the tree-structure as features.

• •

  Bert-based Parser also conducts parsing in a two-stage transition way, but in the second stage it uses a bert-based model, which incorporates BERT [Devlin et al. (2018] with one additional output layer, to identify relation types. It keeps the pre-trained parameters³³3https://github.com/huggingface/transformers and is fine-tuned on our task using Adam.

As for metrics, we use UAS and LAS to measure the dependency tree labeling accuracy without and with relation labels respectively. LAS${}_{\text{O}}$ and LAS${}_{\text{U}}$ denote using the original relation set of each corpus or the predefined unified relation set.

Table 4 compares performance of the benchmark parsers. To give a rough upper bound of the parsing performance, the last row lists the consistency of human annotation by comparing two annotators’ labelling results on 30 documents from each corpus respectively. Division of training, validation and test set is shown in Table 3. The first four methods in the first block show the results of the benchmark parsers which are trained and tested respectively on HIT-CDTB${}_{\text{dep}}$, SU-CDTB${}_{\text{dep}}$ and Sci-CDTB.

Graph-based method is less effective than the others, but it could probably be improved by using other training methods like MIRA. Among the transition-based methods, Bert-based performs the best on SU-CDTB${}_{\text{dep}}$ and Sci-CDTB${}_{\text{dep}}$ with respect to LAS, but is 2.4% lower than Two-stage on HIT-CDTB${}_{\text{dep}}$, probably because HIT-CDTB${}_{\text{dep}}$ is a multi-domain corpus, and feature-based methods are more robust to changes in different domains. Comparing the three corpora, parsing results on Sci-CDTB are the worst because Sci-CDTB is too small for supervised learning. As we expect, SU-CDTB performs the best since it has relatively shallow tree structure and its labeling is highly consistent. With original relations mapped to the unified relation set, LAS${}_{\text{U}}$ results of all the methods on the three corpora have been improved compared with LAS${}_{\text{O}}$, proving that the unified relation set we use is acceptable. Two-stage transition-based method performs better than the vanilla one with respect to LAS due to the addition of tree structural features in relation type prediction.

Two-stage${}_{\text{Uni}}$ and Bert-based${}_{\text{Uni}}$ use Unified${}_{\text{dep}}$ as training data, and are tested on each corpus respectively. From Table 4, we can see that Sci-CDTB has a significant promotion of 7.8% on UAS, while HIT-CDTB${}_{\text{dep}}$ and SU-CDTB${}_{\text{dep}}$ slightly decline. One possible explanation is that documents in HIT-CDTB${}_{\text{dep}}$ are much longer than in SU-CDTB${}_{\text{dep}}$ so their structures are not similar enough to improve each other’s performance. For Sci-CDTB, however, it also has a small numbers of EDUs per text so parsing performance can be improved by learning the augmented data of short trees from SU-CDTB.

For LAS${}_{\text{U}}$, both methods increase on HIT-CDTB${}_{\text{dep}}$, but drop on SU-CDTB${}_{\text{dep}}$. As HIT-CDTB${}_{\text{dep}}$ covers multiple domains and has an average of 28.6 EDUs per text, its annotation is more difficult and the DDS-conversion process is also more complicated. In comparison, with an average of 4.5 EDUs per tree, SU-CDTB only focuses on news domain and its DDS-conversion is relatively easy. As a result, SU-CDTB${}_{\text{dep}}$ keeps a better data consistency than HIT-CDTB${}_{\text{dep}}$, which may explain why the augmented data promotes the results on HIT-CDTB${}_{\text{dep}}$, but introduces noise data for SU-CDTB${}_{\text{dep}}$. As Sci-CDTB is from scientific domain and its relations are much different from those of HIT-CDTB and SU-CDTB, the augmented labeled relations do not bring much improvement on relation labeling.

Overall, Unified${}_{\text{dep}}$ can serve as a discourse dataset for researching cross-domain or long text discourse parsing. However, our unification method may also introduce noise due to difference in text length and domain, so it remains to be considered how to better leverage this unified large corpus to improve the Chinese discourse parsing techniques.