Building a Hebrew Semantic Role Labeling Lexical Resource
from Parallel Movie Subtitles

This work is part of a larger effort to develop a Hebrew FrameNet [Hayoun and Elhadad (2016]. FrameNet is an annotation scheme and a lexical database inspired by frame semantics [Fillmore (1976]. Fillmore presented the concept of a ‘semantic frame’ as

A system of concepts related in such a way that to understand any one of them you have to understand the whole structure in which it fits.

?) introduced a linguistic resource called FrameNet, on which work is ongoing to this day. The purpose of FrameNet is to realize the idea of frame semantics in English, by building a lexical database of annotated examples of how various words are used in actual texts, grouped by semantic frame.

The project defines a formal structure for semantic frames, and various relationships among them ?). The FrameNet lexical database is comprised of the following elements, which defines useful terminology for all SRL resources:

• Frames. Each frame contains a list of frame evoking words, such as “bought” in the sentence “John bought a car from Jane.” These are known as Frame Evoking Elements (FEEs) or Lexical Units (LUs). Additionally, each frame defines a list of participants and a list of constraints on and relationships between these participants. The participants are called Frame Elements (FEs).

• Lexical Units. Formally, an LU is a word lemma paired with a coarse part-of-speech tag and is unique within its frame. For example, both the words bought and buying are represented by the LU buy.v in the Commerce_buy frame.

  In the FrameNet formalism, LUs can have almost any part-of-speech tag. As an example consider the noun purchase (as in “a purchase was made”), which is one of the LUs in the Commerce_buy frame. Verbs, however, are the most common LUs. In this work we only consider verbal LUs.

  A target is any constituent which evokes a frame. It is the instance of an LU in a given piece of text. For example, both buying and bought could be targets which evoke the Commerce_buy frame and they represent the LU buy.v.

• Frame Elements. FrameNet classifies frame elements in terms of how central they are to the frame. The two primary levels of centrality are labeled “core” and “peripheral”.

  An FE is classified as a core FE if it instantiates a conceptually necessary component of a frame, while making a frame unique. For example, in the Commerce_buy frame, the FEs Buyer and Goods are considered core elements, while the FE Money (representing the thing given in exchange for the goods) is not.

  In determining which FEs are core in their frame, a few formal properties are considered which may provide evidence for core status:

  • –

    When an element must be explicitly expressed, it is core. For instance, resemble requires two entities to compare.

  • –

    If an element receives a definite interpretation when omitted, it is core. For example, the sentence “John arrived.” is incomplete if the goal location at which John arrived cannot be inferred from context.

  FEs that do not introduce additional, independent or distinct events from the main reported event are classified as peripheral. Peripheral FEs mark notions such as time, place, manner, degree of the main event represented by the frame.

FrameNet in English is currently at version 1.7 which consists of 1,221 frames, 13,572 LUs, and 11,428 FEs. For training purposes, there are 10,147 sentences in 107 fully-annotated texts.

A great effort is made to expand FrameNet to other languages. As of today, FrameNets have been developed in Finnish [Lindén et al. (2017], Spanish [Subirats and Petruck (2003], German [Burchardt et al. (2006], Japanese [Ohara et al. (2004], Chinese [You and Liu (2005], Korean [Nam et al. (2014], Brazilian Portuguese [Torrent and Ellsworth (2013], Swedish [Borin et al. (2010], French [Candito et al. (2014], Danish [Bick (2011], Polish [Zawisławska et al. (2008], Italian [Tonelli and Pianta (2008], Slovenian [Lönneker-Rodman et al. (2008], and of course, Hebrew [Petruck (2005, Hayoun and Elhadad (2016].

The methods to develop FrameNets presented in these papers are quite similar: assume the universality of the English frame inventory, and under that assumption, tag, either manually or semi-automatically, sentences in the desired language using this inventory. Almost every language has its specific corner cases where the English frames are insufficient. In those cases, usually new frames are created specifically for the language in question. An example of a corner case of Hebrew is multi-word lexical units like give up and turn in - in Hebrew, these LUs might not appear as contiguous words. This case was solved by allowing annotation of discontinuous units.

The idea of annotation projection is used in most research addressing cross-lingual SRL and linguistic annotation [Yarowsky et al. (2001, Padó (2007, Padó and Lapata (2009, van der Plas et al. (2011]. In this line of work, we start with a resource-rich source language, usually English, and a parallel corpus of English and a resource-poor target language. The English sentences are annotated using an automatic tool, a semantic role labeling model in our case, and using word alignment, the annotations are projected to the target language. This process is called “direct projection.” The idea of “filtered projection” is introduced in ?): only alignments which satisfy the suggested filters are kept, while the rest are discarded. Such filters include verb filter (discard sentences where the predicate is not aligned to a verb), and translation filter (discard sentences where the aligned predicate is not a translation of the source predicate). We find in this work that filtered projection is essential to produce reliable annotations in Hebrew.

In the rest of the paper, we present related work in the field of cross-lingual SRL annotation projection and the method we used to construct a Hebrew SRL dataset starting from a large parallel corpus of English/Hebrew sentence pairs. We eventually train a Hebrew SRL system on the data we produce and report on its performance on the automatically generated data.

The OpenSubtitles 2016 dataset [Tiedemann (2012] provides aligned data in 62 languages from movie subtitles. It specifically includes 23.7 million English-Hebrew sentence pairs. Due to the nature of the dataset, some of the pairs are very noisy. Examples of such noise include special Unicode characters unrelated to the text, e.g., musical note symbol to let hearing impaired viewers know that a song is playing, joined words (either typos or OCR artefacts) such “Amanonce” instead of “A man once,” and pairs in which the Hebrew part is actually found in English, not translated.

Given the large number of available sentence pairs, we filter noisy pairs in the dataset. The removal of noise consists of running fastText’s language detection model [Joulin et al. (2016b, Joulin et al. (2016a] on each pair, and keeping only English-Hebrew pairs. After this first filtering, we have 22.4M sentence pairs.

The importance of Hebrew preprocessing in our work is twofold: (a) we use features from the dependency parse tree later on in the pipeline, and (b) it acts as a quality gate for our data.

The preprocessing pipeline consists of morphological analysis, morphological disambiguation, and dependency parsing, all using the YAP system [More et al. (2019]. In Hebrew, common words including prepositions, conjunctions and articles are written in an aggregated manner. For example, the written form for the phrase “in the house” (be ha bayit) will be written as a single token (babayit). For the goal of SRL, it is particularly relevant to segment such compound tokens so that prepositions and conjunctions can be properly aligned with their English counterparts and projected with the corresponding English sentences.

Before segmentation, the data includes 118M tokens from 2.4M word types. We predict that after segmentation, the number of tokens will rise, but the number of word types will be significantly lower. Indeed, after segmentation we are left with 188M tokens and 0.89M word types. This is a surprising number, as one would expect the number of types in Hebrew to be approximately 200-300K, not 890K. A quick check shows that 782K word types appear less than ten times in the entire dataset, leaving us with a more reasonable vocabulary of 112K word types.

Naturally, segmentation also affects sentence length, with 5.13 words before segmentation, and 8.17 after. Both of these numbers indicate that the sentences in the dataset are very short. The genre that this dataset represents is informal spoken language, with a high prevalence of personal pronouns (I, you, he, she) and modals (can, should). The three most prominent parts-of-speech are Nouns with 22M instances, personal pronouns with 13M instances, and verbs with 13M instances (including modals).

Dependency parse tree have an average depth of 2.96, which confirms that the average sentence is very short and simple, containing most often a single predicate with its arguments. We discuss below that as part of the overall preprocessing pipeline, we filter sentences that are either too short or too shallow to be of interest for SRL purposes.

We apply the pre-trained SEMAFOR system [Das et al. (2014] on English sentences to obtain syntax and SRL annotations. As part of SEMAFOR’s pipeline, pre-processing is done on the input sentences using MaltParser [Nivre et al. (2007] pre-trained on sections 02-21 of the WSJ section of the Penn Treebank.

On the English side, the dataset contains 148M tokens consisting of 1.5M types, with sentence length averaging at 6.45. 41M of them are nouns, 36M personal pronouns, and 23M verbs. Compare the 36M personal pronouns in English with 13M in Hebrew. This is a known syntactic aspect in Hebrew where personal pronouns are often unmarked and understood from the morphological inflection of the main verb. Such a mismatch between EN and HE challenges the methods for constituent alignment. The depth of the dependency parse tree in English is slightly smaller than in Hebrew, averaging at 2.44.

The actual output from SEMAFOR consists of 55M instantiated frames across the entire dataset, but only 784 unique frames were evoked out of the 1,020 frames available in FrameNet v1.5, on which SEMAFOR is trained. On average, each sentence contains 2.4 frames and 1.17 frame elements per frame. Before filtering, the most general frame elements are also the most frequent - 3.9M instances of Entity, 3.3M instances of Agent, and 3.1M instances of Theme.

After the complete filtering pipeline, however, the most frequent frames are Statement, Arriving, Becoming, Motion and Killing (reflecting the violent nature of movies) and the most frequent frame elements are Theme, Speaker, Message, Agent and Goal.

The original data obtained from OpenSubtitles provides aligned sentence pairs. In order to project SRL annotations from English to Hebrew, we must compute word alignment. To this end, we use the fast_align method of ?) on the set of aligned sentences. The output provides a directional mapping from English tokens in a sentence to the corresponding tokens in the associated Hebrew sentence. This alignment is not necessarily one to one.

The word alignment process identified 181M word pairs. On average, a single English token is mapped to 1.25 Hebrew after segmentation. Of these 181M pairs, 115M are one-to-one alignments, 29M are cases of 1-n alignments. In the case of a 1-n alignment, the single English token is mapped to multiple Hebrew tokens and these 29M cases cover 66M pairs.

Within the 181M collected pairs, we identify 17.6M distinct word pairs (for the full vocabulary of about 1.5M distinct words in English). Given the fact that Hebrew has rich morphology, the average rate of distinct Hebrew tokens associated to the same English token is as expected (each English form can be mapped to multiple inflected Hebrew forms in addition to the natural expected lexical diversity).

Manual inspection indicated that when one English word is mapped to many Hebrew tokens (1-3 or more), it is an indication of poor data quality. This feature was, therefore, added in the pair quality filter described below.

On the basis of the word alignment, it is possible to project the English SRL annotations onto the Hebrew sentence. The projection process traverses the dependency syntax tree recursively: it identifies the head word of an annotated span in English, maps it to the word aligned in Hebrew, find its subtree in the Hebrew side, and annotates it accordingly.

This simple projection method cannot work well in the presence of noisy word alignments. For example, if the head word of a span in English is aligned with zero or more than one word in Hebrew, the procedure will not produce aligned constituents. Given the abundance of data, we simply skip these problematic alignment cases. Out of the 22M sentence pairs, 6.8M annotated pairs are left after this projection filter.

After preprocessing both English and Hebrew, we filter the 6.8M sentence pair in a way that reflects our confidence in the alignment between the annotated English and Hebrew syntactic structures.

To this end, we built a graphical user interface to visualize the word alignment, dependency tree of both English and Hebrew, the English SRL annotations, and the Hebrew SRL projections as shown in Fig. 2. This diagram summarizes all the NLP annotations available on both languages and the mapping between the SRL spans in a synthetic manner.

We manually annotated 124 sentences as one of the following six options:

• •

  Error in sentence alignment *

• •

  Poor translation *

• •

  Error in word alignment

• •

  Poor syntactic parsing

• •

  Poor frame parsing

• •

  Good

The items marked with an asterisk are problems in the dataset itself, i.e., better tools cannot improve these sentences’ quality.

Using these annotations and random resampling, we train a linear classifier to automatically annotate other sentences. We annotate a new sentence if the classifier gives it more than 80% of being “Good”. For training, we use the following features:

• •

  Sentence lengths (English/Hebrew)

• •

  English-Hebrew sentence length ratio

• •

  Number of frames in the English sentence

• •

  Number of one-to-one word alignments, i.e., which align one English word to one Hebrew word

• •

  Number of one-to-many word alignments

• •

  Dependency parse tree depth (English/Hebrew)

We interpret the prediction returned by the classifier as a score indicating the quality of the SRL match between the English and Hebrew sentences. Higher scores are associated with longer sentences (for example, with a score over 0.80, the average sentence length is 19.74 words vs. 8.2 for the whole data) with good sentence and word alignments when verified manually.

By adjusting the threshold on the reliability score produced by this classifier, we can control how many sentence pairs we use in the training phrase of the pipeline. The distribution of sentence pairs scores and their length is shown in Fig. 3, where the numbers on top indicate for each bin, the number of sentence pairs with a score above a given threshold and the average length of the Hebrew sentences with a score above this threshold. For example, with a threshold above 0.70, there are 422.8K sentence pairs with an average length of 14.25 segmented Hebrew tokens.

Given a Hebrew SRL resource, we can train a Hebrew SRL system. Because there are no available FrameNet parsers for languages other than English, we decided to map the projected FrameNet annotations we collected to PropBank-style annotations in order to make use of existing automatic SRL systems. In this mapping, we only consider annotations with FEs marked as “Core” by FrameNet. We map these Core FEs to PropBank arguments by relying on the order in which they appear in the FrameNet frame definition, and map them accordingly to ARG0, ARG1, etc. For example, the Commerce_buy frame has two core FEs: Buyer and Goods. In the sentence “Abby bought a car from Robin for $5,000”, “Abby” is the Buyer and “a car” is the Goods. This sentence will be labeled as “$[\text{Abby}_{\text{{ARG0}}}]\text{{ bought }}[\text{a car}_{\text{{ARG1}}}]\text{ from Robin for \$5,000}$”.

To apply this mapping to PropBank, we only keep sentences that include only Core Frame Elements²²2Mapping non-core elements such as space or time between FrameNet and PropBank is more interesting semantically, but we found it challenging to perform this mapping automatically. Noisy mapping would increase the risk of producing bad PropBank annotations, and we end up with a good enough number of sentences with the more stringent filter.. Upon further filtering with this criterion, we are left with 423K annotated sentences.

We sample from these sentences a training set of 240K sentences, 30K for development and 30K for test. The statistics of the Hebrew PropBank dataset are shown in Table 1.

We train a Hebrew PropBank semantic role labeler on the dataset we constructed to provide a baseline.

To this end, we use AllenNLP’s [Gardner et al. (2017] implementation of the model by ?), which only uses BERT [Devlin et al. (2018] for argument identification and classification. The specific BERT model we adopt is the multilingual cased version of BERT-Base which was trained, among other languages, on Hebrew. Hyper-parameters are as specified in AllenNLP’s configuration file³³3 https://github.com/allenai/allennlp/blob/master/training_config/bert_base_srl.jsonnet. This model simply encodes the whole sentence using the BERT model and predicts the BIO tags for the arguments given the predicate. A sentence is passed to BERT in the form [CLS] sentence [SEP] pred [SEP] - for example, [CLS] Barack Obama went to Paris [SEP] went [SEP]. ?) report a performance of F1=82.7 on argument identification on CoNLL 2009 in English.

As BERT works with WordPiece tokenization [Sennrich et al. (2016], we hypothesized that the segmented Hebrew words, i.e., the phrase “and from you / ve min ata” is one Hebrew word which is segmented into three words, would perform worse, as BERT has not seen segmented sentences. To this end, we unsegment the sentences back to their original form (joining “and from you / ve min ata” back into one token “vemimkha”) to see if this would improve performance. Table 2 shows that this hypothesis is incorrect, as training with both segmented and unsegmented tokens provides roughly equivalent performance.