JaMIE: A Pipeline Japanese Medical Information Extraction System

We adopt the following medical entity types defined in Yada et al. (2020): Diseases and symptoms <D>, Anatomical entities <A>, Features and measurements <F>, Change <C>, Time <TIMEX3>, Test <T-test/key/val>, Medicine <M-key/val>, Remedy <R>, Clinical Context <CC>. The complete entity and modality definition refers to the original paper.

On the top of the entity and modality annotation above, we designed relation types between two entities. They can be categorized into medical relations and temporal relations. The example of each relation type is presented in Table 1.

In practice, we annotated two datasets: 1,000 radiography interpretation reports of LC and 156 medical history reports of IPF. We annotate all reports with two passes. One annotator conducted the first pass relation annotation for a report. In the second pass, the expert examined the annotation and led the final adjudication by discussing the inconsistency with the first pass annotator. This procedure is to balance the quality and cost, since it does not rely on the full expert annotation.

Table 2 shows the statistics of the relations annotation. Though the number of the medical history reports is relatively smaller, they usually contain more content per report and a wider coverage of entity types. Considering that the popular English 2010 i2b2/VA medical dataset contains 170 documents (3,106 relations) for training, our annotation scale are comparable with or even larger than it. The results show very different relation type distribution in the two types of reports. As the medical history reports of IPF can be viewed as the mixture of several types of reports such as radiography reports, examination reports, test results, etc., they show a more balanced coverage of relation types, while the radiography interpretation reports of LC are more narrowly distributed among the disease-relevant relation types such as ‘region’ and ‘feature’.

Recent medical IE research Si et al. (2019); Alsentzer et al. (2019); Peng et al. (2019) suggests the contextual pre-trained models such as ELMO Peters et al. (2018) and BERT Devlin et al. (2019) markedly outperform traditional word embedding methods (e.g., word2vec, glove, and fastText). In our pipeline system, we adopt the Japanese pre-trained BERT³³3https://alaginrc.nict.go.jp/nict-bert/index.html as the sentence encoder for retrieving token embeddings.

Formally, a sentence $S=[x_{0},x_{1},x_{2},...,x_{n}]$ is encoded by a contextual BERT or word embedding with bidirectional Long Short-Term Memory (LSTM) Hochreiter and Schmidhuber (1997) as:

Medical entity recognition (MER) aims to predict the token spans of entities and their types from the text.

We formulate Medical Entity Recognition as sequential tagging with the BIO (begin, inside, outside) tags. The outputs are constrained with a conditional random field (CRF) Lafferty et al. (2001) layer. For a tag sequence $y=[y_{0},y_{1},y_{2},...,y_{n}]$, the probability of a sequence $y$ given $X$ is the softmax over all possible tag sequences:

where the score function $s(X,y)$ represents the sum of the transition scores and tag probabilities.

The modality classification (MC) component is to classify the modality types of the given entities. For a multi-token entity $E_{i}$ predicted by the MER model, we represents the entity embedding as the element sum of embeddings in the entity span. To enrich the context for predicting assertion, we concatenate the entity embedding with the auxiliary entity type. The $i$-step modality prediction is:

where $E_{i}$ denotes the $i$-th entity embedding, $E_{i}^{type}$ denote the entity type embedding predicted by the MER model.

The relation extraction (RE) component is to predict the relations and their types between two named entities. We formulate the relation extraction problem as the multiple head selection Zhang et al. (2017) of each entity in the sentence. Given each entity $E_{i}$ in the sentence, the model predicts whether another entity $E_{j}$ is the head of this token with a relation $r_{k}$. The probability of a relation is defined as: $P(E_{j},r_{k}|E_{i};\theta)=sigmoid(s(E_{j},r_{k},E_{i}))$, where $s(.)$ denotes a single full-connected layer. An additional ‘N’ relation presents no relation between two tokens. The final representation of an entity $E_{i}$ is the concatenated embeddings of the entity, entity type, and modality type.

For each dataset, we conduct the patient level 5-fold cross-validation to evaluate the performance of our system. 10% training data is split as the validation set. In each stage in the pipeline, the current component is trained with the gold inputs. The Japanese text is segmented into tokens by the morphological analyzer⁴⁴4MeCab Kudo et al. (2004). We adopt PyTorch Transformers⁵⁵5https://github.com/huggingface/transformers to implement the system. The following hyper-parameters are empirically chosen: fine-tuning epoch as 10, batch size as 16, AdamW with learning rate as 5e-5. The best checkpoints on the validation set are saved to produce test results.

Instead of applying the usual pipeline evaluation with the gold inputs at each stage, we are more interested in the practical performance of the system and adopt the joint evaluation Zheng et al. (2017) as described in the following:

• •

  Medical entity recognition identifies medical entity from raw reports. We evaluate each {entity, entity type} to the reference.

• •

  Modality classification classifies the modality types of the entities identified by the former stage. The evaluation is on each {entity, entity type, modality type}.

• •

  Relation extraction extracts the relations between the entities identified by the former stages. The evaluation is on each triplet {entity, relation, entity2}.

We measure micro-F1 of the system prediction to the gold reference in each pipeline stage.

Table 3 shows our system performance on two types of reports: radiography interpretation reports of LC and medical history reports of IPF. The radiography interpretation reports’ performance suggests that by concentrating annotation efforts on a specific report type the system achieves high F1 with sufficient training data. Compared to 95.30 MER F1 reported by Yada et al. (2020), our MER score outperforms their score by 0.35 with the additional CRF layer. The RE model obtains 86.53 F1 of the radiography interpretation reports and 71.04 F1 of the medical history reports.

We offer the baseline encoder with LSTM⁶⁶6LSTM and Word2vec hidden size equal to 256. upon word2vec embeddings Mikolov et al. (2013) trained on Japanese Wikipedia. We observe significant drops in all three tasks, especially in the final relation extraction. In both radiography interpretation reports of LC and medical history reports of IPF, the BERT-based RE models leading ‘LSTM + word2vec’ by approximately 10 points F1. We suggest that solving relation extraction requires long-range information between entities. BERT naturally models such long-range dependency in the self-attention mechanism, while word2vec is trained with a fixed local window and LSTM could also fall in the long sequential actions.

While the medical history reports contain broader relation types and the data size is relatively smaller, the system still obtains satisfactory performance. In addition, we present each relation F1 in Table 5. Except for three rarely appearing relations, i.e. ‘finish’, ‘after’ and ‘before’, the F1 scores on the other types are balanced and match the statistics in Table 2. As for the radiography interpretation report results in Table 5, the major relations of ‘region’ and ‘feature’ relations achieve high performance with 84.59 and 90.16 F1. The moderate ‘change’, ‘compare’ and ‘on’ obtain satisfying 76.23 to 81.92 F1.

One question is whether concentrating annotation efforts on a specific report type can quickly obtain high accuracy to meet the requirements of the practical applications. A valid approach is to compare the RE performance of two report types with the comparable annotation efforts i.e. training data size. The medical history report of IPF contains total 5,432 relations, which is approximately 39% of the radiography interpretation reports. We designed the experiment by reducing the train set of the radiography interpretation reports to the comparable 39% of the origin. The results in Table 6 show that even with comparable training size, the specific radiography interpretation reports lead the performance by 11.29 points F1.

To be clarified, the two results are still not exactly comparable due to the different relation distributions. However, the radiography interpretation reports more densely spread in the relation types such as ‘region’ and ‘feature’ (Table 2), which usually means less number of report needed for achieving the similar overall accuracy compared to the medical history reports. In the scenario of demanding high accuracy for practical medical applications, the results suggest that the annotation strategy of starting from a specific type of report and gradually increasing the coverage of report types is feasible.

JaMIE provides an easy-to-use Command-Line Interface (CLI). We design our training/testing scripts similar to the official Transformers examples, in order to be friendly to the Transformers users. We demonstrate how to train/test a relation model with the following script:

In the case of annotating raw medical reports with our trained model, users need to download our trained models from the JaMIE GitHub beforehand. Users then execute the pipeline ‘test’ scripts to annotate entities, modalities, and relations step by step. At each stage, the model will generate the prediction as the input of the next stage model. The prediction is presented in the same XML-style as shown in Figure 1.

Our medical IE annotation schema serves to encode a wide range of general medical information not limited to any specific disease, report types or languages. Users can manually annotate other types of medical reports by following our guideline. Users can apply the ‘train’ scripts to train the pipeline models on their newly annotated corpus for providing automatic annotation.