Automated Knee X-ray Report Generation

The burden on diagnostic radiologists has been increasing quite rapidly with the advancement of a variety of modern imaging modalities, which, while allowing for higher resolution images in 3D and even 4D, dramatically increase the complexity of the diagnostic process. It has become common for radiologists to rely on various image analysis and automated decision support systems to facilitate the interpretation process. These computer aided diagnostic systems, or CAD, have been able to make great advances with the help of machine learning algorithms and a large amount of clinical imaging data, and have been successfully tested in clinical settings [1, 2, 3, 4].

Many of these machine learning algorithms are supervised learning approaches that require large amounts of annotated image data for training. Gathering suitable data is especially challenging in the medical domain as it is incredibly time consuming for radiologists to generate ground-truths of the standard and volume required for training a predictive model. An alternative approach is to use past clinical images and corresponding radiological reports available through a hospital’s picture archiving and communication system (PACS); the advantage being that, although this data is largely unstructured (free text), it is available to us in high volumes and removes the need for manual annotation.

Hence, it is necessary to develop an appropriate learning framework: one that can take advantage of past radiological examinations and their corresponding textual reports, and formulate a method for prediction that can best expedite the diagnostic process. An additional motivation to learning from clinical reports is that reports provide a much richer context to the pathology than a single class label, such as severity and location.

Here we present a model for captioning knee X-ray exams that is capable of generating radiological reports summarising present pathologies and the contexts within which they are presented. A key contribution is that the proposed model is able to handle an arbitrary number of input image views (which is typical in the context of knee X-ray exams) and is able to learn from free-text reports. The model is based on the neural image caption (NIC) model of [5], where a word-level language generation model is conditioned on image features. In our implementation, the X-ray image features for each input image are derived from a pre-trained classification network, and aggregated into a fixed-sized input to the language model.

In the field of computer vision, the use of human generated visually descriptive text to infer the contents of an image has primarily been applied to image caption generation. Earlier models relied on linking template-based language models to objects and spatial contexts in the image [6, 7, 8]. More recently, interest has moved to the combined potential of convolutional neural networks (CNNs) and recurrent neural networks (RNNs) for describing images using natural language [9, 5, 10, 11, 12]. The advantage of using neural networks for caption generation is that the model is not constricted by hard-coded language templates and is able to learn more freely from the training data.

By contrast, using clinical reports to perform automated pathology detection in medical images has only been explored in a few studies [13, 14, 15, 16]. In addition, these models are limited to learning from carefully curated reports, either medical subject heading annotations (MeSH^® [17]) [13, 14], or templated reports made specifically for the purpose of training [15, 16], which are both very time consuming for radiologists/pathologists to create.

To our knowledge, this is the first study exploring the use of past medical examinations and their corresponding raw reports in order to develop a predictive model for an arbitrary number of input radiological images.

The knee X-ray dataset has been extracted from the PAC system of St Thomas Hospital (part of Guys and St Thomas NHS Foundation Trust) and has been fully anonymised to remove sensitive patient information. It comprises a total of 330 knee X-ray exams collected over the years 2015 and 2016. Each exam consists of a textual report and one or more X-ray images (left/right knees or both, taken from different views: anteroposterior (AP), lateral (L) and skyline (S), and different positions: weight-bearing (WB) and non-weight-bearing(nonWB)). The most common exam consists of both AP and L views of left and right knees separately, making up 42% of total exams. The reports vary in length between 2 and 145 words, with an average of 30 and standard deviation of 18.7; and between 1 and 16 sentences with an average of 2.7 per report. The X-ray images vary in sizes between $420\times 650\times 3$ and $3056\times 3056\times 3$.

Prior to using the raw reports for training an image captioning model, they were cleaned to remove phrases not directly describing the images, for instance, instructions to compare to past exams (which are not available to us due to the anonymisation process). These phrases do not allow any inferences about current images, so are not useful for finding image-text correspondence. In addition, common ‘stopwords’ (such as ‘and’, ‘the’) were removed, and all punctuation was replaced with either commas or full stops where appropriate. All report cleaning was done using regular expressions. Exams were randomly split into 80%-10%-10% for training, testing and validation respectively.

For training the report generation model, the LSTM is unrolled up to 33 time steps (1 for image features, 1 per start and end token, and 30 for the average number of words in the reports). The images were rescaled and padded (preserving the aspect ratio) to $224\times 224$. Words were one-hot encoded, and ones that occurred at a frequency <5 were removed.

To improve the X-ray image features, a simple CNN BBox regressor was built and trained on a subset of the training images and ground-truth BBoxes manually created for this purpose (231 image-BBox pairs). In addition, the training set was augmented eight-fold by random cropping the original images from $256\times 256$ to $224\times 224$, (in the case of the BBox images, small, random translations (between -5 and 5 in x and y) were applied to the BBox of each image), flipping the images along the vertical axis, and shuffling the sentences in the reports.

We then trained the LSTM model for report generation conditioned on the combined image features by minimising the negative log-likelihood between the output and true sequence:

where $p$ is the probability that the predicted word $P_{t}$ equals the true word $T_{t}$ at time step $t$ given image features CNN($I$) and previous words $P_{0}\ldots P_{t-1}$, and $N$ is the LSTM sequence length. At training time, loss is minimised over the training set using stochastic gradient descent (batch size 20, learning rate $1\times 10^{-5}$), and parameters are updated using Adam [25] optimisation.

Reports are generated on the test set by max-aggregating the image features of an exam, using it as the first word input into the LSTM, and sampling consequent words. The quality of the generated reports was evaluated by measuring BLUE [26] and METEOR [27] scores averaged over all the reports, which are a modified form of n-gram precision commonly used for evaluating image captioning and machine translation as they maintain high correlation with human judgement. As a comparison, we trained the model on single image inputs, duplicating the reports for each image in an exam. We also evaluated whether the performance improved if we narrow the input to the CNN to a BBox surrounding the knee joint.

The report generation model performed better when trained on a combination of image features than on single images as the conditional input (see Table 1), suggesting that all images are required, in part, in order to make the pathology assessment. This is in line with expectations since pathologies may only be present in one leg, or may only be seen from a particular view. Contrary to expectation, bounding box extraction did not improve the performance of report generation as it may have instead removed pertinent image features. A sample exam with the original report and two auto-generated reports of contrasting BLEU-1 scores is presented in Figure 2.

We demonstrate how past knee X-ray exams consisting of sets of images and corresponding radiological reports can be used as part of a learning framework to determine image-text correspondence and hence automate the generation of such reports for new X-ray exams. Preliminary results look promising as the auto-generated reports correlate well with true reports, and we hope to train the model on additional knee X-ray exams as these become available to us. Further developments to the model can be made by incorporating the knowledge of the view-type of each image, keeping them as separate inputs, and finding correspondence between each image and parts of text (for instance, through the use of an attention mechanism [28]). We will explore this in a future study.