A Novel Application of Image-to-Image Translation: Chromosome Straightening Framework by Learning from a Single Image

For our framework, we propose a two-step strategy to construct an augmented dataset using only one curved chromosome image.

Step 1. We construct the label of a curved chromosome (Fig. 2(a)) by extracting a pertinent internal backbone. The entire process is summarized in Algorithm 1. Considering the chromosome image to be comprised of rows of pixels, the centers of each row are connected to form an approximate central axis extending from top to bottom (Lines 1 to 8 of Algorithm 1, Fig. 2(b)). To alleviate small-scale fluctuations generated in Line 8, this central axis is then smoothed by a moving average algorithm with an 11-pixel window length [27] (Line 9, Fig. 2(c)). We divide this smoothed central axis equally into 11 parts in the $y$ axis. Since the first and the last parts may not be aligned in the same directions with both sides of the chromosome (red boxes), these two parts are subsequently removed (Lines 10 to 11, Fig. 2(d) to (e)). The remaining splitting points are connected by 33-pixel width sticks, and these 9 sticks are filled with pixel values in series of equal difference (23, 46, 69, 92, 115, 138, 161, 184, and 207) (Line 12, Fig. 2(f)). This stick figure contains the information of curvature, length, and orientation of the original chromosome. Finally, a vertical backbone is constructed with the same length of each stick (Line 13, Fig. 2(g)), and is fed into the fine-tuned image-to-image translation model for synthesizing the straightened chromosome.

Fig. 3 illustrates that the morphological and Stentiford thinning algorithms may cause branches and irregular rings when the chromosome features pronounced widths. Thus the previous work directed at chromosome straightening [14, 15, 6], composed of these thinning algorithms, cannot be utilized here. In contrast, our predicted 10-point central axis are approximately in accordance with the actual chromosome backbone.

Step 2. We improve the performance of deep learning models by generating more augmented chromosomes with different degrees of curvatures. We first apply random elastic deformation [28] and random rotation (from -45 to 45 degree) to the curved chromosome and its backbone simultaneously (Fig. 2(a) and (f)) until a sizeable number of augmented chromosomes and backbones (1000 pairs in this research) are obtained for training and validation (Fig. 4). Note that the setup of the elastic deformation algorithm [28] is $points=3$ and $sigma=18$ for $256\times 256$ images, in order to generate plausible virtual curvatures. Since we utilize 33-pixel width sticks, rather than key points to label internal backbones, the detailed augmentation information, such as stretching, rotation and distortion, is retained and learned by the image-to-image translation models.

Since the objective of this study is to input a straightened backbone of a chromosome for synthesizing the corresponding chromosomes with preserved banding patterns, our novel image-to-image translation models are object specific. Therefore, it is essential to construct an augmented dataset for each image-to-image translation model. Utilizing the approach mentioned in Step 2, we generate 1000 augmented image pairs for each curved chromosome. The augmented dataset is then randomly split using a ratio of 9:1 for training and validation, respectively. Under our framework, we shall utilize two image-to-image translation models, U-Net and pix2pix (Fig. 5(a)). It should be noted that the U-Net utilized in this research is identical to the generator part of pix2pix. The training process of U-Net is a regular supervised learning method achieved by synthesized chromosomes and corresponding ground-truths. In pix2pix, a generator $G_{b}$ synthesizes chromosomes from the augmented backbones to mislead $D_{b}$. Meanwhile, a discriminator $D_{b}$ is trained for discerning “real” images from “fake” images yielded by the generator. The $G_{b}$ and $D_{b}$ is optimized with the objective function:

where $G_{b}^{*}$ represents the optimal generator; $\lambda$ is a coefficient to balance two losses; $\mathcal{L}_{cGAN}(G_{b},D_{b})$ is the adversarial loss (Equation 2); and $\mathcal{L}_{pix}(G_{b})$ is L1 distance to evaluate pixel-wise performance between generated images and ground-truths (Equation 3):

In the above: $x_{B}$ and $y_{B}$ represent augmented backbones and chromosomes, respectively; $B\in\{1,...,K\}$ where $K$ is the number of augmented pairs that we want; and $z$ is the noise introduced in the generator.

To straighten the chromosome, we input its vertical backbone (Fig. 2(g)) into the optimal U-Net or optimal generator $G_{b}^{*}$, which will output the corresponding chromosome (Fig. 5(b)).

To test our framework on real-world images, we extract 642 low-resolution human chromosome images from karyotypes provided by a biomedical company. Images in this research have been cleaned so that connections between these images and their corresponding owners have been removed. Since the chromosomes with relatively long arms and noticeable curvatures require straightening (Figure 1), we collect Type 1 to 7 chromosomes in this research. We invert the color of these grey-scale images and center them in a $256\times 256$ black background. As described in Section II. A, 1000 augmented image pairs were obtained from each curved chromosome image before feeding into the U-Net and pix2pix models. It should be noted here that each augmented dataset is individually trained for straightening since our framework is object specific.

We apply two evaluation metrics to quantitatively measure the performance of these straightening methods. Due to the obvious morphological deformation between straightened results and original curved chromosomes, traditional similarity measurement metrics, such as Euclidean distance, structural similarity index (SSIM) [29] and peak-signal-to-noise ratio (PSNR) [30], designed for evaluating image quality degradation generated by image processing or compression, are not suitable for this task. Instead, Learned Perceptual Image Patch Similarity (LPIPS) [31] was used to evaluate straightening performance of different methods in this paper. The LPIPS is an emergent deep neural network-based method which is able to extract deep features of images for evaluating high-order structure similarity. Compared to the results of these traditional metrics, its results are more in accordance with human perceptual similarity judgment [31].

Apart from LPIPS, to ensure the details of straightened results are preserved in practice, we also assess the effectiveness of different straightening methods based on chromosome type classification. If the banding patterns and edge details of chromosomes are well preserved during straightening, the classification accuracy of straightened chromosomes should not decrease. In contrast, unpreserved details, such as broken bands, may not provide enough information for the classification model. The original images (642 curved chromosomes, Type 1 to 7) are randomly split using the ratio of 3:1 for 4-fold cross-validation. With a fixed random seed, this process is similarly carried out for the straightened chromosomes generated by different methods.

Our experiments are implemented using PyTorch and run on two NVIDIA RTX 2080Ti GPUs. In each training process of chromosome straightening, the training and validation sets are split by a fixed random seed. The input image pairs are first normalized by default values (mean $\mu=$ 0.5 and standard deviation $\sigma=$ 0.5), and these results are fed into image-to-image translation models for learning the mapping dependence from backbones to chromosomes. Models are trained with an initial learning rate $lr=0.00004$. The validation performance is checked three times per epoch, and the weights are saved when the best validation performance is updated. When the validation performance does not improve for 9 consecutive checks, the learning rate is reduced to 80% for fine-tuning. To avoid overfitting, the training process is terminated when there are 27 consecutive checks without updated validation performance. For each chromosome type classification model (Alexnet [32], ResNet50 [33] and DenseNet169 [34]), the training process is initialized with a learning rate of $lr=0.00004$ and corresponding ImageNet pre-trained weights. We utilize 12 and 120 consecutive checks for fine-tuning and avoiding overfitting, respectively. Furthermore, we use identical random seeds, preprocessing and hyperparameter settings for 4-fold cross-validation of the chromosome type classification.

To address the flaws, such as the broken banding patterns in geometric methods and random stretching in elastic deformation algorithms, we propose a chromosome straightening framework which is object specific. Therefore, it is time-consuming to train a separate straightening model for every curved chromosome. In future research, a generalized chromosome straightening model shall be designed. We would design an improved model for disentangling the information of internal backbones and banding patterns.

Under our framework, we notice two types of failure cases. First, the straightening performance hinges on the accuracy of the central axes identified. When the curvature of a chromosome is too large, the extracted internal backbone may not be aligned in a similar direction with the original image (red arrows of Chr_4 in Fig. 8). In this case, the relation between the backbone and corresponding banding patterns are still preserved. As a result, that part may not be well straightened. Second, some irregular chromosomes may still cause small-scale fluctuations of backbones even after the moving average algorithm, resulting in blurred synthesized banding patterns and edge details (Chr_5 in Fig. 8). Because of this, high-quality labels of chromosomes are still deficient in the augmented dataset. A plausible direction would be an improvement of the backbone extraction method. A crowdsourcing database of labeled backbones could be established for developing a powerful deep learning based backbone detector of chromosomes.

Since the results of our straightening framework demonstrate a higher classification accuracy, it is worthwhile to incorporate the framework into automatic karyotyping analysis and cytogenetic map construction. With the development of image-to-image translation research, many advanced modules and architectures, for example, attention-based GANs [37], may be integrated into our framework to further improve its efficacy and robustness.

Since our augmented datasets contain information concerning random deformation and rotation, we observe that fine-tuned generators not only have an ability to straighten chromosomes, but also can synthesize more chromosomes by inputting internal backbones with different curvatures (Fig. 9). Therefore, our framework shows the potentiality for generating augmented chromosomes with highly preserved detail along with customized backbone images.

Compared to regular U-shape networks, cGANs have more potential in the application of high-resolution chromosome straightening with higher translation consistency. In the latest study, Artemov et al. [5] employs PhotoShop for straightening high-resolution chromosomes when developing cytogenetic maps, so an automatic high-resolution chromosome straightening framework is still in demand. Similar to the evolution from pix2pix to pix2pixHD [38], our straightening framework may also be further modified for high-resolution chromosome images.