Image augmentation with conformal mappings for a convolutional neural network

A convolutional neural network (CNN) is a type of deep learning technique that is well-suited for processing images. It receives image data in a matrix format so that each element of the matrix corresponds to the value of one pixel in the image and then transforms this input through several layers by taking into account the spatial relationships between the data points. The CNNs have been noted to be very useful in different areas of research but training even a single CNN often requires a large number of labelled images, which can sometimes be difficult to obtain.

One possible solution to this problem is using data augmentation, which means that the amount of existing data is multiplied by using simple transformations to create new, slightly different versions of images. Typical transformations used for this purpose are rotations, reflections, and translations but they do not suit for all types of data. Namely, if the image is square-shaped, creating a new version of it with a translation always crops out some areas from the image close to its edges and there are only seven new images that can be created with such a rotation or a reflection that fully preserves the original square. Clearly, the issue would not be encountered if the images given to the CNN would be disk-shaped but this is rarely the case.

However, according to the Riemann mapping theorem from classical function theory, any simply connected proper subdomain of the complex plane can be mapped onto the unit disk with a conformal mapping and, in particular, a conformal mapping called a Schwarz-Christoffel mapping can be used to map a two-dimensional disk onto the interior of a simple polygon [8]. Conformal mappings are much studied in complex analysis because they have many desirable properties such as preserving the magnitudes and directions of angles between curves even though they can turn straight line segments into circular arcs and vice versa. Furthermore, another subtype of conformal mappings are Möbius transformations which can be defined so that they map the interior of a disk onto itself but still significantly transform its contents. While conformal mappings could potentially be utilized in image data augmentation, their formulas are generally quite complicated and require integration of complex valued functions that cannot be computed directly with the existing functions in common programming languages.

In this article, we study if image augmentation can be performed by first mapping each square-shaped image onto a disk with a conformal mapping, then applying such rotations and Möbius transformations that preserve this disk, and finally mapping the resulting images back onto their original square shape. First, in Section 2, we present the usual formulas of conformal mappings and other mathematical theory needed. In Section 3, we show how these mappings can be written in Python or other programming languages and how images can be mapped with them. In Section 4, we test this method with one simple example about a CNN predicting a simulated data set. All the codes written in Python and MATLAB are also publicly available so the readers can access to these codes.

Let $G$ be the interior of the square with the vertices $1+\mathrm{i},-1+\mathrm{i},-1-\mathrm{i},1-\mathrm{i}$. In other words, define $G=\{z\in{\mathbb{C}}\text{ }|\text{ }-1<{\rm Re}(z)<1,-1<{\rm Im}(z)<1\}$, where ${\mathbb{C}}$ is the complex plane. Denote the unit disk $\{z\in{\mathbb{C}}\text{ }|\text{ }|z|<1\}$ by ${\mathbb{B}}^{2}$.

Consider the complete elliptic integrals of the first kind ${\mathcal{K}}(r)$ and ${\mathcal{K}}^{\prime}(r)$ defined for $r\in(0,1)$ by [3, 16]

In many references, the complete elliptic integrals of the first kind ${\mathcal{K}}(r)$ are defined as in (2.1), see e.g. [4, 7, 12]. However, in this paper, we will use the notations used in [1]. Thus, the complete elliptic integrals of the first kind are defined by [1, p. 590]

where $m=r^{2}\in(0,1)$ and $m_{1}=1-m$. The incomplete elliptic integral of the first kind is defined by [1]

and then ${\mathcal{K}}(r)=F(\pi/2,m)$. See [1, 3, 4, 7, 12] for more information.

The Jacobian elliptic functions can be defined with the help of the incomplete elliptic integral (2.3), see [1, Ch. 16]. If

then the angle $\varphi$ is called the Jacobi amplitude and we write

The Jacobian elliptic functions are defined by

The exact conformal mapping from the square region $G$ and its inverse can be written in terms of the elliptic functions and the incomplete elliptic integral. It follows from [11, p. 182] and [12, p. 242] that the conformal mapping $f$ from the domain $G$ onto the unit disk ${\mathbb{B}}^{2}$, $f:G\to{\mathbb{B}}^{2}$, is given by

where

Note that ${\mathcal{K}}(m)={\mathcal{K}}(1/2)=\Gamma(1/4)^{2}/(4\pi)$, see [3, (3.19), p. 51]. Then, it follows from (2.4) and (2.5) that the inverse of this mapping, $f^{-1}:{\mathbb{B}}^{2}\to G$, is given

which is a conformal mapping from the unit disk ${\mathbb{B}}^{2}$ onto the square domain $G$.

Denote the extended complex space by $\overline{{\mathbb{C}}}={\mathbb{C}}\cup\{\infty\}$ and choose some $\alpha\in{\mathbb{B}}^{2}$. Define the Möbius transformation $g:\overline{{\mathbb{C}}}\to\overline{{\mathbb{C}}}$,

where $\overline{\alpha}$ is the complex conjugate of $\alpha$. This mapping $g$ fulfills $g({\mathbb{B}}^{2})={\mathbb{B}}^{2}$, $g(\alpha)=0$, and $g(0)=-\alpha$, which means that it preserves the unit disk ${\mathbb{B}}^{2}$ but is not an identity mapping as long as $\alpha\neq 0$.

Finally, we define formally the rotation about the origin for an angle $k\in[0,2\pi)$ as $\upsilon:{\mathbb{C}}\to{\mathbb{C}}$,

Figure 1 represents visually how the functions $f$, $g$, and $\upsilon$ and the inverse function $f^{-1}$ affect the shape of an image, when $g$ and $\upsilon$ are defined with constants $\alpha=0.3+0.3\mathrm{i}$ and $k=\pi/3$, respectively.

Computing the mapping function $f$ by (2.5) and its inverse function $f^{-1}$ by (2.7) requires computing the Jacobian elliptic functions $\operatorname{\mathrm{sn}}(\phi,m)$ and $\operatorname{\mathrm{dn}}(\phi,m)$ for complex $\phi$ as well as computing the incomplete elliptic integral $F(\varphi,m)$ for complex $\varphi$. Thus, the function $f$ and its inverse function $f^{-1}$ cannot usually be computed directly from their definitions (2.5) and (2.7) in the code because the functions used to evaluate the elliptic integrals are not defined for complex numbers in many programming languages. For instance, the complete elliptic integral ${\mathcal{K}}(m)$, the incomplete elliptic integral $F(\varphi,m)$, as well as the Jacobian elliptic functions $\operatorname{\mathrm{sn}}(\phi,m)$, $\operatorname{\mathrm{cn}}(\phi,m)$, and $\operatorname{\mathrm{dn}}(\phi,m)$ can be computed for real arguments, , with the functions ellipk, ellipkinc, and ellipj of the subpackage special of the package Scipy [20] in Python. So, in this paper, we will use the properties of the Jacobian elliptic functions and the incomplete elliptic integral to compute their values without the need to compute elliptic integrals of complex variables. For example, the functions $\operatorname{\mathrm{sn}}(z,m)$ and $\operatorname{\mathrm{dn}}(z,m)$, for a complex variable $z=x+\mathrm{i}y$ where $x$ and $y$ are real variables, can be computed using the following formulas [1, §16.21.1]

where

and

Similarly, for a complex variable $z=x+\mathrm{i}y$, the incomplete elliptic integral $F(z,m)$ can be computed by [1, §17.4.11]

where $x_{1}$ and $y_{1}$ are real variables such that $X=\cot^{2}x_{1}$ is the positive root of the equation

and $\tan^{2}y_{1}$ is given by

Note that, if $b=-\left(\cot^{2}(x)+m\sinh^{2}(y)\csc^{2}(x)-m_{1}\right)$ and $c=-m_{1}\cot^{2}(x)$, then $c\leq 0$ and hence $\sqrt{b^{2}/4-c}\geq b/2$. Thus, we have

Denote below the floor and the ceiling functions with ${\rm floor}()$ and ${\rm ceil}()$. Then, the values of $x_{1}$ and $y_{1}$ can be computed from (3.6) and (3.5), respectively, through (see [14])

At the beginning of the code, fix the constants $r$, $m$, and $L$ as in (2.6). The conformal mapping $f$ from the square $G$ onto the unit disk can now be written with the following pseudo-code.

Finding the inverse mapping $f^{-1}$ requires solving the quadratic equation (3.4). The more unusual trigonometric functions such cot or csc and their inverses are in the package sumpy in Python. Note also that the parameter $\epsilon$ can be chosen to be any small positive number.

After writing the functions $f$ and $f^{-1}$ with the pseudo-codes (3.9) and (3.10), they can be tested by choosing a random number $z$ from the square $G$, computing $w=f(z)$ and $z^{\prime}=f^{-1}(w)$, and printing the difference $z-z^{\prime}$, which should be very close to zero.

An image can be mapped conformally onto a disk with the function squareToDisk presented in (3.11). First, suppose then that there is a square matrix called img which can be read as a square-shaped grayscale image by choosing the colour of each pixel according the corresponding element in img. Let $h$ be the number of rows or columns in the matrix img, imgH a vector containing $h$ evenly spaced points in the interval $[-1,1]$, and $u$ the distance between two adjacent points in imgH. Then we initialize the new image img by creating a zero matrix of the same size as img. After that, we create a loop that goes through each element of img and expresses it as a point $z\in G$ with the help of the vector imgH. If the point $z$ belongs to the unit disk, we use the inverse mapping $f^{-1}$ to find the point $w$ in the square $G$ that becomes $z$ when the domain $G$ is mapped conformally to the unit disk with the mapping $f$. For this point $w$, we find such points $j_{0},k_{0}\in(0,1,...,h-1)$ and $j_{1},k_{1}\in[0,u)$ that $w=j_{0}+j_{1}+(k_{0}+k_{1})i$, which gives us also the closest four points to $w$ that correspond to the pixel locations of the original image matrix img and the distances between $w$ and these locations. Finally, we compute the values of the pixel at the point $w$ by using the weighted means of the four surrounding pixels and, since $z=f(w)$, we have the value of the pixel at the point $z$ in the unit disk. In the pseudo-code below, $v[i]$ refers to the $(i+1)$th element of the vector $v$ as the indexing starts from 0.

The function in (3.11) returns such an image matrix that has the original image mapped onto the largest possible disk fitting inside the square-shaped image matrix, and the other values outside this disk are zeroes. Note that this function can be extended for also colour images of RGB or another similar format by just computing the weighted means for the values of each color channel at the end of the loop. Similarly, by replacing the inverse function $f^{-1}$ with either the Möbius transformation $g$ of (2.8) or the rotation $\upsilon$ of (2.9), we can map the input image so that the part inside the disk is transformed. To create the function that conformally maps the interior of this disk in the square-shaped image matrix onto the whole matrix, we just need to remove the condition $|z|<1$ and replace $w=f^{-1}(z)$ by $w=f(z)$ in the code (3.11). Alternatively, we can use for instance the composed mapping $w=(f^{-1}\circ\upsilon\circ g\circ f)^{-1}(z)=f^{-1}\circ g\circ\upsilon\circ f(z)$ to obtain the image of Figure 1(E).

Here, we build a CNN for predicting how many small black disks an otherwise white image contains. We use Python (version: 3.9.9) [19] with packages TensorFlow (version: 2.7.0) [2], Keras (version: 2.7.0) [6], and SciPy [20] (version: 1.7.3).

Firstly, it must be noted that the experiment above is very simplified example and better results could be obtained by building a suitable algorithm instead of using a CNN. However, there are numerous practical reasons why a CNN that recognizes certain unusual disk-shaped areas are useful. In fact, this experiment was inspired by the CNNs detecting tumors from tomography images of cancer patients in [10, 13, 17].

It also should be taken into account that this method is quite complicated and designed for situations where the typical transformations cannot be used. For instance, if some of the disks would be so close to edges of the images in our experiment that they would be cropped completely or partially out when the square-shaped images are directly rotated and re-calculating their number is not possible, using this new method rather than the usual rotation is justified. The rotation used as a comparison augmentation method in our experiment did not crop the images but rather included more background. Given how commonly CNNs are used nowadays, there are likely many practical examples of cases where information could be lost if the typical rotation is used. Note that these mappings can be also extended to map any rectangle-shaped image into a disk or vice versa because the rectangle can be very easily stretched into a square.

Another reason for using this method is wanting to utilize the properties of the Möbius transformation in particular. Because of this mapping, the images change in more complicated ways under this augmentation method than they do in simple rotations. We can see that the right-most disk in Figure 2(D) is larger than any of the disks in the original image in Figure 2(A). By increasing the absolute value of the point $\alpha$ used to define the Möbius transformation, the differences between the images before and after the composed mapping are also increased. Still, all these disks stay circular because Möbius transformations can only map circles into lines or circles and the disks stay inside the edges of the image. The use of Möbius transformations in image augmentation has been also studied by Zhou et al. [21] but they did not use other conformal mappings and their augmented images therefore contain much empty background outside the original photographs.

One alternative method is using general adversarial network (GAN) augmentation first introduced in 2014 by Goodfellow et al. [9]. GANs are a class of neural networks that generate synthetic samples resembling the real images of the original data set. However, it might be difficult to predict what sort of augmented data GANs produce while the use of conformal mappings only distort the images. This means that the augmentation based on the conformal mappings preserves the number of disks in the images of the data set used here, while GANs might change the number of disks. Furthermore, GANs also require some amount of data so that they can be trained for their work, while the amount of the data does not affect how the images change under the conformal mappings. Our method of augmentation could be also compared with such procedures that use prior information about the data distribution in augmentation [5, 15].

We used conformal mappings to create a new way to augment the image data of a CNN and, according to our result, this method both works and produces better predictions than a CNN trained with non-augmented data set.