Pair-Relationship Modeling for Latent Fingerprint Recognition

Due to the different qualities of the latent and the reference fingerprints, we use different techniques to generate their enhanced fingerprint images and corresponding orientation fields. For the latent fingerprint, the method in Ref. [20] is used to firstly produce an enhanced fingerprint image, an estimated orientation field, and a quality map. Then, the produced quality map is used to segment the region of interest (ROI) of the produced fingerprint image and orientation field. Finally, the segmented orientation field is further regularized by a fingerprint orientation model FOMFE [15, 16] for denoising and estimating orientations for moderately corrupted areas. For the reference fingerprint, because it is usually of good quality, the STFT method [21] is used to produce its enhanced fingerprint image, estimated orientation field, and ROI segmentation. By the same token, its segmented orientation field is further regularized by the FOMFE model. Therefore, for both the latent and the reference fingerprints, their enhanced fingerprint images and FOMFE-based orientation fields are produced.

To generate the tensors FIT and OFT, the two enhanced fingerprint images and the two FOMFE-based orientation fields need to be aligned first. Due to the poor quality of latent fingerprints, we propose to use orientation field for the alignment because it can provide a robust coarse alignment. We have observed that the random hexagon partition in the DLO alignment [17] could generate significant discrepant hexagons over the ROI, leading to poor alignment results. To address this issue, we propose an improved minutia-based DLO alignment named as M-DLO alignment where the hexagons are formed around the most reliable minutia. One or two reliable minutiae will be sufficient to help produce a sound alignment with our proposed M-DLO alignment. Algorithm 1 summarizes the M-DLO alignment.

After aligning the two enhanced fingerprints and the two FOMFE-based orientation fields, they are respectively concatenated to generate the tensors FIT and OFT. There are two ways of the concatenation: 1) the fingerprint image (or orientation field) of the latent fingerprint at front and the one of reference fingerprint at back; and 2) in reverse. We name the tensors generated by these two ways as FIT₁ (or OFT₁) and FIT₂ (or OFT₂), respectively.

For convenience, we define the architecture of a network only by the types of its layers and their connections, such that a network refers to an architecture with specific hyper-parameters, and a model refers to a network with specific inputs.

Fig. 4 illustrates the proposed hybrid deep network. Its inputs are patches (or macro-patches as detailed in Section 3.3.3) taken from the FIT and OFT tensors, thus they are also tensors. In the remaining paper, we refer these patch-level tensors as patches (or macro-patches) unless stated otherwise. It have eight different CNN networks, which are constructed from three architectures I, II, and III, as illustrated in Fig. 5. Architectures I and II are designed to extract features from multi-scale patches and architecture III is designed to learn the neighborhood relationship of a set of atomic patches in a macro-patch. They provide complementary information. Compared to architecture I, architecture II has one more convolution layer and one more pooling layer due to its larger input size. Compared with architectures I and II, architecture III has an additional convolution layer after the softmax for combining the softmax outputs of multiple input atomic patches. We name the CNN networks based on the architectures I and II as cCNNs, and the CNN networks based on the architecture III as pCNNs.

cCNNs: There are four cCNN networks corresponding to four input sizes 64$\times$64, 80$\times$80, 96$\times$96, and 192$\times$192 (detailed in Appendix A). Under each network, two models are trained to extract complementary information from two types of inputs: FIT type input and OFT type input. Hence, there are a total of eight cCNN models. Each model will generate two outputs corresponding to two sets of input patches cropped from FIT₁ and FIT₂ (or OFT₁ and OFT₂), respectively⁹⁹9The input patches to the cCNN models are obtained using a sliding window and thus there are many cropped patches corresponding to a cCNN model. They are all input to the corresponding cCNN model and their outputs are averaged for one output. Thus, for each model, there are two outputs corresponding to two sets of input patches cropped from FIT₁ and FIT₂ (or OFT₁ and OFT₂). . Therefore, there are 16 outputs from cCNNs.

pCNNs: There are also four pCNN networks corresponding to four input sizes 32$\times$32, 48$\times$48, 64$\times$64, and 96$\times$96 (detailed in Appendix B). However, four models are trained under each network to extract complementary information from four types of inputs: FIT type input and OFT type input each generated through two methods $A$ and $B$ (detailed in Section 3.3.3). Hence, there are a total of 16 pCNN models. Similar to cCNNs’ output generation, there are 32 outputs from pCNNs.

A total of 48 outputs are generated by these 24 CNN models, and they pass through a pooling layer which can reduce the prediction variance by averaging the two outputs of each CNN model. Finally, the RBM makes a decision based on the outputs of this pooling layer.

Motivation: Latent fingerprints have major challenges of distorted/corrupted areas and small ROIs. Our hybrid deep network is delicately designed to handle these challenges through the utilization of noise resistant orientation field, multi-scale patches, and neighborhood relationship of macro-patches. The patch partition can help generate some quality inputs to the network and reduce the impact of poor-quality parts due to the distorted/corrupted areas. As the average fingerprint ridge frequency (distance between ridges) is about eight pixels for images with 500dpi resolution [23], the smallest atomic patch is set as 32$\times$32. This would make sure the atomic patch contains at least three ridges/valleys. A smaller size patch will likely contain one or two ridges/valleys, which is less reliable or discriminative. By the same token, the stride of different patch sizes is set to be 16 pixels, which can normally add two new ridges/valleys incrementally. Also, we have surveyed the sizes of the ROI area of all latent fingerprints in SD27 database which is a practical database provided by the FBI and found that the smallest ROI area is larger than the size of 32$\times$32 which fits well with the smallest atomic patch size we set. Accordingly, we have the patch size list 32$\times$32, 48$\times$48, 64$\times$64, 80$\times$80, 96$\times$96, $\dots$, 192$\times$192. The largest patch size is set to be 192$\times$192 because 95% of the latent fingerprints have a ROI area larger than the size of 192$\times$192. A size larger than 192$\times$192 would increase the number of latent fingerprints whose ROI sizes are smaller than 192$\times$192, resulting patches with many void areas. The setting of the numbers and sizes of cCNNs and pCNNs in our deep network can systematically cover almost all the sizes in the above patch list.

The cCNNs are designed to model the pair-relationship from patches. They have different input sizes and thus can encode multi-level local relationships into the pair-relationship. To enhance the network’s discriminative power and reduce the number of parameters, we set the kernel size to be 3$\times$3 for all convolution layers except the last one. The kernel size of the last convolution layer is adaptively decided to ensure that the final output is always of size 1$\times$1. Therefore, the cCNN network corresponding to the input size of 192$\times$192 uses architecture II instead of architecture I (Details of the parameters of the four cCNN networks can be found in Appendix A). A cCNN network takes its corresponding sized patch as input and outputs a two-way softmax which represents the probability distribution over two classes. For all convolution layers except the last one, ReLU active function is adopted. Also, the dropout is employed as a regularizer on the fully-connected layer to prevent over-fitting.

Training Input Generation: The input patches are cropped from the FIT and OFT tensors using a sliding window. In addition, the entire FIT and OFT tensors are resized to 192$\times$192 for use as patches, which ensures that the entire overlap of the two fingerprints is considered. To exclude unreliable patches from these patches, a quality evaluation needs to be conducted. As the FOMFE-based orientation is comparatively reliable, the quality evaluation is not conducted on the patches cropped from the OFT tensor. This also enables the OFT patches to provide complementary information and guidance in the case of missing quality FIT patches. The quality evaluation on the FIT patches is conducted as follows. Firstly, good-quality masks are respectively obtained for the latent and rolled fingerprints in the FIT. Then, only the patch with more than 75% of its area being the overlap of two good-quality masks is treated as quality patch and kept. The good-quality mask for the latent fingerprint in the FIT is its quality map generated in Section 3.2.1. The good-quality mask for the reference fingerprint in the FIT is generated as follows. Firstly, its quality map $\mathcal{Q}$ is obtained by evaluating its orientation coherence [1] which is formulated as:

with

where $(x_{0},y_{0})$ represents a 17$\times$17 window center. The local gradient is calculated as $(g_{x},g_{y})$ in a 3$\times$3 window with $(x,y)$ being its center. $\mathcal{N}$ is a 17$\times$17 window centered at the pixel $(x_{0},y_{0})$. Then, the good-quality mask is generated by applying a threshold 0.9 on $\mathcal{Q}$.

The pCNNs are designed to model the pair-relationship from macro-patches which can be generated by two methods $A$ and $B$, and formulated as:

where $\Phi(A,j)$ and $\Phi(B,j)$ are macro-patches generated by the methods $A$ and $B$, respectively. $\phi_{c}(j)$ is one of the nine atomic patches, $j\in\{32,48,64,96\}$ represents the atomic patch size, and $c$ is the center coordinate of $\phi_{c}(j)$. $d_{x}$ and $d_{y}$ are the Euclidean distances along the $x-$ and $y-$ directions. This way, the macro-patch pair-relationship can be modeled by summarizing its nine atomic patches’ pair-relationships through convolution. Such summarization can increase the robustness of the pair-relationship modeling against the unreliable pair-relationship occurred at some atomic patches.

All four pCNN networks use the same architecture III, which consists of a sub-architecture (marked by the pink dotted rectangle in the middle) identical to the architecture I and a following convolutional layer used to summarize the softmax outputs of nine input atomic patches, as shown in Fig. 5(c). Detailed parameters of these four pCNN networks can be found in Appendix B. The inputs of the pCNNs are nine atomic patches in a macro-patch and the output is a prediction representing the pair-relationship of the macro-patch area. Hence, a pCNN can capture the neighborhood relationship of the central atomic patch and provide complementary information to the cCNNs.

Training Input Generation: The input macro-patches are cropped from the FIT and OFT tensors using a sliding window. Similar to the cCNN training input generation, the quality evaluation is also conducted only on the macro-patches cropped from the FIT to exclude unreliable ones. Also, the same good-quality masks used in the cCNN training input generation are used here. A FIT macro-patch will be kept if it has more than four atomic patches where each one has more than 75% of its area inside the overlap of the two good-quality masks.

RBM is a generative stochastic artificial neural network which learns a probability distribution over its set of inputs. It has been successfully applied to a classification task by Hinton et al. [24], where the joint distribution of an input $\textbf{x}=({x_{1},\ldots,x_{m}})$ and output $y\in({1,\ldots,C})$ is modeled using a hidden layer of binary stochastic units $\textbf{h}=({h_{1},\ldots,h_{H}})$. This is achieved by firstly defining an energy function as:

where W and U are weight matrices. r, s, and t represent the bias weights (offsets) for input x, hidden units h and the representation $\textbf{e}_{y}$, respectively. $\textbf{e}_{y}=(1_{i=y})^{C}_{i=1}$ is the ‘one out of $C$’ representation of $y$. Then, the probabilities of the values of $y$ and x can be assigned as [24]:

where $g(\cdot)=\log(1+\exp(\cdot))$.

The RBM is adopted in the proposed hybrid deep network to combine the outputs of the set of CNNs and make a decision on whether an input fingerprint pair is a genuine pair or not. Since it is conducted in a supervised learning setting and only needs to obtain a good prediction of the target according to the given input, discriminative training is adopted to minimize the objective function $\mathcal{L}$ formulated as:

where $\mathcal{T}={(\textbf{x}_{i},y_{i})}$ represents the training set, and $L(\textbf{x}_{i})=-\log p(y_{i}|\textbf{x}_{i})$ is the RBM loss function.

A challenging problem of applying deep learning to latent fingerprint application is the training database preparation. As the current public databases are either short of the correspondence between latent fingerprints and their true mates or lack of quantity, they are more suitable to be used as the evaluation databases to evaluate the performance of the proposed method. Therefore, for training the hybrid deep network, we propose to generate the training database using the NIST SD14 database [25] as follows.

The NIST SD14 database consists of 54,000 rolled fingerprints, and thus we synthesize 54,000 corresponding latent fingerprints for them. To generate latent fingerprints that better mimic real cases, we propose to add complex and realistic noise to the rolled fingerprints to synthesize their latent ones. This also enables the proposed deep network to learn more effective pair-relationship modeling from tough latent fingerprint situations.

To this end, firstly, a realistic noise image database is created, which consists of noise images with diverse types of noises. Those noise images are cropped from natural images and resized to the same size as the rolled fingerprint images in NIST SD14 database. Then, given a rolled fingerprint $I$, a plastic distortion [26] is added by

where $\textbf{v}=[x,y]^{T}$ is a point in $I$ and $\textbf{v}^{\prime}$ is its distorted point. $l$ is the skin plasticity coefficient. $\Delta(\textbf{v})$ is the torsion and traction amount computed on the basis of a rotation angle $\theta$ and a displacement vector $\textbf{a}=[a_{x},a_{y}]^{T}$, which is given by

where $\textbf{o}_{r}$ is the center of rotation. $g(f(\textbf{v}),l)$ is the gradual transition defined as:

Function $f(\textbf{v})$ returns a measure proportional to the distance between the point and the border of an ellipse centered at $\textbf{o}_{e}$ with semi-axes $s_{x}$ and $s_{y}$, and is formulated as:

These plastic distortion parameters $l$, $\theta$, $(a_{x},a_{y})$, $s_{x}$, and $s_{y}$ are set in the ranges of $[0.5,2]$, $[0,5]$, $[-15,15]$, $[0.2b,0.6b]$, and $[0.2b,1.2b]$, respectively, where $b$ is half the width of the image. Fig. 6 illustrates samples of multiple generated distortions and their related distorted fingerprints for one rolled fingerprint.

Finally, a synthesized latent fingerprint is generated by integrating the distorted fingerprint and a noise image randomly picked up from the created noise image database, with the intensity degree of the noise image ranging from $0.2$ to $0.8$. Fig. 7 shows some examples of the synthesized latent fingerprints. Overall, the training database consists of the rolled fingerprints in the NIST SD14 database and their generated corresponding latent fingerprints.

In the NIST SD14 database, the two fingerprint impressions of each finger are stored with labels ‘f’ and ‘s’. A positive pair of fingerprints is formed by the latent fingerprint derived from ‘f’ and the rolled fingerprint ‘s’, or the latent fingerprint derived from ‘s’ and the rolled fingerprint ‘f’. Thus, there is a total of 54,000 positive pairs of fingerprints for the training. The negative pairs of fingerprints are formed by randomly selecting a non-mated rolled fingerprint for each of the synthesized latent ones, and thus the total number of the negative pairs is also 54,000. We randomly choose 80% of these pairs of fingerprints to train the set of CNNs and use the remaining 20% pairs to train the RBM and tune the entire network.

Latent fingerprint databases: In line with current methods, the latent fingerprint databases NIST SD27 [27] and IIIT-Delhi MOLF [28] are used to evaluate the proposed method.

Reference fingerprint databases: In the literature, many methods use different reference databases. For a comprehensive evaluation, four reference databases are used in this paper and summarized as follows.

Reference database DB-A: It consists of the rolled fingerprints provided in NIST SD27 database.

Reference database DB-B: It consists of the rolled fingerprints provided in NIST SD27 database and the 2,000 fingerprints in NIST SD04 database.

Reference database DB-C: It consists of the rolled fingerprints provided in NIST SD27 database and the 27,000 fingerprints in NIST SD14 database.

Reference database DB-L: In general, the larger the reference database, the closer it will be to the realistic application of latent fingerprint recognition. The latest methods [12] [14] use a large-scale reference database consisting of 100,000 fingerprints. However, it is not publicly available. Therefore, the large-scale reference database DB-L is created using public rolled fingerprint databases: NIST SD04 [29], NIST SD14 [25], FVC2002 DB1A-DB4A [30], FVC2004 DB1A-DB4A [31], FVC2006 DB1A and DB3A [32], Sokoto [33], the database in [34], the database in [35], and the database in [36], and the associated full fingerprints in databases: NIST SD27 [27], IIIT-Delhi MOLF [28], and IIIT-D Latent [37], which has a total of 97,998 fingerprints. All the fingerprint images are resized to 500ppi for the recognition. Note that the NIST SD14 database has been only used for synthesizing latent training data and has not been used in any way for the recognition task, and thus it can be used here to enlarge the reference database.

For the evaluation, all patches and macro-patches cropped from the FIT and OFT are correspondingly input to the trained hybrid deep network. The quality evaluation used in the training input generation is not used here. For the case of the size of the FIT and OFT being smaller than the size of the required patch (or macro-patch), the required patch (or macro-patch) is generated by filling the remaining area with $255$. The performance of the latest method [14] reported in the following is obtained by running its public code.

The NIST SD27 database is provided by the NIST in collaboration with the FBI. It consists of 258 crime-scene latent fingerprints, which are classified based on three different image qualities, namely ‘good’, ‘bad’, and ‘ugly’, with numbers of images 88, 85, and 85, respectively. The resolutions of these fingerprint images are 500ppi. Many methods [4, 6, 7, 8, 9, 10, 12, 13, 14] have been evaluated on this database but their reference databases vary. We evaluated the proposed method (using DLO and M-DLO alignments respectively) over the four reference databases (DB-A, DB-B, DB-C, DB-L). Table I compares the rank-1 accuracies of the proposed method, the recent method [10] which uses manually marked minutiae, and the latest method [14]. It has been reported that using manually marked minutiae could improve the accuracy performance by up to 38% [38]. This is not surprising because it gets the benefit of the assistance from latent fingerprint experts. Compared with these results achieved using manually marked minutiae, our results without using any manual inputs are competitive, with two results being very close and one result being better. Also, our results evaluated using four reference databases are all superior to those of the latest method [14].

Fig. 8 compares the CMC curves of the proposed method and the latest method [14] (labeled as Cao2020 in the figure) evaluated using the reference database DB-L with respect to both the overall and the three quality types of latent fingerprints in the NIST SD27 database. As can be seen, the performance of all algorithms for good latent fingerprints are significantly better than those for bad and ugly latent fingerprints. This is expected because the quality of fingerprints has an important impact on the recognition performance. The low-quality latent fingerprints usually mean seriously corrupted ridges and small regions of fingers, which increase the difficulty of recognition. The proposed method achieves better results than the latest method on all these three quality types of fingerprints.

The IIIT-Delhi MOLF database is publicly provided by [28]. It contains 4,400 latent fingerprints sampled from 10 fingers of 100 individuals. The associated reference fingerprints are captured using three different sensors and partitioned into three sub-databases labeled as ‘L’, ‘S’, and ‘C’, respectively. The resolution of these fingerprint images is 500ppi. These latent fingerprints are very challenging for recognition due to their bad quality, as shown in Fig. 9. The latent fingerprint recognition experiments are conducted according to the testing protocol established in Experiment III in [28]. The recognition performance reported in [28] is used as a baseline.

Fig. 10 compares the CMC curves of the proposed method, the latest method [14] (labeled as Cao2020 in the figure), and the baseline with respect to the three reference sub-databases. As can be seen, the proposed method achieves much better rank-1 accuracy and a continuously rising identification rate with the increase of $N$ for all three reference sub-databases. This is attributed to: 1) the proposed pair-relationship modeling is robust for recognition because it does not rely on extracting reliable representation features from a latent fingerprint image; and 2) the preprocessing provides relatively reliable OFT as a supplement to FIT for the pair-relationship modeling.

To analyze the soundness of the proposed hybrid deep network, latent fingerprint recognition experiments using different architecture variants are conducted as follows.

Experiment I: half of the proposed 24 CNNs are used for the latent fingerprint recognition. This is achieved by removing a total of $12$ CNNs: the cCNN₁ (including cCNN_1,1 and cCNN_1,2 ), cCNN₃ (as cCNN₁), pCNN₅ (including pCNN_5,1, pCNN_5,2, pCNN_5,3 and pCNN_5,4), and pCNN₇ (as pCNN₅), and using the remaining 12 CNNs to form a new architecture which is named as M-Half.

Experiment II: only cCNN-typed CNNs are used for the latent fingerprint recognition. Specifically, cCNN₁ (including cCNN_1,1 and cCNN_1,2), cCNN₂ (as cCNN₁), cCNN₃ (as cCNN₁), and cCNN₄ (as cCNN₁) are used to form a new architecture named as M-C.

Experiment III: only CNNs taking FIT patches or marco-patches as input are used for the latent fingerprint recognition. Specifically, cCNN_1,1, cCNN_2,1, cCNN_3,1, cCNN_4,1, pCNN_5,1, pCNN_5,2, pCNN_6,1, pCNN_6,2, pCNN_7,1, pCNN_7,2, pCNN_8,1, and pCNN_8,2 are used to form a new architecture named as M-F.

These experiments are conducted using the experiment settings described in Section 4.1.2 and 4.1.4. The performance of these three models is evaluated using IIIT-Delhi MOLF database with testing protocol described in Section 4.2.2. Fig. 11 compares the CMC curves of these three models (M-Half, M-C, M-F) and the proposed full model (with $24$ CNNs and named as M). As can be seen, the performance of these three models is worse than that of the proposed full model. Specifically, first, the results of M-Half and M-C support our claim that different sizes and types of CNNs can model multi-level pair-relationship and provide complementary information to contribute to the final decision. This can be further explained by observing the outputs of the CNNs, which are removed in forming the models M-half, M-C, and M-F, in the model M. Fig. 12 shows the outputs (before average, see Footnote 9) of one of these CNNs (cCNN_1,1) for a true pair of fingerprints (a latent fingerprint and its true match) and a false pair of fingerprints. As can be seen, nearly half of the outputs (49%) of the true pair are greater than 0.5, and most outputs (72%) of false pair are less than 0.5. This proves the contribution of cCNN_1,1 (removed and not included in M-Half) to the final decision and explains why the result of M-half is worse than that of M. Second, the result of M-F demonstrates the necessity and effectiveness of incorporating the orientation field for the pair-relationship modeling.

Alignment Accuracy Analysis: Two experiments IV and V are conducted to evaluate the alignment accuracy in terms of two metrics.

Experiment IV: The number of matched genuine minutia pairs after the proposed M-DLO alignment is compared with that of the DLO alignment and the manually marked minutiae. Fig.13 shows the statistical results on the NIST SD27 database. As can be seen, the M-DLO alignment is significantly better than the DLO alignment with larger numbers of matched genuine minutia pairs and comparable with the ground-truth alignment with similar numbers of matched genuine minutia pairs. The MSE values 480.7519 and 7.3178 obtained by the D-LO and M-DLO alignments mean that the average numbers of unmatched genuine minutia pairs are 18.72 and 2.47, respectively.

Experiment V: The orientation errors produced by the DLO and the proposed M-DLO alignments are compared. The orientation error measures the difference between two aligned orientation fields of the latent and its corresponding rolled fingerprints and is calculated using the Eq. (3) in method [17]. Fig. 14 shows the statistical results on the NIST SD27 database. As can be seen, the M-DLO alignment achieves a moderately smaller alignment error than the DLO alignment. The MSE values 84.38 and 80.99 mean that the average differences between two aligned orientations by the DLO and M-DLO alignments are 8.78 and 8.59, respectively.

Alignment Impacts on Recognition Performance: Based on the experiments IV and V, we can conclude that: 1) the M-DLO alignment is more accurate than the DLO alignment. This alignment improvement also leads to the corresponding recognition performance improvement (see Table I). 2) The hexagon partition of the DLO algorithm is constructed around a random point which could produce inconsistent hexagon partitions. Our proposed M-DLO constructs the hexagon partitions based on the most reliable minutia, which leads to better consistent partitions evidenced by more matched genuine minutia pairs. 3) A larger number of unmatched pairs of minutiae will generally lead to a poor recognition performance in many minutia-based algorithms. However, our recognition network depends on the orientation field instead of direct minutia features. Both DLO and M-DLO alignments are orientation-based coarse alignments, but they also provide the advantage of noise resistance due to their averaging characteristics. Also, both the DLO and M-DLO alignments select the alignment point based on the minimum average orientation difference cost among all candidate points. Therefore, the minimum alignment error, in terms of orientation difference cost, is guaranteed even though there exists a certain number of unmatched minutiae. Therefore, our proposed recognition network is less sensitive to the number of matched genuine minutia pairs. This is evidenced by the graceful performance degradation (see Table I) even though there exists a significant variation of the number of matched genuine minutia pairs.

Two ablation experiments VI and VII that using the same preprocessing module but different recognition methods are conducted to evaluate the preprocessing impact on the recognition performance. Results are summarized in Table II, and shown in Fig. 16.

Experiment VI: The same preprocessing module (Cao’s [14]) with different recognition methods (Cao’s and ours) are respectively used for the latent fingerprint recognition on the NIST SD27 database against the reference database DB-L.

Experiment VII: The same preprocessing module (ours) with different recognition methods (Yager’s [17] and ours) are respectively used for the latent fingerprint recognition on the IIIT-Delhi MOLF database with respect to the three reference sub-databases ‘L’, ‘S’, and ‘C’.

In experiment VI, Cao’s preprocessing module plus M-DLO alignment and Cao’s preprocessing module are regarded as the same preprocessing module because: 1) Cao’s system is independent of alignment. Hence, it implies that its system performance is independent of alignment, i.e., Cao’s preprocessing module plus M-DLO is virtually the same as using Cao’s preprocessing module in Cao’s framework. 2) It is impossible to directly connect Cao’s preprocessing module to our recognition network if alignment is not added because our recognition network requires inputs of tensors of latent-rolled fingerprints where a coarse registration is required. Therefore, the best we can do is to use Cao’s preprocessing module plus the alignment component. Also, the enhanced fingerprint image generated by Cao’s preprocessing method is a gray image, and we translate it into the binary image by a threshold $128$. The generated orientation field is also regularized by the FOMFE model for the alignment.

Table II summarizes the results of these two ablation experiments. As can be seen, in the case of using Cao’s preprocessing method with different recognition methods, our proposed recognition method has enhanced the rank-1 accuracy by 2.94%. In the case of using ours preprocessing method with different recognition methods, our recognition method has enhanced the rank-1 accuracy by 41.32%, 40.64%, and 40.86% over sub-databases ‘L’, ‘S’, and ‘C’, respectively. These experimental results strongly support the capability of our recognition network.

Fig. 15 and Fig. 16 compares respectively the CMC curves of the two methods in experiment VI (labeled as ‘CaoPre+CaoRec’ and ‘CaoPre+OurRec’ in the figure) and the two method in experiment VII (labeled as ‘OurPre+YagerRec’ and ‘OurPre+OurRec’ in the figure). As can be seen, the performance achieved by using our recognition network is much better than using the other recognition methods, which demonstrates the soundness and robustness of the proposed hybrid deep network.

An illustration example (see Appendix D) is also provided as follows. For a pair of fingerprints recognized by the proposed recognition method at rank-1, an alignment noise $(\Delta{x}=10,\Delta{y}=-10,\Delta{\theta}=10^{\circ})$ is added to its M-DLO alignment to generate new FIT and OFT tensors input to the hybrid deep network for the recognition. Experimental result shows that it can still be recognized by the proposed recognition method at rank-1, which demonstrate the robustness of the proposed method against inaccurate alignment.

To verify the effectiveness of the joint optimization, we compare the recognition performance of the hybrid deep network trained with and without the joint optimization evaluated on the IIIT-Delhi MOLF database. Fig. 17 shows the CMC curves of these two networks trained with and without the joint optimization, which are labeled as M and M-NoJO in the figure, respectively. As can be seen, the performance achieved by the network without the joint optimization is obviously worse than that of the network with the joint optimization, which supports our choice of the joint optimization.

A visualization of the features learned by the first and second layers of the cCNN_4,1 is shown in Fig. 18. As can be seen, the proposed deep network tends to learn the similarity of two fingerprint images. The higher the similarity, the higher the response in the feature maps. For a genuine pair of fingerprints, most of its feature maps have a high response, and for a false pair of fingerprints, most of its feature maps have a low response. This shows the interpretability and the soundness of the proposed deep network and the learned features.

The hybrid deep network is trained off-line on the Australia National Computational Infrastructure (NCI). The training time for each component and the overall hybrid deep network is provided in Appendix E. The recognition is run on Intel(R) Xeon(R) E5-2670 CPUs (2.6 GHz) powered by a Linux system. On the recognition stage, using the pre-trained network to perform prediction for each fingerprint pair about 0.2s. This time cost is mainly due to the processing of the large number of patches and macro-patches by the set of CNNs. To reduce the computational burden caused by this, we use 24 threads to process the patches and marco-patches for different CNNs in parallel, which can reduce the running time to less than 0.02s.