Novel Nonlinear Neural-Network Layers for High Performance and Generalization in Modulation-Recognition Applications

Identification and classification of wirelessly transmitted digital communication signals into their respective modulation schemes without prior knowledge of signal parameters is a challenging problem that occurs in both military and commercial applications such as spectrum monitoring, signal intelligence, or electronic warfare [1], and conventional approaches to modulation classification employing either likelihood-based methods [2] or CSP-based techniques [3] can prove challenging [4]. As an alternative, in recent years, DL-based techniques have been explored for blind classification of digitally modulated signals using for NN training and signal recognition/classification either the raw I/Q signal components [5, 6, 7, 8], or pre-processing them to extract CC features [9] or other information that characterizes the signal amplitude/phase and frequency domain representations [10, 11, 12, 13].

We note that a common theme of the DL-based approaches proposed for signal classification involves the use of NN layers with structures developed in the context of image processing, which appear to be poorly suited to extract information from time-domain signal samples. Specifically, the DL-based approaches that use raw I/Q data as inputs along with conventional image processing NN layers for modulation classification have been shown to be unable to generalize to new I/Q data when the distribution of the signal parameters of the new data differs slightly from the training data [5, 6]. Furthermore, pre-processing the I/Q data to obtain CCs and using these in conjunction with CAPs having conventional image processing NN layers results in robust signal classification and generalization performance [9].

Motivated by the brittleness of DL-based classifiers that feed I/Q signal components into conventional image processing NN layers, and with the knowledge that DL-based classifiers using CCs are robust, we propose a new approach to DL-based classification of digitally modulated signals, which involves custom-designed NN layers that perform specific nonlinear mathematical functions on the input I/Q data to enable downstream conventional image processing NN capsules to identify specific features that are similar to those implied by the CCs. By combining novel function layers that force the NN to extract periodic (cyclic) features with capsules, our proposed approach enables a DL-based classifier to use the I/Q signal components as inputs, obtain generalization that outperforms other types of NNs in the classification of digitally modulated signals [5], yet obviates the computational expense of estimating CC features. We note that the robustness and generalization abilities of the proposed DL-based digital modulation classifier are assessed on two distinct datasets that are publicly available [14] and include signals with similar digital modulation schemes but which have been generated independently using distinct signal distribution parameters, such that data from the testing dataset is not used in the training dataset.

The remainder of this paper is organized as follows: we provide related background information on CSP and the cycle frequency (CF) features relevant for digital modulation classification in Section II, followed by presentation of the novel NN function layers and proposed NN structure in Section III. In Section IV we provide details on the datasets used for training and testing the NNs, and we present numerical performance results in Section V. We conclude the paper with discussion and final remarks in Section VI.

CSP provides a set of analytical tools for estimating distinct features that are exhibited by digital modulation schemes and that can be used to perform classification of digitally modulated signals in various scenarios involving stationary noise and/or co-channel interference. These tools include higher-order CC estimators [15, 16] and spectral correlation function (SCF) estimators [17, 18] that can be compared to a set of theoretical CCs or SCFs for classifying the digital modulation scheme embedded in a noisy signal.

To accurately estimate CC features for NN classification, specific signal parameters must be either known or accurately estimated from the I/Q data prior to estimating the CC features [9]. These parameters include the symbol interval, $T_{0}$, and symbol rate $1/T_{0}$, the carrier-frequency offset (CFO), $f_{0}$, the excess bandwidth of the signal¹¹1For square-root raised-cosine (SRRC) pulse shaping this is implied by the roll-off parameter, $\beta$., and the signal power level²²2This directly impacts the in-band signal-to-noise (SNR) ratio.. These signal parameters define the CFs needed for CC computation, with a key processing step involving blind estimation of the coarse second-order CF pattern for the signal. CFs $\alpha$ for which a CC is non-zero for typical communication signals include harmonics of the symbol rate $1/T_{0}$, multiples of the carrier frequency $f_{0}$, and combinations of these two sets, such that $\alpha$ can be written as

where $n$ is the order, $m$ is the number of conjugations, and $k$ is the set of non-negative integers typically constrained to a maximum value of $k=5$.

Although the set of possible CFs for all PSK/QAM/SQPSK signals is determined by the symbol rate and the carrier frequency offset in (1), most signals exhibit only a subset of the maximum possible set. Moreover, there are only a few basic patterns: BPSK-like, QAM-like, $\pi/4$-DQPSK-like, 8-PSK-like, and staggered QPSK (SQPSK)-like. If we can find the pattern, we can also determine the actual number of cycle frequencies needed to fully characterize the modulation type through its set of associated CC values.

Two distinct features that are meaningful for identifying which of the five basic CF patterns are present in a signal of interest are [17]:

1. 1.

   The Fourier transform of the squared signal

2. 2.

   The Fourier transform of the quadrupled signal

Furthermore, features that are meaningful in relation to even-order CC estimates are:

1. 1.

   The squared signal

2. 2.

   The quadrupled signal

3. 3.

   The signal raised to a power of six

4. 4.

   The signal raised to a power of eight

Since second and higher even orders in both the time- and frequency-domains are necessary to estimate CC features, we create custom NN function layers that perform generic mathematical equations to provide feature extraction layers that are more consistently meaningful (in that they are proportional to CC features) than the I/Q inputs prior to the trainable branches and final layers with many learnable hyperparameters. The aforementioned list of features is not exhaustive in regards to CCs but only includes orders where no conjugations have been performed ($m=0$). This subset of CC features was chosen for these proposed feature extraction layers to limit the size and complexity of the resulting NN structure.

To consistently extract the second, fourth, sixth, and eighth-order signal features in both time- and frequency-domain using the I/Q signal data, we only need to implement three custom NN layers for the needed nonlinear computations: a squaring layer, a raise to the power of three (or Pow3) layer, and a fast Fourier transform (FFT) layer. These custom layers can then be connected together as shown in Fig. 1, such that their outputs branch out until all eight feature types have been extracted.

Noting that the I/Q signal data is input as pairs of real numbers $(I,Q)$, the squaring and power-of-three layers can be implemented as follows:

– Squaring Layer:

– Pow3 Layer:

For the FFT layer, only the magnitude is needed at the output, with the zero frequency bin in the center, as this information is most likely to reveal CF patterns present in the signal. Therefore, after taking the FFT, its absolute value is obtained to ensure the output magnitude is a real number, suitable for use by downstream trainable NN layers.

The CAP includes eight convolutional neural network (CNN) branches that implement the primary capsules and will be trained to classify eight common digital modulation schemes: BPSK, QPSK, 8-PSK, $\pi/4$-DQPSK, MSK, 16-QAM, 64-QAM, and 256-QAM. The primary capsules contain the majority of the network’s learnable hyperparameters, with each branch having as its input one of the eight desired features. At the output of each branch there is a fully connected layer with eight outputs, which is used to reduce the branch output size to the number of modulation classes in the training dataset; this is done to ensure that each branch has the ability to distinguish between all modulation types in the training dataset. This ability is important because, while not every branch will be able to fully distinguish between all modulation types, each branch will be able to identify which classes it can distinguish between and this determination can be made during NN training. For example, due to the nature of the CF patterns, we expect that the branch with second-order frequency-domain inputs will be able to distinguish between BPSK and MSK, but all other modulation types will not be distinguishable on this branch. Likewise, the branch with fourth-order frequency-domain inputs should be able to distinguish between 8-PSK-like, $\pi/4$-DQPSK-like, and SQPSK-like patterns, but QAM-like and BPSK-like CF patterns may not be distinguishable to this branch. Each branch, having opportunity to distinguish between all eight modulation types, then has its outputs concatenated together with the other branches followed by a final fully connected layer. This last fully connected layer learns the appropriate weights to apply to the outputs of each CNN branch so that each branch’s ability is combined together to obtain the minimum error on the training dataset.

The overall structure of the proposed CAP with custom feature extracting NN layers is shown in Fig. 1, with the characteristics of the subsequent CNN branches for classifying digitally modulated signals outlined in Tables I – III. The number of filters $A$ and filter size $\left[B\times C\right]$ for each convolutional layer are defined in Table I as are the number of output activations (e.g., $X\times Y$) for each layer.

To train and test the performance of the proposed CAP (including its generalization ability) we use two publicly available datasets that both contain signals corresponding to the eight digital modulation schemes of interest (BPSK, QPSK, 8-PSK, $\pi/4$-DQPSK, MSK, 16-QAM, 64-QAM, and 256-QAM). The two datasets are available from [14] as CSPB.ML.2018 and CSPB.ML.2022, and their signal generation parameters are listed in Table IV.

We note that the CFO ranges corresponding to signals in the two datasets are non-intersecting, which allows evaluation of the generalization ability of the trained CAP. Specifically, if the CAP trained on a large portion of the CSPB.ML.2018 dataset displays high classification accuracy on the remaining signals in the dataset, and its performance when classifying signals in the CSPB.ML.2022 dataset is at similarly high levels, then the CAP is robust and has a high ability to generalize³³3Likewise, if its classification accuracy on signals in CSPB.ML.2018 is high, while on signals in CSPB.ML.2022 is low, then its generalization ability is also low..

Prior to training the proposed CAP on these datasets, a blind band-of-interest (BOI) detector [19] is used to locate the bandwidth for signals in the datasets, filter out-of-band noise, and center the I/Q data at zero frequency based on the CFO estimate provided by the BOI detector. Finally, the I/Q data is normalized to unit total power (UTP) so that it does not prevent the activation functions of the NN from converging. To show that using a blind BOI detector to shift the signal to an estimated zero frequency does not necessarily enable an I/Q-trained CAP such as the one in [5] to generalize between the CSPB.ML.2018 and CSPB.ML.2022 datasets, we also use this data to retrain the CAP in [5] as an alternative to provide a point of comparison with the proposed CAP.

The proposed CAP and the alternative CAP in [5] have been implemented in MATLAB and trained on a high-performance computing cluster, such that each CAP was trained and tested two separate times as follows:

• •

  In the first training instance, dataset CSPB.ML.2018 was used, splitting the available signals into $70\%$ for training, $5\%$ for validation, and $25\%$ for testing. The probability of correct classification for the test results obtained using the $25\%$ test portion of signals in CSPB.ML.2018 is shown in Fig. 2.

  The CAPs trained on CSPB.ML.2018 are then tested on dataset CSPB.ML.2022 to assess the generalization abilities of a trained CAP when classifying all signals available in CSPB.ML.2022. The probability of correct classification for this test is shown in Fig. 3.

• •

  In the second training instance, the NN was reset and trained anew using signals in dataset CSPB.ML.2022, with a similar split of $70\%$ signals used for training, $5\%$ for validation, and $25\%$ for testing. The probability of correct classification for the test results obtained using the $25\%$ test portion of signals in CSPB.ML.2022 is shown in Fig. 4.

  The CAPs trained on CSPB.ML.2022 are then tested on dataset CSPB.ML.2018 to assess the generalization abilities of the trained NN when classifying all signals available in CSPB.ML.2018. The probability of correct classification for this test is shown in Fig. 5.

In summary, from the plots shown in Figs. 2 – 5, we note that the proposed CAP with custom feature extraction layers outperforms the alternative CAP from [5] in all training and testing scenarios, displaying high classification performance and generalization abilities.

The overall probability of correct classification ($P_{CC}$) for all experiments performed is shown in Table V and provides further confirmation that the proposed CAP containing the novel custom layers is able to perform feature extraction tailored to the modulated signals of interest more effectively than the alternative CAP in [5], which employs only the conventional NN layers used in the context of image processing. The lowest classification performance seen by the proposed CAP came from training on CSPB.ML.2022 and testing on CSPB.ML.2018 with an overall correct classification probability $P_{CC}=77.1\%$, while the highest classification performance observed for the alternative CAP in [5] came from training on CSPB.ML.2022 and testing on CSPB.ML.2022, resulting in a corresponding overall $P_{CC}=72.1\%$. Thus, even the lowest performance of the proposed CAP exceeds by $5\%$ the best performance of the alternative CAP in [5].

For additional insight into the classification performance of the proposed CAP with custom feature extracting layers, we have also looked at its corresponding confusion matrices, which are shown in Figs. 6 and 7. As can be observed from these figures, the classification performance for individual BPSK, QPSK, 8-PSK, $\pi/4$-DQPSK, and MSK modulation schemes when the CAP is trained and tested on CSPB.ML.2018 dataset is within $1.5\%$ difference of the performance observed when the CAP is trained on CSPB.ML.2018 dataset and tested on CSPB.ML.2022 dataset⁴⁴4Similar confusion matrices are obtained for the cases when the CAP is trained and tested on CSPB.ML.2022 dataset and when the CAP is trained on CSPB.ML.2022 dataset and tested on CSPB.ML.2018, but the corresponding plots are omitted due to space considerations., indicating excellent generalization performance for these five types of digital modulation schemes. Furthermore, the only modulation schemes for which the classification performance of the proposed CAP decreased when the testing dataset was changed from CSPB.ML.2018 to CSPB.ML.2022 are 16-QAM, 64-QAM, and 256-QAM.

The reason the proposed CAP is unable to generalize its training well for the three QAM modulation types is due to the fact that these modulation types are only distinguishable from each other at higher orders when their signal power is known or accurately estimated. Since we normalized the total (signal+noise) power rather than just the signal power, the relative differences between the power levels of the 16-QAM, 64-QAM, and 256-QAM higher-order moments were removed. Thus, the differences between the extracted features for 16-QAM, 64-QAM, and 256-QAM were different enough between CSPB.ML.2018 and CSPB.ML.2022 that the proposed CAP could not obtain good generalization on these three modulation types.

In future work, we plan to investigate additional custom NN function layers that process the I/Q signal data with only signal power normalization, rather than total power normalization, in order to overcome the issue of poor generalization for QAM modulation types, without needing to resort to full estimates of the CCs.

This paper presents a novel DL-based neural network (NN) classifier for digitally modulated signals that uses a capsule (CAP) network with custom feature-extraction layers. The proposed CAP takes as input pre-processed I/Q data, on which blind band-of-interest (BOI) estimation has been applied to enable out-of-band noise filtering and CFO-estimate-based spectral centering. The input data is normalized to unit total power prior to input to the CAP. The proposed custom function layers are essentially homogeneous even-order nonlinear functions, and perform feature generation and extraction on the pre-processed I/Q data more reliably and predictably than convolutional NN layers alone, resulting in very good classification and generalization performance. Prior to this effort, we have been unable to achieve significant generalization using any conventional CAP or convolutional NN that takes I/Q data as input and know of no other published successes.

Future research will involve additional novel NN function layers that enable the use of I/Q data which has been normalized to unit signal power rather than unit total (signal+noise) power, to further improve classification and generalization performance for QAM modulation schemes of interest (16-QAM, 64-QAM, and 256-QAM). Additionally, the inclusion of NN function layers that extract features proportional to non-conjugate CCs ($n=2m$) and conjugate CCs, where the number of conjugations is greater than zero ($n\neq 2m$ and $m\neq 0$), will be considered as this may prove necessary to distinguish between expanded input classes, such as multiple kinds of FSK/CPM/CPFSK, which can have little or no cyclostationarity for $n\neq 2m$. In this way we intend to create neural-network classifiers that can accept I/Q data samples as input, yet simultaneously produce excellent classification performance and high generalization, proving to be highly robust in the face of probability-model departures from the training-set density functions.

The authors would like to acknowledge the use of Old Dominion University High-Performance Computing facilities for obtaining numerical results presented in this work.