Quantum continual learning of quantum data realizing knowledge backward transfer

Promoted by the interaction between quantum mechanics and computer science, quantum computation [1] is a brand new computation scheme that manipulates quantum information units following the rules of quantum mechanics. Due to the exponentially increasing Hilbert space and unique properties like superposition and entanglement, quantum computation possesses tremendous potential computing power. Many prominent research achievements have appeared in the realm of quantum computation since the end of last century. In recent years, quantum computation has received high attention and significant investment from the industry.

One of the research focuses in quantum computation is quantum machine learning (QML) [2], which is the mergence between quantum computation and machine learning [3]. The efficiency of machine learning can be enhanced by quantum algorithms such as the quantum algorithm for linear systems of equations [4], quantum algorithm for data fitting [5], quantum support vector machine [6, 7], quantum principal component analysis [8] and quantum recommendation system [9]. On the other hand, using quantum circuits with learnable parameters (a.k.a. variational quantum circuits or parameterized quantum circuits) as ML models is a feasible way to integrate quantum and classical computation in the NISQ [10] era. Two typical pioneering quantum-classical hybrid algorithms are variational quantum eigensolver [11] and quantum approximate optimization algorithm [12]. Various common ML problems have been considered using variational quantum algorithms such as classification [13, 14, 15, 16, 17], generalization [18, 19, 20, 21, 22, 23] and autoencoder [24, 25, 26, 27]. The structure of the variational quantum circuit (layout of gates) is fixed and the parameters of the gates are optimized at the first place. Recently, variational quantum compiling [28, 29, 30] and quantum architecture search [31, 32] try to optimize the structures of variational quantum circuits to obtain more efficient circuits.

Intelligence creatures have the ability to continually learn new knowledge and skills and incrementally adapt to new scenarios over the duration of their lifetime. However, learning of new tasks for an ML model often results in forgetting of old tasks, which is called the catastrophic forgetting phenomenon. Continual learning (a.k.a lifelong learning, incremental learning, sequential learning) [33] is a research topic that provides solutions for acquiring knowledge from continuously changing data stream of multiple tasks. Recently, continual learning has been explored in the context of quantum machine learning [34]. It’s been shown that a quantum classifier’s performance on old tasks may deteriorate when learning a new task. On the assumption that some parameters in variational quantum classifiers are more important than others, the elastic weight consolidation (EWC) [35] method can help to protect those parameters with great importance to the old tasks from being updated drastically. Numerical experiments demonstrate that a quantum classifier can continually learn three different classification tasks without catastrophic forgetting. Another goal of continual learning is to learn a new task faster by utilizing past knowledge acquired from old tasks. Ref. [36] proposed a continual reinforcement learning framework to deal with the constantly changing environment. In this framework, the agent leverages past policies learned from previous noise environments to generate a state preparation quantum circuit for a new noise environment. Without training from scratch, the convergence speed can be much faster.

In this work, we investigate the continual learning of a sequence of quantum state classification tasks. The training and testing data are all quantum states, challenging to classical neural networks because the exponential increasing resources required to represent them classically. We adopt the quantum-classical hybrid computing scheme to train a variational quantum classifier, the parameters of which are updated by a classical gradient descent optimizer. We incorporate the gradient episodic memory (GEM) method [37] to keep the classifier remembering previously learned tasks. Before the gradient is used to update the parameters, it’s modified to avoid the reduction of the classifier’s performance on old tasks. A merit of this method is the possibility of positive backward transfer, which denotes the benefits previous tasks get from the training of new tasks. Numerical results demonstrate that the continual learning of six quantum state classification tasks with GEM has higher accuracy and backward transfer than EWC.

This paper is organized as follows. In section 2, we present the related work of classical and quantum continual learning. In section 3, we describe the quantum state classification tasks, the structure of our variational quantum classifier and the GEM method. In section 4, the numerical experiment results of quantum continual learning with GEM and other strategies are shown and compared. Section 5 presents the concluding remarks.

Elastic weight consolidation (EWC) [35] is a famous continual learning method overcoming catastrophic forgetting phenomenon in neural networks. It adds a penalty term to the loss function to confine the parameter update, because the deviation of important parameters from the optimal solutions of previously trained tasks will compromise the classifier’s performance on those tasks. More specifically, the diagonal elements of the Fisher information matrix at the optimal point are saved when each task has been trained. This information tells about the importance of the parameters to the task and can be used to constrain the update strength of parameters. When training the $t$-th task, the model’s parameter $\theta$ is optimized to minimize the following loss function

where $L^{t}(\theta)$ is the original loss function for the $t$-th task, $\lambda$ is a hyper-parameter determining how important the old tasks are compared with the new one, $F^{k}_{j}$ is the $j$-th diagonal element of the Fisher information matrix at the optimal point $\theta^{k}$ for the $k$-th task. By adding the quadratic penalty term, each parameter is pulled back toward its old values by an amount proportional to its importance for performance on previously learned tasks.

The gradient episodic memory (GEM) method [37] uses samples of previous tasks kept in the memory to modify the gradient of the current task. Unlike EWC, the performance of the old tasks is more likely to increase because the previous samples are rehearsed in the parameter update. This phenomenon is called positive backward transfer. Ref. [37] proposed an evaluation metric

to measure the influence that learning a task has on the performance on a previous task. $N_{t}$ denotes the number of tasks. $R_{i,j}$ denotes the test accuracy of the model on the $j$-th task after learning the $i$-th task. Positive BWT means learning new tasks increases the performance of old tasks, and negative BWT means learning new tasks decreases the performance of old tasks, i.e., catastrophic forgetting. Another evaluation metric we use in this paper is the average accuracy defined by

which is the average accuracy on all tasks after the last task has been learned. This is the most commonly used metric in continual learning.

Continual learning has been investigated in the QML field. Ref. [34] adopted the EWC method to help a quantum classifier overcome catastrophic forgetting and learn a sequence of three classification tasks successfully. Ref. [36] studied quantum state preparation for changing environment by training a reinforcement learning agent that utilizes past policies learned from previous environments.

Another subject called transfer learning, which is similar to continual learning to some extent, has also been studied in the QML field. In transfer learning, a model is trained on task B taking advantage of the knowledge of a similar task A. Unlike continual learning, the model’s performance on only task B is concerned. A generic quantum transfer learning scheme was considered in Ref. [38]. In this scheme, a pre-trained classical or quantum model for task A is truncated by removing some final layers. A new trainable classical or quantum model is connected to the end of the pre-trained model. While keeping the parameters of the pre-trained model constant, the new model’s parameters are optimized using the dataset of task B. Ref. [39] proposed a quantum deep transfer learning scheme for feature-based knowledge transfer, which is composed of a model for generating quantum feature data based on quantum Boltzmann machine [40], an algorithm for computing kernel matrices of mixed quantum states, and a quantum alignment algorithm for comparing the characters of the kernel matrices.

We use a variational quantum classifier to classify quantum states in the continual learning fashion. As far as we know, it’s the first time a quantum classifier is trained to solve up to six quantum state classification problems. Each of the six tasks contains a dataset of quantum states that belong to the positive class or the negative class. After random initialization of parameters, the classifier reads the training data from one task after another. The classifier switches to the next task only when it has read all training samples of the current task. After being trained through all the tasks, the classifier is expected to have good performance on all tasks, i.e., predict the class label for a testing quantum state corresponding to anyone of the trained tasks as accurately as possible.

We conduct numerical simulation experiments on continual learning of six quantum state classification tasks given in section 3.1. We compare the GEM method and the EWC method, along with plainly training the tasks one by one (denoted as ‘Plain’ in the following). The average accuracy (ACC) and backward transfer (BWT) defined in section 2 are calculated to evaluate the performance.

The simulation is carried out using the Pennylane library [42] alongside the PyTorch [43] interface. The stochastic gradient optimizer Adam (Adaptive Moment Estimation) [44] is used to update the classifier’s parameters with the initial learning rate 0.1. Each task contains 512 samples, 400 of which are used as training samples and the remains are used for testing. The variational quantum classifier has $N_{l}=1$ layer, and is trained on each task for only 1 epoch with a batch size 10. For the EWC method, 50 samples randomly chosen from the training set are used to calculate the diagonal of the Fisher information matrix. For the GEM method, the capacity of each episodic memory $M_{k}$ is 50 samples which are randomly picked from the training set of the $k$-th task.

With regard to the training order of the tasks, we first try the lexical order 123456 in which the tasks are introduced in section 3.1, then we pick another four random orders that can be found in the first column of table 2. For each training order and strategy, we train 5 classifiers with different random initialization, and report the one with the highest test ACC. The results are summarized in table 2. It can be seen that for all task sequences, GEM achieves higher ACC than EWC and plain training. GEM results in comparable ACC higher than $90\%$ for all task sequences except 123456, yet the ACC of EWC is lower than $80\%$ for all task sequences. The BWT is all positive in the case of GEM, but all negative for EWC, and even lower in the case of plain training. We can conclude that the overall performance of GEM is significantly better than EWC and plain training.

The training curve of task sequence 234561 is plotted in figure 1. The catastrophic forgetting phenomenon appears in figure 1(a) which shows the plain training of a sequence of tasks without any continual learning strategy. After the classifier has been trained on a task, the test accuracy of that task will generally drop obviously when the classifier is continually trained on the following tasks. The knowledge acquired from previous tasks are forgotten gradually. It can also be seen that the test accuracy of task 2 and 3 is around $50\%$, which implies the classifier’s performance is not better than a random guess. The catastrophic forgetting phenomenon can also be observed in figure 1(b) corresponding to the EWC method. Although the ACC has risen from 0.6637 to 0.7307, the classifier still fails to classify the states of task 2 and task 3. Figure 1(c) manifests the effectiveness of the GEM method that successfully overcomes catastrophic forgetting. Although 1 epoch of training on task 2 and 3 cannot result in high test accuracy, the continual training of other tasks help to improve the performance of the classifier on task 2 and 3. After the sixth task has been trained, the model’s test accuracy on task 2 and 3 is above $86\%$, much better than the other two plots. Every task’s test accuracy is higher at the end of training the last task than immediately after training that task, thus the BWT is positive. The numerical results for other 4 task sequences are depicted in figure 2, which shows similar behaviors.

Continual learning is a fascinating research topic because it makes artificial intelligence systems learn in a more human-like way. When facing the always changing reality, a machine learning model must be equipped with the capacity of adaptation to new emerging tasks. Previous work [34] has exhibited that a variational quantum classifier can learn a sequence of 3 tasks without catastrophic forgetting. In this work, we move forward to train a variational quantum classifier sequentially on a total of 6 quantum state classification tasks. With resort to the GEM strategy that yields positive knowledge transfer to previous tasks, we have acquired a quantum classifier with better performance. The positive backward transfer explains the improvement brought by GEM. A drawback of GEM is the necessity of computing gradients of previous tasks at each training iteration, while EWC needs to record the diagonal of the Fisher information matrix only at the last training iteration of each task. Another drawback is the episodic memory cost, which would become a heavy burden if the number of tasks is very large. Addressing the two issues will be the focus of our future research. We hope that more exploration and development of quantum continual learning will be done in the future, paving the way to quantum artificial general intelligence.