Evolving Metric Learning for Incremental and Decremental Features

Wasserstein distance [31] is an optimal transportation to transport all the earth from the source to target destination, while requiring the minimum amount of efforts. Formally, given two signatures $P=\{(x_{pi},\mu_{pi})\}_{i=1}^{m}$ and $Q=\{(x_{qj},\mu_{qj})\}_{j=1}^{n}$, the smoothed Wasserstein distance [32] between $P$ and $Q$ is:

where $D(P,Q)=\{d_{L}(i,j)\}_{i,j=1}^{m,n}\in\mathbb{R}^{m\times n}$, and $d_{L}(i,j)$ denotes the cost of transporting one unit of earth from the source sample $x_{pi}$ to the target sample $x_{qj}$. $F=\{f(i,j)\}_{i,j=1}^{m,n}$ indicates the flow network matrix, and $f(i,j)$ represents the amount of earth that is transported from $x_{pi}$ to $x_{qj}$. $\mu_{p}=[\mu_{p1},\cdots,\mu_{pm}]\in\mathbb{R}^{m}$ and $\mu_{q}=[\mu_{q1},\cdots,\mu_{qn}]\in\mathbb{R}^{n}$ are normalized marginal probability mass vectors, and they satisfy $\sum_{i}\mu_{pi}=1$ and $\sum_{j}\mu_{qj}=1$. $\sigma\geq 0$ is a balance parameter, and $h(F)=-\left<F,\log(F)\right>$ is the strictly concave entropic function.

In Eq. (2), the Mahalanobis distance is employed as ground distance to construct smoothed Wasserstein distance. Thus, each element $d_{L}(i,j)$ of $D(P,Q)$ in Eq. (2) represents the squared Mahalanobis distance between the source sample $x_{pi}$ of $P$ and the target sample $x_{qj}$ of $Q$, i.e., $d_{L}(i,j)=\left\|L(x_{pi}-x_{qj})\right\|_{2}^{2}$. Given the online constructed triplet $(P,Q,K)$ via [33], where the samples of $P$ and $Q$ belong to the same class, and the samples of $P$ and $K$ belong to different classes. After substituting Mahalanobis distance in Eq. (1) with the smooth Wasserstein distance defined in Eq. (2), online Wasserstein metric learning could be formulated as follows:

where $\ell_{L}(P,Q,K)=[1+W_{\sigma}(P,Q)-W_{\sigma}(P,K)]_{+}.$ When compared with the triplet $(x_{p},x_{q},x_{k})$, each signature in $(P,Q,K)$ consists of several samples belonging to same class rather than only one sample.

In one-shot scenario where the features of streaming data would only be incremental and decremental by one time, two essential stages (i.e., T-stage and I-stage) of our proposed EML model for steaming data with feature evolution are elaborated below.

I. Transforming Stage (T-stage): As shown in Fig. 2, suppose that $\{X_{i},Y_{i}\}_{i=1}^{r}$ denotes the streaming data in the T-stage, where $X_{i}=[X_{i}^{v},X_{i}^{s}]\in\mathbb{R}^{n_{i}\times(d_{v}+d_{s})}$ and $Y_{i}\in\mathbb{R}^{n_{i}}$ denote the samples and labels in the $i$-th batch, respectively. $r$ is the total batches in T-stage and $n_{i}$ indicates the sample number in the $i$-th batch. Obviously, each instance of $X_{i}$ consists of vanished and survived features, and $d_{v}$ and $d_{s}$ indicate the corresponding dimensions of vanished features $X_{i}^{v}\in\mathbb{R}^{n_{i}\times d_{v}}$ and survived features $X_{i}^{s}\in\mathbb{R}^{n_{i}\times d_{s}}$.

If we directly combine both vanished and survived features to learn a unified metric function, it fails to be utilized in I-stage where some features are vanished and some other new features are augmented. We thus propose to extract important information from vanished features and forward it into survived features by exploring a common discriminative metric space. In other words, we aim to train a model using only survived features to characterize the effective information extracted from both vanished and survived features.

In the $i$-th batch of T-stage, inspired by [33], the triplet $(P_{i}^{s},Q_{i}^{s},K_{i}^{s})$ for survived features is constructed in an online manner, where the samples of $P_{i}^{s}\in\mathbb{R}^{n_{p}\times d_{s}}$ and $Q_{i}^{s}\in\mathbb{R}^{n_{q}\times d_{s}}$ belong to same class while the samples of $P_{i}^{s}$ and $K_{i}^{s}\in\mathbb{R}^{n_{k}\times d_{s}}$ belong to different classes. $n_{p},n_{q}$ and $n_{k}$ are the numbers of samples in each signature. Likewise, we can construct the triplet $(P_{i}^{a},Q_{i}^{a},K_{i}^{a})$ for all features (containing both vanished and survived features) in T-stage, where the samples of $P_{i}^{a}\in\mathbb{R}^{n_{p}\times(d_{v}+d_{s})}$ and $Q_{i}^{a}\in\mathbb{R}^{n_{q}\times(d_{v}+d_{s})}$ belong to same class while the samples of $P_{i}^{a}$ and $K_{i}^{a}\in\mathbb{R}^{n_{k}\times(d_{v}+d_{s})}$ belong to different classes.

Let $L^{s}\in\mathbb{R}^{k\times d_{s}}$ and $L^{a}\in\mathbb{R}^{k\times(d_{v}+d_{s})}$ denote the distance matrices trained on survived features and all features (containing both vanished and survived features) in T-stage. Since the dimensions of $L^{s}$ and $L^{a}$ are different, it is reasonable to add some essential consistency constraints on the optimal distance matrices $L^{s}$ and $L^{a}$ to extract important information from vanished features, and forward it into survived features. Generally, based on the smoothed Wasserstein metric learning in Eq. (3), the formulation of the $i$-th batch in the T-stage could be expressed as follows:

where $\mathcal{L}_{L^{s}}(P_{i}^{s},Q_{i}^{s},K_{i}^{s})$ and $\mathcal{L}_{L^{a}}(P_{i}^{a},Q_{i}^{a},K_{i}^{a})$ denote the triplet losses of smoothed Wasserstein metric learning on survived features and all features (containing both vanished and survived features), respectively. $\mathrm{rank}(\cdot)=\mathrm{rank}(L^{s})+\mathrm{rank}(L^{a})$ denotes the regularization term, which learns the underlying low-rank property of heterogeneous samples. $\rho\geq 0$ and $\lambda\geq 0$ are the balance parameters. $\mathcal{C}_{L^{s},L^{a}}(\cdot;\cdot)$ in Eq. (4) is designed to enable the consistence constraint for $L^{s}$ and $L^{a}$, which aims to use only survived features to characterize the efficient information extracted from both vanished and survived features.

Specifically, $\mathcal{C}_{L^{s},L^{a}}(\cdot;\cdot)$ constructs an essential triplet loss by incorporating smoothed Wasserstein metric learning on different feature spaces, i.e., survived features and all features (containing both vanished and survived features). We attempt to compute the smoothed Wasserstein distance between different heterogeneous distributions based on vanished features and all features. For example, $W_{\sigma}(P_{i}^{a},Q_{i}^{s})=\{d_{L}(u,v)\}_{u,v=1}^{n_{p},n_{q}}\in\mathbb{R}^{n_{p}\times n_{q}}$ denotes the smoothed Wasserstein distance between $P_{i}^{a}$ from all features and $Q_{i}^{s}$ from survived features, where $d_{L}(u,v)=\left\|L^{a}x_{pu}^{a}-L^{s}x_{qv}^{s}\right\|_{2}^{2}$ indicates the Mahalanobis distance between the $u$-th source sample $x_{pu}^{a}$ of $P_{i}^{a}$ and the $v$-th target sample $x_{qv}^{s}$ of $Q_{i}^{s}$. Likewise, $W_{\sigma}(P_{i}^{a},K_{i}^{s}),W_{\sigma}(P_{i}^{s},Q_{i}^{a})$ and $W_{\sigma}(P_{i}^{s},K_{i}^{a})$ have similar definitions with $W_{\sigma}(P_{i}^{a},Q_{i}^{s})$. Formally, the consistence constraint $\mathcal{C}_{L^{s},L^{a}}(\cdot;\cdot)$ is concretely expressed as follows:

II. Inheriting Stage (I-stage): Suppose that $\{X_{r+1},Y_{r+1}\}$ denotes the data samples in the $r+1$-th batch of I-stage, where $X_{r+1}=[X_{r+1}^{s},X_{r+1}^{n}]\in\mathbb{R}^{n_{r+1}\times(d_{s}+d_{n})}$ indicates the samples and $Y_{r+1}\in\mathbb{R}^{n_{r+1}}$ is the corresponding labels, as shown in Fig. 2. $X_{r+1}^{s}$ and $X_{r+1}^{n}$ represent the survived features and new augmented features in the $r+1$-th batch. $d_{n}$ and $n_{r+1}$ are the dimension of the new augmented features and the number of samples. Thus, the goal of I-stage is to use $\{X_{r+1},Y_{r+1}\}$ for training and make the prediction for the $r+2$-th batch data $X_{r+2}=[X_{r+2}^{s},X_{r+2}^{n}]\in\mathbb{R}^{n_{r+2}\times(d_{s}+d_{n})}$ whose number of samples is same as that of $X_{r+1}\in\mathbb{R}^{n_{r+1}\times(d_{s}+d_{n})}$.

To classify the $r+2$-th batch data, we propose to inherit the metric performance of optimal distance matrix $L^{s}$ learned on survived features in T-stage, since a set of common survived features exist in both T-stage and I-stage. Although we could construct the triplets directly from the $r+1$-th batch for training, this trivial strategy has two significant shortcomings: 1) the trained metric model is difficult to be extended into multi-shot scenario; 2) the metric model learned only with the $r+1$-th batch data would have worse prediction performance due to the lack of full usage of data in T-stage.

To this end, we utilize a similar stacking strategy with [34, 35], where [34, 35] focus on forming linear combinations of different predictors to train a unified classifier and achieve improved prediction accuracy. However, we propose to concatenate all feature descriptors as in stacking and train a unified predictor on the stacked features. It could better inherit the metric performance learned in T-stage. Concretely, let $Z_{r+1}^{s}=X_{r+1}^{s}(L^{s})^{\top}\in\mathbb{R}^{n_{r+1}\times k}$ as the transformed discriminative metric space, which can be regarded as the new representation of $X_{r+1}^{s}$ for stacking. $X_{r+1}$ could then be represented as $Z_{r+1}=[Z_{r+1}^{s},X_{r+1}^{n}]\in\mathbb{R}^{n_{r+1}\times(k+d_{n})}$. Likewise, $X_{r+2}$ is characterized as $Z_{r+2}$. Furthermore, we learn an optimal distance matrix $L^{z}\in\mathbb{R}^{k\times(k+d_{n})}$ on $Z_{r+1}$ with online constructed triplet $(P_{r+1}^{z},Q_{r+1}^{z},K_{r+1}^{z})$, and evaluate the performance on $Z_{r+2}$, where the samples of $P_{r+1}^{z}$ and $Q_{r+1}^{z}$ belong to same class while the samples of $P_{r+1}^{z}$ and $K_{r+1}^{z}$ belong to different classes. Formally, at the $t$-th iterative step, the objective function of learning $L^{z}\in\mathbb{R}^{k\times(k+d_{n})}$ in I-stage can be formulated as:

where $\gamma\geq 0$ and $\lambda\geq 0$ are the balance parameters. In our experiments, $\lambda$ and $\gamma$ in both Eq. (4) and Eq. (6) are set as the same value for simplification. $\mathrm{rank}(L^{z})$ denotes the regularization term, which aims to explore the intrinsic low-rank structure of heterogeneous samples in I-stage.

Different from one-shot scenario, the features of streaming data in multi-shot scenario would be incremental and decremental $M$ times. This subsection extends our model from one-shot case into multi-shot scenario, and the illustration example of multi-shot scenario when $M=2$ is depicted in Fig. 3. Specifically, $\{X_{i},Y_{i}\}_{i=1}^{R/2}$ denotes the streaming data in Stage 1, where $X_{i}=[X_{i}^{v},X_{i}^{s}]\in\mathbb{R}^{n_{i}\times(d_{v}+d_{s})}$ and $Y_{i}\in\mathbb{R}^{n_{i}}$ respectively represent the samples and labels in the $i$-th batch. $n_{i}$ indicates the sample number in the $i$-th batch, and $R/2$ denotes the total batches in Stage 1. When the streaming data $\{X_{i},Y_{i}\}_{i=R/2+1}^{R}$ in Stage 2 arriving, it performs features evolution for the first time (i.e., some features are vanished and some new features are augmented), where $X_{i}=[X_{i}^{s},X_{i}^{n}]\in\mathbb{R}^{n_{i}\times(d_{s}+d_{n})}$. Moreover, in Stage 3, the streaming data $\{X_{R+1},Y_{R+1}\}$ performs features evolution for the second time, and we predict the results of our proposed EML model on the $R+2$-th batch data $X_{R+2}$, where $X_{R+1}=[X_{R+1}^{s},X_{R+1}^{n}]$ and $X_{R+2}=[X_{R+2}^{s},X_{R+2}^{n}]$. Note that there are overlapped feature representations between any two adjacent stages. For example, as presented in Fig. 3, the survived features in Stage 1 are regarded as the vanished features in Stage 2, and the augmented feature in Stage 2 are considered as the survived features in Stage 3. Therefore, there are multiple Transforming stages (T-stage) and Inheriting stages (I-stage) in multi-shot scenario. To be specific, our proposed model first regards Stage 1 and Stage 2 as T-stage and I-stage for the first feature evolution. Then, it considers Stage 2 and Stage 3 as T-stage and I-stage for the second feature evolution. Generally, in multi-shot scenario, we have two essential learning tasks:

• •

  Task I: Similar to the prediction task in one-shot case, we aim to classify testing data $X_{R+2}$ in Stage 3 by training our proposed model on previous $R+1$ batch streaming data $\{X_{i},Y_{i}\}_{i=1}^{R+1}$.

• •

  Task II: Different from the prediction task in one-shot scenario, we attempt to make predictions for all stages (i.e., Stage 1, Stage 2 and Stage 3 when $M=2$) by training our proposed model on the streaming data $\{X_{i},Y_{i}\}_{i=1}^{R+1}$ in all stages.

When fixing the variables $L^{s},H^{a}$ and $F$, the optimization problem in Eq. (4) for solving variable $L^{a}$ can be concretely expressed as:

The optimal solution of $L_{t}^{a}$ could be relaxedly achieved by nulling the gradient of Eq. (7):

where $G_{1}={Q_{i}^{a}}^{\top}\mathrm{diag}(\textbf{1}^{\top}F){Q_{i}^{a}}-{K_{i}^{a}}^{\top}\mathrm{diag}(\textbf{1}^{\top}F){K_{i}^{a}}-{P_{i}^{a}}^{\top}F{Q_{i}^{a}}-{Q_{i}^{a}}^{\top}F^{\top}{P_{i}^{a}}+{P_{i}^{a}}^{\top}F{K_{i}^{a}}+{K_{i}^{a}}^{\top}F^{\top}{P_{i}^{a}},G_{2}={P_{i}^{a}}^{\top}\mathrm{diag}(F\textbf{1}){P_{i}^{a}}-{K_{i}^{a}}^{\top}\mathrm{diag}(\textbf{1}^{\top}F){K_{i}^{a}},G_{3}={K_{i}^{s}}^{\top}F{P_{i}^{a}}-{Q_{i}^{s}}^{\top}F^{\top}{P_{i}^{a}},G_{4}={P_{i}^{s}}^{\top}F{K_{i}^{a}}-{K_{i}^{s}}^{\top}F^{\top}{P_{i}^{a}}$.

With the obtained distance matrix $L^{a}$ and flow matrix $F$, the optimization problem for variable $L^{s}$ in Eq. (4) could be formulated as:

Concretely, the updating operator for $L_{t}^{s}$ could be given as:

where $G_{5}={Q_{i}^{s}}^{\top}\mathrm{diag}(\textbf{1}^{\top}F){Q_{i}^{s}}-{K_{i}^{s}}^{\top}\mathrm{diag}(\textbf{1}^{\top}F){K_{i}^{s}}+{P_{i}^{s}}^{\top}F{K_{i}^{s}}+{K_{i}^{s}}^{\top}F^{\top}{P_{i}^{s}}-{P_{i}^{s}}^{\top}F{Q_{i}^{s}}-{Q_{i}^{s}}^{\top}F^{\top}P_{i}^{s},G_{6}={P_{i}^{a}}^{\top}F{K_{i}^{s}}-{P_{i}^{a}}^{\top}F{Q_{i}^{s}},G_{7}={Q_{i}^{s}}^{\top}\mathrm{diag}(F\textbf{1}){Q_{i}^{s}}-{K_{i}^{s}}^{\top}\mathrm{diag}(\textbf{1}^{\top}F){K_{i}^{s}},G_{8}={K_{i}^{a}}^{\top}F^{\top}{P_{i}^{s}}-{Q_{i}^{a}}^{\top}F^{\top}{P_{i}^{s}}$.

When the distance matrices $L^{a}$ and $L^{s}$ are fixed, we split Eq. (4) into some independent traditional smoothed Wasserstein distance subproblems, which could be solved by [33]. We omit the detailed process of solving smoothed Wasserstein distance subproblems for simplification.

Given the fixed $H^{z}$ and $F$, the formulation for $L^{z}$ in Eq. (6) is rewritten as:

By nulling the gradient of Eq. (11), the optimization solution of Eq. (11) for $L^{z}$ could be given as:

where $G_{9}=Q_{r+1}^{z^{\top}}\mathrm{diag}(\textbf{1}^{\top}F)Q_{r+1}^{z}+P_{r+1}^{z^{\top}}FK_{r+1}^{z}-K_{r+1}^{z^{\top}}\mathrm{diag}(\textbf{1}^{\top}F)K_{r+1}^{z}+K_{r+1}^{z^{\top}}F^{\top}P_{r+1}^{z}-P_{r+1}^{z^{\top}}FQ_{r+1}^{z}-Q_{r+1}^{z^{\top}}F^{\top}P_{r+1}^{z}$.

The optimization procedure of variable $F$ in I-stage is same as that in T-stage: with the fixed $L^{z}$, the formulation Eq. (6) is split into some independent traditional smoothed Wasserstein distance subproblems, and we solve the variable $F$ via [33].