Competition over data: how does data purchase affect users?

In this paper, we propose a general competition environment and study what happens when competing ML predictors can actively acquire user data. Our main contributions are as follows.

• •

  We propose a novel environment that can simulate various real-world competitions. Our environment allows ML predictors to use AL algorithms to purchase labeled data within a finite budget while competing against each other (Section 2).

• •

  Surprisingly, our results show that when competing ML predictors purchase data, the quality of the predictions selected by each user can decrease even as competing ML predictors get better (Section 3.1).

• •

  We demonstrate that data purchase makes competing predictors similar to each other, leading to this counterintuitive finding (Section 3.2). Our finding is robust and is consistently observed across various competition situations (Section 3.3).

• •

  We theoretically analyze how the diversity of a user’s available options can affect the user experience to support our empirical findings. (Section 4).

At each round $t\in[T]\mathrel{\mathop{\mathchar 58\relax}}=\{1,\dots,T\}$, we denote a user query by $X_{t}\in\mathcal{X}$ and its associated user label by $Y_{t}\in\mathcal{Y}$. We focus on classification problems, i.e., $|\mathcal{Y}|$ is finite, while our environment can easily extend to regression settings. We denote a sequence of users by $\{(X_{t},Y_{t})\}_{t=1}^{T}$ and assume users are independent and identically distributed (i.i.d.) by some distribution $P_{X,Y}$. We call $P_{X,Y}$ the user distribution.

As for the ML predictor side, we suppose there are $M$ competing predictors in a market. For $i\in[M]$, each ML predictor is described as a tuple $\mathcal{C}^{(i)}\mathrel{\mathop{\mathchar 58\relax}}=(n_{\mathrm{s}}^{(i)},n_{\mathrm{b}}^{(i)},f^{(i)},\pi^{(i)})$, where $n_{\mathrm{s}}^{(i)}\in\mathbb{N}$ is the number of i.i.d. seed data points from $P_{X,Y}$, $n_{\mathrm{b}}^{(i)}\in\mathbb{N}$ is a budget, $f^{(i)}\mathrel{\mathop{\mathchar 58\relax}}\mathcal{X}\to\mathcal{Y}$ is an ML model, and $\pi^{(i)}\mathrel{\mathop{\mathchar 58\relax}}\mathcal{X}\to\{0,1\}$ is a buying strategy. We consider the following setting. A predictor $\mathcal{C}^{(i)}$ initially owns $n_{\mathrm{s}}^{(i)}$ data points and can additionally purchase user data within $n_{\mathrm{b}}^{(i)}$ budgets. We set the price of one data point is one, i.e., a predictor $\mathcal{C}^{(i)}$ can purchase up to $n_{\mathrm{b}}^{(i)}$ data points from a sequence of users. A predictor $\mathcal{C}^{(i)}$ produces a prediction using the ML model $f^{(i)}$ and determines whether to buy the user data with the buying strategy $\pi^{(i)}$. We consider the utility function for $\mathcal{C}^{(i)}$ is the classification accuracy of $f^{(i)}$ with respect to the user distribution $P_{X,Y}$. Lastly, $f^{(i)}$ and $\pi^{(i)}$ are allowed to be updated throughout the $T$ competition rounds. That is, companies keep improving their ML models with newly collected data points.

Before the first competition round, all the $M$ competing predictors independently train their model $f^{(i)}$ using the $n_{\mathrm{s}}^{(i)}$ seed data points. After this initialization, at each round $t\in[T]$, a user sends a query $X_{t}$ to all the predictors $\{\mathcal{C}^{(j)}\}_{j=1}^{M}$, and each predictor $\mathcal{C}^{(i)}$ determines whether to buy the user data. We describe this decision by using the buying strategy $\pi^{(i)}$. If the predictor $\mathcal{C}^{(i)}$ thinks that the labeled data would be worth one unit of budget, we denote this by $\pi^{(i)}(X_{t})=1$. Otherwise, if $\mathcal{C}^{(i)}$ thinks that it is not worth one unit of budget, then $\pi^{(i)}(X_{t})=0$. As for the $\pi^{(i)}$, ML predictors can use any stream-based AL algorithm (Freund et al., 1997; Žliobaitė et al., 2013). For instance, a predictor $\mathcal{C}^{(i)}$ can use the uncertainty-based AL rule (Settles & Craven, 2008)—i.e., $\mathcal{C}^{(i)}$ attempts to purchase user data if the current prediction $f^{(i)}(X_{t})$ is not confident (e.g., the Shannon’s entropy of $p^{(i)}(X_{t})$ is higher than some predefined threshold value where $p^{(i)}(X_{t})$ is the probability estimate at the $t$-th round). In brief, we suppose a predictor $\mathcal{C}^{(i)}$ shows purchase intent if the remaining budget is greater than zero and $\pi^{(i)}(X_{t})=1$. If the remaining budget is zero or $\pi^{(i)}(X_{t})=0$, then $\mathcal{C}^{(i)}$ does not show purchase intent and provides a prediction $f^{(i)}(X_{t})$ to the user. To analyze the complicated real-world competition, we simplify the competition environment and assume that the service price is the same for all companies. This allows users to primarily compare the quality of ML predictions. Given that users select one company within their desired price range, this assumption describes that the options competing companies provide are within this range.

We now elaborate on how a user selects one predictor. At every round $t\in[T]$, the user selects only one predictor based on both purchase intents and prediction information received from $\{\mathcal{C}^{(j)}\}_{j=1}^{M}$. If there is a buyer, then we assume that a user prefers the company with purchase intent to others. We can think of this as a bargain in that the company offers a financial advantage (e.g., discounts or coupons) and the user selects it even if the quality might not be the best. When there is more than one buyer, we assume a user selects one of them uniformly at random. Once selected, the only selected predictor’s budget is reduced by one; all other predictor’s budget stays the same because they are not selected and do not have to provide financial benefits. If no predictor shows purchase intent, then a user receives prediction information $\{f^{(j)}(X_{t})\}_{j=1}^{M}$ and chooses the predictor $\mathcal{C}^{(i)}$ with the following probability.

where $\alpha\geq 0$ denotes a temperature parameter and $W_{t}\in[M]$ denotes the index of selected predictor. Here, $q\mathrel{\mathop{\mathchar 58\relax}}\mathcal{Y}\times\mathcal{Y}\to\mathbb{R}^{+}\mathrel{\mathop{\mathchar 58\relax}}=\{z\in\mathbb{R}\mid z\geq 0\}$ is a predefined quality function that measures similarity between the user label $Y_{t}$ and the prediction (e.g., $\mathbbm{1}(\{Y_{1}=Y_{2}\})$). With the softmax function in Equation (1), users are more likely to select high-quality predictions, describing the rationality of the user selection. Here, the temperature parameter $\alpha$ indicates how selective users are. For instance, $\alpha$ is close to $\infty$, users are very confident in their selection and choose the best company. Afterwards, the selected predictor $\mathcal{C}^{(W_{t})}$ gets the user label $Y_{t}$ and updates the model $f^{(W_{t})}$ by training on the new datum $(X_{t},Y_{t})$. The other predictors $f^{(i)}$ stay the same for $i\neq W_{t}$. We describe our competition system in Environment 1.

Our environment simplifies real-world competition and data purchases, which usually exist in much more complicated forms, yet it captures key characteristics. First, our environment reflects the rationality of customers. Customers are likely to choose the best service within their budget, but they can select a company that is not necessarily the best if it offers financial benefits, such as promotional coupons, discounts, or free services (Rowley, 1998; Familmaleki et al., 2015; Reimers & Shiller, 2019). Such a user selection represents that a user can prioritize financial advantages and change her selection, which has not been considered in the ML literature. Second, our environment realistically models a company’s data acquisition. Competing companies strive to attract more users, constantly purchasing user data to improve their ML predictions. Since the data buying process could be costly for the companies, data should be carefully chosen, and this is why we incorporate AL algorithms. Our environment allows companies to use AL algorithms within finite budgets and to selectively acquire user data. Third, our environment is flexible and takes into account various competition situations in practice. Note that we make no assumptions about the number of competing predictors $M$ or budgets $n_{\mathrm{b}}^{(i)}$, algorithms for predictors $f^{(i)}$ or buying strategies $\pi^{(i)}$, and the user distribution $P_{X,Y}$.

To quantitatively measure the effects of data purchase, we introduce three population-level evaluation metrics. First, we define the overall quality as follows.

where the expectation is taken over the user distribution $P_{X,Y}$. The overall quality represents the average quality that competing predictors provide in the market. Second is the quality of experience (QoE), the quality of the predictor selected by a user. The QoE is defined as

Here, the expectation is over the random variables $(X,Y,W)$ and a conditional distribution of a selected index $P(W\mid X,Y)$ is considered as Equation (1). Given that a user selects one predictor based on Equation (1) when there is no buyer, QoE can be considered as the key utility of users. Note that the overall quality and QoE capture different aspects of prediction qualities, and they are only equal when users select one predictor uniformly at random, i.e., when $\alpha=0$ (See Lemma 1).

Next, we define the diversity to quantify how variable the ML predictions are. To be more specific, for $i\in\mathcal{Y}$, we define the proportion of predictors whose prediction is $i$ as $p_{i}(X)\mathrel{\mathop{\mathchar 58\relax}}=\frac{1}{M}\sum_{j=1}^{M}\mathbbm{1}(f^{(j)}(X)=i)$. Then the diversity is defined as

where the expectation is taken over the marginal distribution $P_{X}$ and we use the convention $0\log(0)=0$ when $p_{i}(X)=0$. Note that the diversity is defined as the expected Shannon’s entropy of competing ML predictions. When there are various different options that a user can choose from, the diversity is more likely to be large.

Our experiments consider the seven real datasets to describe various user distributions $P_{X,Y}$, namely Insurance (Van Der Putten & van Someren, 2000), Adult (Dua & Graff, 2017), Postures (Gardner et al., 2014), Skin-nonskin (Chang & Lin, 2011), MNIST (LeCun et al., 2010), Fashion-MNIST (Xiao et al., 2017), and CIFAR10 (Krizhevsky et al., 2009) datasets. To minimize the variance caused by other factors, we first consider a homogeneous setting in Sections 3.1 and 3.2: for each competition, all predictors have the same number of seed data $n_{\mathrm{s}}^{(i)}$ and budgets $n_{\mathrm{b}}^{(i)}$, the same classification algorithm for $f^{(i)}$, and the same AL algorithm for $\pi^{(i)}$. As for heterogeneous settings in Section 3.3, competitors are allowed to have different configurations of parameters.

Throughout the paper, we set the total number of competition rounds to $T=10^{4}$, the number of predictors to $M=18$, and a quality function to the correctness function, i.e., $q(Y_{1},Y_{2})=\mathbbm{1}(\{Y_{1}=Y_{2}\})$ for all $Y_{1},Y_{2}\in\mathcal{Y}$. We set a small number for seed data points $n_{\mathrm{s}}^{(i)}$, which is between 50 and 200 depending on a dataset. We use either a logistic model or a neural network model with one hidden layer for $f^{(i)}$. As for the buying policy, we use a standard entropy-based AL rule for $\pi^{(i)}$ (Settles & Craven, 2008). We consider various competition situations by varying the budget $n_{\mathrm{b}}\in\{0,100,200,400\}$¹¹1In Section 3, for notational convenience, we often suppress the predictor index in the superscript if the context is clear. For example, we use $n_{\mathrm{b}}$ instead of $n_{\mathrm{b}}^{(i)}$. and the temperature parameter $\alpha\in\{0,1,2,4\}$. Note that a pair $(n_{\mathrm{b}},\alpha)$ generates one competition environment. We repeat experiments 30 times to obtain stable estimates for each pair $(n_{\mathrm{b}},\alpha)$. We provide the full implementation details in Appendix A.

As the evaluation time, we do not perform data buying procedures in order to directly compare the predictability of ML models. As the evaluation metrics are defined as the population-level quantity, it is difficult to compute the expectation exactly. To handle this, we consider the sample averages using the i.i.d. held-out test data that are not used during the competition rounds.