Multi-rubric Models for Ordinal Spatial Data with Application to Online Ratings Data

In recent years, the complexity of data used to make decisions has increased dramatically. A prime example of this is the use of online reviews to decide whether to purchase a product or visit a local business; we refer to the objects being reviewed as items. Consider data provided by Yelp! (see, http://www.yelp.com/), which allows users to rate items, such as restaurants, convenience stores, and so forth, on a discrete scale from one to five “stars.” Additional features of the businesses are also known, such as the spatial location and type of business. Datasets of this type are typically very large and exhibit complex dependencies.

As an example of this complexity, users of Yelp! effectively determine their own standards when rating a local business. We refer to the particular standards a user applies as a rubric. We might imagine a latent variable $Y_{iu}$ representing the utility, or benefit, user $u$ obtained from item $i$. For a given level of utility, however, different users may still give different ratings due to having different standards for the ratings; for example, one user may rate a restaurant 5 stars as long as it provides a non-offensive experience, a second user might require an exceptional experience to rate the same restaurant 5 stars, and a third user may rate all items with $1$ star in order to “troll” the website. Each of these users are applying different rubrics in translating their utility to a rating for the restaurant. In addition we also expect user-specific selection bias in the sense that some users may rate every restaurant they attend, while other users may only rate restaurants that they feel strongly about.

This article makes several contributions. First, we develop a semiparametric Bayesian model which accounts for the existence of multiple rubrics for ratings data that are observed over multiple locations. To do this, we use a spatial cumulative probit model (e.g., see Higgs and Hoeting,, 2010; Berret and Calder,, 2012; Schliep and Hoeting,, 2015) in which the break-points are modeled as user-specific random effects. This requires a flexible model for the distribution $F$ of the random effects, which we model as a discrete mixture. A by-product of our approach is that we obtain a clustering of users according to the rubrics they are using.

Second, we use the multi-rubric model to address novel inferential questions. For example, ratings provided to a user might be adjusted to match that user’s rubric, or to provide a distribution for the rating that a user would provide conditional on having a particular rubric. Utilizing this user-specific standardization of ratings may provide users with better intuition for the overall quality of an item.

This adjustment of restaurant quality for the rubrics is similar to, but distinct from, the task of predicting a user’s ratings. Good predictive performance is required for filtering, which refers to the task of processing the rating history of a user and producing a list of recommendations (for a review, see Bobadilla et al.,, 2013). As a third contribution, we show that allowing for multiple rubrics improves predictions.

The model proposed here also has interesting statistical features. A useful feature of our model is that it allows for more accurate comparisons across items. For example, if a user rates all items with $1$ star, then the model discounts this user’s ratings. This behavior is desirable for two reasons. First, if a user genuinely rates all items with $1$ star, then their rating is unhelpful. Second, it down-weights the ratings of users who are exhibiting selection bias and only rating items which they feel strongly about, which is desirable as comparisons across items will be more indicative of true quality if they are based on individuals who are not exhibiting large degrees of selection bias.

Additionally, the rubrics themselves may be of intrinsic interest. We demonstrate that the rubrics learned by our model are highly interpretable. For example, when analyzing the Yelp! dataset in Section 4, we obtain Figure 7 which displays the ratings observed for users assigned to a discrete collection of rubrics and reveals several distinct rating patterns displayed by users.

Other features of our model are also of potentially independent interest. The multi-rubric model can be interpreted as a novel semiparametric random-effects model for ordinal data, even for problems in which the intuition behind the multi-rubric model in terms of latent utility does not hold. Other study designs in which the multi-rubric analogy may be useful include longitudinal survey studies, or more general ordinal repeated-measures designs. Additionally, the cumulative probit model we use to model latent user preferences includes a spatial process to account for spatial dependencies across local businesses. Recovering an underlying spatial process allows for recommending entire regions to visit, rather than singular items. The development of low-rank spatial methodology for large-scale dependent ordinal data is of interest within the spatial literature, as the current spatial literature for ordinal data do not typically address large datasets on a similar order of the Yelp! dataset (e.g., see De Oliveira,, 2004, 2000; Chen and Dey,, 2000; Cargnoni et al.,, 1997; Knorr-Held,, 1995; Carlin and Polson,, 1992; Higgs and Hoeting,, 2010; Berret and Calder,, 2012; Velozo et al.,, 2014, among others). We model the underlying spatial process using a low-rank approximation (Cressie and Johannesson,, 2008) to a desired Gaussian process (Banerjee et al.,, 2008; Bradley et al.,, 2015).

Starting from Koren and Sill, (2011), several works in the recommender systems literature have considered ordinal matrix factorization (OMF) procedures which are similar in many respects to our model (see also Paquet et al., 2012 and Houlsby et al., 2014). Our work differs non-trivially from these works in that the multi-rubric model treats the break-points as user-specific random effects, with a nonparametric prior used for the random effects distribution $F$. For the Yelp! dataset, this extra flexibility leads to improved predictive performance. Additionally, our focus in this work extends to inferential goals beyond prediction; for example, depending on the distribution of the rubrics of users who rate a given item, the estimate of overall quality for that item can be shrunk to a variety of different centers, producing novel multiple-shrinkage effects. Several works in the Bayesian nonparametric literature have also considered flexible models for random effects in multivariate ordinal models (Kottas et al.,, 2005; DeYoreo and Kottas,, 2014; Bao and Hanson,, 2015), but do not treat the break-points themselves as random effects.

The paper is organized as follows. In Section 2, we develop the multi-rubric model, with an eye towards the Yelp! dataset, and provide implementation details. In Section 3, we illustrate the methodology on synthetic data designed to mirror features of the Yelp! dataset, and demonstrate that we can accurately recover the number and structure of the rubrics when the model holds, as well as effectively estimate the underlying latent utility field. In Section 4, we illustrate the methodology on the Yelp! dataset. We conclude with a discussion in Section 5. In supplementary material, we present simulation experiments which demonstrate identifiability of key components of the model.

The goal of this simulation is to illustrate that we can accurately learn the existence of multiple rubrics in settings where one would expect it would be difficult to detect them. We consider a situation where the data is generated according to two rubrics that are similar to each other. This allows us to assess the robustness of our model to various “degrees” of the multi-rubric assumption. The performance of our multi-rubric model is assessed relative to the single-rubric model, which is the standard assumption made in the ordinal data literature.

We calibrate components of the simulation model towards the Yelp! dataset to produce realistic simulated data. Specifically, we set $\eta$ and $\sigma^{2}_{b}$ equal to the posterior means obtained from fitting the model to the Yelp! dataset in Section 4. We set $\Sigma_{\eta}=0.5\operatorname{I}$, corresponding to a much stronger spatial effect than what was observed in the data, and for simplicity we removed the latent-factor aspect of the model by fixing $\sigma^{2}_{\beta}\equiv 0$. A two-rubric model is used with $\omega_{1}=\omega_{2}=0.5$. We also use the same spatial basis functions and observed values of $(i,u)$ as in the Yelp! analysis in Section 4.

We now describe how the two rubrics $\theta_{1}$ and $\theta_{2}$ where chosen. First, $\theta_{1}$ was selected so that $\{Z_{iu}:C_{u}=1,i=1,\ldots,I\}$ was evenly distributed among the five responses. Associated to $\theta_{1}$ is a probability vector $p_{1}=(0.2,0.2,0.2,0.2,0.2)$. To specify $\theta_{2}$, we use the same approach with a difference choice of $p$. Let $p_{2}=(0,0.25,0.5,0.25,0)$. Then $\theta_{2}^{(\tau)}$ is associated to $\tau p_{1}+(1-\tau)p_{2}$ in the same manner as $\theta_{1}$ is associated to $p_{1}$. Here, $\tau$ indexes the similarity of $\theta_{1}$ and $\theta_{2}$, and it can be shown that the total variation distance between the empirical distribution of $\{Z_{iu}:C_{u}=1\}$ and $\{Z_{iu}:C_{u}=2\}$ is $0.8(1-\tau)$. Thus, values of $\tau$ near $1$ correspond imply that the rubrics are similar, while values of $\tau$ near $0$ imply that they are dissimilar. Figure 2 presents the distribution of the $Z_{iu}$’s with $C_{u}=2$ when $\tau=0,0.8$, and $1$.

We fit a $10$-rubric and single-rubric model for $\tau=0.0,0.1,\ldots,1.0$. Figure 3 displays the proportion of individuals assigned to each rubric for a given value of $\tau$. If the model is accurately recovering the underlying rubric structure, we expect to see a half of the observations assigned to one rubric, and half to another; due to permutation invariance, which of the 10 rubrics is associated to $\theta_{1}$ and $\theta_{2}^{(\tau)}$ vary by simulation. Up to $\tau=0.9$, the model is capable of accurately recovering the existence of two rubrics. We also see that, even at $\tau=0.8$, the model accurately recovers the empirical distribution of the $Z_{iu}$’s associated to each rubric.

Next, we assess the benefit of using the multi-rubric model to predict missing values. For each value of $\tau$, we fit a single-rubric and multi-rubric model. Using the same train-test split as in the our real data illustration, we compute the log likelihood on the held-out data $\text{loglik}_{\text{test}}=\sum_{(i,u)\in\mathcal{S}_{\text{test}}}\log\Pr(Z_{iu}\mid D),$ which is further discussed in detail in Section 4 . Figure 4 shows the difference in held-out log likelihood for the single-rubric and multi-rubric model as a function of $\tau$. Up-to $\tau=0.8$, there is a meaningful increase in the held-out log-likelihood obtained from using the multi-rubric model. The case where $\tau=1$ is also particularly interesting, as this implies that the data were generated from the single rubric model. Here the predictive performance of our model at missing values appears to be robust to the case when the multiple rubric assumption is incorrect.

Displayed above each point in Figure 4 is the proportion of observations which are assigned to the correct rubric, where each observation is assigned to their most likely rubric. When the rubrics are far apart the model is capable of accurately assigning observations to rubrics. As the rubrics get closer together, the task of assigning observations to rubrics becomes much more difficult.

This simulation study suggests that the model specified here is able to disentangle the two-rubric structure, even when the rubrics are only subtly different. This leads to clear improvements in predictive performance for small and moderate values for $\tau$. Additionally, when the multi-rubric assumption is negligible, or even incorrect, our model performs as well as the single-rubric model.

We now apply the multi-rubric model to the Yelp! dataset, which is publicly available at https://www.yelp.com/dataset_challenge. We begin by preprocessing the data to include reviews only between January $1^{\text{st}}$, 2013 and December $31^{\text{st}}$ 2016, and restrict attention to restaurants in Phoenix and its surrounding areas. We further narrow the data to include only users who rated at least 10 restaurants; this filtering is done in an attempt to minimize selection bias, as we believe that “frequent raters” should be less influenced by selection bias.

We first evaluate the performance of the single-rubric and multi-rubric models for various values of the latent factor dimension $L$. We set $M=20$ and induce sparsity in $\omega$ by setting $\omega\sim\operatorname{Dirichlet}(1/20,\ldots,1/20)$. We divide the indices $(i,u)\in\mathcal{S}$ into a training set $\mathcal{S}_{\text{train}}$ and testing set $\mathcal{S}_{\text{test}}$ of equal sizes by randomly allocating half of the indices to the training set. We evaluate predictions using a held-out log-likelihood criteria

where $\mathcal{D}=\{Z_{iu}:(i,u)\in\mathcal{S}_{\text{train}}\}$, $\Pr(Z_{iu}\mid\mathcal{D})$ denotes the posterior predictive distribution of $Z_{iu}$, and $t=1,\ldots,T$ indexes the approximate draws from the posterior obtained by MCMC. Results for the values $L=1,3$, and 5, over 10 splits into training and test data, are given in Figure 6. We also compare our methodology to ordinal matrix factorization (Paquet et al.,, 2012) with learned breakpoints and spatial smoothing, and the mixture multinomial model (Marlin and Zemel,, 2009) with $10$ mixture components. The multi-rubric model leads to an increase in the held-out data log-likelihood (4) of roughly $5\%$ over ordinal matrix factorization and $8\%$ over the mixture multinomial model. Additionally, we note that the holdout log-likelihood was very stable over replications. The single-rubric model is essentially equivalent to ordinal matrix factorization.

The dimension of the latent factors $\alpha_{u}$ and $\beta_{i}$ has little effect on the quality of the model. We attribute this to the fact that $|\mathcal{U}_{i}|$ and $\mathcal{I}_{u}|$ are typically small, making it difficult for the model to recover the latent factors. On other datasets where this is not the case, such as the Netflix challenge dataset, latent-factor models represent the state of the art and are likely essential for the multi-rubric model. In the supplementary material we show in simulation experiments that the $\alpha_{u}$’s, $\beta_{i}$’s, and $L$ are identified.

Figure 5 displays the learned spatial field $\widehat{W}(s)=\psi(s)^{\top}\widehat{\eta}$ where $\widehat{\eta}$ is the posterior mean of $\eta$. The results suggest that the downtown Phoenix business district and the area surrounding the affluent Paradise Valley possesses a higher concentration of highly-rated restaurants than the rest of the Phoenix area. More sparsely populated areas such as such as Litchfield Park, or areas with lower income such as Guadalupe, seem to have fewer highly-rated restaurants.

We now examine the individual rubrics. First, we obtain a clustering of users into their rubrics by minimizing Binder’s loss function (Binder,, 1978) $L(\bm{c})=\sum_{u,u^{\prime}}|\delta_{c_{u},c_{u^{\prime}}}-\Pi_{u,u^{\prime}}$, where $\delta_{ij}=I(i=j)$ is the Kronecker delta, $\bm{c}=(c_{1},\ldots,c_{U})$ is an assignment of users to rubrics, and $\Pi_{u,u^{\prime}}$ is the posterior probability of $C_{u}=C_{u^{\prime}}$. See Fritsch and Ickstadt, (2009) for additional approaches to clustering objects using samples of the $C_{u}$’s.

The multi-rubric model produces interesting effects on the overall estimate of restaurant quality. Consider the rubric corresponding to $m=7$ in Figure 7. Users assigned to this rubric give the majority of restaurants a rating of five stars. As a result, a rating of 5 stars for the $m=7$ rubric is less valuable to a restaurant than a rating of 5 stars from a user with the $m=6$ rubric. Similarly, a rating of $3$ stars from the $m=7$ rubric is more damaging to the estimate of a restaurant’s quality than a rating of $3$ stars from the $m=6$ rubric.

For restaurants with a large number of reviews, the effect mentioned above is negligible, as the restaurants typically have a good mix of users from different rubrics. The effect on restaurants with a small number of reviews, however, can be much more pronounced. To illustrate this effect, Figure 8 displays the posterior distribution of the quantity $\lambda_{i}$ defined in (2) for the restaurants with $i\in\{3356,3809,9\}$. Each of these businesses has $4$ reviews total, with empirically averaged ratings of 4.25, 3.75, and 3 stars. For $i=3809$ and $i=9$, the users are predominantly from the rubric with $m=7$; as a consequence, the fact that these restaurants do not have an average rating closer to five stars is quite damaging to the estimate of the restaurant quality. In the case of $i=3809$, the effect is strong enough that what was ostensibly an above-average restaurant is actually estimated to be below average by the multi-rubric model. Conversely, item $i=3356$ has ratings of $4,5,5$, and $3$ stars, but one of the $5$-star ratings comes from a user assigned to the rubric $m=2$ which gave a $5$-star rating to only 8% of businesses. As a result, the $5$-star ratings are weighted more heavily than they would otherwise be, causing the distribution of $\lambda_{i}$ to be shifted slightly upwards.

Lastly, we consider rescaling the average ratings according to a specific rubric. This may of interest, for example, if one is interested in standardizing the ratings to match a rubric which evenly disperses ratings evenly across the possible stars. To do this, we examine the rubric-adjusted average ratings $\lambda_{im}$ given by (3). Figure 9 displays the posterior density of $\lambda_{im}$ for $i=24$ and $i=44$, for the $9$ most common rubrics. These two restaurants have over 100 reviews, and so the overall quality can be estimated accurately. We see some expected features; for example, the quality of each restaurant has been adjusted downwards for users of the $m=10$ rubric, and upwards for the $m=7$ rubric. The multi-rubric model allows for more nuanced behavior of the adjusted ratings than simple upward/downward shifts. For example, for the mediocre item $i=44$, we see that little adjustment is made for the $m=13$ rubric, while for the high-quality item $i=24$ a substantial downward adjustment is made. This occurs because the model interprets the users with $m=13$ as requiring a relatively large amount of utility to rate an item 5 stars, so that a downward adjustment is made for the high-quality item; on the other hand, users with $m=13$ tend to rate things near a $3.5$, so little adjustment needs to be made for the mediocre item.

In this paper we introduced the multi-rubric model for the analysis of rating data and applied it to public data from the website Yelp!. We found that the multi-rubric model yields improved predictions and induces sophisticated shrinkage effects on the estimated quality of the items. We also showed how the model developed can be used to partition the users into interpretable latent classes.

There are several areas exciting areas for future work. First, while Markov chain Monte Carlo works well for the Yelp! dataset (it took 90 minutes to fit the model of Section 4), it would be desirable to develop a more scalable procedure, such as stochastic variational inference (Hoffman et al.,, 2013). Second, the model described here features limited modeling of the users. Information regarding which items the users have rated has been shown in other settings to improve predictive performance; temporal heterogeneity may also be present in users.

The latent class model described here can also be extended to allow for more flexible models for the $\alpha_{u}$’s and $\beta_{i}$’s. For example, a referee pointed out the possibility of inferring about how controversial an item is across latent classes, which could be accomplished naturally by using a mixture model for the $\alpha_{u}$’s.

A fruitful area for future research is the development of methodology for when MAR fails. One possibility for future work is to extend the model to also model the missing data indicators $\Delta_{iu}$. This is complicated by the fact that, while $\{Y_{iu}:1\leq i\leq I,1\leq u\leq U\}$ is not completely observed, $\{\Delta_{iu}:1\leq i\leq I,1\leq u\leq U\}$ is. As a result, the data becomes much larger when modeling the $\Delta_{iu}$’s.

The authors thank Eric Chicken for helpful discussions. This work was partially supported by the Office of the Secretary of Defense under research program #SOT-FSU-FATs-06 and NSF grants NSF-SES #1132031 and NSF-DMS #1712870.