Do NHL goalies get hot in the playoffs?
A multilevel logistic regression analysis

The hot-hand phenomenon generally refers to an athlete who has performed well in the recent past performing better in the present. Having a “hot goalie” is seen as crucial to success in the National Hockey League (NHL) playoffs. A goaltender who keeps all pucks out of the net for 16 games (4 series of 4 wins) will win his team the Stanley Cup—obviously. In this paper, we use data from the NHL playoffs to investigate whether goaltenders get hot, in the sense that if a goaltender has had a high recent save probability, then that goaltender will have a high save probability for the next shot that he faces.

NHL fans, coaches, and players appear to believe that goaltenders can get hot and that a hot goaltender can be the key to the success of an NHL team. A famous example of a hot goaltender is the legendary playoff journey of Jonathan Quick of the Los Angeles Kings in 2012. With the fifth-best save percentage ($92.86\%$) in the regular season, Quick continued to improve ($95.35\%$) in the first round of the playoffs against the Vancouver Canucks. He remained hot until lifting the first Stanley Cup in franchise history and completed the playoffs with the best save percentage ($94.61\%$) of all goalies who played more than 10 playoff games.

NHL goaltenders let in roughly one in ten shots. More precisely, during the 2018-19 regular season, 93 goaltenders playing for 31 teams faced a total of 79,540 shots, which resulted in 7,169 goals, for an average save percentage of 90.99%. Among goaltenders who played at least 20 games, the season-long save percentage varied from a high of 93.42% (Ben Bishop, Dallas Stars) to a low of 88.58% (Aaron Dell, San Jose Sharks)—a range from 2.41 percentage points (pps) below to 2.43 pps above the overall average save percentage. In the playoffs of the same year, the overall average save percentage was 91.64%. Among goaltenders who played in two or more playoff games, the save percentage varied from 93.56% (Robin Lehner, New York Islanders) to 85.58% (Andrei Vasilevskiy, Tampa Bay Lightning)—–a range from 6.06 pps below to 1.92 pps above the overall average save percentage.

It is crucial to determine whether the hot-hand phenomenon is real, for NHL goaltenders, to understand whether coaches are justified in making decisions about which of a team’s two goaltenders should start a particular game based on perceptions or estimates of whether that goaltender is hot. If the hot hand is real, then appropriate statistical models could potentially be used to predict the likely performance of a team’s two goaltenders in an upcoming game, or even in the remainder of a game that is in progress, during which the goaltender currently on the ice has performed poorly.

Our main finding is the non-existence of the hot-hand phenomenon, that is, no positive influence of recent save performance on the save probability for the next shot. We also find a statistically significant negative slope coefficient for the influence of recent save performance on the save probability for one window size; in other words, we demonstrate that contrary to the hot-hand theory, better past performance may result in a worse future performance. For other window sizes, we find insignificant but negative slope coefficients. A permutation test indicates that our main finding is not simply the result of regression-to-the-mean bias.

The remainder of the paper is organized as follows: in Section 2, we review related literature; in Section 3, we describe our data set; in Section 4, we specify our regression models; in Sections 5 and 6, we present our results; and in Section 7, we discuss our findings.

We summarize five streams of related work addressing the following: (1) whether the hot hand is a real phenomenon or a fallacy, (2) whether statistical methods have sufficient power to detect a hot hand, (3) whether offensive and defensive adjustments reduce the impact of a hot hand, (4) estimation of a hot-hand effect for different positions in a variety of sports, and (5) specification of statistical models to estimate the hot hand.

(1) Is the hot hand a real phenomenon or a fallacy? The hot hand is originally studied in the 1980s in the context of basketball shooting percentages [9, 23, 24]. These studies conclude that even though players, coaches, and fans all believe strongly in a hot-hand phenomenon, no convincing statistical evidence supports its existence. Instead, [9] attribute beliefs in a hot hand to a psychological tendency to see patterns in random data; an explanation that has also been proposed for behavioral anomalies in various non-sports contexts, such as financial markets and gambling [15]. Contrary to these findings, recent papers by [15, 16] demonstrate that the statistical methods used in the original studies are biased, and when their data is re-analyzed after correcting for the bias, strong evidence for a hot hand emerges.

(2) Do statistical methods have sufficient power to detect a hot hand? [9] analyze players individually. This approach may lack sufficient statistical power to detect a hot hand, even if it exists [26, 27]. Multivariate approaches that pool data for multiple players have been proposed to increase power [2]. We follow this approach, by pooling playoff data for multiple NHL goaltenders over multiple years.

(3) Do offensive and defensive adjustments reduce the impact of a hot hand? A hot hand, even if it is real, may not result in measurable improvement in performance if the hot player adapts by making riskier moves or if the opposing team transfers resources to focus on the player. For example, a hot basketball player may attempt riskier shots and the opposing team may double team such a player. The extent to which such adjustments can be made varies by sport, by position, and by the game situation: There is little opportunity for such adjustments for basketball free throws [9] and there is less opportunity to transfer defensive resources towards a single player in baseball than in basketball [10] because the fielding team faces batters sequentially. Resources can only be transferred across time by letting better pitchers face better batters, rather than across space for the defensive team since there is only one batter in the batter’s box at a time. The opportunity for transferring resources is even less for an NHL team facing a hot goalie than for a baseball team facing a hot batter, because the NHL team typically faces the same goalie for the entire game, which minimizes the opportunity to transfer resources across time. Transferring resources across space is only possible through strategies like “crowding the net”. Overall, the opportunities to transfer resources away from other tasks and towards scoring are more limited than in the case of basketball and even baseball. NHL goaltenders thus provide an ideal setting in which to measure whether the hot-hand phenomenon occurs.

(4) Estimation of a hot-hand effect for different positions in a variety of sports. In addition to basketball shooters, the list of sports positions for which we have investigated hot-hand effects includes baseball batters and pitchers [10], soccer penalty shooters [19], soccer in-game scoring [21], dart players [20], and golfers [14].

In ice hockey, a momentum effect has been investigated at the team level [13]. [25] has investigated hot-hand effects for ice hockey shooters but not for goaltenders, except for the study by [17]. The latter study focuses on the duration of NHL playoff series, noted a higher-than-expected number of short series, and proposed a goaltender hot-hand effect as a possible explanation. This study has not analyzed shot-level data for goaltenders, as we do.

(5) Specification of statistical models to estimate the hot hand. Hot-hand researchers have used two main approaches in specifying their statistical models: (1) Analyze success rates, conditional on outcomes of previous attempts [1, 10] or (2) incorporate a latent variable or “state” that quantifies “hotness” [10, 19]. We follow the former approach. With that approach, past performance is typically summarized over a “window” defined in terms of a fixed number of past attempts—the “window size.” It is not clear how to choose the window size. We vary the window size over a range that covers the window sizes used in past work. We also perform the analysis using time-based windows, an approach that complicates data preparation and has not been used by other investigators. Results for the two windows types are consistent (Appendix A).

We contribute to the hot-hand literature by investigating NHL goaltenders. This position has not been studied previously and provides a setting in which there are limited opportunities for either team to adapt their strategies in reaction to a perception that a goaltender is hot. In terms of methodology, we use multilevel logistic regression, which allows us to pool data across goaltender-seasons to increase statistical power, and we use a wide range of shot-based windows to quantify a goaltender’s recent save performance.

Our data set consists of information about all shots on goal in the NHL playoffs from 2008 to 2016. The season-level data is from www.hockey-reference.com [11] and the shot-level data is from corsica.hockey [22]. We have data for 48,431 shots, faced by 93 goaltenders, over 795 games and 9 playoff seasons, with an average of 30.46 shots on goal per team per game and 91.64% of the shots resulting in a save. We divide the data into 224 groups, containing from 2 to 849 shot observations, based on combinations of goaltender and playoff season. The data set includes 1,662 shot observations for which one or more variables have missing values. Removing those observations changes the average save proportion from 91.64% to 91.61% and the number of groups from 224 to 223. We exclude observations with missing values from our regression analysis but we include these observations when computing the variable of interest (recent save performance), as discussed in Section 3.2.

We use multilevel logistic regression with partial pooling, also referred to as mixed effects modeling. We rescale the variable of interest $x_{ij}$ in (1) [6, 12] by subtracting the long-term average performance $\tilde{x}_{ij}$ of the goaltender to obtain deviations from his performance:

We compute $\tilde{x}_{ij}$ as in (1), but with a longer window size: $k=28\times 30=840$. This window size corresponds, roughly, to 28 games, which is the longest possible playoff journey (7 games $\times$ 4 rounds). It is not straightforward to determine whether a positive deviation in player performance should be attributed to “being hot” versus “being a much improved player” [10]. If a player’s performance increases over time, then a long-term performance estimate based on a long time window (say, the career average) will lag behind the player’s current ability, resulting in a positive deviation that should not be attributed to being hot. To avoid this issue, we use a window size that is longer than the largest of the window sizes for calculating recent performance, but short enough so that the player’s ability is unlikely to change much.

We exclude all shots from the regression model for which $\tilde{x}_{ij}$ cannot be computed because the goaltender did not play enough during the current season to compute the long-term average performance. Specifically, (1) for groups in which the goaltender’s regular season play exceeds the long-term window size, we include all shots; (2) for groups in which the goaltender’s regular season and playoff play combined fall short of the long-term window size, we exclude all shots; and (3) for groups that fall between these two extremes, we exclude some shots. In total, we exclude 7,166 shots ($14.80\%$) for the shot-based window analysis.

Our variable of interest becomes:

Our interest is in whether deviations from long-term performance persist over time.

All estimates that we report in Section 5 are obtained using Markov chain Monte Carlo (MCMC), using the rstan and rstanarm R packages. We use the default prior distributions from the rstanarm package. The default distributions are weakly informative—Normal distributions with mean $0$ and scale $2.5$ (for the slope coefficients) and $10$ (for the intercepts) and multivariate Normal distributions with zero mean vector and an unknown covariance matrix to be estimated for the random effects [7]. We use default values for the number of chains ($4$), the number of iterations (1,000), and the burn-in (1,000 iterations).

First, we provide detailed results for the $k=120$ window. Second, we investigate the consistency of our main finding by varying the window size. Third, we illustrate the effect size of all independent variables (the variable of interest and all control variables) on the save probability. Appendix A provides additional results for time-based windows; Appendix B provides diagnostics for the MCMC estimation.

In our rescaling process, we subtract a long-term average over a window that only includes shots that occur before shot $i$. This approach, however, may introduce a regression-to-the-mean effect which causes negative fixed effect estimates: if a goaltender’s performance, after controlling for observable factors, is completely random, then we will expect a period of above-average or below-average recent save performance to be likely to be followed by a period of save performance that is closer to the average. To determine whether the negative $\hat{\bar{\beta}}$ is only from the regression-to-the-mean effect, we present a permutation test to determine whether this rescaling process would introduce significant negative biases.

For this analysis, we use the 120-shot baseline model and perform 100 permutations. In each permutation, we randomly permute the $y_{ij}$ values across the goalie-season groups, and we recalculate the $x_{ij}$ and $\tilde{x}_{ij}$ values using the permuted $y_{ij}$ values. By randomly permuting the $y_{ij}$ values, we eliminate any correlations among the shot outcomes. If rescaling $x_{ij}$ values does not introduce any biases, then we expect the empirical distribution of estimated fixed effects to be centered around 0.

Figure 6 shows the results. The mean and median $\hat{\bar{\beta}}$ values are $-0.51$ and $-0.55$, suggesting a small negative bias. However, the recent save performance coefficient for the un-permuted data, of $-1.82$, is outside a $90\%$ interval defined by the 5th and 95th percentiles of the permutation distribution. We conclude that the negative fixed effect estimate that we obtained cannot be explained solely by a negative bias from the re-scaling process.

We employ multilevel logistic regression to investigate whether the performance of NHL goaltenders during the playoffs is consistent with a hot-hand effect, using data from the 2008–2016 NHL playoffs. We measure past performance using shot-based windows. Our window sizes span a wide range: from, roughly, 1 game to 5 games. We allow the intercept and the slope for recent save performance to vary across goaltender-season combinations.

We find no evidence of hot-hand effects and a significant negative impact of recent save performance on the next-shot save probability, for one window size. The estimates were negative for all window sizes and goaltender-season groups. Our findings are contrary to the findings for baseball in [10], who use a window size similar to our longest window, and hypothesize that hot hand effects should generally be observed for skilled activity.

Referring back to Section 6, if the effect we find were entirely due to regression to the mean, then we would expect that as we increase the window size used to measure recent save performance, the average amount by which performance moves toward the average to decrease. The fact that we observe an increase in magnitude followed by a decrease (see Figure 3), together with our permutation test results, argues against our finding being driven by regression to the mean.

A motivation effect provides one possible explanation for our findings. That is, if a goaltender’s recent save performance has been below his average for the current playoffs, then his motivation increases, resulting in increased effort and focus, causing the next-shot save probability to be higher. Conversely, if the recent save performance has been above average, then the goaltender’s motivation, effort, and focus decrease, leading to a lower next-shot save probability. [3] find support for the first of these effects (greater performance after failure) for “obsessively passionate individuals” but have not found support for the second effect (worse performance after success) for such individuals. The study has found support for neither effect for “harmoniously passionate individuals.” These findings are consistent with Hall-of-Fame goaltender Ken Dryden’s (\citeyeardryden2019life) sentiment that “if a shot beats you, make sure you stop the next one, even if it is harder to stop than the one before.” The psychological mechanisms underlying our findings warrant further study.

Although the estimated recent save performance coefficients are consistently negative, their magnitude varies. In particular, the magnitude increases sharply with the window size up to $k=120$. We expect to see more reliable estimates with longer window sizes, but the increase in magnitude is surprising, given that we define the recent save performance as a scale-free save percentage.

A literal interpretation of our negative finding suggests a strategy that prefers a goalie with below-average recent save probability because his save probability for the next few shots is expected to improve. One should be cautious about using such a strategy, keeping in mind Goodhart’s Law, that “any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes” [4]. For one thing, such a strategy would provide an incentive to underperform in non-essential games to earn more playing time in big games.

One limitation of our study is that, in defining windows, we ignore the time that passes between games. Past research, such as [10], shares this limitation. This limitation can be particularly serious for backup goaltenders, for whom the interval between two successive appearances may be several days long.