Written by John McCool (@Desertrose28)
- There appears to be significant coefficient variability in the angle and rebound coefficients from 2007-08 through 2015-16 using bootstrap analysis
- The distance and rush coefficient remain relatively stable across seasons
- This suggests that current expected goal (xG) models could be biased if trained on coefficients from past seasons
In the previous article, we showed that there is some potential evidence that expected goals models may not be appropriate for player evaluation across seasons in the NHL. Specifically, for expected goals models trained via logistic regression, the rebound and rush coefficients shifted across seasons, and other coefficients (e.g. shot distance and shot angle) had small, steady shifts over time. These shifts could lead to biased and inaccurate expected goal (xG) projections in future seasons, if/when there are systematic changes in goalies’ abilities to stop certain shots (or players’ abilities to convert certain shots).
The second part of this series explores the coefficient variability using a bootstrap analysis. Bootstrapping allows us to get an approximate distribution of the distance, angle, rebound, and rush coefficients for each of the 2007-08 through 2015-16 seasons.
As an overview, bootstrapping is a statistical method that draws repeated samples from a population of interest with n observations (in our case, all unblocked shot attempts in a season). Bootstrapping gives us the sampling distributions of each of the coefficients in the model. In other words, it helps to answer the question, “If we replayed the 2015-16 season, how different would our expected goal model be?”
Bootstrapping also allows us to estimate the confidence interval for the unknown parameter estimate (θ) and the probability that θ lies within the bounds of the interval, without any of the parametric assumptions from logistic regression. In other words, it helps to answer the question, “How much variability would there be in the rush shot coefficient if we replayed the 2015-16 season many times?”
We’ll use the following procedure for our bootstrap analysis:
- Resample dataset (usually more than 100,000 unblocked shot attempts per season, marked as goal or non-goal) with replacement 1000 times. Each of the 1000 (re-)samples of the data is called a “phantom” sample (or bootstrapped sample).
- Estimate the expected goals model with logistic regression on each sample, recording the coefficients for each iteration.
- Create histograms of each coefficient. We’ll call these the bootstrapped distribution of the coefficients.
- Repeat this for all seasons for which we have data on unblocked shot attempt locations (2007-2016).
We performed bootstrap analysis on each xG coefficient with 1,000 re-samples over the past nine seasons. For reference, we compared the mean coefficient from the 2007-2015 seasons to the 2015-16 bootstrapped distributions as a measure of coefficient variability across NHL seasons.[i]
- 2015-16: Black
- 2014-15: Blue
- 2013-14: Yellow
- 2012-13: Red
- 2011-12: Purple
- 2010-11: Green
- 2008-09: Orange
- 2009-10: Violet
- 2007-08: Brown
From the previous article, we found that increasing the angle of a shot slightly decreases the log odds of a goal. The coefficient, however, marginally increased to -0.0134 last season, which is significantly higher than the previous eight seasons. The bootstrapped coefficient distribution (2015-16) is approximately normally distributed, with 95% of the coefficients between (roughly) -0.015 and -0.012.
Aside from 2011-12, none of the angle coefficients from previous seasons fall inside last season’s 95% bootstrap confidence interval for the 2015-16 season. This means that if we used an xG model trained on the 2014-15 season to obtain player and team evaluation metrics for the 2015-16 season, the model would be biased in how it assigns goal expectancies for the 2015-16 season. Finally, four angle coefficients lie completely outside of the bootstrapped distribution, the furthest of which are the -0.0162 and 0-.0161 values from 2007-08 and 2008-09 respectively.
The bootstrapped distance coefficients exhibit less variability than that of the angle coefficients. Last season’s distance coefficient compares similarly to 2014-15, 2011-12, and 2013-14. The distribution also shows no evidence of an increasing or decreasing trend in the coefficient. Furthest from the center of 2015-16 distance distribution are the 2007-08, 2010-11, and 2013-14 mean coefficients. That being said, the distance coefficient range is just -0.01 indicating that it is a relatively stable xG predictor.
The rebound coefficient (0.934) in 2015-16 fell to lowest value over the past nine seasons. This bootstrapped distribution is approximately normally distributed with 95% of the coefficients ranging between about 0.82 to 1.05. On the right tail of the histogram lie the 2013-2015 rebound coefficients, which are outside of the 95% confidence interval for 2015-16. Again, this means that an xG model trained on the 2014-15 season would be biased if applied to the 2015-16 season.
The rebound coefficients follow a decreasing trend in the rebound coefficient since the start of the 2013-14 campaign, as we saw in Part I of this series. For example, the 2012-13 coefficient was 1.18 compared to just 1.03 and 1.04 in the following seasons. All of this might suggest that rebound shots are currently having less influence on expected goal models. Although this is not rule out the possibility xG rebound coefficient could jump all the way to 1.10 or even 1.20 this season, shattering this assumption.
On the more extreme end are the 2008- 2011 rebound coefficients with a combined 1.25 average. In other words, rebounds had more weight in expected goal models during this period. For reference, the 2010-2011 confidence interval (1.15, 1.37) for the bootstrapped rebound coefficient lies completely outside of the last season’s interval!
All together, the bootstrapped rush coefficients from 2015-16 are similar to those from previous seasons. The increased variation in the rush coefficient is likely due to the reduced sample size of rush shots in each season. From 2008-2010, the rush coefficient averaged 0.743 before increasing to 0.879 over the next three seasons. Since then, the coefficient has fallen between 0.786 to 0.839.
In summary, by using the above bootstrap analyses, we provide a more rigorous statistical analysis that suggests that bias likely exists in some xG models, especially those trained on prior seasons and applied to subsequent seasons. Significant deviations in the angle and rebound coefficients could potentially alter the accuracy of xG projections that fail to adjust for these changes. We again encourage analysts to be detailed and forthright in describing how and on which seasons their xG models are trained, so that any potential bias in the resulting xG statistics can be accounted for.
[i] For reference, 2007-2015 refers to NHL seasons beginning in the October of 2007 through April of 2015.