Evaluation of creating scoring opportunities for teammates in soccer via trajectory prediction

First, we describe the base model of our evaluation method called OBSO [45] and then propose the potential score model. OBSO evaluates off-ball players by computing the following joint probability

where $D$ is the instantaneous state of the game (e.g., player positions and velocities). The details in OBSO are given in Appendix 0.B. $P(S_{r})$ is the probability of scoring from an arbitrary point $r\in R\times R$ on the pitch, assuming the next on-ball event occurs there. $P(C_{r})$ is the probability that the passing team will control a ball at point $r$. $P(T_{r})$ is the probability that the next on-ball event occurs at point $r$. Here, for simplicity, we can assume that $P(S_{r}|D),P(T_{r}|D),P(C_{r}|D)$ are independent if the parameter $\alpha=0$ in the original work implementation (Eq. (6) in [45]). Then, the joint probability can be decomposed into a series of conditional probabilities as follows:

$P(C_{r}|D)$ is the probability that the attacking team will control the ball at point $r$ assuming the next on-ball event occurs there, which is called the potential pitch control field (PPCF). $P(T_{r}|D)$ is defined as a two-dimensional Gaussian distribution with the current ball coordinates as the mean. $P(S_{r}|D)$ is simply calculated as a value that decreases with the distance from the goal. We used the grid data and computed $P(C_{r}|D)$ and $P(T_{r}|D)$ based on the code at https://github.com/Friends-of-Tracking-Data-FoTD/LaurieOnTracking.

In the original OBSO [45], the scoring probability was calculated as the output $P(S_{r}|D)$ of the score model as a function of the distance from the goal. However, the scoring probability may depend on the angle to the goal and the defensive position of the opponent. Therefore, in this paper, we propose a score model that reflects the angle to the goal and the position of multiple defenders. Here, we consider the shot-blocking distribution of the defenders who can block shots in the field, and propose a potential model where the scoring probability decreases when defenders exist. The basic idea shown in Fig. 1B is to calculate the scoring probability from the angle to the goal at which the shot tends to be scored, considering the mixed distribution of the positions of multiple defenders. The proposed scoring probability $P(S^{p}_{r}|D)$ at a certain point $r$ is calculated as the sum of the shot value $V_{shot}$ as follows:

where $n$ is determined by the angle from the shooting position to the goal, and $\vec{s}$ is a shot vector per degree ($n$ is larger when the shot from the center and smaller from the side). The shot value $V_{shot}$ is calculated by subtracting the shot block value $V_{block}$ from a certain constant $c$ ($c,C$ are parameters determined from data to be adjusted so that $V_{shot}\geq 0$ and $P(S^{p}_{r})\in[0,1]$). Let $V_{block}$ be the sum of the shot block distribution values along the shot vector $\vec{s}$. The shot blocking distribution is the sum of the normal distributions (variance $\sigma^{2}=0.5+l_{d}$) assigned to each defender on the goal side of the shooting position (shot blockable players using legs), where $l_{d}$ is the distance between the shooting position and the defender. We consider that goalkeepers have a shot blocking distribution with twice the value of normal defenders because of a higher shot-blocking ability. Here, we assume that the block distribution is not changed with the distance from the ball. A defender near to the ball may affect the ball, but far players use the flight time of the ball for their movement. This formulation is left for future work.

Here, we describe the base model of our trajectory prediction method called GVRNN [51] and then describe our C-OBSO framework. Our contribution here is to evaluate how the actual “off-ball” movement contributes to scoring opportunity compared to the predicted movement (or trajectory) as a reference. In our method, we use GVRNN [51], which is a VRNN [9] combined with a graph neural network (GNN [26]). For the details in VRNN and GVRNN, see Appendices 0.C and 0.D. In GVRNN, the graph encoder-decoder network models the relationship between players as a graph, which is one of the best performing models for predicting player trajectories in team sports [51]. This is a probabilistic model which can sample multiple possible trajectories.

Based on the trajectory prediction, we propose an evaluation index C-OBSO of players who create scoring opportunities for teammates. The basic idea is to evaluate an off-ball player from the difference in the modified OBSO values between the predicted and actual movements of the players. The C-OBSO value of a player $i$ without the ball can be expressed as follows.

where the player $k$ is the ball carrier who performs a final action (e.g., shot), $V^{k}_{OBSO}$ is the modified OBSO in the actual game situation, and $V^{\prime k}_{OBSO}$ is the modified OBSO based on the predicted trajectory as a reference. For example, in Fig. 1A, C-OBSO is positive and the player to be evaluated (A1) contributes more to the shooter (A2) than the referenced (predicted) player. Specifically, A1 has created a more advantageous situation for A2 by attracting D1 more than expected. C-OBSO can evaluate a player in such situations with an interpretable value (i.e., the increase in scoring probability). Theoretically, if perfectly predicted, C-OBSO is always zero, but actually, if we apply this to a test data, the perfect prediction is impossible. In other words, we assume the imperfect trajectory prediction in this framework.

In this study, we used all 34 games data of Yokohama F Marinos in the Meiji J1 League 2019 season to perform specific player-level evaluations in limited data. Note that the tracking data for all players and timesteps were not publicly shared in such amounts. The dataset includes event data (i.e., labels of actions, e.g., passing and shooting, recorded at 30 Hz and the simultaneous xy coordinates of the ball) and tracking data (i.e., xy coordinates of all players recorded at 25 Hz) provided by Data Stadium Inc. The company was licensed to acquire this data and sell it to third parties, and it was guaranteed that the use of the data would not infringe on any rights of the players or teams. For annual salaries, we used the salaries of the same team (Yokohama) in 2019 [44] because they were different valuation criteria for different teams and the transfer of the players took place during the season. The goals for each player in each match were collected from [23]. The rating by experts in each match [43] was also used for verification, which was scored in 0.5 point increments with a maximum of 10 points.

We used the attacking data of Yokohama F Marinos for the test and those of the opponent teams for training the model or parameter fitting. Again, since the data was limited in this study, we split the data in such a way, and if we have more data, we can analyze all teams with the training data with the same team. Here we describe the processing of the potential score model, the trajectory prediction model, the C-OBSO, and their statistical analyses.

First, we validated the potential score model needed to calculate the C-OBSO. The RMSE with the actual scores was 0.324 $\pm$ 0.014 for the conventional score model [45] without considering the defenders and goal angles, and 0.309 $\pm$ 0.0014 for the potential score model ($p<10^{-10}$). This result suggests that the proposed method models the scores more accurately.

Figure 2 shows an example of the two methods in two actual situations where a shot is attempted from a similar distance. In the existing method, the probabilities were the same (both 0.1237) because the shots were taken from almost the same distance. The proposed method had a lower scoring probability with more defenders (upper: 0.0489, lower: 0.1202). We indicate that the proposed method reflects the position of multiple defenders and can model the score accurately.

Next, we show the results of the trajectory prediction model for computing C-OBSO. Endpoint errors (MAE and standard error, [m]) in GVRNN were $0.608\pm 0.014,0.867\pm 0.022,1.701\pm 0.045,1.606\pm 0.042$ in 4, 6, 8, 10 s prediction. In GVRNN, longer predictions show larger prediction errors except for the difference between 8 s and 10 s. Since the 4 s prediction of GVRNN achieved a low the MAE of less than 0.7 m, the GVRNN trajectory prediction of 4 s was used in the next C-OBSO. For details, see also Appendix 0.E.

Verification of C-OBSO is challenging because of no ground truth values or player ratings. Therefore, we analyzed the relationship with the annual salary, the goals, and the game rating by experts, whereas we admit that these variables include various confounding factors. The relationships between the average C-OBSO and OBSO values of each player of Yokohama F Marinos in 2019 and the annual salary of each player in 2019 are shown in Fig. 3 (note that the tracking data for all players and timesteps were not publicly shared). Here we analyzed 15 players with more than 10 sequences under evaluation. As a result, there was a significant positive correlation between annual salary and C-OBSO ($\rho=0.45,p=0.046$). In addition, the two players with the higher evaluation values but lower salaries (in red in Fig. 3A) were highly evaluated players, who won the individual awards (the most valuable player and valuable player award). In fact, their annual salary for the following year (2020) was also increased (valuable player: increased from 11 million yen to 40 million yen; the most valuable player: increased from 20 million yen to 60 million yen).

We found that these tendencies were similar to the C-OBSO and OBSO without the potential score model (see Appendix 0.F). On the other hand, there was no significant correlation ($\rho=-0.28,p=0.154$) for OBSO, which evaluates a player’s own scoring opportunities (Fig. 3B). We also examined the relationship between annual salary and goals (Fig. 3C), and found no significant correlation ($\rho=-0.23,p=0.208$). Therefore, there was no relationship between annual salary and goals. There were many players with zero goals, and it is difficult to evaluate them only with the goals.

Next, in order to examine the relationship with player performance in more detail, we show the relationship between C-OBSO and the rating by experts of the top three scorers (Nakagawa with 15 goals, Marcos with 15 goals, and Edigar with 11 goals in this season) in Fig. 4. We analyzed the games in which there were two or more C-OBSO evaluations using the average of C-OBSO values on each game (17 games for Nakagawa, 14 games for Marcos, and 10 games for Edigar). A strong positive correlation was found only for Nakagawa ($\rho=0.75,p=0.0003$) but not for Marcos ($\rho=0.27,p=0.174$) and Edigar ($\rho=-0.37,p=0.145$). In Appendix 0.G, we show the results of the other four players who played seven games or more and had two related scoring opportunities or more (for C-OBSO). Similarly, there were no significant correlations between them for all players ($\rho s<0.190,~{}ps>0.05$). In addition to the number of scoring opportunities for teammates (17 times), the results found that Nakagawa would be subjectively and quantitatively an outstanding player.

For reference, we also show the relationship between the goals of the top three scorers and the ratings by experts in Fig. 5. We analyzed the games in which each player played (33 games for Nakagawa, 33 for Marcos, and 16 for Edigar). For each player, there were strong correlations between the goals and the rating (Nakagawa $\rho=0.63,p=4.33\times 10^{-5}$, Marcos $\rho=0.71,p=1.98\times 10^{-6}$, Edigar $\rho=0.91,p=4.40\times 10^{-7}$). We found that the rating of each game depends on a rare event (i.e., goals). In Appendix 0.G, we show the results of the other four players who scored two points or more. Similarly, there were significant correlations between them for all players ($\rho s>0.516,~{}ps<0.018$). Recall that there was a stronger correlation between C-OBSO and Nakagawa’s rating than for the other two players. Nakagawa also had higher average ratings than the other players (6.26 for Nakagawa, 5.97 for Marcos, and 6.09 for Edigar), and he was the player who won the most valuable player award. The game rating by experts would depend on the goals, but it may also evaluate the creation of scoring opportunities only for Nakagawa. From these results, we speculate that Nakagawa was highly evaluated not only for his scoring but also for his contribution to other attacking players. Our method can also evaluate players difficult to be evaluated by conventional indicators, which is crucial for assessing teamwork and player salary, player recruitment, and scouting.