Decisions, Decisions

Finding the balance of limitations in talent identification and selection in team sports

By Dr Sam McIntosh

Whilst often accepted that with experience comes knowledge, it is important that we’re not overconfident to think that experience alone will always lead us to making better decisions. For example, subjective biases can lull us into a false sense of security, potentially leading us to confuse what we’re observing, taking it to be ‘fact’, when the reality is far from. Thus, finding the balance between expert and empirical knowledge is crucial for those responsible for making decisions in sport.

Talent scouts are often individuals who’ve spent years immersed in the sport they love. Some bring knowledge from their experiences as a former player or coach, whilst others are just intense lovers of the game. One thing that all scouts have in common is that they’re employed because they’re thought to understand what attributes a player needs to make it at the elite level. But what if what they think they know is biased or drastically different from the data?

Like many areas of elite sport, talent identification processes have embraced the ongoing increase in data availability. However, despite incorporating data as part of standard practice, the amount to which data is used to inform decision making remains varied between sports and individual organisations. Whilst there is a general acceptance amongst many decision makers that data driven ways of thinking is a step in the right direction, fully committing to a process which prioritises it is still relatively uncommon. My take on this is twofold: On one hand, utilising data gives us an ability to uncover patterns and causative associations that as a human we could not possibly comprehend without the data. On the other hand, even the most complex algorithm based on an extensive dataset will still have difficulty accounting for all of the erratic contextual factors that exist within team sports. This then poses the rebuttal, but what if when you look at the data, it doesn’t tell the whole story? Or worse, what if those interpreting the data are unclear what story the data is telling?

The allure of the ‘Moneyball’ concept leads many to believe that telling the whole story about a player, and accurately valuing their worth, is a process that’s attainable in elite sport. On one end of the scale, we have a sport like baseball where this belief is more ‘true’. A baseball game is completely structured, and outcomes emanate directly from player actions. Before the batter steps into the box we know exactly what innings it is, how many outs there have been, and if there are any batters currently on base. Then, once the batter steps into the box, there are only a finite number of outcomes that can occur. The same can be said for the pitcher. On the other end of the scale, we have a sport like Australian football where this belief is less ‘true’. The game is almost completely unstructured, there is an absence of quantifiable outcomes which originate directly from player actions, and there is rarely a situation where all of the tangibles are the same.

In a study published in the Public Library of Science last year, some colleagues and I looked to investigate how match statistics and data driven measures of a player’s performance can be used to better understand a human’s subjective perception of performance in the Australian Football League. The findings of this study provided some valuable insights for implementing decision support strategies. Most notably, overall evaluations of player performance should consider both subjective and objective assessments in a complementary manner, particularly in instances where there’s conflicting evaluations. Whilst the balance of input which is considered to make final decisions is dependent on the nature of the sport itself, evaluating multiple perspectives as a way to challenge our thought processes can assist in improving the accuracy of decisions made.

Challenging our perceptions can come in a variety of ways. We should take an objective look at our subjective perceptions. Can we utilise data to give us any insights into whether a scout's perception is biased? For example, do they overrate marginal players in a strong team, or alternatively, do they underrate strong players in a weaker team? Does the perception of a players kicking ability change because of its perceived likeness to that of a currently elite player? Can we uncover that an action or a trait which is traditionally thought to be effective, might actually be unproductive or irrelevant (or vice versa)?

On the contrary, we should also take a subjective look at our objective perceptions. If the data doesn’t seemingly tell the whole story, can we use expert knowledge to piece together a bigger picture? For example, assessing the value of defensive actions is something that has been traditionally difficult to quantify objectively in team sports. The same goes for elements such as a player’s ability to be a leader/organiser on the field, and the effect that has on the value of their teammate’s performances. As such, some subjective context can aid towards the overall evaluation of these traits.

Common sense tells us that we should always strive to implement strategies which allow us to make better decisions more often than the current processes which are in place. Whilst the introduction of data analysis and objective methods into talent identification processes is increasing, one constant which is fundamental to successful implementation is the ability of those making the decisions to be able to interpret and comprehend the data and analyses accurately. Whilst embracing data is a proactive step forward in striving to implement a better talent selection strategy, it is in essence only the first step. Poor interpretation can be misleading, resulting in decisions that can be more detrimental than not embracing data in the first place. Ensuring that the data analysis is interpretable for the end user, and is compatible with the existing processes of the organisation, will allow for the implementation of a strategy which endeavours to find the balance between the elements of human performance what only we can comprehend, and those which we could not possibly comprehend without the data.