Heatherly Carlson & Joo Eng Lee-Partridge, Central Connecticut State University
For many years, sports analytics have demonstrated a robust relationship between NFL draft measurements and NFL success. In particular, NFL drafts have been harnessing the power of sports analytics to predict the future value of the quarterback. However, most of these pre-draft metrics deal with the physical prowess or prior physical achievements of the quarterback.
While these measurements may provide some estimate of the quarterback’s predicted value in the NFL, we are proposing looking at other intangible variables to provide an alternate indicator of future value. Some potential intangible variables that could predict to QB success include character risk, injury resilience, psychological variables, environmental variables, adversity, cognitive ability, motivation and leadership experience.
Heatherly explores the relationship between the player intangibles and quarterback outcome measures such as number of playoff game appearances, number of years as starting QB and whether they have ever taken their team to a playoff game. They use both supervised and unsupervised machine learning to provide insights in QB success and QB intangible variables and provide recommendations for future QB drafts.
00:04 [music] So good afternoon, everyone. I want to talk a little bit about how we decided sort of where our dataset was going to come from. So in the NFL, most of you maybe‐‐ show of hands, anybody watch the Super Bowl about a week ago? Okay, so we have some fans. Most of you know that when the NFL Super Bowl happens, that that determines sort of which teams draft in what order. So the teams that get eliminated from the playoffs first get to sort of pick first, and then the last team to get eliminated or the team that wins gets to pick sort of last in every round. So when some teams are doing really poorly in the season, they actually kind of go over to their draft mode knowing, “Okay, the playoffs are not going to happen for us. Perhaps it’s best if we just don’t win anymore because that will sort of increase our chance of equalizing when the draft comes.” So the teams that go to the Super Bowl like Kansas City and the 49ers, when they’re done with that game, they’re actually kind of five weeks behind the other teams, right, because they didn’t get a chance to go to the Senior Bowl. They didn’t get a chance to sort of look at the players longer. They were so busy preparing for the playoffs that they’re just sort of behind.
01:21 Our interest in the NFL is sort of trying to find out, if all the teams go to the NFL Combine, and they all watch the drills and the player workouts, and they kind of are there for all of the interviews, how is it that all 32 teams get the same data set, in a sense, but there seems to be sort of a competitive advantage? Meaning some teams are masters at drafting while other teams don’t really do that well. And how does that work? I mean, how is it that some teams have that sustained competitive advantage, while others struggle? So we decided to take a look at it in terms of intangibles versus tangibles. So these tests here‐‐ we’re going to concentrate on the physical tests, and we’re going to call those the tangibles, right?
02:14 So at the NFL Combine, players are basically poked and prodded and measured, and they put them through a series of tests. So we’re just going to run through these so that you can recognize them when we present them in the regression model and the decision trees. So there’s the 40‐yard dash, the vertical jump, the broad jump, the shuttle run, and the three‐cone drill. There are other ones, but we’re only interested in looking at the quarterbacks because asking the question, “What advantage do teams have?” Is so broad. It’s so hard to parse it down to all the possible permutations of coaching trees and strategies and that kind of thing. So we’re literally just going to refine our NFL look to just the quarterbacks. And I’ve only put up the ones that the quarterbacks compete in. So, for example, they don’t compete in the bench press, for example. The linebackers do, but not the quarterbacks. So on February 23rd, there’ll be 337 prospects that are going to attend the NFL Combine this year. And we’re interested in the intangibles. So the intangibles are as follows. The yellow highlight is sort of our impression of, “My goodness, if I could get my hands on this intangible.” I’m not‐‐ I don’t know anyone. Maybe somebody knows somebody who would have that medical grade. But that’s basically the data point that we’re interested in. We want to know what is on the line. When you select a player, how likely are they to get injured? How many years are you going to get out of them? Are you only going to get a few seasons? What is the risk and what are you getting? Sort of that cost‐benefit ratio.
03:57 The other variable we’re interested in, and there’s a multitude of ones‐‐ again, we couldn’t go back and ask the players to assess themselves, but we were interested in character risk and we really couldn’t find anything that sort of mapped onto that. It’s very hard to pull together their college transcript in terms of how they did on the team. Were they kind of coachable? Were they high‐risk in terms of their character? So we just kind of skipped over that and went straight to the psychological, mental health intangible. So the Wonderlic, for those of you who don’t know what this is, is sort of the IQ test. It was, I think, developed in the ’30s. And most people probably know it as having been used by the military to sort of figure out and grade personnel for World War II, trying to figure out who was going to be a pilot. This is just a sample of the battery; there was actually the 50 questions. And the way that this is scored ‐ and they’re under duress for how long they have time ‐ is they literally just get one point for every correct answer. So you have to move along at a very fast pace, at about four questions a minute, or you literally won’t get exposed.
05:13 So then getting back to the actual RapidMiner environment, now that you’re familiar with the Combine data, we also took in data from their actual NFL debut. And we didn’t really know exactly where to start looking at it because we have some quarterbacks that have been around for 20 years, and then we have other quarterbacks that are in our data set that have only been around a year. But just to sort of put them on a level playing field, we kind of took a cut off, like, “We’re going from the NFL draft from the year 2000.” So that was sort of our starting date, so that we took only the quarterback subset. We inputted all values we could find for the Wonderlic, and we also inputted an injury value as seen that was similar to the one you saw, but it wasn’t the actual NFL expressed values. So when we ran our model, we looked at classification first. But there were problems with our data. First, we found out that half those players in our data set hadn’t actually been drafted. So we kind of wondered, “What do we do with these players?” We decided, ultimately, to impute their draft pick number, which on average for quarterbacks, was going to be a little earlier in the draft than later. So we imputed that with the average. And we also found that the probability of injury value that we took from, I think, sportspredictor.com‐‐ sportsinjurypredictor.com, sorry‐‐ we found that those values were missing. So we had to impute those as well. So a lot of limitations in what we were trying to look at.
06:51 When we ran the supervised values through the classification, we found that‐‐ sorry. Here we go. We found that about 95% were optimized using the Gini index for accuracy. And then when we ran the Naive Bayes, we found that it predicted almost as well, 92%. And then the ROC curves kind of give us a graphical representation of that. When we ran a linear regression, and this time, our‐‐ sorry, I should back up. I should tell you the label for these analyses for classification was “success.” Success was defined as reaching the postseason. So if you played in any postseason game at all as of about December 30th, if we put you into the successful “Yes” category. And if you did not play, then you were considered not successful. So irrespective of how long you were in the NFL. And then for the linear regression, we obviously had to go with a continuous variable. So we took the total count of NFL postseason games started, and we predicted to that using all of those college variables, like their passing metrics in college, and their Combine values. And we also put in the intangibles. And lo and behold, actually what came out, which was shocking to me‐‐ although, with all the imputation, you can’t take it too seriously‐‐ would be that the Wonderlich, which is really sort of my favorite thing ‐ that’s the one that I’m personally invested in ‐ actually came out significantly predicting the NFL postseason games.
08:39 And then in terms of the unsupervised, let’s take a look at the clusters. This is what our clusters came out with, so we specified we wanted five, and then we got a nice little break. And when we looked at the actual players‐‐ does anyone want to guess which cluster here was probably the best set of quarterbacks? The ones that you know, the name‐brand quarterbacks that you might be familiar with? Does anyone want to take a guess? Well, you have to‐‐ which cluster number? Who do you think the Patriots would‐‐ which cluster would the Patriots’ quarterbacks fall under? Okay, anybody else have a guess? Okay, let’s take a look. So we have this nice distribution, so the players are all in one cluster. And then when we look in three, lo and behold, you’re correct. The best quarterbacks, sort of the ones that you think of as being the franchise quarterbacks that are usually pretty guaranteed with their contract‐‐ although I know Matthew just left. Tom Brady, Ben, Drew, all of our favorite quarterbacks kind of shook out in cluster three. So that was pretty exciting. So this is just a preliminary look at our NFL success variable, but we’re hoping that when we go forward, we can optimize our model and look at different parameters and kind of see through various techniques back in our design. Using some of these subprocesses and using some cross-validation methods, we’ll be able to see if we’re on the right path as far as figuring out what are those intangibles that NFL teams seem to be able to harness. [music]