Georgia Tech Football: Decision Making - Fourth Downs

For the better part of a decade, Georgia Tech built its football reputation on a schematic penchant for doing more with less. Powered by the flexbone option, the Yellow Jackets were able to close the gap between themselves and more talented rosters. But in a new era of college football where offenses have coalesced into the same blob of “pro-style spread option RPO”, coaches must find new ways to find winning edges during the course of a game. In the NFL — a league notorious for its institutionally-enforced parity, teams have started to find these edges in two ways:

1. Fourth-down decision making
2. Two-point conversion decision making

Let’s focus our discussion to just fourth-down decision making. Why is this important to winning games? The answer here is simple: a drive you can continue is a drive your opponent can not start. However, that cursory analysis might beg the question (perhaps backed with a bit of snark): “well, why don’t coaches go for it on fourth down every time then?” Well, the answer to that is also simple: why give your opponent a better chance to score if you fail? Thus, to optimize fourth-down decision making, a coach has to balance the reward of keeping his opponent’s offense off the field and the penalty of putting his opponent in prime field position.

But here’s the catch: coaches are notoriously bad at this. Football traditionalists (which make up the vast majority of coaches) frequently preach conservatism and “trusting your defense” when it comes to fourth down. In years past, no one has ever been (seriously) hung out to dry in a press conference for punting on fourth down. Risk aversion is often rewarded, while risk taking is often criticized. Coaches will often argue that their gut told them that punting was the right decision, despite the butterfly effect that decision may have had in the game.

But in recent years, more and more data has become available to evaluate these decisions objectively and analytically. Why rely on gut feeling to make decisions when we can put numbers to work? Some might say “numbers can’t account for the variety of variables at play in an 11v11 on field battle”, but why can’t they? The concept of going for a fourth down conversion is inherently binary: you either go for it or you don’t; we have almost twenty years worth of play-by-play data (thanks to the work of sites like collegefootballdata.com and the people behind cfbscrapR) that provide context to in-game activity; and it’s not like we’re trying to predict what happens in the given fourth down — we’re only trying to answer whether or not a coach should or should not attempt a conversion on said fourth down.

It turns out that — in true Tech fashion — figuring out optimal conversion situations is entirely possible using a bit of advanced math powered by machine learning. Using tools developed by Jared Lee (and pioneered by Ben Baldwin for the NFL), we can effectively evaluate how college football coaches have (or have not) taken advantage of advantageous fourth down situations to win games.

Evaluations

The technical details that power the fourth down model are far beyond my pay-grade, so I’ll defer to Jared’s introduction to his model on UteZone, where he goes in depth on how it all works.

Here’s the quick TL;DR:

...you start by seeing how likely every possible number of yards gained is based on past fourth down attempts and on a number of different factors such as distance to the first down, field position, strength of the offense, and strength of the defense...Adding in 3rd down plays helps to tell how teams perform when they’re trying to get a first down. Once you have the probabilities for each possible outcome, you do the same thing as field goals and punts and calculate what the win probability would be for that result and average the results together....Once you know the expected win probability of each decision, it is as simple as picking the play call that gives you the highest chance of winning the game.

If you’re interested in reading through the code, you can check that out here.

ACC Decision Making

Let’s look at decision making across the conference a few different ways to set some context.

Note: “Toss-up” includes all recommendations with a potential added win probability between -0.5% and 0.5%, regardless of the available decisions (Punt vs Go; Field Goal vs Go, etc).

Overall

By season

By team in 2020:

Note: Split by division.

By team, in “strong go” situations, in 2020:

Note: sample size listed inside each bar.

Expected vs Actual “Go” Rates in “Obvious Go” Situations in 2020

Note: gray line is where expectation equals actual performance; gold line is conference average; blue lines are national averages in expectation and performance, respectively.

Note: gray line is where expectation equals actual performance; gold line is national average; blue lines are national averages in expectation and performance, respectively.

Let’s talk about these figures:

• The conference’s overall aggressiveness (IE: “go”-ing when the model says to “go”) hasn’t really changed over the last four seasons. However, compare those numbers to the national averages. The ACC has slipped as the rest of the nation has modernized; its aggressiveness compared to the rest of the nation has declined: +1.6% in 2017, +1.7% in 2018, -0.6% in 2019, -1.6% in 2020. Why? My theories include the noticeable lack of Paul Johnson after 2018 and/or coaching churn over the last few off-seasons. Whatever the reason, let’s reverse this trend, ladies and gents.
• However, it’s incredibly interesting (and heartening, I might add) to see that the nation as a whole has become more aggressive on fourth down over time.
• Your five most aggressive ACC teams on fourth down in 2020: Louisville (went for it in “go” situations 58.6% of the time), Virginia (57.1%), Florida State (56.5%), Georgia Tech (51.5%), and Boston College (45.8%). All five of these teams are in the “mediocre muck” in the center of the conference table, and four of them have recent coach hires. Why do these items matter? Well, teams that are evenly matched may get aggressive to find an edge to win a game, and more recent hires may be more numerically-inclined.
• However, your five most analytically-optimized ACC teams on fourth down in 2020: Louisville, Boston College, North Carolina, Clemson, and Virginia. The appearances of the Cardinals, Eagles, and Tar Heels may be explained this the same way as above: more recent hires may be more analytically-inclined and therefore more likely to optimize their decision-making processes. However, this comes with the caveat of UNC recent re-hire of head coach Mack Brown, but his offense is run by Phil Longo, who was known for his offensive aggressiveness while at Ole Miss. Clemson has more football resources than any other program in the conference, so they may be investing in analytics staff to optimize their decision-making. Finally, there’s Virginia, whom we know have analytics staff. More on analytics staffing later.
• It’s interesting that Dave Clawson’s high-tempo offense doesn’t also encourage him to be aggressive. The Clawfense hates numbers, I guess.
• Pittsburgh had a lot of opportunities to be aggressive and put themselves in places to succeed on fourth down. The Panthers...chose poorly.
• Only 13 of 130 FBS teams (127 of them actually playing in 2020) met or out-performed their expected “go” rates in “obvious go” situations, but the one I found most intriguing was Kent State, who were expected to go for it 60% of the time in these situations and did just that. Golden Flashes head coach Sean Lewis isn’t getting public calls for other jobs, but people in the industry that do podcasts that I listen to — yes, this is a long game of telephone — have heard his name whispered in and around better jobs. This seems like a reason why.

Georgia Tech

One of the things that I’ve wanted to take a look at for a while (and really the central conceit for this whole piece) is former coach Paul Johnson’s tenacity on fourth downs. Even when the distance, time left, or scoreline wasn’t in his favor, one always got the feeling that Johnson had a trick up his sleeve. We’ll never know whether Johnson’s penchant for fourth downs was guided by in-house modeling or just a good gut feeling, but we can evaluate how good said decisions were as a baseline versus Collins’ decisions. Here’s how that comparison shakes out (note: Johnson’s data is limited to the 2017 and 2018 seasons):

It seems that Johnson was indeed the riverboat gambler we all remembered, but we can’t sleep on Collins’ aggressiveness either (+3.1% advantage over the four-year national average). But did either go for it when he should have? Well, here’s the data (again, sample size in each bar):

Overall

By year

Compared to national average (“go”-ing on 52% of “obvious go” situations in 2020), Collins has been conservative during his tenure, sitting at 47% likely to go for it in an “obvious go” situation. One might think that given Tech’s offensive quality in 2019 (or lack thereof, considering their 117th ranking in offensive SP+), Collins may have been more reticent to push his luck, but even with improvements in offensive performance in 2020 (final offensive SP+ rank of #81), he wasn’t more comfortable doing so. Square these results with those from Paul Johnson’s final two years at the helm: with top-40 offenses in both years (#29 in 2017 and #40 in 2018), Johnson felt very confident going for it frequently.

But how did both coaches do when they went for it like they were supposed to? Was their risk-taking rewarded? Well, here’s how things shake out (again, sample size in each bar):

Overall

By season

Assessment: unclear. Based on the other charts and data points, it seems reasonable to guess that Johnson had more success in “go”-ing in these “obvious go” situations, but given the small sample sizes in the dataset we do have in front of us, it is tough to make that claim.

Conclusion

Bottom line: what do we make of all of these charts? Is there an analytics revolution brewing in college football coaching circles? Well, it’s unclear. Unlike in the NFL, there is no promise of parity in college football — you can quite literally [Styx] out wins based on talent differential alone (see: Alabama versus the vast majority of its opponents). It’s possible that coaches don’t believe they need to scrap at wins on the margins when winning games on the rest of the dang sheet of paper is more or less the same as winning them on the field. However, that approach only works 100% of the time for a certain subset of top teams. The rest of us still have to find some way forward.

And truth be told, it seems like some teams are using data-driven thinking as that way forward — the national decision-making chart above seems to indicate that there’s an increasing trend towards trusting the data in the sport in the last four years, much like there has been across sport in general over the same timeframe. It seems grandiose to stake that claim, especially given that we established that only 10% of 130 FBS programs are truly optimizing their fourth-down decision-making. However, seeing the underlying trend towards higher and higher “go” rates gives me hope that there are football programs out there that are putting the copious amounts of data they have to work to find and optimize not just fourth down performance, but also other general program inefficiencies. It doesn’t matter whether those inefficiencies are found in the weight room, on the field, on video, or in spreadsheets: identifying and fixing poor processes wherever they may be provide immense value to any sort of organization. It sounds like a lot of played-out corporate [Badness][Styx] (and believe me, I hate that it does), but it’s true.

Part of fixing these processes in a football program is resource identification: what pieces do we have or do we need to bring in to solve our problem? Oftentimes, programs may bet on a more expensive or expansive software suite, new athletic technology, or a new external hire to save the day. However, by exploring these options over all else, they may miss out on another way to close the gap: investing in bright, technically-competent, and sports-loving students at their own universities. These students, typically from data science, computer science, industrial engineering, math, or analytics programs, can blend findings from raw data with intensive domain knowledge to translate trends into sporting insights and employ those insights to help drive program decision-making.

If you want evidence of the competency of the average student that falls into this category, just check Twitter on the average college football Saturday or college basketball Wednesday. There is no shortage of numbers-oriented armchair quarterbacks and point guards who throw whatever publicly-available data they can find into Jupyter Notebooks or RStudio to analyze team performance and make fancy-looking (but easily-digestible) charts to post on the bird app™️. They’re often highly motivated to produce results and insights because of their fan-hood or career interests. In any other industry, one might consider this combination of interested, competent, and widely available talent a generous gift from their deity of choice. College sport programs must take better advantage of this homegrown talent, period. As the saying goes, adapt or die.

---

Special thanks to those that helped put the data behind this piece together: Jared Lee for his fourth-down model/bot, Saiem Gilani and Meyappan Subbiah for their work on cfbscrapR, and Bill Radjewski for his efforts organizing CollegeFootballData.com.

Interested in recreating some of these charts yourself? You can find the code to create them here.