Georgia Tech Football: Advanced Stats Review Part 5 - Checking in on Goals, Predictions, and the Binion Index

Our offseason Advanced Stats Review is not complete without some self-scouting. Throughout last offseason and the 2021 season, we offered a multitude of predictions for GT and the wider CFB world, and we also set some goals that would provide a benchmark to evaluate GT’s progress this past season. Here, we try and put all of that on the table to see where we went right, where we went wrong, and how we can do better next year.

Georgia Tech Season Goals

In my offseason preview work last summer, I identified some key stats to pay particular attention to for tracking GT’s progress this year. I specifically honed in on the offense’s Completion Percentage above Expectations and Defensive EPA/Pass as numbers that would go a long way in telling the story of the season. We were quite correct about those numbers telling the story of the season, but that didn’t turn out to be a feel-good story.

As you can see above, we did not exceed a single one of the goals I set (for the second year in a row, in fact). What ongoing issues does this point to? Quarterback play isn’t good enough. The offense lacks a game-breaking receiver who can catch slightly off-target balls. Offensive and defensive line struggles manifest in the below average pressure rate on both sides of the ball. The abysmal defensive havoc rate continues to point to a lack of top end talent throughout the defense. The passing EPA/play allowed obviously led to wholesale changes on the defensive coaching staff and in personnel.

Mea Culpa

I had identified increasing the first down passing rate as a preseason goal and harped on that repeatedly throughout the year. I was basing that on research that has shown that in the NFL, passing more on first downs produces more efficient offense. However, in the offseason work I have done, I have found that relationship simply does not hold in CFB.

Above, you can see that there is not a statistically significant relationship between first down pass rate and first down EPA/play, either last season or overall for the four previous seasons.

I rescind my criticisms for CDP in this department, as it seems the inconsistency in QB play in college and the relatively higher value of the running game nullifies any statistical correlation between first down pass rate and offensive efficiency. This won’t be part of our goal-setting for next season, although we will continue to monitor GT’s play-calling trends, such as in 2nd and long situations.

Game by Game Predictions

I went back through a number of the predictions I offered leading up to the season, and I need to self-evaluate. In our final season preview post before the first game, I offered a game by game breakdown of what I expected.

For these final predictions, I set the spreadsheets aside and made an assessment that included data I had looked at but ultimately rested on more of my intuition. Comparing these predictions to how our model performed throughout this season shows the value of sticking closer to projections coming from a well-calibrated mode. The Duke and Notre Dame games are the clearest examples of how ignoring the direction the numbers pointed can lead to some really bad predictions.

Leading up to the start of the season, it’s easy to give in to the desire to make more optimistic predictions. For example, in my offseason article predicting improved play at the QB position, part of my reasoning was based off of some sound bites from a CDP press conference. Sometimes, we are so desperate for any information leading up to the season that we get blinded into thinking that communication like that is pointing towards guaranteed improvement. In the end though, we need an empirical reason to project improvement if we want the projections to ultimately stand up to scrutiny. That’s probably my foremost lesson learned from last offseason’s predictions.

But what about a more empirical game by game set of predictions?

Georgia Tech Win Probabilities

Georgia Tech ended up at 3-9, which was, of course, much worse than average fan expectations coming into the season. But we can look more closely to measure the performance on the scoreboard against model expectations and the actual underlying play by play results during the game. Was three wins more than expected, less than expected, or about right based on how the team actually played?

We can begin answering these questions by using our TBI numbers, closing Vegas spreads on each game, and a win expectancy metric that I have developed incorporating success rate, yards per play, and EPA/play margin in each game.

Vegas and computer models, like TBI, saw GT as a 5 win team in 2021. The actual on-field performance of the team would normally produce 4 wins. GT closed things out with 3. Here’s one way we can interpret that: the team quality that Vegas and other models saw should have led to a 5 win season. The actual play-by-play performance in those games (accounting for factors that are generally governed mostly by luck) was that of a 4.3 win team. The actual result was 3-9. Actual wins versus expected wins evens out over time in most programs, but this is the second straight year that GT has underperformed its post-game win expectation by about 1.5 wins. Two seasons are not enough data points to draw strong conclusions, but we do have another pointer towards the failures of this staff in on-field, game-day coaching situations.

Underperforming win expectations on the field points to the deployment and tactical weaknesses on the coaching staff that we have observed repeatedly. The good news is that we are not likely to continue to win less games in coming years than our post-game win expectancy predicts. That means it is fair to look at this team as closer to a 4.5 win team heading into the offseason. Being a 4 win team in 2022 won’t exactly ease the concerns of the fanbase, but hitting the magical talisman of 6 wins is a lot more likely for a team with an underlying 4.5 win quality than 3 win quality.

Other Predictions

I also offered a few more specific predictions in that series of posts:

• Georgia Tech to play in the Birmingham Bowl on the day after Christmas: Incorrect. We were comfortably home on Christmas.
• Jeff Sims to be the third best quarterback in the ACC and put himself on the national radar for 2022: Incorrect.
• Jahmyr Gibbs to live up to the hype, finishing the year with 1500 yards from scrimmage and 15 touchdowns: We were a little high on touchdowns, but his all-purpose yardage exceeded expectations. Alas, he now resides in Tuscaloosa.
• Jordan Domineck to hit double digit sacks, GT’s first such performance since Derrick Morgan in 2009: Incorrect.
• The overall statistical profile will show progress but still lag behind the better years under CPJ. The recruiting class will hold together and finish 35th nationally, and 2022 will lurk as the year to make a leap: Incorrect across the board.

Unfortunately, here was my commentary on the worst case scenario for the season:

I rewatched the 2020 GT-Boston College game recently. That’s what the worst case looks like: repeatedly gashed on inside zone runs, looking helpless to stop explosive passing plays, and turning the ball over repeatedly to kill promising drives. The worst case is finishing 3-9 again, winning only one of the final 10 games, and the goodwill given to the rebuild all but dissipates. The recruiting class loses two key members and ends up about 50th nationally.

That is almost exactly what happened. Oops.

TBI Analysis

Finally, let’s take a big picture look at the model for predicting CFB outcomes that we introduced this year. Overall, we came in just below 55% in ATS results, which is excellent, but our absolute error was over 13 points per game, which needs work. Below is a team by team visual of how the model did, showing ATS win percentage and average absolute error for every FBS team in 2021.

The model really struggled with the triple-option running service academies, as well as some programs that have had significant coaching changes over the past few years. To work towards improvement in 2022, we’ve been analyzing some different metrics in comparison to how the model performed on each team. So far, we have been able to identify six factors that had a statistically significant influence on error, and I’m excited to utilize those results as we work towards building our final 2022 model.

As a side note, we will have a preliminary 2022 preseason model out in the next few weeks, but the fully updated model won’t be ready until closer to the summer.

That wraps up our Advanced Stats review looking back on the 2021 season. I’m expected for an offseason of improving our modeling, searching for more insight on GT’s 2022 season, and looking broadly at the ever-evolving CFB landscape.

Anything else you’re hoping to see from an advanced stats perspective this offseason?