Johnny Manziel ranked 16th in 2012 in NCAA passer efficiency with a value of 155.3. When Manziel won the 2012 Heisman Trophy as the best college football player, no one cited his passer efficiency to support it. Manziel's value came from more than his ability to throw the ball, which is what passer efficiency attempts to capture. Manziel led Texas A&M to 11 wins by running, avoiding the pass rush, taking care of the ball and moving the chains. The A&M system brought out the best features in Manziel, and its offense was the best in college football, especially accounting for the good SEC defenses it faced ...
... So the traditional numbers lied. That's what football people will tell you. And that's what sports analytics people will tell you when you use the wrong numbers. When you use numbers that don't accurately reflect football performance, they will provide wrong conclusions. The right numbers incorporate the context of the game, can give better conclusion, and provide reasons they get to those conclusions. It is overcoming the misperceptions created by numbers such as "Passer Efficiency" that is the job of sports analytics.
What Total QBR Captures
In particular for college football, we introduce this year the Total Quarterback Rating, also known as Total QBR or QBR. And, since college football has imbalanced schedules, QBR will be seen in both a form that adjusts for defenses faced - often called Opponent-Adjusted QBR or Adjusted QBR - and in a form that doesn't adjust for defense, often called Raw QBR or Unadjusted QBR.
The scale of QBR is from zero to 100, where 50 is average. Top quarterbacks are in the upper 80s and 90s for a season. Manziel, in fact, ranked first in QBR in 2012 with a value of 90.5. His unadjusted value was 86.4, also the best among FBS schools. The increase from unadjusted to adjusted reflects that he did face good defenses overall.
Unlike NCAA Passer Efficiency, which uses only box score statistics, Total QBR accounts for what a quarterback does on a play-by-play level, meaning it accounts for down, distance, field position, as well as the clock and score. A 5-yard gain on third-and-4 is a good play, whereas a 5-yard gain on third-and-14 isn't. A 20-yard touchdown pass when tied in the second quarter means more than a 20-yard touchdown pass when down 30 points late in the fourth quarter. QBR accounts for those things using analysis that turns traditional productivity into points on the scoreboard and wins in the standings.
It also accounts for a quarterback's ability to scramble, his ability to run on designed rush plays, how well he avoids sacks, drawing and committing penalties, and all-important fumbles, which can be significant for quarterbacks. If Texas A&M gained 5 yards on third-and-4, Manziel's contribution to that play is captured.
• If he threw it the full 5 yards and the receiver was immediately downed, Manziel gets a fair amount of credit for the throw, splitting it with pass blockers and the receiver.
• If he threw it a couple of yards behind the line of scrimmage to a running back, Manziel gets less credit because the receiver and any blockers in front of him did more of the work.
• If Manziel avoided a sack and scrambled for the 5 yards, he gets a lot of credit because his line gave up pressure and his receivers weren't open.
• If it was a designed rush for Manziel to get those 5 yards, he gets less credit than with a scramble because the offensive line often clears a couple of yards for a runner.
• If the defense was drawn offside, Manziel gets a modest amount of credit for that.
• If Manziel just turned to hand the ball off to a back for the 5 yards, QBR doesn't give Manziel any credit or even count it as an "action play" - our term for plays in which the QB gets some portion of the credit or debit.
These should make sense intuitively, and our analysis of the data supported these intuitions.
Opponent-Adjustment and Charted Data
The concepts of QBR are the same as those that were included in the NFL version. The biggest difference is that NFL QBR doesn't need opponent adjustment like college's does. We have checked the impact of adjusting for defenses in the NFL, and it is usually quite small. This is not the case in college, where Geno Smith, Tajh Boyd and AJ McCarron had unadjusted QBR values in 2012 that ranked outside the top 10 but had opponent-adjusted QBR values that were in the top five. We particularly emphasize that QBR adjusts for the defenses that a quarterback faced - in particular, the strength of those defenses versus opposing QBs. Although Texas A&M was a good team in 2012, its pass defense was only a little better than average, meaning that quarterbacks who faced them would not see significant adjustment upwards.
Another difference is that the NFL version incorporates extra data that is charted at ESPN, including how many pass-rushers there were, whether a quarterback scrambled or was on a designed rush and how many yards in the air a pass traveled. This extra information is not universally available for college football -- not for all those lower-level FBS schools, in particular. QBR for college football was built to incorporate this charted data, but it cannot be done quickly in the course of a season. Although historic tables of QBR do incorporate charted information when it's available, that info will not be incorporated throughout the current season until we can find ways to do it faster. As a result, 2013 QBR is based solely on play-by-play information until the end of the season. By using play-by-play information, we are capturing a lot more than ever before and we have spent a great amount of time to ensure that ESPN has the most accurate college football play-by-play information.
Benefits of QBR
There are several benefits of QBR for assessing college football.
For instance, ESPN college football analyst Brad Edwards looked before the 2013 season at what Notre Dame would gain and lose by having to go with Tommy Rees instead of Everett Golson after Golson was suspended. QBR highlighted the difference in how they contribute (passing versus running), but also that they're not terribly different in terms of overall efficiency. Being able to break down why -- play types, skills, situations -- leads to a greater understanding of the quarterback's QBR, and that is useful in analyzing a game.
The opponent-adjustment on QBR also applies on a game level, not just on a season basis. For instance, Georgia's Aaron Murray recorded an unadjusted 33 QBR against Alabama in the 2012 SEC title game but was adjusted to 68 based on Alabama's defense; Washington's Keith Price recorded an unadjusted 65 QBR against awful Colorado but was adjusted down to 35. Quality games are much easier to identify this way. (Note that opponent adjustment will begin about a month into the season, when there is enough information to do it well. It will be done regularly after that point, trying to take information from recent games to better evaluate opponents. This can change past "adjusted QBR" values somewhat, but to nowhere near the level that the QB's own performance can.)
A benefit of the zero-to-100 scale is that it strongly relates to a winning percentage. For example, Manziel's unadjusted QBR of 86.4 implies that Texas A&M should win 86.4 percent of its games given his performance and an average supporting cast (defense, special teams and non-QB rush offense). In fact, the Aggies' defense was pretty close to the FBS average and they won 11 of 13 games, or 85 percent. If Texas A&M had had a quarterback whose rating was 50, they probably would have lost an extra 36.4 percent of their games. Instead of 11 wins, they would have had six or seven. That's a big difference.
Further, analysts often say that winning the turnover battle usually means you win the game. We've run the numbers here and winning the turnover battle in college football wins 73 percent of games. Winning the QBR battle wins 85 percent of the games. So it is a key statistic, representing the importance of overall quarterback play to winning.
We also envision numerous future studies that use QBR. An obvious one is assessing the best NFL prospects. We do see a correlation between QBR at the college level and QBR at the NFL level, but we are sure that other factors -- height, hand size, arm strength, etc. -- will allow for better predictions of what will happen in the NFL. Maybe, with that info and QBR, we could tell the Raiders not to draft someone like Jamarcus Russell next time. Other possible studies include:
• What happened with Matt Barkley in his senior year at USC?
• Can Manziel and Marcus Mariota get better? How much does a quarterback improve from his freshman to his sophomore year? How much do typical quarterbacks continue to improve?
• What happens to a quarterback's performance when he loses good offensive linemen? AJ McCarron lost a couple of first-round picks from his offensive line at Alabama. Will that impact his QBR?
• Are there styles of quarterbacks that are truly more efficient in the red zone or in short-yardage situations? Are those running quarterbacks like Manziel and RGIII more valuable there?
Even more than in the NFL, where watching all quarterbacks is possible, we think that QBR will help people get a better understanding of not only how good college quarterbacks are, but what style they play and why they are good or bad.