Question: How Many Touchdowns Did Malcolm Butler Allow?

By Nick Webster

Malcom Butler

A few years back Malcolm Butler was the man who saved the Super Bowl for the Patriots famously jumping the route on the game’s decisive play and bursting onto the scene from relative anonymity.  

This year he will be known – consistently – as the most targeted CB in the NFL. Just look where everyone else does on Pro Football Reference (PFR) of Pro Football Focus (PFF) and you will learn he was the most picked on. PFR will tell you that Butler was Targeted 127 times in 2020, allowing 80 Completions for 873 Yards and 5 TDs. Now PFR does not compile these numbers they merely present these ‘Advanced’ defensive stats which are compiled by Sportradar. 

However, if PFF is your preferred source you will agree that he was the most targeted, but nothing else. PFF credits Butler with 116 Targets, 73 Completions for 892 Yards and 4 TDs.  

Now spiritually the numbers are similar as the PFR rating is 83.2 and the PFF rating is 83.7; nobody cares about this difference (it's close enough for rock and roll)  but when we look closer and some interesting questions arise.

It’s tempting, evaluating Butler’s numbers to suggest that Sportradar simply credited him with being in primary coverage on 11 plays where PFF did not. On those plays 7 were completions and 1 was a touchdown; however, somehow PFF has credited Butler with allowing 19 more receiving yards than Sportradar. 

The only way this can happen is if there were more than 11 targets and 7 completions that were differently assigned, there must have been at least one more Target not assigned to Butler and one that Sportradar assigned to someone else assigned to Butler which went for big yardage. Likely, it was not just one incrementally not assigned to Butler and one incrementally assigned to him but more.

Plotting the number of Targets Sportradar and PFF credited each top CB you see an extremely strong relationship. In fact, the R^2 of .92 suggests that one can ‘explain’ 92% of the other. However, our observation about Butler suggests that the gap is actually larger than that as Targets can be misassigned in both directions.

Interestingly, if you generally compare the Sportradar tracked Targets versus PFF they tend to be high.  Note the below chart which shows, for every CB Targeted at least 32 times in 2020 their Sportradar Target figure minus their PFF Target figure.  

Any of the Blue bars represent Sportradar having more targets than PFF—as you move in the direction of the Blue arrow Sportradar reflects more and more until you get to Steve Nelson, Rasul Douglas, and Tre Flowers, each credited with 22 more targets by Sportradar as PFF.  

The Black bar on the chart has them showing the same number of targets and the two sources agree on just 7 of 113 qualifying CB’s. Finally, the Green bars represent PFF compiling more Targets than Sportradar, increasing in the direction of the green bar until you reach Mackenzie Alexander and Chandon Sullivan with 11 more targets by PFF than Sportradar.

There’s a clear bias for Sportradar to have more Targets than PFF. Of the 113 qualifying CB’s, 89 or 79% have more Targets credited by Sportradar and just 17 or 15% have more Targets by PFF. On average Sportradar documents 5.4 more Targets than PFF does, which raises an interesting question, where are all the other passes?  

Possibly when coverage is totally blown Sportradar credit the CB who blew it and PFF just classifies it as an open receiver or a blown coverage. What about double coverage, maybe one credits both players as in coverage and the other, likely PFF, requires that the assignment be credited to a single CB.  Perhaps things like screen passes or hot-routes to RB’s are treated as uncovered by one and covered by another. We just don't know for sure we just know there is a statistical difference.

So, if PFF and Sportsradar cannot agree on Targets, then certainly they cannot agree on other aspects of pass coverage. And in fact, Targets are where the two sources agree most with the highest R^2 of any other metrics tracked . . . the worst, believe it or not, is Touchdowns.  

The following are the charts for the other data points in the individual defensive passer rating formula—

Somehow Sportradar (SR) and PFF were so different in their views of coverage responsibility that the former accounted for Troy Hill giving up 4 TD passes in 2020 and the later just 1. Given how important TD’s are as part of passer rating this means Hill’s SR passer rating is 91.2 and PFF rating is 74.3.

Fortunately, at the level of the top performers, there is general agreement. Of the Top 10 CBs—ranked by Passer Rating against 5 occupy the exact same position in the rankings. Of Sportradar’s Top 10 ranked CB’s 8 are also Top 10 in PFF. 

What does this tell us?  It tells us that these stats are generally reliable and a reasonable indicator, but they are not perfectly accurate either. We should not be splitting hairs between a completion here and there or a couple point s on passer rating.  

But we can probably say that, for example, Marcus Peters with Ratings of 78.9 and 88.1 by SR and PFF was probably better in 2020 than Patrick Peterson at 98.2 and 100.8.  

It also reminds us that as much as we love to follow the numbers, they are just one piece of player evaluation, always trust the tape by watching it yourself—"The Eye in the Sky Does Not Lie".