- Home Posts filed under draft2
Bayesian Draft Analysis Tool
For details on how the model works, please refer to these write-ups:
- A full description of the purpose and capabilities of the model
- A discussion of the theoretical basis of Bayesian inference as applied to draft modeling
- More details on the specific methodology
If you want to jump straight to the results, here they are. But I recommend reading a little further for a brief description of what you'll find.
The interface consists of a list of prospects and two primary charts. Selecting a prospect displays the probabilities of when he'll likely be taken. You can filter the selection list by overall ranking or position.
The top chart plots the probabilities the selected prospect will be taken at each pick #. I think this chart is pretty cool because it illustrates the Bayesian inference process. You can actually see the model 'learn' as it refines its estimates with the addition of each new projection. Where there is a firm consensus among experts, the probability distribution is tall and narrow, indicating high confidence. When there is disagreement, the distribution is low and wide, indicating low confidence.
The lower chart is the bottom line. It's the take-away. It depicts the cumulative probability that the selected prospect will remain available at each pick #. For example, currently there's an 82% chance safety HaHa Clinton-Nix is available at the #8 pick but only a 26% chance he's available at #14. A team with an eye on a specific player could use this information in deciding whether to trade up or down, and in understanding how far they'd need to trade.
Hovering your cursor over one of the bars on the chart provides some additional context, including which team has that pick and that team's primary needs (according to nfl.com).
The box in the upper right gives you the player's vitals - school, position, height, weight. The expert projections used as inputs to the model are also listed. Currently those include Kiper (ESPN), McShay (Scouts, Inc.), Pat Kirwan(CBS Sports), Daniel Jeremiah (former team scout, NFL Network), and Bucky Brooks (NFL Network). Experts were selected for their reputation, historical accuracy, and independence--that is, they don't all parrot the same projections. Not every prospect has a projection from each expert.
Link to the tool.
Cade Massey on Flipping Coins and the NFL Draft
Massey has continued research into the draft. His presentation at the 2012 MIT Sloan Sports Analytics Conference outlines his recent findings. (I recommend using IE to view the presentation. Chrome didn't play nice with the video.) The slides from the brief can be viewed here.
If I understand things correctly, Massey has found that:
WP: Moving beyond Grossman
This week's post at the Washington Post's Redskins Insider site takes a look at the Redskins quarterback situation going into the 2012 off-season.
...with so many needs heading into the off-season, it might be tempting for the Redskins to think that Grossman is a medium-term solution at quarterback. But this is a trap...
“Best season ever”, “31st in the league”, and “leads the league in interceptions” is a combination that does not bode well for the future of a franchise that would consider standing pat at quarterback.
WP: Rebuilding with Moneyball for Football
The Redskins need to restock the cupboard with talent. Here's how a team can build and sustain a winning roster.
Playing Moneyball in the NFL is about jettisoning expensive and under-producing veterans, rejecting the big-splash free agent, and stockpiling draft picks. There are two ways of generating those picks. First, you can trade away soon-to-be free agents to other teams in return for picks or allow restricted free agents to sign elsewhere in return for compensatory picks. For too long, the Redskins have been on the wrong end of those transactions.
The second way is to trade picks for more picks. Overconfidence and urgency run rife in personnel departments around the league, and smart teams can take advantage of this. There are always teams willing to overpay for a pick that they are so certain will immediately turn their team into a Super Bowl winner. A team can sell its first-round pick for a second-round pick this year, plus a first-round pick next year. In the next draft, that team will have an additional first-round pick that could be sold for another second-rounder, plus another future first rounder. Presuming there are enough buyers, a team could generate an additional second-round pick in perpetuity by foregoing its first-round pick in only one year.
There's one team in the league that understands this, and they've been phenomenally successful doing it:
What Happened to the First Round RB?
The graph below illustrates the trends in how teams favor each position over the past 41 years. Most positions are fairly stable. Click to expand.
Draft Needs According to 2010 EPA - Defense
The table below lists the 2010 regular season +EPA totals for each team by position. Although it doesn't consider free agent losses or injuries, this might be considered a good starting point for determining defensive draft needs.
Unfortunately, defensive positions are not so cut-and-dry. Depending on the base scheme, there are varying numbers of DEs, DTs, and LBs. (The 49ers don't even have a DE position.) Plus, players can sometimes be designated different positions from week to week. Advanced NFL Stats ultimately classifies players according to their most frequent designation in each game's official playbook.
Draft Needs According to 2010 EPA - Offense
For example, consider a team whose production from the offensive line, quarterback, running back, and tight end positions all rank somewhere in the top third of the league. But its wide receiver production ranks in the bottom third. It should be no secret where the team should look to improve.
Although it's doubtful the stats will tell us much we don't already know about team needs, they can confirm, underscore, or possible refute the common perceptions.
The table below lists each team's EPA rankings by position. Within each position, the top third of the league is shaded in green, and the bottom third is shaded in red. EPA stats for the QB, RB, TE, and WR positions are straightforward aggregations of each player's EPA by team. But offensive line EPA is measured indirectly, using the concept of -EPA. Each column is sortable.
Steven Pinker vs. Malcolm Gladwell and Drafting QBs
Last season you might recall a dust-up between Harvard evolutionary psychologist Steven Pinker and popular science author Malcolm Gladwell over whether teams really have any ability to predict which college QBs will pan out into good pros. You might be wondering what the heck a psychologist and a pop-science author have to do with NFL football.
In his book What the Dog Saw, Gladwell wrote about how hard it is for school administrators to discriminate the better teacher candidates from the lesser candidates. Gladwell used the NFL draft to illustrate how difficult it is for anyone to predict human performance, even in a sport where there is ample performance metrics and every step, throw, and catch is videotaped from 12 different angles. Gladwell was referring to what was reported by economists Dave Berri and Rob Simmons as a "very weak" correlation between draft order and per-play performance by QBs.
In an exchange of letters following Pinker's critical review of What the Dog Saw, Pinker took issue with Gladwell's claim that there was "no connection" between when a QB is taken in the draft and his per-play performance. Pinker wrote that this is "simply not the case."
As has been pointed previously, the problem with the weak correlation cited by Gladwell is that it excludes players who are not judged good enough by coaches during their development to warrant much if any playing time. At its core, the NFL draft is a process of selection, and we should expect selection bias will taint most attempts at analysis. Gladwell looked at the draft process and (correctly) said:
"Coaches and GMs turn out to be good decision-makers when it comes to drafting quarterbacks when you consider the fact that the quarterbacks who never played aren’t any good. And how do we know that the quarterbacks who never play aren’t any good? Because coaches and GMs are good decision-makers!”
But Gladwell's argument cuts both ways. The only way to see that coaches and GMs aren't any good at drafting QBs is to assume they're no good at choosing which QB on their roster to play in games!
In this post I'll attempt to settle the question of whether NFL scouts really have any ability to identify the better QBs. Do the QBs picked higher in the draft turn out to be better performers on a per-play basis? Is Pinker correct that they do, or is Gladwell correct that they do not?
Are Top Draft Pick QBs Any Better Than Late Round Picks?
Is quarterback performance related to where a passer is taken in the draft? It may seem like a silly question, but the answer is more complicated (and controversial) than it first seems. What if I said that the correlation between a QB’s draft rank and his career adjusted Yards Per Attempt (AdjYPA) is only -0.07? You’d think that’s amazingly small. (The correlation coefficient would be -1 if the relationship between draft order and performance is perfectly proportional, and it would be 0 if there is no relationship at all.)
What if I told you the correlation coefficient is -0.72? That’s more like it, you’d think. But which correlation coefficient is correct? They both are.
Rethinking the Massey-Thaler Draft Study
Economists Richard Thaler and Cade Massey authored a widely-read research paper analyzing the value of NFL draft picks, and they've recently published an updated version. The paper's primary finding was that teams are overconfident in their ability to choose the best players. In essence, the very top picks are overvalued relative to later picks, both in terms of what teams are willing to trade to move up in the draft and in terms of salary.
Recently Richard Thaler penned an article for the NYT discussing his paper. But puzzlingly, he goes on to make a claim that clearly contradicts his own research. He writes, "it makes absolutely no sense to be giving so much money to unproven rookies, many of whom turn out to be busts." Further, he writes, "veteran players would probably agree with the principle that eight-figure salaries should be reserved for players who have already proved themselves on the field."
Are Safeties Risky Top Picks?
One of the Chiefs' most dire needs this off-season is a dynamic safety, but GM Scott Pioli is reluctant to take a safety with the 5th pick in the draft. Falcons GM Thomas Dimitroff is apparently on the same wavelength. There seems to be a growing conventional wisdom that safeties are high-risk picks at the top of the draft. As Peter King pointed out recently, the three best safeties of the decade--Ed Reed, Bob Sanders, and Troy Polamalu--have missed 78 games due to injuries in their combined 21 NFL seasons.
The thinking is that safety (ironically) is a fundamentally dangerous position. The nature of the position, launching head-first at high rates of closure toward oncoming ball carriers, may carry a systematically higher risk of injury than most other positions. Reed, Polamalu, and Sanders suggest this may be the case, but a sample size of three is small to say the least. Are Pioli and Dimitroff rightfully concerned?
Are Rookies Overpaid?
I recently looked at what might explain why the top draft picks are paid disproportionately to their expected performance compared to later picks. But that doesn't address the larger issue--are rookies overpaid compared to their veteran counterparts?
A 2005 research paper called The Loser's Curse by economists Cade Massey and Richard Thaler tackled that question. In a nutshell, the paper compares rookie pay to the pay of a 6th-year veteran who could be expected to deliver the same performance as a rookie from each slot in the draft. (Performance is defined by a mix of measures including: being on a team roster, starts, and Pro Bowls.)
The conclusion of the paper is that team executives and scouts overpay for the top picks in the draft relative to the later picks, likely due to overconfidence in their ability to identify the best players. But what might surprise some readers is that rookies at every level of the draft are bargains compared to equivalently performing veterans.
This graph from the paper is the study's bottom line. The red 'compensation' line is the average annual pay for each draft pick. The blue 'performance' line is the salary a team would have to pay a 6-year veteran free agent for the same expected performance. The green 'surplus' line is the difference between the two pay levels.
The surplus performance peaks shallowly at the bottom of the first round and through the second round. That's where teams get the biggest bang for the buck. But still, the surplus is strongly positive throughout the entire draft. According to Massey and Thaler, rookies are a bargain compared to veterans.
There's a good explanation why rookies would be underpaid. Veterans are known quantities while there is a tremendous amount of uncertainty with draft picks. Think of it this way--Peyton Manning has been to nine Pro Bowls and Ryan Leaf to zero, for an average of 4.5 between the two players. Four Pro Bowls--that's not bad. But would a GM pay more for a guaranteed 4.5 Pro-Bowl-type player or for a 50/50 shot between a total bust and Hall of Famer? Just about every modern economic and psychological theory tells us that people will pay a premium for the sure average.
Unfortunately, that's not an option in the draft. Peyton Leaf just doesn't exist. But 6-year veterans do, and GMs will be willing to pay a premium for the reduced uncertainty in performance.
One note of caution on the paper. The draft years studied were 2000-2002, and rookie salaries have increased substantially since then. But veteran salaries have too. The question is whether rookie pay increases have outpaced veteran pay increases since then. However, rookie pay would needed to have increased over 15-20% faster than veteran pay to change the conclusions of the paper.
Career Success by Draft Order
In response to some requests from commenters I’ve put together a few more graphs of how draft picks tend to pan out according to when they were picked. These graphs are for all players and not broken out by position. Mostly it’s just some food for thought heading into the big day tomorrow.
As always the data are from Pro-Football-Reference.com. All draft picks from 1996-2008 are considered.
First is a graph of the likelihood of Pro Bowl selection according to draft order (overall pick number). The graph is grouped into sets of five picks, with the first data point as picks #1 to #5, the second as #6 to #10, etc. Without the grouping the graphs are too noisy to be helpful.
Notice the small spike for picks #11 - 15. I'm not sure if we can read anything into that or not, but it might be worth investigating.
Next is a graph of the average years as a starter by draft order. The picks are grouped into sets of 5.
Phil B. raised an important consideration. Player success has a lot to do with opportunity, and that needs to be factored into the discussion. He suggested that top picks will get the opportunities to start, (ostensibly because teams have the most invested in them). So regardless of differences in ability, top picks would naturally be expected to become Pro Bowlers more often simply due to opportunity. Phil suggested a graph plotting success divided by opportunity.
I think this graph is what he was suggesting. Below is a plot of number of Pro Bowl selections divided by years as starter (#PBs/St Yrs), by draft order.
There is a distinct downward slope. The top picks are more likely to become successful even accounting for opportunity (at least in terms of Pro Bowls, an admittedly imperfect measure). In fact, my hunch is that this would over-account for opportunity because Pro Bowl selection and being a starter are both directly proportional to player talent. So we’re really dividing talent by talent + opportunity. The “excess” Pro Bowl selections of the top picks suggests their success has to do with more than just opportunity. But it might not be all due to talent--top picks certainly get their share of notoriety, which can be a factor in Pro Bowl selection.
Notice the plateau from about pick 51 to pick 80. I'm not sure if it means anything, but perhaps this suggests 3rd round picks are better in terms of talent than their opportunities allow. Or on the other side of the coin, maybe 2nd round picks are given more opportunities than their talent merits. But it might be just noise in the data.
A quick note regarding the use of Pro Bowl selections as a measure of success. I've pointed to the flaws in using Pro Bowl selections several times, so perhaps I should explain why I do think it can be useful. Even though Pro Bowls aren't purely performance-based, the best players do rise to the top in the aggregate. Plus, being named to at least a single Pro Bowl at some point in a player's career, at the very least, confirms a pick is not a bust.
Draft Picks: Bricklayers or Gladiators?
With the draft upon us, there is a lot of chatter about ballooning rookie salaries for top picks. The consensus seems to be that top picks are not worth the cost and salaries should be capped. But there’s a good reason why the top players’ salaries are so high, and the explanation can be found in economic ‘tournament theory.’ A short example by economics professor Robert Schenk explains it nicely:
Say you’re a contractor and your company builds brick walls. Most of your bricklayers lay about 3 bricks per minute and make about 8ドル per hour. (You can think of this as the replacement level.) But along comes a guy who lays bricks twice as fast--6 bricks per minute. How much would you be willing to pay him? Simple fairness suggests 16ドル per hour. Certainly no more than that because you could just replace him by hiring two replacement-level guys and get the same production. So in this example rewards are based on absolute differences in productivity. Large differences in productivity result in large differences in rewards. Similarly, small differences in bricklaying ability would result in small differences in hourly pay.
Now consider two ancient gladiators entertaining the emperor in combat. Even if one gladiator is only slightly better than the other, he’ll very likely win, and the differences in rewards could be extreme. Here, in a winner-take-all system, absolute differences in ability do not matter, only relative differences.
What about sports like football? First, in many ways the NFL is a winner-take-all system. Whoever wins the game earns 100% of the win while the loser eats all of the loss, and there is only one champion left standing at the end of the season.
Second, football players are not like bricklayers. You cannot replace an All-Pro QB by sending two average QBs out on the field and expect the same productivity. When there is a constraint on the number of people that can be employed at one time, the value of the most productive people rapidly increases.
And when there is a constraint on the number of contributors combined with a winner-take-all reward structure, the value of the top performers will skyrocket. This is why the top NFL draft picks make so much more than the lower picks. Even if the abilities of the top picks are only marginally better than those of the picks in later rounds, there will be very large differences in pay.
It’s not much different than CEO compensation. If a company is in competition with other companies for market share, the shareholders should want the best CEO they can get--especially because a competitor with a slightly more visionary CEO will likely steal market share, even if your guy is still top-notch. And since you can’t replace a single chief executive with two average guys or a whole mob of slackers, the CEO’s pay is going to end up being wildly disproportionate to his actual ability. There can be only one guy at the top, only one winner of the tournament.
Note that I’m not claming that rookie salaries should be this high, just trying to understand why they’re so high. And I’m not comparing rookie pay to veteran pay. That’s another topic for another day.
Drafting Linebackers
In some circles, the conventional wisdom is that great linebackers can be found anywhere in the draft, and that teams should think twice before taking a LB in the first round. This post will take a look at whether this is true by looking at LB performance according to draft round and pick order.
We've seen how performance varies by draft position in QBs, RBs, DEs, WRs, and DBs. How do LBs compare? Based on the careers of all LBs taken in the 1980 through 2001 NFL drafts, we'll see how scarce top LBs typically are, and what kind of performance teams can expect from their picks.
I'll start by looking at Pro Bowl selections, and I'll repeat my standard disclaimer. Pro Bowl selection is a very imperfect measure of a player's value for a lot of different reasons, but it does identify the top players at each position which is what much of the draft is about. One other advantage it offers is that player value can be compared across positions. For example, we can compare LB draft picks to QB draft picks using Pro Bowl selections, but using passing yards or tackles wouldn't work to well.
The first graph looks at the rate of Pro Bowl selection by draft round. The blue line illustrates the likelihood a pick from each round will be selected to at least one Pro Bowl at some point in his career. The red line is for two or more Pro Bowls, and the green line is for three or more.
The next graph looks at Pro Bowl selections by draft order within position. The scouts must be doing their job because first linebacker taken really outshines the second, third, etc, at least in terms of going to at least one Pro Bowl.
The third graph is the average number of years a player is a starter for his team, broken out by draft order. The careers of the first LBs taken appear to have more longevity.
The graphs remind me a lot of the ones for wide receivers. There is a relatively large drop off after the very top players taken.
The continuing theme in this series is that the best players really do come from the top of the draft. No surprise there. But the top players have more than just an incrementally higher chance of great success, but double or triple the chance. The scouts and GMs do have an ability to recognize the players with the most potential at every position we've looked at so far. It's interesting to see just how steep the drop off really is after the first few players.
Drafting Defensive Backs
Continuing the series of analyzing the NFL draft by position from last year, this post will look at defensive backs. How likely do the top picks outperform the later ones? How often do later picks turn out to be solid contributors? How do they compare to other positions? Using data from all defensive backs taken from 1980 through 2001 I'll answer those questions.
Unfortunately, the draft data at Pro-Football-Reference.com does not distinguish between cornerbacks and safeties (for understandable reasons). They are all considered defensive backs, so this analysis will have to do the same.
First, let's look at Pro Bowl selections. As I've mentioned before, Pro Bowl selection is a very limited measure of a player's value. In fact, no single measure can be perfect. But Pro Bowl selection does tend to signify the top players at their positions, and that's really what much of the draft about.
The graph below illustrates the likelihood of a defensive back taken in each round to be named to 1 or more, 2 or more, and 3 or more Pro Bowls in his career. As you'd expect, there is a steep drop off after the first round.
The next graph illustrates the same thing, but by draft order--in other words, regardless of round or overall pick number, was the player the 1st, 2nd, 3rd and so on defensive back taken? After the top couple DBs taken each year, there's a steep drop off in the chances the player will turn out to be a star.
Notice anything about the shape of the distributions? They're power law probability distributions, the signatures of many natural complex systems with all the implications that come with them. The next graph isolates just the '2 or more' Pro Bowls likelihoods and fits a power law curve.
Next is a graph of years as a starter by draft position. No, this time it's not a power law distribution but an exponential one, which is a little easier to interpret. Every subsequent DB taken will have an average career as a starter 7.9% shorter than the earlier DB taken. There may not be many stars in the later picks, but there are plenty of durable starter-quality players to be had.
Although it's not the only measure of a DB by a long shot, we still want them to intercept passes as often as possible. The next graph looks at the total number of interceptions by draft order. Just like the years as starter graph, the curve is exponential with an average 7.4% difference in total interceptions from any given DB pick to the subsequent pick. I'm not sure this tells us anything more than the last graph of years as starter. The two graphs closely mimic each other for obvious reasons.
The last graph we'll look at is the likelihood the a DB draft pick will turn out better than the subsequent DB chosen. To define 'better' I used years as a starter rather than any of the other measures. All the measures are flawed, but I think years as a primary starter is the safest measure because it captures so much other information, both quantitative and qualitative, about a player's performance.
This graph will tell us how well scouts and personnel executives identify the better prospects. But looking at DBs in isolation doesn't tell us much, so I added RBs and QBs to the graph for comparison.
All three positions are fairly noisy, but it looks like scouts can identify the superior DBs slightly better than RBs and QBs, particularly deeper in the draft. What this means is that GMs can have slightly better confidence when picking a DB than the other selected positions. We should expect an inverse relationship between the ability of scouts to identify the superior prospects and how deep into the draft teams can expect to find solid starter-quality players.
And that's really the point of all this analysis. In order to eventually build a sound comprehensive model of draft strategy, we'd need to know all of the likelihoods of success for the various positions in each round and at each pick order. In the meantime, it's useful to know how deep into the draft a team can find viable contributors at the various positions.
Draft Analysis by Position
It’s draft season again, and one of the most important aspects of draft strategy is assessing the value of players taken at the various spots throughout the draft. How well to 1st round QBs pan out compared to 2nd or 3rd rounders? How sure can a team be that the 1st QB taken will in fact turn out better than the 2nd QB taken? Can a team really find good RBs in the 3rd round compared to the 1st? Do some positions tend to be gambles compared to others which may tend to be sure things?
The purpose of this analysis is to quantify the scarcity of quality players in the various positions and the ability of scouts to actually identify who the better players are going to be. Knowing these things, teams can construct better draft strategies. For example, if the better RBs actually come from the first few taken each year and generally can’t be found in the later rounds, then teams looking for a RB need to plan accordingly. Additionally, if it’s found that top RB draft picks tend to be ‘sure things’ compared to other positions, then teams may prefer to fill other needs with ‘known-quantity’ veterans rather than relatively uncertain draft picks.
Last year I looked at most of the skill positions because they tend to be the focus of the most attention and they provide easier measures of performance with their stats. My analysis of each position usually follows a similar pattern.
I looked at how likely players taken in the various rounds would attend one, two, or three or more Pro Bowls. Although Pro Bowls are an imperfect measure of a player for several reasons, they do signal a player’s overall achievement in ways that individual performance stats cannot capture. Pro Bowl appearances also indicate whether a player was a ‘home run,’ something GMs are certainly looking for in the early rounds.
I also looked at other indications of a successful draft pick, such as numbers of years in the league and number of years as a team’s primary starter. Although a player may never make a Pro Bowl, he may still be a solid above average player, and that’s certainly of great value to the team who drafts him.
For positions like QB or RB that offer obvious performance measures such as yards per attempt (or even DE with sacks per game), it’s very helpful to look at those distributions too. Plus, a good measure of how well scouts can predict the better prospect is the likelihood that an earlier pick will actually turn out better than the subsequent pick at the same position.
Each one of these measures is imperfect for measuring the value of a draft pick in some way, but together they give us a very good idea of how scarce the various positions really are and how well scouts identify the better players.
One more thing I’ll point out is that draft round doesn’t tell the whole story. It’s also important to look at the overall order a player was taken within his position. In other words, not all first round picks are the same. If 3 QBs were taken in the first round, they’re going to have very different likelihoods of becoming a top passer.
I’ll recycle the positions I analyzed last year, partly because there are so many new readers this off-season compared to last:
QBs Part 1
QBs Part 2
RBs
WRs
DEs
Next, I’ll be looking at defensive backs, but I’ll also tie in an interesting observation about how career success is distributed among NFL players. And if anyone is waiting for part 2 of the Passing Predictability article, I’m going to put that on hold until after the draft.
Lastly, I’ll mention that most data comes from Pro-Football-Reference.com’s draft database, which is complete with great career data and features a great query tool.
Drafting Wide Receivers
If Lions GM Matt Millen has contributed anything to the world of football player evaluation, it's that wasting a top pick on a wide receiver is very dumb idea. The fragile-egoed temperamental prima donnas that they are, WRs are known as a notoriously undependable lot. Worse, their productivity seems completely unpredictable. But it's not true.
Using data from the draft database at Pro-Football-Reference.com, all WR draft picks from 1980-2000 were evaluated by Pro Bowl selection, receiving yards per year, and years as a starter.
Pro Bowls
Receivers were classified as being selected one or more, two or more, or three or more Pro Bowls. The likelihood of each by round and by position draft order is illustrated in the two graphs below.
After the 4th WR taken in the average draft class, it becomes drastically less likely a team will find a star player. It looks like that the drop-off usually happens around halfway through the 2nd round. By the 5th WR taken and the 3rd round, WRs appear to be about equally as likely as much later picks of becoming an all-pro at about a 5-10% probability.
Receiving Yards Per Year
I normally prefer rate stats when comparing players, but comparing the records of receivers is an exception. Yards per reception is a helpful stat, but receivers create their own receptions by getting open, so yards per reception would not capture a very important part of playing WR. Yards per "target" might be better, but targets are very subjective and unstandardized statistic. For this comparison, I'll use career yards per year broken out by round and by position draft order.
We see a similar pattern as with Pro Bowl selection. After the 4th WR taken, prospects for finding a solid contributor drop significantly.
Scout Accuracy
How well to NFL team scouts and player evaluators predict the performance of WR draft picks? The table below lists the likelihood that a WR will end up having more Yds/Yr than the next WR taken in the draft. For example, the 1st WR taken ends up as better than the 2nd WR taken only 38% of the time.
It seems to be very difficult for teams to identify the better player between two closely matched potential draft picks, even for the #1 WR taken. But overall, they seem to have selected the better receivers in the top round. These results suggest a good strategy for picking a good WR with a very high draft pick is to trade down to pick the second or third WR available in the draft. Teams should have very little confidence that the highest rated WR will turn out any better than the second. On the other hand the #1 WRs tend to become the biggest stars, being selected to three or more Pro Bowls in one out of three cases.
Years as Starter
Another way to evaluate draft picks is to look at how many years they are starters for their teams. The tables below list the averages for each round and draft order.
Conclusion
As temperamental as they may be, top WR draft picks really do turn out to be stars far more often than later picks. They seem to be a lot like QBs. There is a real scarcity of talent at both positions, and it is difficult to predict with much certainty which ones will pan out. A team's chances of finding a highly productive player are still better with a top pick.
Drafting Defensive Ends
Where can NFL teams find quality defensive ends in the draft? This post looks at DEs drafted in the first seven round from 1980-2000. Data comes from the great draft database at Pro-Football-Reference.com. Picks from each round and at each draft order are compared by Pro Bowl selection, sacks per year, and number of years as a primary starter.
Although sacks are obviously not the only contribution a DE makes, and some DEs benefit from certain defensive schemes more than others, I believe that it is still a valid measure. First, sacks are only used as one measure. Overall DE performance can be reflected in Pro Bowl selections and years as primary starter. Additionally, sack rate isn't used to rank individual players, just the aggregate from each round and at each spot in the draft order. Other DE contributions to the pass rush such as QB hits or hurries (or even soaking up a double team) are very difficult to quantify. However, sacks can be used as a proxy measure for the overall contribution to the pass rush. In other words, a player with a lot of sacks very probably also has a lot of hits and hurries. Across so many seasons and with so many players, the effects of scheme and other considerations largely wash out.
Unlike QBs and RBs, players who did not make a roster or make many game appearances did not pose a problem when scoring their contribution. If they didn't have any sacks, they simply got a zero.
Pro Bowl Selection
The graphs below show the likelihood that a drafted DE will be selected to the Pro Bowl. The first graph is broken out by draft round. There is a steady decline from the top rounds to the last.
The second graph is broken out by draft order, that is, which DE was the player taken as in his draft class. This graph is much more erratic.
To me, the erratic nature of the graph suggests that compared to QBs and RBs it is easier for later round picks to shine. My guess is that later pick DEs get far more opportunities than later pick QBs or RBs, partly because a team needs two DEs on the field. Plus, they're more likely to be platooned.
Sacks Per Year
Despite its limitations as a comprehensive measure of DE performance, there is a clear relationship between draft round and order and sacks. First round picks significantly outperform subsequent picks in particular.
Notice the similarity in shape from the sacks graph to the Pro Bowl graph. This suggests that DEs are being picked for the Pro Bowl on the basis of their sacks. No surprise there.
Scout Accuracy
How certain can scouts and GMs be that they're selecting the better DE compared to the next guy down the draft board? The tables below list the likelihood that the higher player selected will turn out to have a better sack rate than the next DE taken in the same draft.
Years as Primary Starter
Each DE's total years as one of their team's primary starter is a good indication of a draft pick's worth. The tables below break out years as primary starter by round and draft order.
Conclusion
It appears that, on average, each draft provides two DEs in the first round who have a high probability of being an above-average player. But compared to QBs and RBs, DEs taken in later rounds may have a better opportunity to excel.
Drafting RBs
Can solid running backs really be found anywhere in the NFL draft? Years ago the conventional wisdom seemed to be that a team needed a superstar RB from the first round to win consistently. Now it seems that the conventional wisdom is that teams still need a star ball carrier, but one can be found deep in the draft. So which is it?
Data
The data consists of RB draft picks from the 1980 through 2000 drafts found at Pro-Football-Reference.com. Running back career performance was judged three ways. First, I averaged the likelihood a RB would be selected to one or more Pro Bowls by round and draft order. Second, I averaged career Yards Per Carry (YPC) by draft round and by draft order. (I also tried various ways of including receiving yards, but the variance in Yards Per Reception is very large and it distorted the data, particularly for players with relatively few receptions. Ultimately, simple YPC worked best and aligned closest with how most people see RBs. That is, Steve Sowell and Dave Meggit aren't ranked above Barry Sanders, Thurmond Thomas, or Emmit Smith.) And lastly, I averaged the number of years as the primary starter by round and by draft order.
Pro Bowl Selection
Although Pro Bowl selection is a flawed measure of career performance in many ways, it can indicate that a draft pick has "panned-out." If you sort the data by PB selection, it very quickly separates the generally productive RBs from the "three yards and a cloud of dust" guys. After looking at PBs for a number of positions now, it seems that 2 or more PB selections is a particularly good measure of career productivity, especially when judging top draft choices.
The graphs below illustrate the likelihood that a RB will be selected to one or more, two or more, and three or more PBs. The first graph is by draft round, and the second graph is by RB draft order (i.e. 1st RB taken, 2nd RB taken, etc.)
I wouldn't read too much into the spike at the 5th RB taken. It's likely just a statistical quirk, but it might be one reason why many experts believe that later round RBs are as good as early round picks.
[Edit: Some have asked why I brushed off the spike of Pro Bowls at the 5th RB taken as a quirk. If we analyze enough draft picks for various positions, as I'm in the process of doing, we're bound to see a significant bunching like this by chance once or twice. The graph is relatively continuous except in one place, where there is a depressed result at the 4th and 6th pick and the spike at the 5th. What is likely at work is that positive results in the 5th pick "bin" have randomly "stolen" positive results from the 4th and 6th bin. There were 20 RBs taken as the 5th RB in the data set, so it would only take 2 or 3 RBs who would otherwise have been the 4th or 6th pick to be bunched into the 5th pick to give us this result. Unless we had a reason to believe there is some special quality about the 5th RB taken before seeing the results, we should not interpret the data to say there is something magical about being the 5th RB taken.]
Yards Per Carry
There is probably no simpler and truer measure of running back performance than yards per carry. Of course, YPC does not belong to the RB alone. For any one RB's season, offensive line ability has a tremendous influence on his stats. But over 490 careers and over 24 years of data, the abilities of offensive lines will average itself out to a great degree, leaving career YPC a reliable estimate of true RB performance when grouped by round or draft order.
As with QBs, the biggest question is how to score draft picks with no carries or very few carries. RBs with fewer than 200 career carries tended to have extreme YPC stats. I assigned them the YPC of the 5th percentile qualifying RB, which was 3.58.
The two graphs below break out career YPC by draft round and by draft order.
The first round RBs, particularly the first couple taken, tend to significantly outperform later picks. By the 3rd or 4th round and the 7th RB taken, teams are likely getting sub-replacement level special teams fodder.
Also notice the nearly 1:1 relationship between career YPC and PB selections, including the spike at the 5th RB taken. This suggests that PB selection is merit-based and is a reasonable proxy for grading career performance.
Scouting Accuracy
How often do the scouts get it right? In other words, how often does the higher pick turn out to be better than the next pick? The two table below lists the likelihood that the higher pick will have a better career YPC than the next RB taken in the same draft. We shouldn't expect the scouts to be perfect, but this table tells us how difficult it is to predict the better player.
Years as Primary Starter
Conclusion
Top picks solidly outperform subsequent picks. The top two RBs taken tend to almost be in a class to themselves, then there is a steady decline in expected performance until the 8th RB taken, at which point there is very little to be expected from a pick.
So do teams need a superstar #1 pick RB to win, or can they find a premier runner deep in the draft? Which conventional wisdom was right? My theory is it's neither.
I think most people still grade RBs in terms of total yards, whether it's for a single game or for a season. Even though it should be well known now that winning leads to running, rather than vice versa, commentators and analysts continue to count 100 yard games or 1000 yard seasons as measures of RB effectiveness.
But even below-average RBs on winning teams with good passing games and good defenses will tend to accumulate large chunks of total yards due to frequent carries. Even a RB who was a 5th round pick on a great passing team will appear much better than he truly is. I believe that might explain the perception that solid RBs can be found anywhere in the draft.
The better RBs really do come from the top picks. It's just that they're not that important, or at least they're not as important as they were in the 1970s before the NFL became a passing league. Plus, our understanding of which RBs are truly the very good ones is distorted by analysts who insist on total yards as the best measure of RB performance.