Nate Silver’s FiveThirtyEight is one of my favorite sites. Shortly after they launched, he wrote a really interesting article on March Madness brackets and the difference between accuracy and skill in prediction.
In 1884, a scientist named John Park Finley set the standard for being accurate but not skillful in his predictions. Over three months, Finley predicted whether the atmospheric conditions in the U.S. were favorable or unfavorable for tornadoes over the next eight hours, and then compared whether his prediction was accurate. By the end, Finley had made 2,806 predictions and 2,708 of them proved accurate, for a success rate of 96.5 percent. Not bad. But two months later, another scientist pointed out that if Finley had just said that there wouldn’t have been a tornado every eight hours, he would have been right 98.1 percent of the time. In forecasting, accuracy isn’t enough. Being a good forecaster means anticipating the future better than if you had just relied on a naive prediction.
This same idea has an impact on how we approach daily fantasy sports. Namely, we need to be less concerned with our rates of accuracy and more concerned about how our accuracy measures up to what should be expected, or what others can realistically achieve. A 25 percent hit rate on a min-priced player might be awesome; for a top-flight quarterback, not so much.
I’m going to give two examples showing how humans are really poor at understanding stats and, in most cases, can be beaten by very simple rules-of-thumb when making predictions.
Let’s go back to the NCAA tournament. When I was in high school, a lot of my friends would get on me because I picked almost all of the favorites in our March Madness pool every year. “You’re an idiot, a 12-seed always beats a 5-seed.”
There were probably 50-plus students in this pool every year and I won two of my four years in high school, but my reputation was that of someone afraid to take risks. In actuality, I’d say I’m pretty strongly risk-seeking, but only when that risk is accompanied by upside (and the upside outweighs the risk relative to the probability of good/bad things occurring). But when the risk comes with no upside—as in aimlessly and arbitrarily choosing a 12-seed to beat a 5-seed—yeah, I’m not going to take a needless risk.
The problem comes in falsely believing that greater accuracy is always achieved through greater skill. Knowledge equals power, but more knowledge doesn’t always equate to more power. Put my March Madness brackets next to 100 ESPN experts, and I’ll probably beat the majority of them with no knowledge of NCAA basketball whatsoever, just picking mainly favorites and throwing in a little game theory.
The question people should be asking themselves whenever they’re dealing with predictions isn’t just “what is the probability of X occurring?” but also “what’s my probability of correctly predicting X?” When it comes to a 12-seed beating a 5-seed, yes, that will probably happen in a given year, but only because there are four such games. The 5-seed will always be favored to win the game, and the chances of you predicting a 5-seed to lose, then have it happen as you predicted, are smaller than the chances of every 5-seed winning.
Let’s look at it another way. Instead of asking if a 5-seed will lose, ask yourself which possible combination of wins/losses in the four games is the most probable. Here’s how the breakdown of outcomes for the No. 5 seeds can look in a given year: WWWW, WWWL, WWLW, WLWW, LWWW, WWLL, WLWL, LWWL, WLLW, LWLW, LWLW, LLWW, WLLL, LWLL, LLWL, LLLW, LLLL.
Now let’s assume that each No. 5 seed has an 80 percent chance to win. What are the chances that all four win? Just under 41 percent. That’s less than a coin flip, meaning odds are one will lose.
But it’s still the most likely individual outcome. Even if we assume that only one 12-seed can win—so it’s either none or one—the remaining 59 percent would be split among four scenarios: WWWL, WWLW, WLWW, and LWWW. So if the probability of each 5-seed winning is 80 percent, chances are one will lose. But the odds of none losing (41 percent) are significantly higher than the probability of one losing and you picking that loser (14.8 percent). By arbitrarily picking low seeds to beat high seeds in the NCAA tournament, you’re drastically cutting into your odds of winning.
Another example: “at least six new teams make the NFL playoffs every year.” That idea leads people to remove playoff-caliber teams in favor of shitty ones just to make sure there’s enough turnover in their playoff predictions. But they’re forgetting they not only need to predict how many of the same teams will make the postseason, but also which teams will be replaced, and by whom. That prediction becomes way, way more difficult.
If you’re projecting playoff teams, you shouldn’t just blindly copy what happened the previous year because the best teams don’t always make it. But you shouldn’t remove a certain number of teams, either; just pick the six best teams from each conference, because that’s the individual path most likely to occur.
Finding the Exception
I read a really well-written piece by Shawn Siegele on a similar idea:
Many people subscribe to the theory that you can’t grade a draft for at least three years. This is partially due to the bizarre yet somewhat prevalent theory that it’s a scout’s job to find the exceptions to the rules instead of finding players who fit the established models of prospects who successfully transition to the NFL. There are two key reasons why it doesn’t work to wait three years to see if longshots like Tavon Austin or Marquise Goodwin pay off. First, if you wait that long to self-evaluate, you will make many more mistakes in the interim. Second, it encourages the lottery ticket idea. A lottery ticket purchaser is not vindicated in his strategy simply because a given ticket pays off.
I’ve always had a problem with grading drafts years after they occur. The NFL Draft is governed by probabilities, in which case we can know the quality of the decision immediately. A poker player doesn’t assume he made a poor choice because he suffers a bad beat on the river. The decision is either good or bad when it’s made, and you live with the results. The same goes for the draft.
Within that excerpt is an interesting phrase: “it’s a scout’s job to find the exceptions to the rules.” That really says a lot about the state of NFL scouting and decision-making. As it stands right now, NFL teams are trying to figure out when they should take the 12-seed to beat the 5-seed. The answer is basically never, but they continue to do it again and again.
As a daily fantasy player, your job isn’t to find exceptions to rules. It’s to identify the rules, use them as a foundation to build a team, and deviate from that path only when there’s a wealth of evidence that you should.
Are you always going to be right? No, just in the same way that we won’t always see every 5-seed beat every 12-seed. But as I detailed to start this article, great forecasting isn’t only about being accurate, but being more accurate than what we can expect with a simple rule-of-thumb.
In many cases, daily fantasy players outsmart themselves. They try to identify situations in which a player or team is going to deviate from what they know is the most likely outcome. Sometimes they’ll even be right, just as going all-in with 2-7 off-suit will sometimes result in winning a pot, but that doesn’t make the decision the right one.
