“The story begins with why I decided to leave my job at Google, where I was a product manager, and decide to join the Obama campagin, where I was the Director of Analytics.”
— Dan Siroker, Co-Founder & CEO of Optimizely
I go through periods in my life when I work out, get almost enough sleep, eat healthy-ish, drink basically nothing but water, read a lot of thought-provoking blogs, essays, and books, and listen to as many ‘smart’ (non-sports) podcasts as I can. My wife has a particular name for these periods: “The NFL offseason.”
Starting in February I get into a good routine — and then the NFL season comes along, and from August to January I don’t work out, I get no sleep, I eat like a pig, I drink stuff that’s definitely not good for me, I read almost nothing, and I listen only to NFL podcasts.
It’s probably not a surprise to anybody that I enjoy the NFL offseason more than the pre-, regular, and postseason. On the one hand, the NFL season is my busiest professional time of the year, and even though I like working hard and being busy I must admit that grinding nonstop for six months is a tall task.
Also, I’m about 97 percent sure that’s the first time I’ve ever used the phrase “tall task” in writing.
On the other hand, it’s possible that perhaps I don’t enjoy the NFL season as much as I otherwise would if I proactively made time to exercise, sleep, etc.
Either way, I’ve gone through this cycle — and been human — long enough to know that my life is better when I’m more purposeful with the way I live. I’m more productive, and I’m happier. As much as someone can informally A/B test oneself, I’ve done it over at least the last five years. (I actually have notebooks and spreadsheets documenting my NFL/non-NFL A/B results. I have some facts and data.)
Did I need to go through this back-and-forth process of healthiness and slothfulness to know that I’d enjoy life more when I’m healthy? Probably not, but I did — and that experimentation has given me evidence.
Why am I telling you this?
- I find myself fascinating. Why wouldn’t you?
- I think more daily fantasy sports players need to A/B test themselves.
I’m not saying that you should A/B test your life, but you probably should A/B test your lineups, strategies, contest selection, etc.
This piece is about how to become a better DFS player with A/B testing.
NPR’s Planet Money
In the summer of 2015 one of the podcasts I enjoyed listening to (in my healthy phase) was Planet Money by NPR . . . and then the NFL season started, I stopped listening to the podcast, and episodes of the show piled up in my queue. In December of that year I finally decided to delete from my podcast app all of the shows I hadn’t listened to in months — but before I deleted Planet Money I listened to the most recent episode. Randomly, it was about A/B testing:
The episodes starts with an anecdote about Dan Siroker, who worked at Google in 2007 and heard Barack Obama speak there as a candidate. Siroker was impressed with Obama’s vision for big data and technology in the government, so he left Google and started working for the Obama campaign, first as a volunteer and eventually as the Director of Analytics.
With the campaign, Siroker noticed that the “Sign Up” button on the splash page of the website didn’t get a lot of clicks, so he started experimenting, creating two versions of the page — one with the original “Sign Up” button (Version A), the other with a new “Learn More” button in its place (Version B). Some people who went to the site saw Version A, others saw Version B, and then Siroker collected data to see which version performed better. He found that Version B generated more clicks, and so the campaign pivoted from “Sign Up” to “Learn More.”
What’s A/B Testing?
What Siroker did — that’s A/B testing: Create two versions of something, see how those versions perform in the market, monitor that performance, and then make informed and impartial future decisions based on the data. Ideally, it removes bias from the decision-making process and makes into a science that which might seem like an art. A/B testing can be done on almost anything and across a variety of factors as long as the means to track performance exist.
By the way, some people might think of “means” as a singular noun, but to me it’s plural. That position might seem weird given that I treat “data” as a singular noun instead of a plural, but that’s how I roll. I A/B tested the singularity and plurality of “means,” and people prefer it as a plural.
A/B testing of course can be much more complicated than a simple one-to-one comparison — it can involve multivariate tests in which a variety of factors are all tested against each other at the same time — but the basic principle of A/B testing is that the relative performance of competing options can be evaluated and leveraged to improve production in the future.
Can A/B Testing Really Improve Performance?
After working on the Obama campaign, Siroker started Optimizely, an experimentation (A/B testing) platform that many companies now use to help them optimize their user experience. Here’s an Optimizely video in which Siroker talks about the best practices and lessons he and his team have learned from more than 30,000 A/B and multivariate tests:
In the video, Siroker notes that all of the controlled tests he did on the Obama splash page improved the click rate by 40.6 percent — which meant that ultimately the campaign was able to get 2.88 million more email subscriptions, 288,000 more volunteers, and $57 million more in contributions.
Yeah, A/B tests can really improve performance.
A/B Testing and DFS
By improving our methods of self-evaluation — by making them more systematic — we can better ourselves as DFS players.
Here are five ways to improve through an A/B testing perspective.
Subscribe to FantasyLabs
Seriously, subscribe to Labs — we have a low-cost trial available — and use our suite of Tools to build lineups. Do everything else as you normally would in terms of bankroll management, contest selection, etc., but do research with our Trends tool and create lineups in our Player Models. And then compare your Labs performance with your pre-Labs performance.
Clearly, not every new Labs subscriber is going to become a noticeably better player within a matter of just a few days, but we’re confident that over an extended period of time in the aggregate subscribing to Labs has positive expected value. We want you to have that same confidence — based on your own self-research.
Build Multiple Lineups for Cash Games
It’s not uncommon for people to have questions about how to build cash game lineups:
- Is a balanced approach better than a stars-and-scrubs strategy?
- Which position is optimal in the flex?
- What’s the proper balance of consistency and upside?
In order to start to answer these questions, experiment with multiple lineups for cash games. I’m not saying that you need need to build 10 lineups or that you even need to play more than one lineup in an actual contest.
I am saying, though, that for maybe 10 consecutive slates you could build two lineups for cash: One balanced, the other stars-and-scrubs (for example). Even if you don’t play both of them in actual contests, you can track the performance of these different strategies over time.
By building multiple lineups and then tracking their performance via A/B testing, you eventually could make your cash game process more optimal.
Shotgun Lineups for Guaranteed Prize Pools
Some people really like the art of building GPP lineups by hand (or excel). There’s nothing wrong with that — but it’s possible that it might be more +EV to shotgun DFS lineups.
Do an A/B test to find out what actually works best for you. Make 10 lineups the old-fashioned way, and record how long it takes you to make them. Then go into the Models, adjust all of the settings as you see fit, and create 10 lineups with our Lineup Builder. Again, record the time it takes you to make them. Do this for a series of slates, and then compare the performance of the two methods for GPP lineup construction.
Specifically, compare the two methods on the basis of overlap, time, production, and enjoyment. Even if you prefer to craft GPP lineups, you might find that shotgunning lineups with our tools is faster, more productive, and more enjoyable. Additionally, once you learn how the settings impact the lineups, you might find substantial overlap between the lineups you craft and those you shotgun. At a minimum, you can use the Lineup Builder to get ideas and create lineups to adjust further. Either way, it’s possible that you can incorporate the Lineup Builder into your GPP process in a beneficial way. If you have doubts, do an A/B test.
Enter Multiple Kinds of Contests
I’ve written recently about contest selection. Some people prefer cash games; other people prefer GPPs. Within cash games, some people prefer head-to-head contests; others, 50/50s and double- and triple-ups. Within GPPs, some people prefer multi-entry, large-field, or high-stakes tournaments; others, single-entry, small-field, low-stakes tournaments. All of that is fine. Some players just prefer particular contests to others.
At the same time, it doesn’t entirely matter which contests a player prefers. What (probably) matters most is which contests are best for that player’s productivity. Focused A/B testing can help players determine the ideal contests for them.
Play Across Multiple Sites
Some people play exclusively on FanDuel, while others play exclusively on DraftKings. I’m fairly convinced there’s an edge to platform diversification, which enables you to leverage Bargain Ratings and our other metrics in a maximal way across all slates. If you play on only one platform, you’ll miss all the value another platform affords — and in any given slate it’s possible that most of the value resides on that other platform. In effect, if you play on only one platform you could lose potential production.
Of course, you can A/B test this theory. If you play on only one platform, continue to make your single-platform lineups as usual — but also make partner lineups on DraftKings and FanDuel with the idea in mind that you’re investing in assets in a coordinated way based on where they provide the most value across platforms.
While it’s possible that some DFS players are better on one platform than another, they’ll never know unless they go through the effort of building synchronized lineups across sites and then comparing their multi- and single-site production via the systematized process of A/B testing.
A/Bout Finished
While A/B testing might seem like a -EV gesture in the short term — “What if a test yields negative results? — there are numerous factors in its favor:
- Many of the A/B tests don’t even require risk. All you need to do is make extra lineups and track how they would’ve done in particular contests if you’d in fact played those lineups. In these cases, A/B testing offers only upside.
- When it comes to science, negative results aren’t necessarily bad.
- Sometimes that which is -EV now is +EV in the long term. In general, it’s ultimately beneficial to analyze and invest in one’s process.
I should probably have one more clever sentence here to finish the piece.
The Labyrinthian: 2017.30, 125
This is the 125th installment of The Labyrinthian, a series dedicated to exploring random fields of knowledge in order to give you unordinary theoretical, philosophical, strategic, and/or often rambling guidance on daily fantasy sports. Consult the introductory piece to the series for further explanation. Previous installments of The Labyrinthian can be accessed via my author page.
