tl;dr Market incentives and the wisdom of crowds better predict the Oscars.
Prediction, understanding causal inputs to forecast outcomes, drives big business. From Google predicting ad clicks, to Nate Silver’s success predicting Barack Obama’s dramatic victory over pundits closer view, to Climate Corp predicting the weather. With enough information on the right inputs, data scientists can forecast real world events.
My first foray into the world of prediction was my undergraduate thesis, Forecasting Motion Picture Box-Office Returns and Analysis of the Hollywood Stock Exchange. After examining the academic literature, I built a more accurate model to predict film box-office returns. I then went to show that even the best academic model paled in comparison to wisdom of crowds. Prediction markets like the Hollywood Stock Exchange enable people to trade in futures markets tied to real-world events. The process motivates movie buffs to assess the adoption risk and wager to win “Hollywood Dollars” representing kudos but not real money. The price of a film’s shares reflects the crowds’ best assessment of their ability to gross. While it’s hard to hold to an efficient market hypothesis, markets have been repeatedly demonstrated to be better than any other mechanism for incorporating information. Incentive is a powerful thing.
In a new study from Microsoft Research, an economist, David Rothschild, compared the performance of a variety methods in predicting last year’s Oscar winners. Like my thesis, David found predictions based on a market, in this case a real money prediction market Betfair, outperformed pundits and best in class statistical models.
This result is good as even Nate Silver has said predicting the Oscars is harder than calling a presidential election. As statisticians like to say, your model can only be as good as your data, garbage in garage out. No poll or sampling of the academy is taken. The best in class standard is to use box office receipts, critic ratings, and release dates (producers are thought to hold Oscar contenders until late in the year, to be fresh for voting). Silver’s own attempts at prediction yielded only four correct of the top six categories.
The Results
The HSX shot 100% and correctly forecast all six major categories. As an example, just before the Oscars, 12 Years was trading at ~$14 in the market for best picture, with Gravity a distant second, priced at less than $6.
- Best Picture: 12 Years a Slave - 55% to win
- Best Director: Alfonso Cuarón, Gravity — 70%
- Leading Actor: Matthew McConaughey, Dallas Buyers Club — 70%
- Leading Actress: Cate Blanchett, Blue Jasmine — 69%
- Supporting Actor: Jared Leto, Dallas Buyers Club — 65%
- Supporting Actress: Lupita Nyong’o, 12 years a Slave — 60%