Statistical and Financial Evaluation of Subjective Probability Forecasts: Empirical Applications in Betting Markets
In this thesis we consider how a decision maker can optimally utilise the predictions of analysts in a subjective probability forecasting competition. Proper scoring rules (Savage, 1971) are used as part of the elicitation process, but we proclaim the benefits of using different types (statistical, categorical, and economic) of scoring rule as part of the evaluation process. Forecasts of competition participants are used, in isolation and in aggregation, for betting against a bookmaker. Evaluating probability forecasts by their realised return in betting markets improves upon the basic economic evaluation methods discussed in literature (e.g. Granger and Pesaran, 2000a, b), because the betto's degree of belief can be more accurately reflected by the size of their stake.
The Kelly (1956) criterion is used to find the optimal wager for a bettor, based on the analyst's forecast and the market odds. We extend the Kelly criterion to allow for the simple calculation of optimal wagers when games occur simultaneously. The Kelly bettor can realise the same returns by betting on the same games simultaneously or sequentially by following our strategy. Using this strategy, we find that individual forecasters are unable to provide predictions for a (full or fractional) Kelly bettor to earn positive profits, but using the pooled forecasts of analysts selected using the different types of scoring rule is a potentially profitable strategy.
The rank-order performance of forecasters in competition is highly dependent on the scoring rule used for evaluation, which is contrary to the findings of Winkler (1971). Each of the different rules captures a different desirable attribute in a forecaster. Unless a competition organiser has a strong preference for one rule over another, the low rank-order correlations across scoring rules suggest that the task of finding the 'best' forecaster (and therefore the competition) is futile. Overall, the results show that it is possible to use probability scoring rules to find experts that are able to provide profitable predictions against a market, provided the forecasts are combined and used with an optimal betting strategy.
David Johnstone and Alex Frino