Out-of-Sample Forecasting Performance

A general way to assess predictive performance is to use receiver operating characteristic (ROC) analysis. A ROC curve is a plot of the true-positive rate (‘sensitivity’) against the false-positive rate (1–true-negative rate, or ‘1–specificity’). The area under the ROC curve (AUC) represents the relative forecasting success for true-positives reduced by the proportion of false-positives generated for any given forecasting threshold. An AUC of .5 indicates prediction no better than chance, while an AUC of 1.0 indicates perfect prediction. The AUC statistic for the out-of-sample period 1988-2003 indicates that our model forecasting genocide / politicide onset one year into the future captures around 89% of the area under the ROC curve. The model parameters are trained on the period 1974-1987.

Out-of-Sample Forecasting Performance Graph