How Well did the Presidential Election App Perform?

[DrDan]

One and a half months have passed since the 2012 US Presidential election was held, and President Barack Obama will be sworn in for his second term this Monday. Given this context, it is an appropriate moment to examine how well Alteryx's Presidential Election App performed. In this blog post I compare the Election App's predictions to actual election results at the county level. While the app was not a crystal ball, it generally did a very good job of predicting election results at the county level. Moreover, an examination of where and why the model went astray provides useful insights into how to improve models like this one in the future.

A Re-Cap of the Presidential Election App

The purpose of this app was to predict voter preferences in the presidential election down to small geographic areas (e.g., zip codes and drive-time areas defined around a particular address), areas much smaller than can effectively addressed using traditional polling data based approaches alone. The practical application of this type of model for a political campaign is to determine the small, closely contested areas where that campaing should focus its resources.

Developing the predictions that underlie the app involved multiple steps. First, a candidate preference predictive model was developed using data from three waves of the USA Today / Gallup Poll (provided to us by the Roper Center at the University of Connecticut). The model related polled registered voter's candidate preference to their demographic characteristics and party identification.

The important variables in the predictive model, in their order of importance, are:

1. Party identification
2. Race
3. Household income
4. Age

The next step of the process involved manipulating Census block group level demographic data (2012 current year projections from Experian's CAPE data files that are packaged with Alteryx's Designer Desktop, Professional Edition) and voter party identification information (provided by a political consulting firm that wishes to remain anonymous) that was matched to the definitions of the measures used in the USA Today / Gallup polling data. Finally, the local block group level data is combined with the polling data based predictive model to generate local area voter preference predictions. A more detailed explanation of the process used to develop the app's predictions are available in an earlier post and the technical appendix that accompanies that post.

While we made predictions down to the block group level, actual election returns are not reported down to that level. The lowest level that election returns data is readily available, since it is reported in the media, is at the county level for every state except Alaska (Alaska does not have an official county level of government, so returns are only readily available at the state level). Our county level predictions are obtained by aggregating the block group level predictions up to the county level. A map of these results are shown in the first figure on the right-hand side of the page.

 

 

 

A Comparison with the Actual County Level Election Results

The actual election returns are from the Associated Press's reported county level election results as published by Politico. The actual county level results for the election are provided in the map on the left-hand side of the page. A comparison of the two maps reveals a very high degree of overlap, albeit the maps are not identical images. However, given that this represents 3,111 different predictions, for some counties with small populations, based on a limited database of national polling data that was projected down to local areas that required a fairly high level of data "massaging" to "line-up" with the measures used in the polling data, one would need to be very suspicious if the two images were identical.

While comparing the maps is very informative, numeric measures are the preferred way to judge the predictive accuracy of a forecast. One measure that is both commonly used, and has desirable statistical properties, is the mean absolute percentage error, which is commonly referred to by its acronym (MAPE). The MAPE for the county level percentage error in the predicted share of the vote going to Obama is 17.7%, while the comparable figure for Romney is 15.5%. It would be ideal if these figures were a bit lower, given the nature of the application, they are still very good.

More importantly, it does indicate that voter preferences in small geographic areas can be reasonably predicted using a combination of national polling data and local area information. While it does not have the accuracy of traditional polling data, it can be implemented for geographic areas that are simply too small or cost prohibitive for conducting traditional polling research.

The last thing we wanted to understand is whether there were any spatial patterns in our prediction errors. In this case we are trying to make three different predictions, the share of vote going to Obama, the share going to Romney, and the share going to any third party candidate (a very small percentage in this election). Given this situation, and the fact that the vote shares sum to 100 in every county, the errors for one of the major party candidates is a mirror image of the errors for the other. For example, if the error in the predicted vote share for Obama in a county is negative 5 percentage points (e.g., the app predicted that Obama would receive 50% of the vote in a county, but the actual percentage for Obama was 45%) then the error for Romney was likely to be very close to plus 5 percentage points. As a result, to do this analysis we only looked at the difference in the vote share for Obama on a county by county basis. The results of this analysis are shown in the map on the lower right of this page. In the map, blue colors indicate a higher share of votes for Obama than was predicted, while red colors indicate a lower share of the vote for Obama than was predicted.

An examination of this map indicates that the model errors are not randomly distributed across the country. In particular, the Upper Midwest, Upstate New York, and the coastal counties of the West Coast voted more heavily for President Obama than the model predicted. In contrast, the Lower Midwest, Appalachia, the Florida Pan Handle and Georgia, and a stretch that runs from eastern New Mexico and through Texas and Louisana had actual vote shares for Obama that were lower than predicted. The existance of spatial patterns in the errors is known as spatial autocorrelation, and is common in this type of analysis. In the FiveThirtyEight blog model, Nate Silver controls for possible spatial autocorrelation by using both a state level measure of the the historic difference between polling and actual results for that state and measures of local economic conditions. We did attempt to address some of these issues through state level norming, but this analysis suggests that more could be done in this regard. Moreover, the errors found in this model provide a means of addressing spatial autocorrelation issues in the future.

 

Summary

The goal of the Presidential Election App was to provide users with predictions of the relative preferences of voters at lower geographical levels than is possible using traditional polling data. Based on our comparison of the actual and predicted results at the county level for the 2012 Presidential election, we are confident that the app delivered on its intended objective. Having said this, by doing the comparison of the predicted and actual results, we have developed a number of ideas that can be used to further improve the predictive efficacy of the app. The good news, we have nearly four years to refine those ideas for our next attempt.