What Policymakers Should Ask Modelers

On April 8, New York Governor Andrew Cuomo announced that his state was "flattening the curve" of the COVID-19 pandemic. But only two weeks earlier, various models had projected that peak hospitalizations in New York could be several times higher than they in fact turned out to be. Juxtaposing the actual number of COVID-19 hospitalizations with those projections, Cuomo wondered, "How do you come up with an actual curve that is so much different than what those experts predicted?"

Cuomo's question encapsulates the challenge that decision-makers face when dealing with predictive models. When the stakes are high, and model-based projections are a primary guide, how should policymakers proceed?

It's a relevant question not only during a pandemic. The 2008 financial crisis highlighted the power (and risks) of economic and financial models, and reliance on such tools will only continue to grow in an age of big data and big computing power. As scientists who routinely build models for policy analysis, we propose four questions that decision-makers should ask when using such models' results.

First, why was the model created?

Every model simplifies reality, and its creator's decisions regarding what to simplify (and how), what to include, and what to leave out are based primarily on the questions the model was built to explore. A model's specific purpose guides the choice of mathematical equations and methods used....