news
International Relations/Political Science
30 April 2020

Evidence-Driven Computational Modelling

Mirko Reul, PhD candidate in Political Science at the Graduate Institute, discusses in this interview a chapter he co-authored in the recently published SAGE Handbook of Research Methods in Political Science and International Relations. The chapter provides guidelines to researchers on how to develop evidence-driven models (EDM).

How does EDM address some of agent-based computational modelling’s (ABM) shortcomings?

The evidence-driven modelling (EDM) framework rests on three methodological pillars: agent-based modelling (ABM), contextualisation using geographical information systems (GIS) and empirical validation. ABM lend themselves well to the analysis of complex social phenomena; however, a common shortcoming concerns the absence of relevant theoretical and empirical anchors, resulting in an unrealistic or arbitrary model specification. 

The EDM approach places these anchors at the core of an iterative model development process, which comprises many of the standard elements of research design in political science. EDM are based on existing, often context-specific theoretical and empirical knowledge, seeded and validated with empirical data, including primary data collection from constituent “agents” whose behaviour the model aims to represent, and refined as the modeller’s understanding of the situation develops.

A validated EDM can be used to conduct scenario-based counterfactual analysis, to offer an indication of what the world could look like if empirically observed trends were to change. That said, a model initially developed for one context need not be applicable to other contexts, highlighting the classic trade-off between external and internal validity. We nonetheless suggest that the EDM approach can incorporate contextual differences in a systematic and transparent manner, while retaining internal validity for specific cases.

How did the case of Karamoja, Uganda, help develop and validate your model?

In the chapter, we use the MERIAM project to illustrate the EDM approach. We were tasked with developing a model to identify how, in response to conflict and climate shocks, household-level decisions affect acute malnutrition in various Sub-Saharan regions – effectively unpacking the “black box” of household behaviour.  Karamoja in Uganda was one of the regions, another was West Pokot in Kenya.
 
Our model is seeded with empirical data on the spatial distribution and characteristics of households in these regions. We are then able to compare simulated outcomes from the model to observed rates of malnutrition, analogous to testing in- and out-of-sample predictive power for statistical models. 

It is important to note that a model specification that “fits” empirical outcomes need not necessarily capture the dynamics that occur on the ground. This is why we utilise case-specific knowledge to tailor the model to particularities of each context. For the MERIAM project, we developed preliminary versions of the model with reference to prior studies on food security and nutrition in Karamoja, and further refined the model based on interviews and focus group discussions in the region. We then repeated case-specific theoretical grounding, data construction, and fieldwork for West Pokot, assessing historical and cultural differences relative to Karamoja, and implementing changes to the model to capture them. 

What is the importance of “ground-truthing” the model and how did it take place?

Primary research is important for any theory-building exercise and plays a central role in our EDM approach. As noted, a distinct advantage of EDM is the close integration between theoretical specification and empirical validation. “Ground-truthing” ensures that we “get the story right,” so to speak. Given that multiple mechanisms may connect any set of initial conditions to outcomes, the exercise ensures that we have identified, to the extent possible, the set of mechanisms most relevant to the problem we seek to model. For the MERIAM project, we conducted interviews and focus group discussions with the people whose behaviour we were modelling. We also interviewed NGO and government experts in the domain of nutrition and food security, in an effort to make their “mental models” explicit. Interviewing both experts and community members was particularly useful, given that community interviews helped us interpret what we learned from experts, and vice-versa.

Are there data challenges that are more specific to the EDM approach and how did you overcome them?

EDM can accommodate data at varying levels of granularity: we can connect model components with data at different levels of resolution based on availability, rather than using the lowest common level of resolution. And there are ways to mitigate spatial imprecision using imputation. But EDM can be relatively demanding on data quality, particularly if the goal is to validate a model that simulates behaviour at the level of households or individuals. For the Karamoja case, we spent a considerable amount of time searching for better data on household behaviour (e.g. on food acquisition strategies and decision-making), perhaps more than we would have for a statistical model at the district level. Ultimately, data quality shapes our ability to seed and validate an EDM the same way as for any model.

What are the key differences between the two “what-if” scenarios in the counterfactual analysis?

We distinguish between: (1) parameters that are comprehensively explored, to identify the combination that maximises agreement between simulated and empirically observed outcomes; and (2) fixed model inputs based on empirical data. Once a model is calibrated and validated, it is possible to conduct counterfactual analysis either by varying a single parameter while keeping the others at optimal values, or by treating model inputs as variable parameters.

Varying a parameter helps us understand its influence, which can be interesting particularly if a small change in value results in a significant change in the outcome of interest. For example, how might levels of acute malnutrition change if households were better equipped to adapt to shocks? Perhaps more interesting for practitioners is the second type of counterfactual with empirical inputs. How much would the distribution of improved seeds prior to a drought have decreased malnutrition, compared to the distribution of humanitarian aid once the drought destroyed less resilient crops? EDM are well suited for this type of scenario-based analysis, with the caveat that underlying assumptions are transparently communicated. Forecasting, for instance, is not feasible if model parameters or empirical inputs change in unexpected ways.

What do you see as areas of further research in EDM and how they can benefit from the experience documented in this chapter?

The chapter mainly provides guidelines to researchers to develop an EDM, walking them through the process from start to finish. EDM are especially useful in issue areas where there is an abundance of theoretical knowledge, outcomes are driven by complex interactions between numerous factors, and it is possible to leverage empirical data, either to seed, validate the model, or both. Numerous organisations seek to model human behaviour with a view towards improving outcomes – whether these pertain to early warning, to more accurately identify communities at risk, or to resilience against environmental shocks. The EDM approach, being data-driven, has a number of advantages over more abstract models. Ideally, EDM provide evidence-driven results that decision-makers can use to evaluate alternative policy options in a systematic and transparent manner.

*  *  *

Complete citation of the chapter:
Bhavnani, Ravinder, Karsten Donnay, and Mirko Reul. “Evidence-Driven Computational Modeling.” In The SAGE Handbook of Research Methods in Political Science and International Relations, 2-vol. set edited by Luigi Curini and Robert Franzese. London: SAGE, 2020.

Interview by Guilherme Suedekum, PhD Candidate in International Economics.
Banner image: excerpt from an image by Liu zishan/Shutterstock.com.