Conceptualizing research outcomes and impacts
The Consortia Advancing Standards in Research Administration Information (CASRAI) is coordinating a working group to develop terminology and information tools that will enable domain-sensitive and standards-based implementation of impact discussions and assessment in the social sciences and humanities (SSH). The working group will also engage the SSH community in further discussions on realistic and appropriate approaches to impact assessment. Brian Belcher is a member of the working group. He recently submitted a brief on how we conceptualize research outcomes and impacts in the Sustainability Research Effectiveness Program. The brief follows:
Typical measures of “research impact” focus on indicators of influence on the research agenda (e.g., publications, citations, impact factors) via research outputs and communications. We are more interested in influence beyond the academic, particularly in social change and the ways that research contributes to change processes. The Higher Education Funding Council for England (HEFCE) definition of “impact” captures this as an “Effect on, change or benefit to the economy, society, culture, public policy or services, health, the environment or quality of life, beyond academia”. However, this is still quite a broad definition. The effect, change, or benefit could be small or large. The change at the scale of society, public policy, etc. could be realized early or much later. Importantly from the perspective of evaluation, the research in question could be sufficient to cause the change or, much more likely, it could serve as one of many contributing factors within a complex system.
Our program uses impact pathways and theories of change as fundamental conceptual tools. An impact pathway models the main steps and the main actors in a process of change (who does what differently). A theory of change adds theoretical understanding to the model (who does what differently and why). An impact pathway can be represented in a typical flow diagram as a sequence of boxes that represent actors, actions, and results. A theory of change explains the arrows between the boxes.
In thinking about research design, monitoring, evaluation, and learning, we also use the concept of spheres of control, influence, and interest. A research project has a relatively high level of control over research definition, design, and implementation through to the production of outputs (though it is important to note that in more engaged transdisciplinary projects, some control is shared). This can be evaluated with standard tools of project evaluation. With more engaged research and deliberately adaptive management approaches, evaluation needs to consider whether and how a project has achieved its goals and not just whether it followed its original plan.
Beyond the project boundary, there are many other actors and processes at work. The project cannot control what happens but it can exert influence in many different ways. The Dutch SIAMPI project concept of “productive interactions” partially captures this idea. Moreover, as researchers and research managers get a better understanding of how research contributes to change and design projects/programs more deliberately to engage in change processes, it will be possible to improve research effectiveness. The degree of influence is likely to be highest closer to the project boundary, among partners and stakeholders and users of research products and services. This is the sphere of influence. It can be evaluated by looking for evidence that the research has contributed to changes in knowledge, attitudes, skills, and/or relationships (KASR) among key actors within the sphere of influence in the impact pathway.
If research is successful at stimulating or contributing to change within the sphere of influence, it is reasonable to expect further knock-on changes. That is, if key actors do something differently as a result or partially as a result of the research, that may in turn contribute to further changes that will help transform systems and ultimately lead to social, economic, and environmental benefits. These kinds of changes are typically well outside the sphere of control and the sphere of influence of a research project or program. It is important to monitor change in the sphere of interest to know if things improving or not (and to respond accordingly), but it is difficult or impossible to attribute those changes to research because there are so many other factors involved.
The figure shows the same stages in a generic research impact pathway within the relevant sphere. It tries to illustrate the typical sequence of a research project vertically (top to bottom in the sphere of control), but also reflects the idea that each step can have influence. The research contributes to changes in KASR, leading to individual and organizational changes in the sphere of influence which in turn contribute to further changes, ideally leading to benefits in the sphere of interest.
Table 1 (below) shows stages of a generic research impact pathway, from problem identification through to the ultimate intended benefits (impacts), with an illustrative list of the kinds of contributions that can be made during a research process and the kinds of changes that might result. It is important to note that work done in a research context to define and conceptualize a problem, for example, may contribute to outcomes; impact pathways do not necessarily flow only through research results and analysis, dissemination, uptake etc., but also through engagement processes as well.
Regarding terminology, we have found it useful to define outcomes and impacts absolutely, by the nature of change, and not relatively (e.g. short term vs long term; immediate vs distal). Changes in KASR are outcomes. If changes in KASR lead to changes in behaviour (including sharing knowledge, for example) by an actor in the system, that may contribute to change other actors’ KASR and behaviour. These changes are also outcomes. Ultimately, it is hoped that the change process to which the research has contributed will result in tangible social, economic, and/or environmental benefits, which we call impacts.
Table 1. Illustrative list of research contributions by stage in impact pathway.
KASR = Knowledge, attitudes, skills, and/or relationships
MEL = Monitoring, evaluation, and learning