Conceptualizing research outcomes and impacts

Sep, 2018

In conceptualizing better research impact, the Consortia Advancing Standards in Research Administration Information (CASRAI) coordinated with experts to establish a working group. The working group aims to develop terminology and information tools to enable domain-sensitive and standards-based implementation of impact discussions and assessment in the social sciences and humanities (SSH). The working group will also engage the SSH community in further discussions on realistic and appropriate approaches to impact assessment. Dr. Brian Belcher joined as a member of the working group. He recently submitted a brief on conceptualizing research outcomes and impacts based on the Sustainability Research Effectiveness Program’s definitions. We summarize the brief below.

For more information on conceptualizing the elements of research impact, check out the following journal article.

Conceptualizing Research Impact

Typical measures of “research impact” focus on indicators of influence on the research agenda (e.g., publications, citations, impact factors). They focus primarily on indicators of research outputs and communications. We hold greater interest in influence beyond the academic, particularly in social change. More specifically, we aim to uncover the ways that research contributes to change processes. The Higher Education Funding Council for England (HEFCE) definition of “impact” captures this as an:

“Effect on, change or benefit to the economy, society, culture, public policy or services, health, the environment or quality of life, beyond academia”.

However, this is still quite a broad definition. The effect, change, or benefit could be small or large. The change at the scale of society, public policy, etc. could be realized early or much later. Importantly from the perspective of evaluation, the research in question could be sufficient to cause the change. Alternatively, and much more likely, it could serve as one of many contributing factors within a complex system.

Our program uses impact pathways and theories of change as fundamental conceptual tools. An impact pathway models the main steps and the main actors in a process of change (who does what differently). A theory of change adds theoretical understanding to the model (who does what differently and why). An impact pathway can be represented in a typical flow diagram as a sequence of boxes that represent actors, actions, and results. A theory of change explains the arrows between the boxes.

Key Concepts

Reflecting on research design, monitoring, evaluation, and learning, we also use the concept of spheres of control, influence, and interest. A research project holds relatively high level of control over research definition, design, and implementation through to production of outputs. However, we note the importance that some control is shared in more engaged transdisciplinary projects. This can be evaluated with standard tools of project evaluation. With more engaged research and deliberately adaptive management approaches, evaluation needs to consider whether and how a project achieved its goals and not just whether it followed its original plan.

Beyond the project boundary, many other actors and processes interact. The project cannot control what happens but it can exert influence in many different ways. The Dutch SIAMPI concept of “productive interactions” partially captures this idea. Moreover, as researchers and research managers get a better understanding of how research contributes to change, they can design projects/programs more deliberately. Design that intentionally engages in change processes increases potential to improve research effectiveness. Closer to the project boundary, projects can exert a higher degree of influence among partners, stakeholders, and users of research. We characterize this as the sphere of influence. We can therefore evaluate by seeking evidence that the research contributed to changes in knowledge, attitudes, skills, and/or relationships (KASR) among key actors in the impact pathway.

Ripple Effects

If research succeeds at stimulating or contributing to change within the sphere of influence, it is reasonable to expect further knock-on changes. That is, if key actors do something differently as a result (partially or fully) of the research, that may in turn contribute to further changes that will help transform systems and ultimately lead to social, economic, and environmental benefits. These kinds of changes typically occur well outside the spheres of control and influence of a research project or program. It is important to monitor change in the sphere of interest to know if things improving or not (and to respond accordingly), but it is difficult or impossible to attribute those changes to research because there are so many other factors involved.

Figure 1. Pathways to Research Outcomes and Impacts conceptualized in a spheres model.

The figure shows the same stages in a generic research impact pathway within the relevant sphere. It tries to illustrate the typical sequence of a research project vertically (top to bottom in the sphere of control), but also reflects the idea that each step can have influence. The research contributes to changes in KASR, leading to individual and organizational changes in the sphere of influence. In turn, these contribute to further changes, ideally leading to benefits in the sphere of interest.

Table 1 shows stages of a generic research impact pathway, from problem identification through to the ultimate intended benefits (impacts). The table offers an illustrative list of the kinds of contributions that can be made during a research process and the kinds of changes that might result. It is important to note that work done in a research context to define and conceptualize a problem, for example, may contribute to outcomes; impact pathways do not necessarily flow only through research results and analysis, dissemination, uptake etc., but also through engagement processes as well.

Regarding terminology, we find it useful to define outcomes and impacts absolutely, by the nature of change, and not relatively (e.g., short term vs long term; immediate vs distal). Changes in KASR are outcomes. If changes in KASR lead to changes in behaviour (including sharing knowledge, for example) by an actor in the system, that may contribute to change other actors’ KASR and behaviour. These changes are also outcomes. Ultimately, it is hoped that the change process to which the research contributed will result in tangible social, economic, and/or environmental benefits, which we call impacts.

Table 1. Illustrative list of research contributions by stage in impact pathway.

conceptualizing

Abbreviations:
KASR = Knowledge, attitudes, skills, and/or relationships
MEL = Monitoring, evaluation, and learning

Read more news posts from the Sustainability Research Effectiveness Program.

0 Comments

Submit a Comment

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.