Publications

To support knowledge-sharing and access, all our peer-reviewed publications are published open-source and can be found linked below. Published case study reports can be found here.

How to build Theories of Change for transdisciplinary research: Guidance and considerations

Citation: Claus, R., Davel, R., Heykoop, C., Pinto, D., & Belcher, B. M. (2023). How to build Theories of Change for transdisciplinary research: Guidance and considerations. GAIA: Ecological Perspectives for Science and Society, 32(1): 186-196. https://doi.org/10.14512/gaia.32.1.18

Abstract

Transdisciplinary research (TDR) aims to solve problems in complex systems by drawing from a range of methods and expertise to contribute to change processes. Theories of Change (ToCs) are well-suited to support TDR design and implementation, but they rarely achieve their full potential. In practice, ToCs are often compromised by insufficient engagement with the context, weak theoretical bases, poor articulation, and a lack of iteration.

This paper presents a process for ToC design based on the authors’ experience facilitating ToC development for research planning and evaluation. We illustrate the process using an in-progress TDR example on patient-oriented cancer care research. The approach begins by framing the social and research problems and then identifies activities and outputs, key actors, outcomes, and underlying causal assumptions. Skilled facilitation and strong conceptual familiarity are key to effectively mobilize ToC concepts into a cohesive and testable model to refine a strategy with TDR stakeholders. Key considerations and resources are offered to enhance ToC development planning and facilitation.

Evaluating and improving the contributions of university research to social innovation

Citation: Belcher, B. M., Claus, R., Davel, R., & Jones, S. M. (2021). Evaluating and improving the contributions of university research to social innovation. Social Enterprise Journal, 18(11): 51-120. https://doi.org/10.1108/SEJ-10-2020-0099 

Abstract

Purpose

The overall purpose of this study is to assess the contributions of graduate research to social innovation and change for learning and improved transdisciplinary practice. Universities, as centers of teaching and research, face high demand from society to address urgent social and environmental challenges. Hence, faculty and students are keen to use their research to contribute to social innovation and sustainable development. As part of the effort to increase societal impact, research approaches are evolving to be more problem-oriented, engaged and transdisciplinary. Therefore, new approaches to research evaluation are also needed to learn whether and how research contributes to social innovation. Moreover, these lessons need to be applied by universities to train and support students to do impactful research and foster an impact culture.

Design, Methodology, Approach

This paper uses a theory-based evaluation method to assess the contributions of three completed doctoral research projects. Each study documents the project’s Theory of Change (ToC) and uses qualitative data (document review, surveys and interviews) to test the ToC. We use a transdisciplinary research (TDR) quality assessment framework (QAF) to analyze each projects’ design and implementation. We then draws lessons from the individual case studies and a comparative analysis of the three cases on, namely, effective research design and implementation for social transformation; and training and support for impactful research.

Findings

Each project aimed to influence government policy, organizational practice, other research and/or the students’ own professional development. All contributed to many of their intended outcomes, but with varying levels of accomplishment. Projects that were more transdisciplinary had more pronounced outcomes. Process contributions (e.g. capacity-building, relationship-building and empowerment) were as or more important than knowledge contributions. The key recommendations are for: researchers to design intentional research, with an explicit ToC; higher education institutions (HEI) to provide training and support for TDR theory and practice; and HEIs to give more attention to research evaluation.

Originality and Value

In summary, this is the first application of both the outcome evaluation method and the TDR QAF to graduate student research projects, and one of very few such analyses of research projects. It offers a broader framework for conceptualizing and evaluating research contributions to social change processes. It is intended to stimulate new thinking about research aims, approaches and achievements.

Conceptualizing the elements of research impact: Towards semantics standards

Citation: Belcher, B., & Halliwell, J. (2021). Conceptualizing the elements of research impact: Towards semantics standards. Humanities and Social Sciences Communications, 8: 1-6. http://doi.org10.1057/s41599-021-00854-2

Abstract

Problem Statement

Any effort to understand, evaluate, and improve the impact of research must begin with clear concepts and definitions. Currently, key terms to describe research results are used ambiguously, and the most common definitions for these terms are fundamentally flawed. This hinders research design, evaluation, learning, and accountability as a result.

Discussion

Specifically, the terms outcome and impact are often defined and distinguished from one another using relative characteristics, such as the degree, directness, scale, or duration of change. Instead, we propose to define these terms by the kind of change rather than by the degree or temporal nature of change.

In general, research contributions to a change process are modeled as a series of causally inter-related steps in a results chain or results web with three main kinds of results: firstly, the direct products of research, referred to as outputs; secondly, changes in the agency and actions of system actors when they are informed/influenced by research outputs, referred to as outcomes; and thirdly, tangible changes in the social, economic, environmental, or other physical condition, referred to as realized benefits.

Complete definitions for these terms are provided, along with examples. In conclusion, this classification aims to help focus research evaluation appropriately and enhance appreciation of the multiple pathways and mechanisms by which scholarship contributes to change.

Leading inter- and transdisciplinary research: Lessons from applying theories of change to a strategic research program

Citation: Deutsch, L., Belcher, B., Claus, R., & Hoffmann, S. (2021). Leading inter- and transdisciplinary research: Lessons from applying theories of change to a strategic research program. Environmental Science & Policy120: 29-41.
https://doi.org/10.1016/j.envsci.2021.02.009

Abstract
Problem Statement

Theory of Change (ToC) is a useful tool in sustainability research for visioning, planning, communication, monitoring, evaluation and learning. Generally, it involves a mapping of steps towards a desired long-term goal supplemented with continuous reflection on how and why change is expected to happen in a particular context. However, there is limited reported experience with the development and application of ToCs in inter- and transdisciplinary research contexts. While some previous publications focused on ex-post application, there is little discussion about the process of developing and using ToCs in strategic planning and monitoring in large inter- and transdisciplinary research programs.

Case Study

This article reports challenges and lessons learned from the experience of developing and using ToCs in the inter- and transdisciplinary research program Wings (Water and sanitation innovations for non-grid solutions). In this case, the challenges include: (1) managing time constraints, (2) balancing between concrete and abstract discussions, (3) ensuring diversity in group composition, (4) fluctuating between reservations and appreciation, and (5) fulfilling both service and science roles while leading the ToC process.

Results

In summary, the experience highlights the importance of alternating formal and informal interaction formats throughout the process, ensuring heterogenous group formation, involving early career scientists, being responsive to emergent needs and making the added value of developing and using ToCs explicit and tangible for all participants. Although these lessons are mainly derived from developing ToCs within the interdisciplinary program team, they can support other programs in both their inter- and transdisciplinary research endeavors.

Understanding and evaluating the impact of integrated problem-oriented research programmes: Concepts and considerations

Citation: Belcher, B. M., & Hughes, K. (2020). Understanding and evaluating the impact of integrated problem-oriented research programmes: Concepts and considerations. Research Evaluation, 30(2): 154-168. https://doi.org/10.1093/reseval/rvaa024

Abstract

Problem Statement

In general, researchers and research organizations endure increasing pressure to demonstrate that their work contributes to positive change and helps solve pressing societal challenges. Simultaneously, there is a trend towards more engaged transdisciplinary research that is complexity-aware and appreciates that change happens through systems transformation, not only through technological innovation. Therefore, we need appropriate evaluation approaches to evidence research impact and generate learning for continual improvement. This is challenging in any research field, but especially for research that crosses disciplinary boundaries and intervenes in complex systems. Moreover, evaluation challenges at the project scale are compounded at the programme scale.

Case Study

In this case, the Forest, Trees and Agroforestry (FTA) research programme serves as an example of this evolution in research approach and the resulting evaluation challenges. FTA research responds to the demand for greater impact with more engaged research following multiple pathways. However, research impact assessment in the CGIAR (Consultative Group on International Agricultural Research) emerged in a technology-centric context where counterfactual approaches of causal inference (experimental and quasi-experimental) predominate. Relying solely on such approaches is inappropriate for evaluating research contributions that target policy and institutional change and systems transformation.

Discussion

Instead, we propose a multifaceted, multi-scale, theory-based evaluation approach. This includes nested project- and programme-scale theories of change (ToCs); research quality assessment; theory-based outcome evaluations to empirically test ToCs and assess policy, institutional, and practice influence; experimental and quasi-experimental impact of FTA-informed ‘large n’ innovations; ex ante impact assessment to estimate potential impacts at scale; as well as logically and plausibly linking programme-level outcomes to secondary data on development and conservation status.

A refined method for theory-based evaluation of the societal impacts of research

Citation: Belcher, B. M., Davel, R., & Claus, R. (2020). A refined method for theory-based evaluation of the societal impacts of research. MethodsX, 7: 100788. https://doi.org/10.1016/j.mex.2020.100788

Abstract
Problem Statement

With high and increasing expectations for research to have social and environmental impact, there is a corresponding need for appropriate methods to demonstrate (for accountability) and analyze (for learning) whether and how research projects contribute to change processes. Evaluation is especially challenging for problem-oriented research that employs inter- and transdisciplinary approaches and intervenes in complex systems, where experimental and statistical approaches to causal inference are inappropriate. Instead, theory-based evaluation can be applied to identify and test causal processes.

Step-by-Step Guidance on Application of the Method

This paper presents a detailed explanation of the Outcome Evaluation approach applied in Belcher et al. (2019). It draws on concepts and approaches used in theory-based program evaluation in addition to the more limited experience of theory-based research evaluation, providing a brief overview of conceptual strengths and limitations of other methods. The paper offers step-by-step guidance on application of the Outcome Evaluation approach, detailing how to: firstly, document a theory of change; secondly, determine data needs and sources; thirdly, collect data; fourthly, manage and analyze data; and lastly, present findings. In summary, this approach provides a clear conceptual and analytical framework in addition to actor-specific and impact pathway analyses for more precision in the assessment of outcomes.

Summary

Specifically, the Outcome Evaluation approach:

Conceptualizes research within a complex system and explicitly recognizes the role of other actors, context, and external processes;
Utilizes a detailed actor-centred theory of change (ToC) as the analytical framework; and
Explicitly tests a set of hypotheses about the relationship between the research process/outputs and outcomes.

Linking transdisciplinary research characteristics and quality to effectiveness: A comparative analysis of five research-for-development projects

Citation: Belcher, B. M., Claus, R., Davel, R., & Ramirez, L. F. (2019). Linking transdisciplinary research characteristics and quality to effectiveness: A comparative analysis of five research-for-development projects. Environmental Science &  Policy, 101: 192-203. https://doi.org/10.1016/j.envsci.2019.08.013

Abstract

Problem Statement

More and more effective research is needed to help address complex sustainability problems. Many research approaches adopt more transdisciplinary characteristics as a way to improve effectiveness. However, empirical evidence of the extent to which and how transdisciplinary research design and implementation contribute to (more) effective scientific and social outcomes remains limited.

Methods

This paper reports a comparative analysis of five research-for-development projects implemented in Peru and Indonesia to: firstly, characterize the extent to which projects employed transdisciplinary principles; secondly, assess the extent to which and how intended project outcomes were achieved; thirdly, analyze the relationship between transdisciplinary research approaches and outcomes; and lastly, provide lessons from the experience of using a theory-based approach to evaluate a set of case studies.

Results

In summary, our analysis demonstrates that the projects employing more transdisciplinary principles in their design and implementation make more diverse contributions and have a greater breadth of influence.

Evaluation for Social Impact: A Theory of Change Approach

Citation: Thexton, T., Belcher, B. M., Claus, R., & Davel, R. (2019). Evaluation for Social Impact: A Theory of Change Approach. In Evaluating Changemaker Education: A Practitioner’s Guide (1st ed., pp. 57-74). AshokaU.

Summary

In order to build a strong changemaker foundation for their business programming, Royal Roads University underwent a comprehensive redesign of their learning outcomes, theory of change, as well as evaluation practices. In this chapter, authors detail how they approached this process. Of particular note is their work to assess new learning outcomes by drawing on existing data collection practices, instead of reinventing the wheel.

Stakeholder perceptions of scientific knowledge in policy processes: A Peruvian case-study of forestry policy development

Citation: Ramirez, L. F., & Belcher, B. M. (2019). Stakeholder perceptions of scientific knowledge in policy processes: A Peruvian case-study of forestry policy development. Science and Public Policy, 46(4): 504-517. https://doi.org/10.1093/scipol/scz003

Abstract

Purpose

There is a need to better understand scientific knowledge use in decision-making. This is especially true in the Global South where policy processes often occur under high political uncertainty and where a shift toward multilevel governance and decision-making brings new opportunities and challenges.

Methods

This study applies knowledge-policy models to analyse a forestry research project that succeeded in influencing national policy-making. We investigate how decisions were made, what factors affected and shaped the policy process, and the use of scientific knowledge.

Results

The results highlight the complexity of policy processes and the related challenges in crossing the science-policy interface. Perceptions of scientific knowledge differed greatly among stakeholders. In particularly, those perceptions strongly influenced how scientific knowledge was valued and used. The findings suggest a need for researchers to better understand the problem context to help design and implement research that will more effectively inform decision-making.

Getting forest science to policy discourse: A theory-based outcome assessment of a global research programme

Citation: Halimanjaya, A., Belcher, B., & Suryadarma, D. (2018). Getting forest science to policy discourse: A theory-based outcome assessment of a global research programme. International Forestry Review, 20(4): 469-487. https://doi.org/10.1505/146554818825240638

Abstract

Purpose

This paper presents an assessment of the outcomes of research carried out under the Sustainable Wetlands Adaptation and Mitigation Programme (SWAMP). SWAMP aimed to inform national and international climate policy and practice. In addition, SWAMP developed tools and methods to quantify greenhouse gas (GHG) emissions, carbon stocks and flux in tropical wetlands due to land use, land-use change and forestry (LULUCF).

Methods and Results

This assessment modelled SWAMP’s intended outcomes as a theory of change (ToC) and used qualitative methods to test the ToC and to evaluate whether and how the outcomes were achieved. It found that SWAMP research helped raise academic and policy interest in wetlands, mangroves and peat forests as carbon reservoirs. Moreover, SWAMP’s recommendations informed policy discourse and supported the development of technical guidance and strategies of sustainable wetland management.

However, the research had a weak effect on international and Indonesian climate change policies compared to other factors. For example, the Paris Agreement and Indonesia’s nationally determined contribution (NDC) do not include the quantification of carbon stocks from mangroves, which are not all located in the forest areas. Knowledge translation occurred through a variety of mechanisms, with direct engagement identified as particularly important. Overall, the outcome evaluation approach proved useful as a way of conceptualising and organising the analysis of research impact on development outcomes.

Crossing the science-policy interface: Lessons from a research project on Brazil nut management in Peru

Citation: Ramirez, L. F., & Belcher, B. M. (2018). Crossing the science-policy interface: Lessons from a research project on Brazil nut management in Peru. Forest Policy and Economics114: 101789.
https://doi.org/10.1016/j.forpol.2018.07.018

Abstract

Problem Statement

There are high expectations for contemporary forestry research, and sustainability research more broadly, to have impact in the form of improved institutions, policy and practice and improved social and environmental conditions. As part of this trend, there is an evolution of research approaches that move beyond isolated, reductionist, disciplinary science toward approaches that integrate disciplines (interdisciplinary) and that engage a wider range of research stakeholders (transdisciplinary) as a way to be more effective. While these approaches evolve, there are good opportunities to learn from the experience of projects that have had impact at some level.

Case Study and Methods

This paper presents lessons from a case-study of a research project that succeeded in crossing the science-policy interface. Our study characterizes the design and implementation of a research project on the influence of timber harvesting on Brazil nut production using transdisciplinary research (TDR) design principles. In addition, the paper empirically assesses project outputs and outcomes in relation to a project theory of change (ToC) based on document review and key informant interviews.

Results

In summary, the Brazil Nut Project included some TDR elements and realized a substantial part of its ToC. The interviews identified mixed perceptions of the research design, implementation and the extent of outcomes achievement from different stakeholder perspectives. Our analysis suggests that limited stakeholder engagement crucially affected perceptions of legitimacy and relevance, the two main TDR principles underpinning the overall research effectiveness in our study. In addition, the application of the TDR analytical framework indicates substantial scope to improve research effectiveness, even without striving for a TDR theoretical ideal.

A response to Hansson and Polk (2018)

Citation: Belcher, B. M., Ramirez, L. F., Davel, R., & Claus, R. (2018). A response to Hansson and Polk (2018) “Assessing the impact of transdisciplinary research: The usefulness of relevance, credibility, and legitimacy for understanding the link between process and impact”. Research Evaluation, 28(2): 196–201.
https://doi.org/10.1093/reseval/rvy037

Abstract

Hansson and Polk (2018, Research Evaluation, 27/2: 132–44) aim to assess the usefulness of the concepts of relevance, credibility, and legitimacy for understanding the link between process and impact in transdisciplinary (TD) research. However, the paper seems to misunderstand and misrepresent some of the ideas in the two main reference articles.

Discussion

Consequently, the paper uses definitions of the concepts it aims to test that are inconsistent with the definitions offered by the reference papers. In addition, the methods description is also insufficient to know what data were collected or how they were analyzed. Most important, the effort to understand relationships between process and impact in TD research needs more careful definitions of the concepts outcome and impact. We also need more objective ways to assess outcomes and impact. Hence, this letter discusses shortcomings in the article and makes suggestions to improve conceptual clarity and methods for empirically assessing TD research effectiveness.

Outcomes and impacts of development interventions: Toward conceptual clarity

Citation: Belcher, B., & Palenberg, M. (2018). Outcomes and Impacts of Development Interventions: Toward Conceptual Clarity. American Journal of Evaluation39(4): 478–495.
https://doi.org/10.1177/1098214018765698

Abstract

Problem Definition

The terms “outcome” and “impact” are ubiquitous in evaluation discourse. However, there are many competing definitions that lack clarity and consistency and sometimes represent fundamentally different meanings. This leads to profound confusion, undermines efforts to improve learning and accountability, and represents a challenge for the evaluation profession.

Method

Consequently, this article investigates how the terms are defined and understood by different institutions and communities. It systematically investigates representative sets of definitions, analyzing them to identify 16 distinct defining elements. This framework is then used to compare definitions and assess their usefulness and limitations.

Discussion

Based on this assessment, the article proposes a remedy in three parts: firstly, applying good definition practice in future definition updates; secondly, differentiating causal perspectives and using appropriate causal language; and lastly, employing meaningful qualifiers when using the terms outcome and impact. The article draws on definitions used in international development, but its findings also apply to domestic public sector policies and interventions.

Evaluating policy-relevant research: Lessons from a series of theory-based outcomes assessments

Citation:  Belcher, B., Suryadarma, D., & Halimanjaya, A. (2017). Evaluating policy-relevant research: Lessons from a series of theory-based outcomes assessments. Palgrave Communications, 3: 1-16.
https://doi.org/10.1057/palcomms.2017.17

Abstract

Problem Definition

The increasing external demand from research funders and managers to assess, evaluate, and demonstrate the quality and the effectiveness of research is well known. However, less discussed but equally important, is the evolving interest and use of research evaluation to support learning and adaptive management within research programmes. This is especially true in a research-for-development context where research competes with other worthy alternatives for overseas development assistance funding. Moreover, highly complex social, economic and ecological environments add to evaluation challenges. Thus, researchers and research managers need to know whether and how their interventions are working to be able to adapt and improve their programmes and satisfy their funders.

Methods

This paper presents a theory-based research evaluation approach developed and tested on four policy-relevant research activities:

  1. a long-term forest management research programme in the Congo Basin;
  2. a large research programme on forests and climate change;
  3. a multi-country research project on sustainable wetlands management; and
  4. a research project of the furniture value chain in one district in Indonesia.

The first used Contribution Analysis and the others used purpose-built outcome evaluation approaches that combined concepts and methods from several approaches. Each research evaluation began with documentation of a theory of change (ToC) that identified key actors, processes and results. Meanwhile, we analysed data collected through document reviews, key informant interviews and focus group discussions to test the ToCs against evidence of outcomes in the form of discourse, policy formulation and practice change.

Results

As a result, the approach proved valuable as a learning tool for researchers and research managers. Moreover, it facilitated communication with funders about actual and reasonable research contributions to change. Evaluations employing a participatory approach with project scientists and partners noticeably supported team learning about past work. Furthermore, it supported discussions about possible adaptations for the future. To summarize, in all four cases, the retrospective ToC development proved challenging and resulted in overly-simplistic ToCs.

Conclusion

Evidently, further work to draw on social scientific theories of knowledge translation and policy processes to develop and further test more sophisticated theories of change will be crucial. This theory-based approach to research evaluation provides a valuable means of assessing research effectiveness (summative value) and supports learning and adaptation (formative value) at the project or programme scale. The approach is well suited to the research-for-development projects represented by the case studies, but it should be applicable to any research that aspires to have a societal impact. This article constitutes part of a collection on the future of research assessment.

Defining and assessing research quality in a transdisciplinary context

Citation: Belcher, B. M., Rasmussen, K. E., Kemshaw, M. R., & Zornes, D. A. (2016). Defining and assessing research quality in a transdisciplinary context. Research Evaluation, 25(1): 1-17.
https://doi.org/10.1093/reseval/rvv025

Abstract

Problem Statement

Research increasingly seeks both to generate knowledge and to contribute to real-world solutions, with strong emphasis on context and social engagement. As we cross boundaries between disciplines, and as research engages more with stakeholders in complex systems, traditional academic definitions and criteria of research quality are no longer sufficient. Therefore, there is a need for a parallel evolution of principles and criteria to define and evaluate research quality in a transdisciplinary research (TDR) context.

Methods

We conducted a systematic review to help answer the question: What are appropriate principles and criteria for defining and assessing TDR quality? We selected and reviewed articles seeking: arguments for or against expanding definitions of research quality, purposes for research quality evaluation, proposed principles of research quality, proposed criteria for research quality assessment, proposed indicators and measures of research quality, and proposed processes for evaluating TDR. We used the information from the review and our own experience in two research organizations that employ TDR approaches to develop a prototype TDR quality assessment framework, organized as an evaluation rubric.

Results

We provide an overview of the relevant literature and summarize the main aspects of TDR quality identified there. Four main principles emerge: relevance, including social significance and applicability; credibility, including criteria of integration and reflexivity, added to traditional criteria of scientific rigor; legitimacy, including criteria of inclusion and fair representation of stakeholder interests; and effectiveness, with criteria that assess actual or potential contributions to problem solving and social change.

Finding appropriate definitions and measures of research quality for transdisciplinary and applied natural resource management research

Citation: Belcher, B. M., Rasmussen, K. E., Kemshaw, M. R., & Zornes, D. A. (2013). Finding appropriate definitions and measures of research quality for transdisciplinary and applied natural resource management research: A systematic review protocol. Occasional Paper No.99. Center for International Forestry Research: Bogor, Indonesia.

Abstract
Problem Statement

Research increasingly seeks not only to generate knowledge, but also exert impact. In this context, traditional academic definitions of research quality may be insufficient.

Methods

This article presents a protocol for the systematic review of new and emerging definitions, criteria and indicators of research quality in applied, inter- and transdisciplinary contexts. Additionally, it seeks to clarify arguments for or against expanding the definitions of research quality. Moreover, it aims to identify appropriate definitions and measures, with an emphasis on natural resource management research.

The primary research question is: What are appropriate criteria and indicators for defining and measuring the quality of transdisciplinary research in natural resource management research?

The proposed review will be based on literature sourced from a search of Scopus, Web of Knowledge and Google Scholar. We used search terms and strings  developed and tested iteratively, based on a benchmark set of references and incrementally refined searches designed to be comprehensive and to reduce irrelevant results.

To select relevant articles, two reviewers independently performed three rounds of screening by scanning titles, abstracts and articles. Once selected, the articles will then be reviewed for the following: arguments for or against expanding definitions of research quality; purposes for research quality evaluation; proposed principles of research quality; proposed criteria for research quality assessment; proposed indicators and measures of research quality; and proposed processes for evaluating transdisciplinary research.

Conclusion

To conclude, the results will be synthesized to provide an overview of the literature, and summarize the arguments and approaches for expanding definitions of research quality. In effect, this will identify and discuss the main purposes, principles, indicators and measures of research quality in transdisciplinary and applied contexts.