Publications: Refined tools, concepts and methods

Problem Statement

Any effort to understand, evaluate, and improve the impact of research must begin with clear concepts and definitions. Currently, key terms to describe research results are used ambiguously, and the most common definitions for these terms are fundamentally flawed. This hinders research design, evaluation, learning, and accountability as a result.

Discussion

Specifically, the terms outcome and impact are often defined and distinguished from one another using relative characteristics, such as the degree, directness, scale, or duration of change. Instead, we propose to define these terms by the kind of change rather than by the degree or temporal nature of change.

In general, research contributions to a change process are modeled as a series of causally inter-related steps in a results chain or results web with three main kinds of results: firstly, the direct products of research, referred to as outputs; secondly, changes in the agency and actions of system actors when they are informed/influenced by research outputs, referred to as outcomes; and thirdly, tangible changes in the social, economic, environmental, or other physical condition, referred to as realized benefits.

Complete definitions for these terms are provided, along with examples. In conclusion, this classification aims to help focus research evaluation appropriately and enhance appreciation of the multiple pathways and mechanisms by which scholarship contributes to change.

Problem Statement

Theory of Change (ToC) is a useful tool in sustainability research for visioning, planning, communication, monitoring, evaluation and learning. Generally, it involves a mapping of steps towards a desired long-term goal supplemented with continuous reflection on how and why change is expected to happen in a particular context. However, there is limited reported experience with the development and application of ToCs in inter- and transdisciplinary research contexts. While some previous publications focused on ex-post application, there is little discussion about the process of developing and using ToCs in strategic planning and monitoring in large inter- and transdisciplinary research programs.

Case Study

This article reports challenges and lessons learned from the experience of developing and using ToCs in the inter- and transdisciplinary research program Wings (Water and sanitation innovations for non-grid solutions). In this case, the challenges include: (1) managing time constraints, (2) balancing between concrete and abstract discussions, (3) ensuring diversity in group composition, (4) fluctuating between reservations and appreciation, and (5) fulfilling both service and science roles while leading the ToC process.

Results

In summary, the experience highlights the importance of alternating formal and informal interaction formats throughout the process, ensuring heterogenous group formation, involving early career scientists, being responsive to emergent needs and making the added value of developing and using ToCs explicit and tangible for all participants. Although these lessons are mainly derived from developing ToCs within the interdisciplinary program team, they can support other programs in both their inter- and transdisciplinary research endeavors.

Problem Definition

The terms “outcome” and “impact” are ubiquitous in evaluation discourse. However, there are many competing definitions that lack clarity and consistency and sometimes represent fundamentally different meanings. This leads to profound confusion, undermines efforts to improve learning and accountability, and represents a challenge for the evaluation profession.

Method

Consequently, this article investigates how the terms are defined and understood by different institutions and communities. It systematically investigates representative sets of definitions, analyzing them to identify 16 distinct defining elements. This framework is then used to compare definitions and assess their usefulness and limitations.

Discussion

Based on this assessment, the article proposes a remedy in three parts: firstly, applying good definition practice in future definition updates; secondly, differentiating causal perspectives and using appropriate causal language; and lastly, employing meaningful qualifiers when using the terms outcome and impact. The article draws on definitions used in international development, but its findings also apply to domestic public sector policies and interventions.

Problem Statement

Research increasingly seeks both to generate knowledge and to contribute to real-world solutions, with strong emphasis on context and social engagement. As we cross boundaries between disciplines, and as research engages more with stakeholders in complex systems, traditional academic definitions and criteria of research quality are no longer sufficient. Therefore, there is a need for a parallel evolution of principles and criteria to define and evaluate research quality in a transdisciplinary research (TDR) context.

Methods

We conducted a systematic review to help answer the question: What are appropriate principles and criteria for defining and assessing TDR quality? We selected and reviewed articles seeking: arguments for or against expanding definitions of research quality, purposes for research quality evaluation, proposed principles of research quality, proposed criteria for research quality assessment, proposed indicators and measures of research quality, and proposed processes for evaluating TDR. We used the information from the review and our own experience in two research organizations that employ TDR approaches to develop a prototype TDR quality assessment framework, organized as an evaluation rubric.

Results

We provide an overview of the relevant literature and summarize the main aspects of TDR quality identified there. Four main principles emerge: relevance, including social significance and applicability; credibility, including criteria of integration and reflexivity, added to traditional criteria of scientific rigor; legitimacy, including criteria of inclusion and fair representation of stakeholder interests; and effectiveness, with criteria that assess actual or potential contributions to problem solving and social change.