Publications

Purpose

The overall purpose of this study is to assess the contributions of graduate research to social innovation and change for learning and improved transdisciplinary practice. Universities, as centers of teaching and research, face high demand from society to address urgent social and environmental challenges. Hence, faculty and students are keen to use their research to contribute to social innovation and sustainable development. As part of the effort to increase societal impact, research approaches are evolving to be more problem-oriented, engaged and transdisciplinary. Therefore, new approaches to research evaluation are also needed to learn whether and how research contributes to social innovation. Moreover, these lessons need to be applied by universities to train and support students to do impactful research and foster an impact culture.

Design, Methodology, Approach

This paper uses a theory-based evaluation method to assess the contributions of three completed doctoral research projects. Each study documents the project’s Theory of Change (ToC) and uses qualitative data (document review, surveys and interviews) to test the ToC. We use a transdisciplinary research (TDR) quality assessment framework (QAF) to analyze each projects’ design and implementation. We then draws lessons from the individual case studies and a comparative analysis of the three cases on, namely, effective research design and implementation for social transformation; and training and support for impactful research.

Findings

Each project aimed to influence government policy, organizational practice, other research and/or the students’ own professional development. All contributed to many of their intended outcomes, but with varying levels of accomplishment. Projects that were more transdisciplinary had more pronounced outcomes. Process contributions (e.g. capacity-building, relationship-building and empowerment) were as or more important than knowledge contributions. The key recommendations are for: researchers to design intentional research, with an explicit ToC; higher education institutions (HEI) to provide training and support for TDR theory and practice; and HEIs to give more attention to research evaluation.

Originality and Value

In summary, this is the first application of both the outcome evaluation method and the TDR QAF to graduate student research projects, and one of very few such analyses of research projects. It offers a broader framework for conceptualizing and evaluating research contributions to social change processes. It is intended to stimulate new thinking about research aims, approaches and achievements.

Problem Statement

Any effort to understand, evaluate, and improve the impact of research must begin with clear concepts and definitions. Currently, key terms to describe research results are used ambiguously, and the most common definitions for these terms are fundamentally flawed. This hinders research design, evaluation, learning, and accountability as a result.

Discussion

Specifically, the terms outcome and impact are often defined and distinguished from one another using relative characteristics, such as the degree, directness, scale, or duration of change. Instead, we propose to define these terms by the kind of change rather than by the degree or temporal nature of change.

In general, research contributions to a change process are modeled as a series of causally inter-related steps in a results chain or results web with three main kinds of results: firstly, the direct products of research, referred to as outputs; secondly, changes in the agency and actions of system actors when they are informed/influenced by research outputs, referred to as outcomes; and thirdly, tangible changes in the social, economic, environmental, or other physical condition, referred to as realized benefits.

Complete definitions for these terms are provided, along with examples. In conclusion, this classification aims to help focus research evaluation appropriately and enhance appreciation of the multiple pathways and mechanisms by which scholarship contributes to change.

In order to build a strong changemaker foundation for their business programming, Royal Roads University underwent a comprehensive redesign of their learning outcomes, theory of change, as well as evaluation practices. In this chapter, authors detail how they approached this process. Of particular note is their work to assess new learning outcomes by drawing on existing data collection practices, instead of reinventing the wheel.

Purpose

There is a need to better understand scientific knowledge use in decision-making. This is especially true in the Global South where policy processes often occur under high political uncertainty and where a shift toward multilevel governance and decision-making brings new opportunities and challenges.

Methods

This study applies knowledge-policy models to analyse a forestry research project that succeeded in influencing national policy-making. We investigate how decisions were made, what factors affected and shaped the policy process, and the use of scientific knowledge.

Results

The results highlight the complexity of policy processes and the related challenges in crossing the science-policy interface. Perceptions of scientific knowledge differed greatly among stakeholders. In particularly, those perceptions strongly influenced how scientific knowledge was valued and used. The findings suggest a need for researchers to better understand the problem context to help design and implement research that will more effectively inform decision-making.

Problem Statement

There are high expectations for contemporary forestry research, and sustainability research more broadly, to have impact in the form of improved institutions, policy and practice and improved social and environmental conditions. As part of this trend, there is an evolution of research approaches that move beyond isolated, reductionist, disciplinary science toward approaches that integrate disciplines (interdisciplinary) and that engage a wider range of research stakeholders (transdisciplinary) as a way to be more effective. While these approaches evolve, there are good opportunities to learn from the experience of projects that have had impact at some level.

Case Study and Methods

This paper presents lessons from a case-study of a research project that succeeded in crossing the science-policy interface. Our study characterizes the design and implementation of a research project on the influence of timber harvesting on Brazil nut production using transdisciplinary research (TDR) design principles. In addition, the paper empirically assesses project outputs and outcomes in relation to a project theory of change (ToC) based on document review and key informant interviews.

Results

In summary, the Brazil Nut Project included some TDR elements and realized a substantial part of its ToC. The interviews identified mixed perceptions of the research design, implementation and the extent of outcomes achievement from different stakeholder perspectives. Our analysis suggests that limited stakeholder engagement crucially affected perceptions of legitimacy and relevance, the two main TDR principles underpinning the overall research effectiveness in our study. In addition, the application of the TDR analytical framework indicates substantial scope to improve research effectiveness, even without striving for a TDR theoretical ideal.

Hansson and Polk (2018, Research Evaluation, 27/2: 132–44) aim to assess the usefulness of the concepts of relevance, credibility, and legitimacy for understanding the link between process and impact in transdisciplinary (TD) research. However, the paper seems to misunderstand and misrepresent some of the ideas in the two main reference articles.

Discussion

Consequently, the paper uses definitions of the concepts it aims to test that are inconsistent with the definitions offered by the reference papers. In addition, the methods description is also insufficient to know what data were collected or how they were analyzed. Most important, the effort to understand relationships between process and impact in TD research needs more careful definitions of the concepts outcome and impact. We also need more objective ways to assess outcomes and impact. Hence, this letter discusses shortcomings in the article and makes suggestions to improve conceptual clarity and methods for empirically assessing TD research effectiveness.

Problem Definition

The increasing external demand from research funders and managers to assess, evaluate, and demonstrate the quality and the effectiveness of research is well known. However, less discussed but equally important, is the evolving interest and use of research evaluation to support learning and adaptive management within research programmes. This is especially true in a research-for-development context where research competes with other worthy alternatives for overseas development assistance funding. Moreover, highly complex social, economic and ecological environments add to evaluation challenges. Thus, researchers and research managers need to know whether and how their interventions are working to be able to adapt and improve their programmes and satisfy their funders.

Methods

This paper presents a theory-based research evaluation approach developed and tested on four policy-relevant research activities:

1. a long-term forest management research programme in the Congo Basin;
2. a large research programme on forests and climate change;
3. a multi-country research project on sustainable wetlands management; and
4. a research project of the furniture value chain in one district in Indonesia.

The first used Contribution Analysis and the others used purpose-built outcome evaluation approaches that combined concepts and methods from several approaches. Each research evaluation began with documentation of a theory of change (ToC) that identified key actors, processes and results. Meanwhile, we analysed data collected through document reviews, key informant interviews and focus group discussions to test the ToCs against evidence of outcomes in the form of discourse, policy formulation and practice change.

Results

As a result, the approach proved valuable as a learning tool for researchers and research managers. Moreover, it facilitated communication with funders about actual and reasonable research contributions to change. Evaluations employing a participatory approach with project scientists and partners noticeably supported team learning about past work. Furthermore, it supported discussions about possible adaptations for the future. To summarize, in all four cases, the retrospective ToC development proved challenging and resulted in overly-simplistic ToCs.

Conclusion

Evidently, further work to draw on social scientific theories of knowledge translation and policy processes to develop and further test more sophisticated theories of change will be crucial. This theory-based approach to research evaluation provides a valuable means of assessing research effectiveness (summative value) and supports learning and adaptation (formative value) at the project or programme scale. The approach is well suited to the research-for-development projects represented by the case studies, but it should be applicable to any research that aspires to have a societal impact. This article constitutes part of a collection on the future of research assessment.

Problem Statement

Research increasingly seeks both to generate knowledge and to contribute to real-world solutions, with strong emphasis on context and social engagement. As we cross boundaries between disciplines, and as research engages more with stakeholders in complex systems, traditional academic definitions and criteria of research quality are no longer sufficient. Therefore, there is a need for a parallel evolution of principles and criteria to define and evaluate research quality in a transdisciplinary research (TDR) context.

Methods

We conducted a systematic review to help answer the question: What are appropriate principles and criteria for defining and assessing TDR quality? We selected and reviewed articles seeking: arguments for or against expanding definitions of research quality, purposes for research quality evaluation, proposed principles of research quality, proposed criteria for research quality assessment, proposed indicators and measures of research quality, and proposed processes for evaluating TDR. We used the information from the review and our own experience in two research organizations that employ TDR approaches to develop a prototype TDR quality assessment framework, organized as an evaluation rubric.

Results

We provide an overview of the relevant literature and summarize the main aspects of TDR quality identified there. Four main principles emerge: relevance, including social significance and applicability; credibility, including criteria of integration and reflexivity, added to traditional criteria of scientific rigor; legitimacy, including criteria of inclusion and fair representation of stakeholder interests; and effectiveness, with criteria that assess actual or potential contributions to problem solving and social change.