Publications: Lessons about what makes research effective
To support knowledge-sharing and access, all our peer-reviewed publications are published open-source and can be found linked below. Published case study reports can be found here.
Evaluating and improving the contributions of university research to social innovation
Citation: Belcher, B. M., Claus, R., Davel, R., & Jones, S. M. (2021). Evaluating and improving the contributions of university research to social innovation. Social Enterprise Journal, 18(11): 51-120. https://doi.org/10.1108/SEJ-10-2020-0099
Abstract
The overall purpose of this study is to assess the contributions of graduate research to social innovation and change for learning and improved transdisciplinary practice. Universities, as centers of teaching and research, face high demand from society to address urgent social and environmental challenges. Hence, faculty and students are keen to use their research to contribute to social innovation and sustainable development. As part of the effort to increase societal impact, research approaches are evolving to be more problem-oriented, engaged and transdisciplinary. Therefore, new approaches to research evaluation are also needed to learn whether and how research contributes to social innovation. Moreover, these lessons need to be applied by universities to train and support students to do impactful research and foster an impact culture.
Design, Methodology, Approach
This paper uses a theory-based evaluation method to assess the contributions of three completed doctoral research projects. Each study documents the project’s Theory of Change (ToC) and uses qualitative data (document review, surveys and interviews) to test the ToC. We use a transdisciplinary research (TDR) quality assessment framework (QAF) to analyze each projects’ design and implementation. We then draws lessons from the individual case studies and a comparative analysis of the three cases on, namely, effective research design and implementation for social transformation; and training and support for impactful research.
Findings
Each project aimed to influence government policy, organizational practice, other research and/or the students’ own professional development. All contributed to many of their intended outcomes, but with varying levels of accomplishment. Projects that were more transdisciplinary had more pronounced outcomes. Process contributions (e.g. capacity-building, relationship-building and empowerment) were as or more important than knowledge contributions. The key recommendations are for: researchers to design intentional research, with an explicit ToC; higher education institutions (HEI) to provide training and support for TDR theory and practice; and HEIs to give more attention to research evaluation.
Originality and Value
In summary, this is the first application of both the outcome evaluation method and the TDR QAF to graduate student research projects, and one of very few such analyses of research projects. It offers a broader framework for conceptualizing and evaluating research contributions to social change processes. It is intended to stimulate new thinking about research aims, approaches and achievements.
Linking transdisciplinary research characteristics and quality to effectiveness: A comparative analysis of five research-for-development projects
Citation: Belcher, B. M., Claus, R., Davel, R., & Ramirez, L. F. (2019). Linking transdisciplinary research characteristics and quality to effectiveness: A comparative analysis of five research-for-development projects. Environmental Science & Policy, 101: 192-203. https://doi.org/10.1016/j.envsci.2019.08.013
Abstract
More and more effective research is needed to help address complex sustainability problems. Many research approaches adopt more transdisciplinary characteristics as a way to improve effectiveness. However, empirical evidence of the extent to which and how transdisciplinary research design and implementation contribute to (more) effective scientific and social outcomes remains limited.
Methods
This paper reports a comparative analysis of five research-for-development projects implemented in Peru and Indonesia to: firstly, characterize the extent to which projects employed transdisciplinary principles; secondly, assess the extent to which and how intended project outcomes were achieved; thirdly, analyze the relationship between transdisciplinary research approaches and outcomes; and lastly, provide lessons from the experience of using a theory-based approach to evaluate a set of case studies.
Results
In summary, our analysis demonstrates that the projects employing more transdisciplinary principles in their design and implementation make more diverse contributions and have a greater breadth of influence.
Stakeholder perceptions of scientific knowledge in policy processes: A Peruvian case-study of forestry policy development
Citation: Ramirez, L. F., & Belcher, B. M. (2019). Stakeholder perceptions of scientific knowledge in policy processes: A Peruvian case-study of forestry policy development. Science and Public Policy, 46(4): 504-517. https://doi.org/10.1093/scipol/scz003
Abstract
There is a need to better understand scientific knowledge use in decision-making. This is especially true in the Global South where policy processes often occur under high political uncertainty and where a shift toward multilevel governance and decision-making brings new opportunities and challenges.
Methods
This study applies knowledge-policy models to analyse a forestry research project that succeeded in influencing national policy-making. We investigate how decisions were made, what factors affected and shaped the policy process, and the use of scientific knowledge.
Results
The results highlight the complexity of policy processes and the related challenges in crossing the science-policy interface. Perceptions of scientific knowledge differed greatly among stakeholders. In particularly, those perceptions strongly influenced how scientific knowledge was valued and used. The findings suggest a need for researchers to better understand the problem context to help design and implement research that will more effectively inform decision-making.
Getting forest science to policy discourse: A theory-based outcome assessment of a global research programme
Citation: Halimanjaya, A., Belcher, B., & Suryadarma, D. (2018). Getting forest science to policy discourse: A theory-based outcome assessment of a global research programme. International Forestry Review, 20(4): 469-487. https://doi.org/10.1505/146554818825240638
Abstract
This paper presents an assessment of the outcomes of research carried out under the Sustainable Wetlands Adaptation and Mitigation Programme (SWAMP). SWAMP aimed to inform national and international climate policy and practice. In addition, SWAMP developed tools and methods to quantify greenhouse gas (GHG) emissions, carbon stocks and flux in tropical wetlands due to land use, land-use change and forestry (LULUCF).
Methods and Results
This assessment modelled SWAMP’s intended outcomes as a theory of change (ToC) and used qualitative methods to test the ToC and to evaluate whether and how the outcomes were achieved. It found that SWAMP research helped raise academic and policy interest in wetlands, mangroves and peat forests as carbon reservoirs. Moreover, SWAMP’s recommendations informed policy discourse and supported the development of technical guidance and strategies of sustainable wetland management.
However, the research had a weak effect on international and Indonesian climate change policies compared to other factors. For example, the Paris Agreement and Indonesia’s nationally determined contribution (NDC) do not include the quantification of carbon stocks from mangroves, which are not all located in the forest areas. Knowledge translation occurred through a variety of mechanisms, with direct engagement identified as particularly important. Overall, the outcome evaluation approach proved useful as a way of conceptualising and organising the analysis of research impact on development outcomes.
Crossing the science-policy interface: Lessons from a research project on Brazil nut management in Peru
Citation: Ramirez, L. F., & Belcher, B. M. (2018). Crossing the science-policy interface: Lessons from a research project on Brazil nut management in Peru. Forest Policy and Economics, 114: 101789.
https://doi.org/10.1016/j.forpol.2018.07.018
Abstract
There are high expectations for contemporary forestry research, and sustainability research more broadly, to have impact in the form of improved institutions, policy and practice and improved social and environmental conditions. As part of this trend, there is an evolution of research approaches that move beyond isolated, reductionist, disciplinary science toward approaches that integrate disciplines (interdisciplinary) and that engage a wider range of research stakeholders (transdisciplinary) as a way to be more effective. While these approaches evolve, there are good opportunities to learn from the experience of projects that have had impact at some level.
Case Study and Methods
This paper presents lessons from a case-study of a research project that succeeded in crossing the science-policy interface. Our study characterizes the design and implementation of a research project on the influence of timber harvesting on Brazil nut production using transdisciplinary research (TDR) design principles. In addition, the paper empirically assesses project outputs and outcomes in relation to a project theory of change (ToC) based on document review and key informant interviews.
Results
In summary, the Brazil Nut Project included some TDR elements and realized a substantial part of its ToC. The interviews identified mixed perceptions of the research design, implementation and the extent of outcomes achievement from different stakeholder perspectives. Our analysis suggests that limited stakeholder engagement crucially affected perceptions of legitimacy and relevance, the two main TDR principles underpinning the overall research effectiveness in our study. In addition, the application of the TDR analytical framework indicates substantial scope to improve research effectiveness, even without striving for a TDR theoretical ideal.
Evaluating policy-relevant research: Lessons from a series of theory-based outcomes assessments
Citation: Belcher, B., Suryadarma, D., & Halimanjaya, A. (2017). Evaluating policy-relevant research: Lessons from a series of theory-based outcomes assessments. Palgrave Communications, 3: 1-16.
https://doi.org/10.1057/palcomms.2017.17
Abstract
The increasing external demand from research funders and managers to assess, evaluate, and demonstrate the quality and the effectiveness of research is well known. However, less discussed but equally important, is the evolving interest and use of research evaluation to support learning and adaptive management within research programmes. This is especially true in a research-for-development context where research competes with other worthy alternatives for overseas development assistance funding. Moreover, highly complex social, economic and ecological environments add to evaluation challenges. Thus, researchers and research managers need to know whether and how their interventions are working to be able to adapt and improve their programmes and satisfy their funders.
Methods
This paper presents a theory-based research evaluation approach developed and tested on four policy-relevant research activities:
- a long-term forest management research programme in the Congo Basin;
- a large research programme on forests and climate change;
- a multi-country research project on sustainable wetlands management; and
- a research project of the furniture value chain in one district in Indonesia.
The first used Contribution Analysis and the others used purpose-built outcome evaluation approaches that combined concepts and methods from several approaches. Each research evaluation began with documentation of a theory of change (ToC) that identified key actors, processes and results. Meanwhile, we analysed data collected through document reviews, key informant interviews and focus group discussions to test the ToCs against evidence of outcomes in the form of discourse, policy formulation and practice change.
Results
As a result, the approach proved valuable as a learning tool for researchers and research managers. Moreover, it facilitated communication with funders about actual and reasonable research contributions to change. Evaluations employing a participatory approach with project scientists and partners noticeably supported team learning about past work. Furthermore, it supported discussions about possible adaptations for the future. To summarize, in all four cases, the retrospective ToC development proved challenging and resulted in overly-simplistic ToCs.
Conclusion
Evidently, further work to draw on social scientific theories of knowledge translation and policy processes to develop and further test more sophisticated theories of change will be crucial. This theory-based approach to research evaluation provides a valuable means of assessing research effectiveness (summative value) and supports learning and adaptation (formative value) at the project or programme scale. The approach is well suited to the research-for-development projects represented by the case studies, but it should be applicable to any research that aspires to have a societal impact. This article constitutes part of a collection on the future of research assessment.