Impact pathway evaluation: an approach for achieving and attributing impact in complex systems
Introduction
The way people behave, including scientists, is determined to a large extent by their belief structures formed during their education. In science another word for ‘belief structures’ is paradigm, which is defined as: “a set of assumptions, concepts, values, and practices that constitutes a way of viewing reality for the community that shares them” (Houghton Mifflin, 2000).
Professor Niels Röling of Wageningen University believes that the paradigms that people are operating from should be made explicit rather than implicit or tacit (from the Foreword of Douthwaite, 2002). One reason for this, as Thomas Kuhn (1970) points out in his highly influential book The Structure of Scientific Revolutions, is that research is not about discovering the unknown, but rather “a strenuous and devoted attempt to force nature into the conceptual boxes supplied by professional education”. This creates what Rogers (1995) describes as ‘invisible colleges’ of researchers who have similar educational backgrounds, quote each other's work, use each other's methods while remaining largely unaware of what lies outside their ‘college’.
The dominant paradigm within the 16 research centres that constitute the Consultative Group on International Agricultural Research (CGIAR) is positivism (Douthwaite et al. 2001a). The CGIAR system is described by Horton and McKay in this issue. This set of belief structures sees innovation as a rather simple and linear process. Positivism has successfully underpinned the CGIAR's early successes in breeding high yielding rice and wheat varieties that helped catalyze the Green Revolution, and much of the CGIAR's other work. As a result impact assessment in the CGIAR System now largely takes place within an ‘invisible college’ with positivism as the dominant paradigm, and agricultural economics as the dominant discipline. However, emergent understanding of rural development as a social and complex process is leading to positivism giving way to constructivism. Within the CGIAR the constructivist movement is largely housed within Integrated Natural Resource Management (INRM). INRM is defined as “an approach to research that aims at improving livelihoods, agro ecosystem resilience, agricultural productivity and environmental services” (Anon 2002). As such INRM includes Integrated Pest Management which has been one of the pioneers of a constructivist outlook in the CGIAR System. In this paper we show that INRM, and constructivist-based research in general, require different types of evaluation methods to complement existing economic impact assessment methods. These approaches can be found outside agriculture in the field of evaluation.
Section snippets
Why shifting research paradigm changes evaluation requirements
Positivism is associated with “hard” science, that is, science that sets up hypotheses and tests them with repeatable and quantifiable experiments. ‘Hard’ scientists (e.g., most natural scientists and economists) are trained to believe that the world they experience has an independent reality which they are discovering in their experiments. From the repeatability principle follows the idea that knowledge gained in this way is independent from context and separate from the knower, hence
Learning evaluation from others
The idea that impact assessment in the CGIAR system might have much to learn from the field of evaluation is not new. Horton (1998) asks: “Why is cost-benefit practically the only systematic method used? And why have broader program theory and expertise been virtually ignored?” His answer agrees with our analysis. “The answers appear to stem fundamentally from the ‘hard science’ culture of agricultural research organisations.”
In May 2000 the Standing Panel on Impact Assessment (SPIA) of the
Conclusions
Research aimed at sustainably improving rural peoples’ livelihoods is largely based on the constructivist paradigm, which is different to the positivist one that was used as the basis of the research that successfully catalyzed the Green Revolution. Hence, such research is conducted with a different set of assumptions, concepts, values, and practices, that also require a different type of evaluation approach, than the economic ones that represent current ‘best practice’ in the CGIAR system.
Acknowledgements
An earlier version of this paper was presented at the international conference “Why Has Impacts Assessment Research Not Made More of a Difference?”, held 4-7 February, 2002 in San José, Costa Rica. The conference was co-organized by the Standing Panel on Impact Assessment (SPIA) of the Consultative Group for International Agricultural Research (CGIAR) and by the Economics Program of the International Maize and Wheat Improvement Centre (CIMMYT).
References (32)
- et al.
Why promising technologies fail: the neglected role of user innovation during adoption
Research Policy
(2001) - et al.
Science under Scarcity
(1995) - Anon, 2002. Current definition of INRM. 1st March, downloaded from...
- et al.
The social construction of facts and artifacts: or how the sociology of a science and the sociology of technology might benefit each other
Introduction
Enabling Innovation: a Practical Guide to Understanding and Fostering Technological Change
(2002)- Douthwaite, B., de Haan, N.C., Manyong, V., Keatinge, D. 2001a. Blending “hard” and “soft” science: the...
- Funnel, P. 2000. Developing and using a program theory matrix for program evaluation and performance monitoring. In:...
- GAO (General Accounts Office), 1987. Case study evaluation. Program Evaluation and Methodology Division, Transfer Paper...
Disciplinary roots and branches of evaluation: some lessons from agricultural research
Knowledge and Policy
(1998)