Elsevier

Agricultural Systems

Volume 78, Issue 2, November 2003, Pages 243-265
Agricultural Systems

Impact pathway evaluation: an approach for achieving and attributing impact in complex systems

https://doi.org/10.1016/S0308-521X(03)00128-8Get rights and content

Abstract

Agricultural development is fundamentally a social process in which people construct solutions to their problems, often by modifying both new technologies and their own production systems to take advantage of new opportunities offered by the technologies. Hence, agricultural change is an immensely complex process, with a high degree of non-linearity. However, current ‘best practice’ economic evaluation methods commonly used in the CGIAR system ignore complexity. In this paper we develop a two-stage monitoring, evaluation and impact assessment approach called impact pathway evaluation. This approach is based on program-theory evaluation from the field of evaluation, and the experience of the German development organization GTZ (Deutsche Gesellschaft für Technische Zusammenarbeit GmbH). In the first stage of this approach, a research project develops an impact pathway for itself, which is an explicit theory or model of how the project sees itself achieving impact. The project then uses the impact pathway to guide project management in complex environments. The impact pathway may evolve, based on learning over time. The second stage is an ex post impact assessment sometime after the project has finished, in which the project's wider benefits are independently assessed. The evaluator seeks to establish plausible links between the project outputs and developmental changes, such as poverty alleviation. We illustrate the usefulness of impact pathway evaluation through examples from Nigeria and Indonesia.

Introduction

The way people behave, including scientists, is determined to a large extent by their belief structures formed during their education. In science another word for ‘belief structures’ is paradigm, which is defined as: “a set of assumptions, concepts, values, and practices that constitutes a way of viewing reality for the community that shares them” (Houghton Mifflin, 2000).

Professor Niels Röling of Wageningen University believes that the paradigms that people are operating from should be made explicit rather than implicit or tacit (from the Foreword of Douthwaite, 2002). One reason for this, as Thomas Kuhn (1970) points out in his highly influential book The Structure of Scientific Revolutions, is that research is not about discovering the unknown, but rather “a strenuous and devoted attempt to force nature into the conceptual boxes supplied by professional education”. This creates what Rogers (1995) describes as ‘invisible colleges’ of researchers who have similar educational backgrounds, quote each other's work, use each other's methods while remaining largely unaware of what lies outside their ‘college’.

The dominant paradigm within the 16 research centres that constitute the Consultative Group on International Agricultural Research (CGIAR) is positivism (Douthwaite et al. 2001a). The CGIAR system is described by Horton and McKay in this issue. This set of belief structures sees innovation as a rather simple and linear process. Positivism has successfully underpinned the CGIAR's early successes in breeding high yielding rice and wheat varieties that helped catalyze the Green Revolution, and much of the CGIAR's other work. As a result impact assessment in the CGIAR System now largely takes place within an ‘invisible college’ with positivism as the dominant paradigm, and agricultural economics as the dominant discipline. However, emergent understanding of rural development as a social and complex process is leading to positivism giving way to constructivism. Within the CGIAR the constructivist movement is largely housed within Integrated Natural Resource Management (INRM). INRM is defined as “an approach to research that aims at improving livelihoods, agro ecosystem resilience, agricultural productivity and environmental services” (Anon 2002). As such INRM includes Integrated Pest Management which has been one of the pioneers of a constructivist outlook in the CGIAR System. In this paper we show that INRM, and constructivist-based research in general, require different types of evaluation methods to complement existing economic impact assessment methods. These approaches can be found outside agriculture in the field of evaluation.

Section snippets

Why shifting research paradigm changes evaluation requirements

Positivism is associated with “hard” science, that is, science that sets up hypotheses and tests them with repeatable and quantifiable experiments. ‘Hard’ scientists (e.g., most natural scientists and economists) are trained to believe that the world they experience has an independent reality which they are discovering in their experiments. From the repeatability principle follows the idea that knowledge gained in this way is independent from context and separate from the knower, hence

Learning evaluation from others

The idea that impact assessment in the CGIAR system might have much to learn from the field of evaluation is not new. Horton (1998) asks: “Why is cost-benefit practically the only systematic method used? And why have broader program theory and expertise been virtually ignored?” His answer agrees with our analysis. “The answers appear to stem fundamentally from the ‘hard science’ culture of agricultural research organisations.”

In May 2000 the Standing Panel on Impact Assessment (SPIA) of the

Conclusions

Research aimed at sustainably improving rural peoples’ livelihoods is largely based on the constructivist paradigm, which is different to the positivist one that was used as the basis of the research that successfully catalyzed the Green Revolution. Hence, such research is conducted with a different set of assumptions, concepts, values, and practices, that also require a different type of evaluation approach, than the economic ones that represent current ‘best practice’ in the CGIAR system.

Acknowledgements

An earlier version of this paper was presented at the international conference “Why Has Impacts Assessment Research Not Made More of a Difference?”, held 4-7 February, 2002 in San José, Costa Rica. The conference was co-organized by the Standing Panel on Impact Assessment (SPIA) of the Consultative Group for International Agricultural Research (CGIAR) and by the Economics Program of the International Maize and Wheat Improvement Centre (CIMMYT).

References (32)

  • B.J. Douthwaite et al.

    Why promising technologies fail: the neglected role of user innovation during adoption

    Research Policy

    (2001)
  • J.M. Alston et al.

    Science under Scarcity

    (1995)
  • Anon, 2002. Current definition of INRM. 1st March, downloaded from...
  • W. Bijker et al.

    The social construction of facts and artifacts: or how the sociology of a science and the sociology of technology might benefit each other

  • M.P. Collinson

    Introduction

  • B. Douthwaite

    Enabling Innovation: a Practical Guide to Understanding and Fostering Technological Change

    (2002)
  • Douthwaite, B., de Haan, N.C., Manyong, V., Keatinge, D. 2001a. Blending “hard” and “soft” science: the...
  • Funnel, P. 2000. Developing and using a program theory matrix for program evaluation and performance monitoring. In:...
  • GAO (General Accounts Office), 1987. Case study evaluation. Program Evaluation and Methodology Division, Transfer Paper...
  • D. Horton

    Disciplinary roots and branches of evaluation: some lessons from agricultural research

    Knowledge and Policy

    (1998)
  • Houghton Mifflin Company 2000. The American Heritage Dictionary of the English Language (on-line). fouth ed. Downloaded...
  • IIRR (International Institute of Rural Reconstruction). 2000. Going to scale. Can we bring more benefits to more people...
  • Johnson, Nancy, van de Fliert, Elske, Lilja, Nina, 2001. The CIP development of integrated crop management (ICM)...
  • Kuby, T., 1999. Innovation as a Social Process: What Does this Mean for Impact Assessment in Agricultural Research....
  • Kuby, T. 2000. Turning Attention Towards Results: how GTZ is Building its Impact Evaluation Capacity. Internal GTZ...
  • T.S. Kuhn

    The Structure of Scientific Revolutions

    (1970)
  • Cited by (0)

    View full text