Skip to main content
Log in

Multi-Criteria Methodology Contribution to the Software Quality Evaluation

  • Published:
Software Quality Journal Aims and scope Submit manuscript

Abstract

Industrial evaluations of COTS software largely used the quality models provided by the international standards. But the context and objectives of COTS evaluations are fundamentally different than those primarily defined by the standards. Several key issues are often forgotten: (1) the existence of several evaluators and several quality models sharing common factors, criteria and measures, (2) the purpose of the evaluation model, (3) measures of different types, and (4) the recursive nature of the model since each node is an evaluation model itself. We had the occasion to study the results of real standard-based COTS evaluations. Faced with the difficulties to exploit them, we experimented the use of multi-criteria methodology. This work allows us to understand some of the problems generated by the application of the standards to COTS evaluations, and to propose new principles for evaluating software quality that should be considered in an evolution of the standards. This paper reports our experiment.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Anderson, E.E. and Chen, Y.-M. 1997. Microcomputer software evaluation: An economic model, Decision Support Systems 19, 75-92.

    Google Scholar 

  • Arrow, K. 1951. Social Choice and Individual Values, New York, Wiley.

    Google Scholar 

  • Blin, M.J., and Tsoukiàs, A. 1999. Evaluation of COTS using multi-criteria methodology, Proc. 6th European Conf. Software Quality, pp. 429-438.

  • Boehm, B.W. 1978. Characteristics of Software Quality, Amsterdam, North Holland Publishing Company.

    Google Scholar 

  • Bouyssou, D., Marchant, Th., Perny, P., Pirlot, M., Tsoukiàs, A., Vincke, Ph. 2000. Evaluation and Decision Models: A Critical Perspective, Kluwer Academic, Dordrecht.

    Google Scholar 

  • Erikkson, I. and McFadden, F. 1993. Quality function deployment: a tool to improve software quality, Information and Software Technology 35(9): 491-498.

    Google Scholar 

  • Fenton, N. and Schneidewind, N.F., 1996. Do Standards Improve Quality? IEEE Software January: 22-24.

  • Feuk, N., Whitty, R., and Ilruka, Y. 1995. Applying the Goal/Question Metric Paradigm in the Experience Factory, International Thomson Computer Press, London, pp. 23-44.

    Google Scholar 

  • ]The Institute of Electrical and Electronics Engineers. 1992. Standard for a Software Quality Metrics Methodology, December.

  • International Organization for Standardization, 1991. ISO 9126: Information TechnologySoftware Product EvaluationQuality Characteristics and Guidelines for Their Use.

  • Keeney, R. and Raiffa, H. 1976. Decision with Multiple Objectives: Preferences and Value Trade-Offs, New York, Wiley.

    Google Scholar 

  • Kitchenham, B., and Pfleeger, S.L. 1996. Software quality: The elusive target, IEEE Software January: pp. 12-21.

  • Le Blanc, L., and Jelassi, T. 1994. An empirical assessment of choice models for software selection: a comparison of the LWA and MAUT techniques, Revue des Systèmes de Décision 3(2): 115-126.

    Google Scholar 

  • MacCall, J.A. 1977. Factors in software quality, J. General Electric C15-OL (77).

  • Meskens, N. 1994. A knowledge-based system for measuring the quality of existing software, Revue des Systèmes de Décision 3(3): 201-220.

    Google Scholar 

  • Morisio, M. and Tsoukiàs, A. 1997. IusWare: a methodology for the evaluation and selection of software products, IEE Software Eng. 144: 162-174.

    Google Scholar 

  • Paschetta, E. and Tsoukiàs, A. 1999. A real world MCDA application: evaluating software, J. Multi-Criteria Decision Analysis 9: 205-226.

    Google Scholar 

  • Paulussen, R.M.C. 1995. The Quint Approach to the Speci.cation of Software Quality, 1st World Congress for Software Quality, San Francisco, June 20–22, 1995.

  • Roberts, F.S., 1979. Measurement Theory with Applications to Decision Making, Utility and the Social Sciences, New York, Addison-Wesley.

    Google Scholar 

  • Roy, B. 1991. The outranking approach and the foundations of ELECTRE methods, Theory and Decision 31: 49-73.

    Google Scholar 

  • Roy, B. 1996. Multicriteria Methodology for Decision Aiding, Dordrecht, Kluwer Academic Publishers.

    Google Scholar 

  • Stamelos, J. and Tsoukiàs, A. 1998. Software evaluation problem situations, cahier du LAMSADE No 156, Université Paris Dauphine, submitted.

  • Verner, J., Moores, T., and Barrett, A.R., 1996. Software quality: perceptions and practices in Hong Kong, Working Paper Series, City University of Hong Kong, Faculty of Business Department of Informations Systems, WP96/02, April, 1996.

  • Vincke, Ph. 1992. Exploitation of a crisp relation in a ranking problem, Theory and Decision 32: 221-240.

    Google Scholar 

  • Von Winterfeldt, D. and Edwards, W. 1986. Decision Analysis and Behavioral Research, Cambridge, MA, Cambridge University Press.

    Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Rights and permissions

Reprints and permissions

About this article

Cite this article

Blin, MJ., Tsoukiàs, A. Multi-Criteria Methodology Contribution to the Software Quality Evaluation. Software Quality Journal 9, 113–132 (2001). https://doi.org/10.1023/A:1016626919680

Download citation

  • Issue Date:

  • DOI: https://doi.org/10.1023/A:1016626919680

Navigation