Abstract
Industrial evaluations of COTS software largely used the quality models provided by the international standards. But the context and objectives of COTS evaluations are fundamentally different than those primarily defined by the standards. Several key issues are often forgotten: (1) the existence of several evaluators and several quality models sharing common factors, criteria and measures, (2) the purpose of the evaluation model, (3) measures of different types, and (4) the recursive nature of the model since each node is an evaluation model itself. We had the occasion to study the results of real standard-based COTS evaluations. Faced with the difficulties to exploit them, we experimented the use of multi-criteria methodology. This work allows us to understand some of the problems generated by the application of the standards to COTS evaluations, and to propose new principles for evaluating software quality that should be considered in an evolution of the standards. This paper reports our experiment.
Similar content being viewed by others
References
Anderson, E.E. and Chen, Y.-M. 1997. Microcomputer software evaluation: An economic model, Decision Support Systems 19, 75-92.
Arrow, K. 1951. Social Choice and Individual Values, New York, Wiley.
Blin, M.J., and Tsoukiàs, A. 1999. Evaluation of COTS using multi-criteria methodology, Proc. 6th European Conf. Software Quality, pp. 429-438.
Boehm, B.W. 1978. Characteristics of Software Quality, Amsterdam, North Holland Publishing Company.
Bouyssou, D., Marchant, Th., Perny, P., Pirlot, M., Tsoukiàs, A., Vincke, Ph. 2000. Evaluation and Decision Models: A Critical Perspective, Kluwer Academic, Dordrecht.
Erikkson, I. and McFadden, F. 1993. Quality function deployment: a tool to improve software quality, Information and Software Technology 35(9): 491-498.
Fenton, N. and Schneidewind, N.F., 1996. Do Standards Improve Quality? IEEE Software January: 22-24.
Feuk, N., Whitty, R., and Ilruka, Y. 1995. Applying the Goal/Question Metric Paradigm in the Experience Factory, International Thomson Computer Press, London, pp. 23-44.
]The Institute of Electrical and Electronics Engineers. 1992. Standard for a Software Quality Metrics Methodology, December.
International Organization for Standardization, 1991. ISO 9126: Information Technology—Software Product Evaluation—Quality Characteristics and Guidelines for Their Use.
Keeney, R. and Raiffa, H. 1976. Decision with Multiple Objectives: Preferences and Value Trade-Offs, New York, Wiley.
Kitchenham, B., and Pfleeger, S.L. 1996. Software quality: The elusive target, IEEE Software January: pp. 12-21.
Le Blanc, L., and Jelassi, T. 1994. An empirical assessment of choice models for software selection: a comparison of the LWA and MAUT techniques, Revue des Systèmes de Décision 3(2): 115-126.
MacCall, J.A. 1977. Factors in software quality, J. General Electric C15-OL (77).
Meskens, N. 1994. A knowledge-based system for measuring the quality of existing software, Revue des Systèmes de Décision 3(3): 201-220.
Morisio, M. and Tsoukiàs, A. 1997. IusWare: a methodology for the evaluation and selection of software products, IEE Software Eng. 144: 162-174.
Paschetta, E. and Tsoukiàs, A. 1999. A real world MCDA application: evaluating software, J. Multi-Criteria Decision Analysis 9: 205-226.
Paulussen, R.M.C. 1995. The Quint Approach to the Speci.cation of Software Quality, 1st World Congress for Software Quality, San Francisco, June 20–22, 1995.
Roberts, F.S., 1979. Measurement Theory with Applications to Decision Making, Utility and the Social Sciences, New York, Addison-Wesley.
Roy, B. 1991. The outranking approach and the foundations of ELECTRE methods, Theory and Decision 31: 49-73.
Roy, B. 1996. Multicriteria Methodology for Decision Aiding, Dordrecht, Kluwer Academic Publishers.
Stamelos, J. and Tsoukiàs, A. 1998. Software evaluation problem situations, cahier du LAMSADE No 156, Université Paris Dauphine, submitted.
Verner, J., Moores, T., and Barrett, A.R., 1996. Software quality: perceptions and practices in Hong Kong, Working Paper Series, City University of Hong Kong, Faculty of Business Department of Informations Systems, WP96/02, April, 1996.
Vincke, Ph. 1992. Exploitation of a crisp relation in a ranking problem, Theory and Decision 32: 221-240.
Von Winterfeldt, D. and Edwards, W. 1986. Decision Analysis and Behavioral Research, Cambridge, MA, Cambridge University Press.
Author information
Authors and Affiliations
Rights and permissions
About this article
Cite this article
Blin, MJ., Tsoukiàs, A. Multi-Criteria Methodology Contribution to the Software Quality Evaluation. Software Quality Journal 9, 113–132 (2001). https://doi.org/10.1023/A:1016626919680
Issue Date:
DOI: https://doi.org/10.1023/A:1016626919680