Isabel Lopes Margarido

EQualPI: a Framework to Evaluate the Quality of the Implementation of the CMMI Practices

Thesis document at the reporsitory of Universidade do Porto,

Abstract

The Capability Maturity Model Integration® (CMMI) allows organisations to improve the quality of their products and customer satisfaction; reduce cost, schedule, and rework; and make their processes more predictable. However, this is not always the case, as there are differences in performance between CMMI organisations, depending not only on the context of the business, projects, and team, but also on the methodologies used in the implementation of the model practices. CMMI version 1.3 is more focused on the performance of the organisations than previous versions. However, the Standard CMMI Appraisal Method for Process Improvement℠ (SCAMPI) is not focused on evaluating performance.

To evaluate practices performance it is necessary to consider the goal of executing the practice, and the quality of implementation of a practice is reflected in its outputs. Therefore, if we can establish a relationship between the methods used to implement a practice and the performance of its results, we can use such relationship in a framework to evaluate the quality of implementation of the practice. We consider that it is possible to objectively measure the quality of implementation of CMMI practices by applying statistical methods in the analysis of organisations’ data, in order to evaluate process improvement initiatives and predict their impact on organisational performance.

In this research we develop a framework to evaluate the quality of the implementation of the CMMI practices that supports the comparison of the quality of the implementation before and after improvements are put in place. Considering the extent of the CMMI model, we demonstrate the framework in the Project Planning’s Specific Practice 1.4 "Estimate Effort and Cost". We consider that the quality of implementation of this practice is measured by the Effort Estimation Accuracy, defined by a set of controllable and uncontrollable factors, and it can be improved by acting on the controllable factors. To implement and validate our framework we conducted literature reviews, case studies on high maturity organisations, data analysis of a survey performed by the Software Engineering Institute (SEI) and on the Team Software Process℠ (TSP) Database, which we used to build a regression model, and conducted an experiment with students to define a process improvement.

This Ph.D. thesis provides to software development organisations a framework for self-assessing the quality of the implementation of the CMMI practices, EQualPI. The framework is also useful to the CMMI Institute, in order to evaluate the performance of the organisations from one SCAMPI A to the next. The framework is already populated with recommendations to support organisations willing to implement CMMI to avoid a set of problems and difficulties, factors to consider when implementing Measurement and Analysis for CMMI high maturity levels, a procedure based on the scientific method to conduct process improvements, a performance indicator model to evaluate the quality of implementation of the effort estimation process, and indicators related with effort estimation accuracy. Additionally, with the implementation and validation of the EQualPI framework, we provide the procedure we used to analyse data from the SEI TSP Database and define process variables, and by applying the process improvements procedure, we contribute with a defects classification specific for requirements.

Thesis structure

• Chapter 2 Fundamental Concepts - presents the concepts necessary to understand this research and the remaining chapters of the thesis
• Chapter 3 Background and Related Work - provides the necessary information to delimit the problem and the contributions of other researchers to help solve some of the problem components. We present our contribution to solve open points identified on prior research
• Chapter 4 The EQualPI Framework - is the core of the research, it details the framework to evaluate the quality of implementation of the CMMI practices and how organisations can use it
• Chapter 5 EQualPI Validation - is the Framework validation
• Chapter 6 Conclusions - guides the reader as to how the framework is extended to other practices, indicates our achievements and their impact in the problem resolution, and defines the boundaries of this research. We leave the research open and point to directions for future work that needs to be done in this area