To analyze the current CMM level of an organization I would conduct a series of tests, interviews and sample for evidence. Based on the results of those I would be able to evaluate an organization’s CMM level with some level of accuracy.
Interviews: the goals of these interviews would be to evaluate both internal and external perceptions on what methodologies and tracking processes are in place for the organization.
- An Executive that is not related to IT
- A Business user not related to IT
- Executive in charge of IT (could be a CIO or Director of IT)
- Manager of Network Operations
- Software Development Manager
- Software Development Project Manager
- Software Development Techical Lead
- Software Developer
- IT Support Desk Rep
Tests: The function of these tests will be to evaluate how competent the users are of their process. These tests may include further interviews asking specific questions about the organizations processes and comparing those to the documentation obtained. These test may also extend into watching users of the process actually enact tasks in the process.
Evidence: It’s one thing to talk the talk. But only with evidence can you prove that the organization is in fact using the process. Evidence will be asked for with little to no warning and will need to be pulled from the system with the evaluator present. The sample set for evidence to be pulled from will include all projects for the last 18 months plus at least 6 completed projects. We will be sampling for evidence that the software process is executed consistently and documented as such. Evidence may include documents, emails, helpdesk tickets, reports derived from any in use Methodware, meeting notes and Groupware archives of scheduled meetings.
Evaluation: Once all of the data is received we would evaluate the results comparing interview notes, tests & evidence against the organizations published process. We would also compare the organizations published and actual process against industry best practices below is what we are looking for to reach a specific CMM level.
1) Initial :
a. The organization has completed at least one project and it has been deployed to production.
b. The project includes at least basic documentation to validate that the application was requested and meet the business users needs
2) Repeatable:
a. The organization has demonstrated at least 60% of their projects use the same basic methodology.
b. The organization has demonstrated that both the internal and external users are aware of the organizations endeded software development methodology and its general process as it affects them
c. The organization has completed at least 3 projects that follow the accepted methodology with minimal deviation without documented explanations for deviation.
3) Defined :
a. Specific process documentation and or Methodware was delivered to the Review team at the beginning of the evaluation.
b. The organization has demonstrated that all project have used the methodology since its approval date or documentation is provided with excepted explanations as to which projects deviated and why.
c. At least 50% of all projects demonstrated execution of the documented process.
d. All users of the process know where documentation is stored in the organizations systems and have a strong knowledge of its methods without referring to the documentation.
e. The documented process has been in effect for at one quarter (3 months)
4) Managed:
a. The documented process has been in effect for at least 4 quarters (one year)
b. Process evidence is primarly provided by a single system rather than a mixture of emails, meeting notes and other ancillary evidence.
c. The organization has reports that demonstrate quantifiable details regarding a projects success or failure. This should include estimates vs. actual for man hours, cost, schedule, administrative overhead, defect tracking, customer feedback.
d. All current projects are presented in the organizations tracking systems and are accurate to within a reporting period.
e. At least 5 now complete projects used the tracking system from kickoff to completion of the project
f. All users of the tracking system have logins to the system.
5) Optimized:
a. The documented process has been in effect for the entire sampling period
b. Process evidence is provided solely from the organizations tracking system.
c. Evidence is provided that the organizations process has been evaluated in accordance with IT policies.
d. All users of the tracking system demonstrate consistent and repeated use of the system.
e. The tracking system has enough data to provide baseline reports for comparison.
f. The process includes a lessons learned and these are archived in the tracking system for every closed project
g. Evidence is proved that lessons learned from all closed projects have feed into validate-able changes to the organizations processes (if applicable).
h. Tracking System metrics & reports are ties to individual and organizational reviews not just process reviews.
Comparing the tests results to the above level qualifications would allow us to know which phase an organization is currently in and how close they are to the next phase of CMM.
So ya, I’ve been thru a few SOX audits (4 to be exact) so I am leaning a lot on how they test our IT department to make sure that we were meeting Sarbanes-Oxley controls as they relate to IT (and there are a surprising number of them that do!).
Whitten, Jeffery L., & Bentley, Lonnie D. (2008). Introduction to System Analysis and Design.New York: McGraw-Hill Irwin.