Using Return on Investment to Evaluate Project Management Training

Using Return on Investment to Evaluate Project Management Training
Gina Westcott-Abudi, MBA

Abstract

Executives today are demanding to see the monetary value that project management training programs bring to the organization. This means they need to see a dollar value placed on the benefits of the training, not just that participants believe the training to be valuable to them or they enjoyed the training overall. They want proof that the training program benefited the organization in a positive manner. This paper examines the use of Return on Investment (ROI) to evaluate project management training programs. It provides a background on ROI, along with a description of the Phillips ROI Methodology™ and takes the reader through a process of evaluating a project management training program.

Introduction

Return on Investment (ROI) is a monetary measurement that is used to evaluate the efficiency and effectiveness of an investment made by an organization. Investments take many forms – financial, human capital, equipment, and training programs – to name just a few. This paper will focus on the use of ROI to measure the effectiveness of project management training programs in particular. In the 1970’s, Jack Phillips of the ROI Institute, Inc. developed a Level 5 ROI evaluation, as a follow on to the Kirkpatrick Levels 1 – 4 Evaluation Model (Kirkpatrick and Kirkpatrick, 2005) for the measurement of training programs. Phillips’ Level 5 takes the measurement of the effectiveness of training program to a higher level – converting the benefits of the training to a monetary value, thereby demonstrating its value to the bottom line. The 5 levels of evaluation (Donald Kirkpatrick, 1959 Levels 1 – 4; Jack Phillips, 1970 Level 5) include:

• Level 1: Reactions (the “smiley” sheet)
  • Did participants like the training they received?
• Level 2: Learning
  • Are participants confident that they have learned something from the training program?
• Level 3: Behavior/Application
  • Are participants able to apply what they learned in the training program back on the job?
• Level 4: Results/Business Impact
  • Did the training show improvement in efficiencies, productivity, profits, costs, reduced turnover?
• Level 5: ROI
  • Did the training program show a positive ROI?

According to Training Magazine (February 2006), aligning the training department, and therefore the training programs offered within an organization, with business goals was a top priority for 2006. This alignment is becoming more of a priority for increasing numbers of organizations each year.

Why the Interest in ROI for Project Management Training Programs?

The challenges surrounding training have become more complex over the years. In the late 1990s the challenge was to ensure that participants found training interesting and they were able to learn new skills. In early 2000, the challenges became greater and trainers were asked to show that participants in training classes actually learned new skills – they needed to be able to apply that training back on the job. (Kirkpatrick and Kirkpatrick, 2005, p. 10 – 11)

Today, the challenge is even more significant for learning and development professionals. Return on Investment (ROI) as a tool for evaluating project management training is becoming an expectation of senior executives within organizations. In today’s tight economy with reduced resources and tighter budgets, learning and development professionals are finding it increasingly necessary to show the monetary value of the project management programs they are bringing to the organization. Today, the success of project management training programs is measured by the financial contribution of the program to the organization.

There are only a handful of processes in organizations that have grown as much as project management has over the past couple of decades. (Phillips and Phillips, 2007, p. 7) According to the Project Management Institute, as of May 31, 2008, there are 278,618 active Project Management Professionals (PMP^®s). (PMI Today, July 2008, p. 10) It is not surprising then that ROI measurements for project management training programs are often requested. With such a large focus on project management in organizations, there is an increasing desire to show the monetary benefit of investing in project management training programs which often reach a wide variety of employees and represent a significant investment for organizations.

Phillips ROI Methodology™

The ROI Methodology™ was developed by Jack Phillips in the 1970’s, refined through application and use in the 1980’s, and implemented globally during the 1990’s. The first impact study was done in 1973 for Lockheed-Martin. (Phillips, Phillips, Stone and Burkett, 2007, p.12)

The Phillips ROI Methodology™ encompasses four major phases:

1. Evaluation Planning
2. Data Collection
3. Data Analysis
4. Reporting

See Exhibit 1, Phillips ROI Methodology™.

ROI Process

Exhibit 1 – Phillips ROI Methodology™, ©ROI Institute, Inc.

When Should You Use Level 4 (Business Impact) and Level 5 (ROI) Evaluation?

Level 4 (business impact) and Level 5 (calculating ROI) evaluation is not used for every training program within the organization. Only those training programs that meet the following criteria should be followed through to Levels 4 and 5 evaluations. Attempting to complete such high level evaluations on every training program within the organization would begin to get quite costly and time consuming. (Phillips and Stone, 2002, p. 29 – 30):

• A long program life cycle
• Linkage of the program to organizational goals and strategic objectives
• Executive interest in the evaluation
• High cost programs
• Programs with visibility throughout the organization
• Programs with a sizeable target audience

As can be seen from the criteria listed above, not all programs are candidates for Level 5 evaluation. Only 5 – 10% of an organization’s training programs are evaluated at Level 5. (Phillips, Phillips, Stone, and Burkett, 2007, p. 13)

The following graphic (Exhibit 2) shows evaluation targets for each level of evaluation. The higher the level a program is evaluated, the more valuable the information gathered and analyzed. Assessments become more difficult to accomplish at higher levels, but the impact to the organization is certainly greater. The more strategic a program within the organization, the more likely a company will want to evaluate it at a higher level.

Exhibit 2 – Setting Evaluation Targets, ©ROI Institute, Inc.

Using the ROI Methodology™ to Evaluate Project Management Training Programs

The focus of this section is to provide a general overview of the use of the ROI Methodology™ specifically around the evaluation of a project management training program. The following assumptions are made for the purposes of this example:

• This is a new project management training program – one which is not already in place within the organization – which, if successful, will be rolled out to the rest of the organization
• There will be a pilot group for the initial project management training program of no more than 20 individuals
• The reason for project management training is to improve the project management technical skills within the organization in order to ensure that projects are on time, within budget and formal processes are followed throughout the organization.

This example follows the Phillips ROI Methodology™ shown above in Exhibit 1. It is provided from a high level overview and does not get into all the details.

Phase 1: Evaluation Planning

Phase 1 is a large component of the ROI Methodology™ process and entails a great deal of project planning. The first step in this phase is to develop the objectives of the project management training program (the “solution”). Think through:

• What is the purpose of the project management training?
  • What problem(s) are you trying to solve?
    • Ensure projects are delivered on time
    • Ensure projects are delivered within budget
    • Develop more efficient processes

Secondly, begin to develop your evaluation plans and determine your baseline data. Develop your data collection plan, ROI analysis plan, and your project plan for moving forward with the evaluation.

• Determine the purpose of the evaluation of the project management training program. This will help you determine what data needs to be collected and reported back to the executives. In this example, the purpose of the evaluation of the project management training program is to:
  • Ensure that participants are learning the skills necessary to have an impact on the business, specifically around:
    • Delivery of projects
    • Adherence to budgets
    • Increased efficiencies
• Based on the information you determined above, determine what data you need to collect now (baseline) to measure against later (after the training program is completed). For this example, data may include:
  • The number of projects completed in a given year and the number of those projects which are delivered on time, delivered late, not delivered, etc.
  • The number of project completed in a given year and the number of projects which are over budget, within budget, under budget
  • Current processes in place and identification of gaps within processes
• Once you have determined the purpose of the project management training program and the data points you want to collect and measure, determine how you are going to measure at levels 1 – 4.
  • Complete a data collection plan that includes information on the objectives to be measured at each level, what is being measured, the instruments used to collect the data (e.g., ERP system, PMO tracked statistics), the source to collect data (e.g., PMO), the timeline for data collection and who has responsibility for getting that data collected.
• Develop your ROI analysis plan based on the data items determined for level 4 – business impact.
  • Complete a ROI analysis plan that includes information on the level 4 objectives – data items, how you will isolate the effects of the project management training program, what methods you will use to convert data to monetary values, the costs categories to be used for capturing the costs of the project management training program, what intangible benefits are expected/will be measured, and to whom you will communicate the final results.
  • Set your ROI target (what percent of ROI are you hoping to achieve) based on the following options (Phillips and Stone, 2002, p. 29)
    • As with other investments, e.g., 15%
    • Slightly above other investments, e.g., 25%
    • At breakeven – 0%
    • At client expectations
• Develop your project plan – this is a project and should be treated as such. Develop your WBS, assign resources, determine start and end dates and manage your ROI study just as you would any other project.

Phase 2: Data Collection

During phase 2, you are collecting two sets of data; the first just after the project management training program has been completed and the second, a period of time after the individuals involved in the program have had the opportunity to apply their skills and you have had the opportunity to measure the impact on the business. Collect data at level 1 through level 4, with level 1 and level 2 data collected immediately upon the end of the project management training program and level 3 and 4 collected sometime after the program has ended. For the purposes of this example, let’s assume that the 20 project managers who participated in the training will immediately be assigned projects at the completion of the training and each project is expected to be completed within a 3 month time period. We will be able to see the impact on the business within 3 months after the projects have been completed.

Collect your data based on the data collection methods you determined you would use during your phase 1 planning stage.

Phase 3: Data Analysis

During this phase, the data collected from phase 2 is processed and analyzed and the effects of the project management training program, using the method we determined when we completed the data collection plan, are isolated (phase 1).

It is desirable to isolate the effects of a program to ensure credibility in the ROI process. In this example, it is unlikely that only the project management training will be responsible for all improvement that is recognized. Let’s assume, for this example, that in addition to the project management training program, a system is being put in place by the application development/IT department to better track projects in progress. This would certainly have an impact on the results – specifically – it will help with tracking timelines and the project budget. It should be estimated how much this system impacts the business and those results should be isolated from the project management training. Additionally, in this example, we have a pilot group going through the training and this will function as a control group for this study. The control group provides an isolation method for the ROI study.

During phase 3, we convert our data to a monetary value, using the methods listed in the ROI analysis plan (phase 1). There is likely to be some intangible (“soft”) data that we will want to capture as well. In this example, let’s assume our identified intangibles include: less stress, increased satisfaction with their roles, and project managers being more proactive and less reactive. We want to capture the costs of our project management training program solution, along with the costs of the ROI study. All costs (fully loaded) should be included in the analysis. For example, for the time participants are out of the workplace and in the training program, use their daily salary including the company fringe benefit rate for purposes of calculating all costs of the program.

During phase 3, we calculate the ROI of the project management training program, using the ROI formula:

ROI = Net Program Benefits / Program Costs x 100

As an example, let’s assume that the benefits of the project management training program, isolated from other factors occurring within the organization as described above, are $350,000. Let’s also assume that the total cost of the program is $125,000. The net program benefits are $350,000 – $125,000 = $225,000. Following the formula above:

ROI = $225,000/$125,000 x 100

ROI = 180%; thus for every $1.00 invested, $1.80 is returned after the costs of the project management training program are recovered.

Phase 4: Reporting

During phase 4, the results of the ROI study will be reported to the executives (you determined who will receive your report when you developed your ROI analysis plan in phase 1). The report should be a detailed report including details on the process used to analyze the program along with the data collected and analyzed. Plan for this communication just as you would when communicating on other projects to stakeholders.

Summary

For training and learning departments, the PMO, or lines of business, showing the ROI of the project management training programs conducted within the organization, whether in-house developed programs or vendor-provided programs, enables you to show the value the programs have on the business in monetary terms that executives can understand, expect, and champion.

The six types of data (Phillips and Phillips, 2007, p. 5) collected during the ROI Methodology ™ process (see list below) enable for consistency in the measurement of project management training programs, thereby bringing credibility to the process.

To begin measuring your project management training programs, you might start with a brand new program or review the current programs that are in place – following the criteria for measuring project management training programs discussed earlier in this paper. If you review a current program, gather data on the initial purpose of the program when it was first developed and implemented. Does it still meet that purpose? What value do you perceive the individuals taking the project management training program are achieving? How much has been invested in the program since the beginning? Should this program remain in place “as is,” or should it be updated, or possibly be eliminated from your offerings in its entirety or in part to accommodate new needs among your project managers? An ROI study will enable you to look at the value of your current programs and enable you to better plan new programs.

References

Hall, B. (2006, February) Missing Link? Training Magazine, 22

Kirkpatrick, D. and Kirkpatrick, J (2005) Transferring Learning to Behavior, San Francisco: Berrett-Koehler Publishers, Inc.

Phillips, J. and Phillips, P. (2007) Show Me the Money: How to Determine ROI in People, Projects and Programs, San Francisco: Berrett-Koehler Publishers, Inc.

Phillips, J., Phillips, P., Stone, R., and Burkett, H. (2007) The ROI Field Book: Strategies for Implementing ROI in HR and Training, Massachusetts: Butterworth-Heinemann/Elsevier

Phillips, J. and Stone, R., (2002) How to Measure Training Results: A Practical Guide to Tracking the Six Key Indicators, New York: McGraw-Hill

PMI Today, Supplement to PM Network (July 2008) PMI Fact File, 10

©2008, Gina Abudi, MBA Originally published as part of 2008 PMI Global Congress Proceedings – Denver, Colorado, USA