Mar 6, 2021
Measuring Coaching, Training, Consulting Intervention Efficacy and Needs
Practicing Empiricism in Learning Interventions
As learning professionals, we coach, train, consult and facilitate our client teams to measure their activities and create feedback loops so that they can adapt and iterate to learn. We call it empiricism or evidenced based management.
It would be hypocrisy if we coaches, trainers, consultants don’t uphold ourselves to the same standards of inspection: we are accountable for measuring the efficacy of the learning interventions we provide.
The following are two frameworks that we use to help ourselves continuously learn from the quality and impact of our work at Agile Organization Development.
Phillips ROI Model
The Phillips ROI model is an extension to the Kirkpatrick model, the original training and educational evaluation model popular since the 1950s. From the four levels (reaction, learning, behavior, results) in the Kirkpatrick model, Jack Phillips extended it to five levels and also added awareness to intangible benefits and chain of impact to the learning intervention’s evaluation.
Level 1: Reaction, Satisfaction
How did the learners like the program? Seek feedback on intervention contents, format, duration, coach/trainer/consultant’s effectiveness etc. Would they be willing to recommend the program to others?
Level 2: Learning
Measure or assess to what extend the learners have learned the desired knowledge, skills, processes, techniques etc. from the training program, or for coaching programs, new knowledge and insights on their strengths and areas of development etc. What are their level of confidence for applying the new knowledge and skills to their jobs?
Level 3: Job Application, Implementation
Measure or assess behavioral changes observed on the job. To what extent are the learners effectively applying their new knowledge, skills and insights to their work? Any barriers or impediments that prevent the application of the learnings observed? Assess to feedback into the learning program for improved knowledge transfer.
Level 4: Business Impact
Identify areas of business improvement and impact created by the learners as a result of applying their learnings from the program. Collect both objective and subjective data as evidence; the former may include output increases (including productivity and sales), quality improvements, time and cost savings, and the latter may include customer satisfaction, employee satisfaction, customer retention etc. (Requires before and after data, and steps to isolate the learning program’s effect from other factors influencing the business impact.)
Level 5: ROI, Cost-Benefit Ratio
Convert the business impact data from the previous level into monetary value and compare it with the cost of the intervention.
Intangible Benefits
Many of the subjective business impact data are hard to convert into monetary values. It is important to raise the awareness of these intangible benefits as they may not be adequately represented in the ROI. For example, improved energy, motivation, commitment, trust and teamwork, reduced conflicts and stress, increased psychological, people more smiling — it would be hard to assign a dollar value to each of these factors, yet when assessed as a chain of impact (see next), they become apparent as major contributors to business impact.
Chain of Impact
While a client statement such as “Definitely the atmosphere has changed and we can really feel the difference!” (correlation) is much appreciated and a valid measure of success (level 1 “reaction”), a demonstration of success in the form of an “As a result of the program…” statement (causation) would be more tangible evidence for assessing the efficacy of the intervention. We can work together with the client in summarizing the “chain of impact” into a story or narrative following the level 1 to 5 impacts. For example:
“Throughout the program we observed [level 1 reaction, satisfaction] among participants and it was impressive how quickly they picked up [level 2 learnings]. The impact was evident in the workplace as we observed increased application of [level 3 job application of new learnings] in both the marketing and sales teams, and that translated into [level 4 business impact] in the three months subsequent to the program, a year-on-year increase of x% which we can attribute mostly to the program because other market factors haven’t changed. As a result of y$ improvement in combined sales increase and customer acquisition cost reduction, we estimate the ROI of the program as z%.”
Intervention Needs Scales Test
While we use the Phillips ROI model for inspecting and adapting our intervention quality and impact throughout the time we are engaged with a client organization, we need a separate tool to continuously assess whether or not the intervention we are providing to a client is producing value and is needed by the client. We consider it our fiduciary duty to inform clients when it is appropriate to pause, conclude or terminate an intervention.
As we found no such tool to assess such condition, we developed one. We call it the Intervention Needs Scale Test:
On the left hand side, we ask the client (sponsor, leader, team) their confidence level of achieving the organizational goal / objective set for the intervention, WITHOUT the help of the intervention team (coaches, consultants).
On the right hand side, we ask the client their Confidence level of achieving the organizational goal / objective WITH the help of the intervention team.
We collect this data as a time series by doing the scales test at the beginning of the intervention and continuously at regular intervals during the intervention.
It may be the case that the without/with confidence level starts against a one year goal at 30%/60%, and then one month into the intervention 40%/70%, three months 50%/80% and 70%/90% at the six month point. At this point we can suggest to the client that we pause the coaches’ intervention so that the client team can attempt to complete the project and achieve the goal by themselves without the help of the coaches. If at the nine month point the confidence level drops to 60%/90%, the coaches can re-engage for support. Or it may be the case that the client completes the project ahead of target at 9 months, i.e. reaches 100%, and we happily conclude the intervention.
Or it may be the case that one month in to the project the confidence level is 40%/60%, and even after looking into why the coaches have not been positive impact to the client team and the coaches adapt to the findings, the confidence level of the client team may be at 50%/60% at the three month point. As the intervention team is not creating incremental value to the client team, at this point it may be in the best interest for the client that we suggest termination of the particular intervention, and explore other means of support.
Transparency, Courage and Commitment
Committing to transparency by continuously measuring ourselves requires courage. At Agile Organization Development, we consider this a hallmark of our professionalism.
