Evaluate

Was it a Success?

There’s no point in conducting lessons that don’t work. But the only way to know if they did, is to evaluate yourself and the training content. There are a few factors to consider in this process.

This grid illustrates the basic Kirkpatrick evaluation structure at a glance.

Level Evaluation type Description & Characteristics Tools & Methods Relevance & Value
1 Reaction Reaction evaluation is how the delegates felt about the training or learning experience. ‘Happy sheets’, feedback forms.

Verbal reaction, post-training surveys or questionnaires.

Quick and very easy to obtain.

Not expensive to gather or to analyze.

2 Learning Learning evaluation is the measurement of the increase in knowledge – before and after. Typically assessments or tests before and after the training.

Interview or observation can also be used.

Relatively simple to set up; clear-cut for quantifiable skills.

Less easy for complex learning.

3 Behavior Behavior evaluation is the extent of applied learning back on the job – implementation. Observation and interview over time are required to assess change, relevance of change, and sustainability of change. Measurement of behavior change typically requires cooperation and skill of line-managers.
4 Results Results evaluation is the effect on the business or environment by the trainee. Measures are already in place via normal management systems and reporting – the challenge is to relate to the trainee. Individually not difficult; unlike whole organization.

Process must attribute clear accountabilities.

1. Trainee Reactions

Did students enjoy themselves? Was it interesting and engaging? Were there things that students feel worked particularly well? That should change? Periodically checking in with students helps build class solidarity and community and keeps you refreshed with new ideas and possibilities.

2. Trainee Learning

Of course, the real measure of success is—did the student actually learn the skills? If we go back to step 2 of the ADDIE process where we designed learning objectives and ways to assess student learning, we should be able to tell if students have learned or not by reviewing the results of their assessments. If we’ve paid careful attention to allowing students to demonstrate learning in concrete ways, then we can be relatively certain that they’ve learned what we wanted them to. If students haven’t learned, then we need to ask ourselves why. What could we do differently the next time? Do we need to address issues of classroom culture? Do we need to try a different strategy? While not every student will be successful, there is always something that we can change in how we teach that can improve learning for more students.

3. Trainee Behavior

Behaviour evaluation is the extent to which the trainees applied the learning and changed their behaviour, and this can be immediately and several months after the training, depending on the situation:

  • Did the trainees put their learning into effect when back on the job?
  • Were the relevant skills and knowledge used
  • Was there noticeable and measurable change in the activity and performance of the trainees when back in their roles?
  • Was the change in behaviour and new level of knowledge sustained?
  • Would the trainee be able to transfer their learning to another person?
  • Is the trainee aware of their change in behaviour, knowledge, skill level?

Observation and interview over time are required to assess change, relevance of change, and sustainability of change. Arbitrary snapshot assessments are not reliable because people change in different ways at different times. Assessments need to be subtle and ongoing, and then transferred to a suitable analysis tool. Assessments need to be designed to reduce subjective judgement of the observer or interviewer, which is a variable factor that can affect reliability and consistency of measurements. The opinion of the trainee, which is a relevant indicator, is also subjective and unreliable, and so needs to be measured in a consistent defined way. 360-degree feedback is useful method and need not be used before training, because respondents can make a judgement as to change after training, and this can be analysed for groups of respondents and trainees. Assessments can be designed around relevant performance scenarios, and specific key performance indicators or criteria. Self-assessment can be useful, using carefully designed criteria and measurements.

4. Business Results

Results evaluation is the effect on the business or environment resulting from the improved performance of the trainee – it is the acid test. Measures would typically be business or organisational key performance indicators, such as: Volumes, values, percentages, timescales, return on investment, and other quantifiable aspects of organizational performance, for instance; numbers of complaints, staff turnover, attrition, failures, wastage, non-compliance, quality ratings, achievement of standards and accreditations, growth, retention, etc.

It is possible that many of these measures are already in place via normal management systems and reporting. The challenge is to identify which and how relate to to the trainee’s input and influence. Therefore it is important to identify and agree accountability and relevance with the trainee at the start of the training, so they understand what is to be measured. This process overlays normal good management practice – it simply needs linking to the training input. Failure to link to training input type and timing will greatly reduce the ease by which results can be attributed to the training. For senior people particularly, annual appraisals and ongoing agreement of key business objectives are integral to measuring business results derived from training.

Individually, results evaluation is not particularly difficult; across an entire organisation it becomes very much more challenging, not least because of the reliance on line-management, and the frequency and scale of changing structures, responsibilities and roles, which complicates the process of attributing clear accountability. Also, external factors greatly affect organisational and business performance, which cloud the true cause of good or poor results.

Globalinx Corp