(Medical
Simulation
TRaining TEchnology
EvAluation
DesIgner)
Formative and Summative
Evaluation Methods
Formative Evaluations
Ideally, training evaluation should be conducted during three critical
stages of the
Simulation Research & Development Process.
The evaluation goal during the
Design and
Development
stages is largely
Formative — to
provide feedback to simulation and instructional designers intended to
improve the training product while it is still under development. After the
project has started and the requirements analysis has been conducted by
the development team, it can be evaluated to ensure the design
objectives are
well defined, that they meet a legitimate need, and that assessment
instruments are matched to those objectives. Once the simulation
design has been
storyboarded an evaluation of both the basic user-simulation interface and the
instructional strategy can be evaluated with additional reviews and
early feedback provided. Upon implementation of a working prototype
there are several methods that can be applied to evaluate the now more
realistic representation of the target simulation based training system.
In this video, Lloyd
Rieber, Instructional Technologist and Professor of
Learning, Design & Technology at The University of Georgia,
overviews Formative Evaluation for one of his classes and
highlights how at the formative stage it is
questions of learning effectiveness, learning efficiency,
motivation, and usability that are considered
to help improve the design,
the importance of involving
the training audience early in the design process,
and that matching objectives to
assessment instruments early in the development
process is recommended.
It isn't
as simple as counting eggs!
Summative Evaluations
The
evaluation goal for an end product or delivered simulation is
Summative—to
determine if the training had it’s intended impact. A
Simulation Outcomes & Impact Evaluation examining the
effect of the final
Simulation on
training outcomes and organizational effects should occur once the
simulation is fully developed. Note: in some cases it may also be
useful to apply summative evaluation methods while still at the
prototype stage. For instance, when the prototype has an
innovative component to it and an experimental approach is desired to be
confident what the impact is of that innovation, or if the prototype is
sufficiently advanced and it is not practical due to timing/resource
constraints to complete a summative evaluation after final delivery.
Any
of the following commonly used TEE methods can potentially be used, adapted, and
combined. This table provide brief descriptions for each method
and when it tends to be applied in the simulation development process:
Development Stage
Purpose
Method
Description
Design
Formative
Instructional Design Review
An inspection method
where categorized true/false questions are answered that ask if good instructional
design strategies have been followed and additional
comments are recorded where appropriate
Strengths: Relatively inexpensive, quick, can be applied
early
An inspection method
conceptually similar to an instructional design review, but where the
questions ask if usability heuristics (“rules of thumb”) have been
followed and additional comments are recorded during the inspection
Strengths: Relatively inexpensive, quick, can be applied
early
Weaknesses:
Requires human factors expertise, prediction
Prototype
Pluralistic Walkthrough
Inspection by a team of
experts (and possibly users) with different backgrounds walking through scenarios
of users interacting with the training system
Weaknesses:
Requires scenario development & experts, prediction
Focus Group
Trainees or other stakeholder groups (e.g., instructors) are
asked to provide feedback (e.g., satisfaction, usability) on
a technology through group discussion. Participants may be
afforded an opportunity for hands on exploration of the
simulation prior to the discussion
Strengths: Real users, efficient, relatively cheap
Weaknesses:
Group think, prediction
Observation
Observe practitioners performing their work by shadowing or
observing from an unobtrusive location
Strengths: Performance in context, real users
Weaknesses:
Possibly little or no control
Survey - questionnaire or
interview
Questionnaire:
Participants are given text questions on paper or online.
Questions ask for the participant to select from a
predefined set of alternatives or are open ended where the
participant responds in his/her own words
Interview: Interviewer asks questions. Can be
structured (fixed script) or semi-structured
Strengths: Simple, inexpensive, relatively easy to
analyze
Weaknesses:
Opinion based, if all fixed questions not very flexible
Summative
Quasi or True Experiment
A Quasi (semi) experiment follows the experimental
design, but lacks random assignment and/or manipulation of independent
variable
Strengths: May be more practical to execute than true
experiment
Weaknesses:
IRB, Less confidence in results
In a true experiment research design is
characterized by a high degree of control over the research setting to
allow for the determination of cause-and-effect relationships among
variables
Strengths: Validity of results
Weaknesses:
IRB,
Not always practical, may limit fidelity
Final Simulation
Non-Experiment (e.g., correlation)
A research design that examines the relationship among or
between variables as they naturally occur
Strengths: Real-world data
Weaknesses:
Many uncontrolled variables
References
Nielsen, J. (1993). Usability
engineering. Boston, MA: AP Professional.
MST-READI is a collaborative research effort among US Army RDECOM-STTC,
OSDi and CWS, funded by RDECOM-STTC
