MST-READI

(Medical Simulation TRaining TEchnology EvAluation DesIgner)

MST-READI website home page A primer on the process and challenges of TEE research Preparing to conduct TEE research Guidance on evaluating the requirements analysis & design for a simulation based training system Guidance on evaluating a prototype simulation based training system Guidance on evaluating a final simulation based training system Additional resources

Goals for the MST-READI website

     Background

Military and civilian organizations are frequently charged with evaluating simulation based training with limited access to guidance in selecting among the myriad of approaches to Training Effectiveness Evaluation (TEE). With this in mind, the MST-READI website is intended to provide the following online: 

1. Education for training effectiveness researchers on the process and challenges of TEE research  

This site offers a TEE primer, Understanding Training Evaluations that includes an overview of the TEE process, general principles to follow, specific methods that are commonly used, and challenges to be aware of in conducting valid research.  This can be found by clicking under the Background  tab on the main navigation bar.

2. Guidance in developing and executing TEE research that follows the MST-READI methodology

This site describes the MST-READI methodology for conducting a training evaluation that may be used at any point from the initial training design stage through to the final product delivery.  An overview of this methodology is shown in the figure below and described in detail through the following tabs:

Getting Started Requirements & Design Evaluation Prototype Evaluation Final System Evaluation

 

 

In brief, the MST-READI methodology recommends three evaluation phases:

Requirements and Design Evaluation is conducted in the earliest stages of research and development to provide formative feedback to training technology developers.  Design Evaluation methods include requirements review, and instructional system design and usability inspections.

Simulation Prototype Evaluation is typically formative – to provide final design feedback to developers; however, initial summative information can be gathered as well. ‘Prototype Evaluation’ refers to a collection of methods for obtaining limited performance information and participant feedback on a given medical training simulation. These methods include Observation, Focus Group, and Survey, and are to be used in sequence within a single session involving small groups (6-10 participants).  

Final System Evaluation focuses on trainee learning outcomes and longer term organizational impacts of the medical training simulation. Depending upon the training evaluation situation and the characteristics of the simulation, the final evaluation may include a true experimental design, quasi-experiment or non-experimental design.

A training effectiveness evaluation may involve a single evaluation or multiple coordinated evaluations as a training system evolves and feedback is utilized.  Regardless, the basic steps, described on this website, are the same: 1. Getting Started; 2. Developing the Protocol; 3. Collecting the Data; and 4. Reporting the Results

The research process for each evaluation phase above begins with planning and then follows the steps below:

Developing the Protocol - The protocol or research plan assists you in preparing to 1) conduct valid research and 2) to navigate the practical challenges of collecting data in military environments.  

Collecting the Data - Data collection involves executing the procedures defined in the protocol to gather and store information in support of the research goals. Data is then analyzed to determine results. Data analysis will differ depending upon the type of data collected.  

Reporting the Results - The final step involves writing up and disseminating the results of the research. Research documentation should be shared for immediate consumption and when appropriate added to the MST-READI database on Army Knowledge Online (AKO).

3. Downloadable MST-READI Tools & Resources for use offline

 

MST-READI provides tools and resources for each of the steps 1-4 that can be used for each type of evaluation. These resources are referenced in context on various pages throughout the website and also under the Resources tab.  Of particular value are the worksheets that can be used to help manage and document an evaluation.

 

Additionally there is a companion printable MST-READI Workbook that contains essentially the same content as the website, organized into the same sections, and with the same worksheets that may be completed while preparing for research, working in the field, or analyzing the results, and preparing reports.

 

       MST-READI is a collaborative research effort among US Army RDECOM-STTC, OSDi and CWS, funded by RDECOM-STTC     

 

Background