MST-READI

(Medical Simulation TRaining TEchnology EvAluation DesIgner)

At the start of a TEE, a research team must be assembled, initial consensus must be reached regarding the purpose of the evaluation, and logistical information must be gathered about the evaluation situation.

Project Planning

An initial project plan should reflect a blueprint of sorts for ensuring the training effectiveness of a training simulation product; that is, identifying which components of the methodology-a single evaluation or multiple evaluations- will be used to evaluate the medical training simulation of interest. In addition, the project plan should include an initial estimate of scope, logistical considerations and a projected timeline for accomplishing the simulation evaluation goals.  A Project Planning Worksheet is provided in the MST-READI workbook to assist the program manager. The program manger and evaluation team should review and update the plan as necessary.

This is an iterative process—you will likely collect much of this information over time. Additionally, initial answers to some questions may change as you answer others. An updated worksheet serves to make the situation concrete as well as to ensure team members have a common understanding of the TEE conditions within which they are working.

It is important to note that project requirements and constraints often evolve or become more concrete over time (e.g., defining the location for the data collection may impose additional or different considerations than originally anticipated); therefore it will be important to maintain communication among the team and between the team and the site point of contact during each phase of the project.

This is an iterative process—you will likely collect much of this information over time. Additionally, initial answers to some questions may change as you answer others. An updated worksheet serves to make the situation concrete as well as to ensure team members have a common understanding of the TEE conditions within which they are working.

The Project Planning worksheet helps you 1) consider issues relevant to your evaluation, 2) organize your approach to examining those issues, and 3) maintain a running record of the evaluation plan and changes to it.  Specific considerations include:

Type of Simulation - The characteristics of the simulation themselves will have an impact on the evaluation planning. Factors such as the simulation type, level of maturity, system objectives, instructional strategy, and assessment methods help to define appropriate evaluation approaches and for summative research will impact your research protocol.

Purpose of the evaluation - The purpose or goal of the evaluation(s) is very important – for a given phase of evaluation research it influences what kind of information you will collect, how you will conduct the evaluation, and how the results should be presented to support use by stakeholders so it must be specified early in the process.

Resources – in planning for several potential evaluation phases, it is important to get an initial idea of the availability of significant resources and constraints.

Evaluation research team – A training evaluation research team should be assembled consisting of representatives of multiple stakeholder communities and including relevant expertise. This worksheet will help you identify the expertise required for specific evaluations and give you an opportunity to identify training evaluation team members who fit each role. Note that the purpose of the evaluation is important in identifying requirements of team members, for example, what types of knowledge and expertise will be needed, so define your purpose in parallel or iteratively with team development.

Consider Institutional Review Board (IRB) issues

Institutional Review Boards exist to ensure the protection of human subjects during biomedical, behavioral and social research. Many IRBs exist.  IRBs are responsible for reviewing, evaluating and either approving or disapproving research protocols—description of the purpose and benefits of the research, the methodology involved, the informed consent process, and any materials used (e.g., questionnaires, training simulation, performance assessments). IRBs also monitoring ongoing research, especially in the case of long term research projects.  DoD research frequently requires two levels of review: a local level and a Headquarters level.

The IRB submission and approval process can be a long one (e.g., 3-6 months), so begin thinking about your IRB submission as early as possible.

Your research is considered human subjects research by The United States Department of Health & Human Services if:

(The research involves collecting data about living people through an intervention/interaction

             OR

The activity involves identifiable, private information)

            AND

The activity is a systematic investigation and is designed to contribute to generalizable knowledge.

If you are following the MST-READI methodology, Requirements & Design Evaluations involving inspection (even those involving target user walkthroughs) typically will not require IRB approval. Prototype evaluations may or may not require IRB approval. Final Simulation Evaluations however will likely require going through the IRB process. 

Researchers conducting Behavioral, social and/or medical research involving human participants must undergo training in the protection of human subjects. Training and certification can be obtained through the Collaborative Institutional Training Initiative (CITI). 

At the CITI site, you must register / create a login. At the prompt to select the Participating Institution with which to affiliate, select t the U.S. Army Medical Research and Material Command (MRMC).

Download the MST-READI IRB Forms to assist you in meeting your IRB requirements.

         MST-READI is a collaborative research effort among US Army RDECOM-STTC, OSDi and CWS, funded by RDECOM-STTC