MST-READI

(Medical Simulation TRaining TEchnology EvAluation DesIgner)

When considering alternate designs for your final evaluation – experimental or quasi-experimental – you will need to understand the four basic building blocks: Treatment, People, Observations and Time. In this section, these elements are first described, followed by the presentation of standard ‘equations’ that succinctly capture how they can be combined in particular ways to form basic designs.

1.  People (N=nonrandom; R=random; C=cut score-differentiate expert/novice) participating in the research. This can include individuals or existing groups of people who will participate in the various treatments for comparison. These individuals or groups will experience a treatment (one form of training or another). 

2.  Observations (O) or measures being influenced by the technology. The set of one or more measures is often simply written as O, but if multiple measures are being assessed the individual measures may be identified through subscript values, such as O₁ for skill, O₂ for knowledge, O₃ for confidence, O₄ for efficiency, O₅ for affect, etc. There are different methods for conducting measurements (e.g. performance or self-report survey).  If there is variation in these methods being used this notation could be extended with a second subscript value (e.g. O_1,1).

3.  Treatment (X) or intervention of interest (e.g., training). The simulation or training technology being evaluated is considered a treatment. More than one form of the experimental training would be denoted X₁, X₂,… X_n. Control groups can be denoted X₀.

4.  Time between the training involving the new technology and the subsequent measurement of outcomes. For instance, a simple pre/post treatment, by definition, involves measurement before and after a treatment to support calculating the change that treatment may have made.  More complex designs could involve even more measures over time or a sequence of treatments and measurement over time.

Experimental design equations consist of one row per treatment, with the elements on each row specified in the order in which those actions would be taken.  For instance, the following represents a pre-post test randomized control group design where one experimental group treatment is being compared to the control group treatment:

R  O  X₀  O

R  O  X₁  O

pre-post test randomized control group design

The four columns represent four steps:

1.     participants are randomly (R) assigned to either the experimental group (X₁) or the control group (X₀)

2.     the same pretests (O) are administered to both groups

3.     participants in X₁ are given the experimental treatment (e.g. trained using the new technology), while participants in X₀ are given the control treatment (e.g. trained using an existing training approach)

4.     upon completion of training, participants in both groups are administered post tests (O).

These way in which these key components are put together gives rise to additional design concepts and considerations which are important in selecting alternative research designs.

·         Pre-Post Test Designs - Pretest-posttest designs are the preferred method to compare participant groups and measure the degree of change occurring as a result of treatments or interventions.

·         Post test only designs - If no pretest is given, it is difficult to ascertain the degree or amount of change attributable to the treatment. However, if the goal is to determine if criterion level of performance can be obtained, the degree of change is not necessary.

·         Control group designs - Control groups are intended to provide a group against which to compare the treatment group to see examine changes related to the treatment. Groups are compared in which one group, the control group may have no treatment or they may have some neutral treatment (e.g., when a placebo is used in a clinical trial).

·         Repeated measures designs – The repeated measures design uses the same subjects with every condition of the research, including the control. This design can be used with smaller sample sizes, however It is susceptible to practice or test taking effects, as well as order effects. The order of the treatment and control conditions must be counterbalanced across participants (i.e., some participants would get the experimental treatment first followed by the control while others received the control condition first followed by the experimental treatment)

·         Within versus between subjects designs – in within subject designs, ever participant is receives every condition; while between subject designs maintain independence. Subjects only participate in one condition. In general within subjects designs allow you to use smaller sample sizes, but you must be wary of testing or carryover effects from repeated measurement. Between subjects designs require more participants and avoid potential problems associated with carryover effects.

·         Randomized versus non-equivalent groups – Randomization is the preferred approach. The goal of random assignment is to ensure equivalence of the two (or more) groups at the start of the experiment. This helps us make inferences that any changes in the post training measures are due to the training and not to any differences that may have existed prior to the treatment (e.g., that one group already had higher levels of knowledge / skill before the experimental training occurred).  In field experiments, random assignment may not always be easy or possible. If participants cannot be randomly assigned to groups, a non-equivalent group design can be used. Other design concepts such as pre-post testing can help you address potential differences that may exist prior to the treatment.

·         Randomized Block designs - If specific differences among groups of subjects exist a randomized block design may be appropriate. Blocking is a strategy for grouping people in your data analysis in order to reduce noise -- it is an analysis strategy. In a block design, data for experimental subjects are divided into homogeneous blocks (e.g., based on different locations). Blocking doesn't necessarily affect anything that you do with the research participants.

       MST-READI is a collaborative research effort among US Army RDECOM-STTC, OSDi and CWS, funded by RDECOM-STTC