MST-READI

(Medical Simulation TRaining TEchnology EvAluation DesIgner)

Home Background Getting Started Requirements & Design Evaluation Prototype Evaluation Guidance on evaluating a final simulation based training system Resources

Pluralistic Walkthrough Focus Group

Focus Group

      Home General Principles

The Focus Group session involves: (1) observation of real-user performance followed by (2) group discussion and finally (3) a survey.

Observation

Participants should be presented with the training material (this may include the entire training content or may be limited to specific modules) individually or in small groups (depending upon the ratio of available simulations to participants) and asked to complete particular training tasks. Researchers have an opportunity to watch the user-simulation interaction and make observations about the usability of the system, accuracy of instructional content and instructional strategies.  (Key differences between this stage in the evaluation and the pluralistic walkthrough are the more realistic pace of user interaction and the fact that developers are no longer participants.)

This methodology complements asking users for their opinions about a system because it actually involves interacting with the system and as such may reveal issues the user wouldn't have predicted.  One challenge when observing users is that thoughts running through the mind could be important data that isn't collected.  Therefore the technique of think-aloud can be used to ask users to simply "say what is going through your mind as you interact with the system". 

The video to the right demonstrates the technique of think aloud.  It also shows how computer screen recording can be used to capture the annotated user interactions with a software interface.  Several programs are available to facilitate this, but popular ones are Techsmith's Camtasia Studio and Beepa Pty's Fraps.

 

If direct interaction with the simulation based training system is not possible due to logistical constraints a video recording of the system being used could be used as a substitute.  For instance, the following video demonstrates CLS training using Americas Army and could be shown to participants if the actual software were not available:

 

Group Discussion

After becoming familiar with the medical simulation based training system, a group discussion will be conducted. This  methodology is a qualitative approach commonly used in research where small groups of people are encouraged to disclose their attitudes and perceptions of a product through group interaction.

In the following video recording of a group discussion session for a "Dream Bank" notice:

1.

A video recording of the session has been created (with a well-placed camera) to help capture the data being collected and to potentially show segments to stakeholders so that attitudes such as enthusiasm or dislike are clearly communicated

2.

The moderator is working from a script and asking specific people for their input rather than just observing an open-ended discussion

 

 

Survey

Finally, each participant will be asked to complete a survey capturing self-reports of usability of the device and prediction of successful use in practice as a training system.

Generic questions about a training simulation:

1.

What did you like best about the training simulation?

2.

What did you like least about the training simulation?

3.

What aspects of the training simulation do you think make it an effective or ineffective training tool?

4.

What training content is missing and should be included in the training simulation?

5.

How do you think the training simulation could best be integrated into the current program of instruction? (How would you like to see it used?)

6.

Did you like the performance feedback capabilities of the training simulation?

Simulation specific questions:

1.

Was the menu system for selecting your response easy to use?

2.

How practical is the simulation to use—is it too heavy (or too fragile, or too low fidelity, etc.) to provide practical value?

http://usability.gov/templates has example questionnaires including the System Usability Scale (SUS) from http://www.usabilitynet.org/trump/documents/Suschapt.doc.  

General considerations for forming well-formed questions are given by Walonick, D. (2004) Designing and Using Questionnaires in Survival Statistics.  StatPac, Bloomington, MN.  Available at: http://www.statpac.com/surveys/

The MST-READI methodology includes procedures for conducting a Focus Group session.

References

http://grouplab.cpsc.ucalgary.ca/saul/hci_topics/tcsd-book/chap-1_v-1.html Task-Centered User Interface Design: A Practical Introduction, by Clayton Lewis and John Rieman.

Ericsson, K., & Simon, H. (May 1980). "Verbal reports as data". Psychological Review 87 (3): 215–251. doi:10.1037/0033-295X.87.3.215. 

 

       MST-READI is a collaborative research effort among US Army RDECOM-STTC, OSDi and CWS, funded by RDECOM-STTC     

 

 Home General Principles