|
Threats to
Research Validity |
 
 |
As mentioned in the
Evaluation Methods section, an experimental design may be applied at
either or both the prototype evaluation and the final system evaluation
stages. The
validity of an
experimental TEE
reflects the accuracy of the research results, that is, how confident we
can be that we can attribute the observed changes to the training. A
number of factors can potentially compromise the interpretation of
results.
Threats to Validity
Internal
Validity includes factors other than the training that may have caused the
results
External
Validity refers
to how generalizable the findings are to other groups
|
Threats to Internal Validity |
|
Threat |
Description |
|
Selection of participants |
concerns related to groups being unequal
prior to the training |
|
History |
events other than the training that may occur
between the first and second measurement |
|
Maturation |
changes
that may naturally occur over time, such as becoming older
or gaining experience that may effect performance regardless
of training |
|
Repeated testing |
potential learning and / or practice effects |
|
Instrumentation |
the changes in instruments (e.g., the
reliability of an instrument to deliver the independent
variable or to assess the dependent variable), or in
observers or raters which may occur over the course of the
experiment and produce changes in outcomes |
|
Statistical regression |
AKA regression to the mean. Selection of
participants based on extreme scores of the dependent
variable (e.g., really poor performers) tends to result in
scores being closer to the mean on a second measure |
|
Experimental mortality |
the loss of subjects systematically, i.e., a
particular subgroup (e.g., experienced soldiers) dropping
out. |
|
Selection-maturation
interaction |
interaction of subject-related variables and
time-related variables may lead to confounding outcomes, and
erroneous interpretation that the treatment caused the
effect |
|
John Henry effect |
being aware of comparison to a machine during
an ‘experiment’, John Henry outperformed the machine |
|
Threats to External Validity |
|
Threat |
Description |
|
Aptitude-Treatment-Interaction |
participants characteristics that
interact with the independent variable, limit generalizability |
|
Situation |
all situational specifics (e.g. treatment
conditions, time, location, lighting, noise, treatment
administration, investigator, timing, scope and extent of
measurement, etc. etc.) of a study potentially limit generalizability |
|
Test effects |
if cause-effect relationships can only be
found when pre-tests or post-tests are used, generalizability is limited. For instance, a
pretest might increase or decrease a subject's sensitivity or
responsiveness to the experimental variable |
|
Hawthorne effects |
effects
only found as a result of studying the situation (vs the training) |
|
