TY - JOUR

T1 - How to get really smart: Modeling retest and training effects in ability testing using computer-generated figural matrix items

AU - Freund, Philipp Alexander

AU - Holling, Heinz

PY - 2011/7

Y1 - 2011/7

N2 - The interpretation of retest scores is problematic because they are potentially affected by measurement and predictive bias, which impact construct validity, and because their size differs as a function of various factors. This paper investigates the construct stability of scores on a figural matrices test and models retest effects at the level of the individual test taker as a function of covariates (simple retest vs. training, use of identical vs. parallel retest forms, and general mental ability). A total of N=189 subjects took two tests of matrix items that were automatically generated according to a strict construction rationale. Between test administrations, participants in the intervention groups received training, while controls did not. The Rasch model fit the data at both time points, but there was a lack of item difficulty parameter invariance across time. Training increased test performance beyond simple retesting, but there was no large difference between the identical and parallel retest forms at the individual level. Individuals varied greatly in how they profited from retest experience, training, and the use of identical vs. parallel retest forms. The results suggest that even with carefully designed tasks, it is problematic to directly compare scores from initial tests and retests. Test administrators should emphasize learning potential instead of state level assessment, and inter-individual differences with regard to test experience should be taken into account when interpreting test results.

AB - The interpretation of retest scores is problematic because they are potentially affected by measurement and predictive bias, which impact construct validity, and because their size differs as a function of various factors. This paper investigates the construct stability of scores on a figural matrices test and models retest effects at the level of the individual test taker as a function of covariates (simple retest vs. training, use of identical vs. parallel retest forms, and general mental ability). A total of N=189 subjects took two tests of matrix items that were automatically generated according to a strict construction rationale. Between test administrations, participants in the intervention groups received training, while controls did not. The Rasch model fit the data at both time points, but there was a lack of item difficulty parameter invariance across time. Training increased test performance beyond simple retesting, but there was no large difference between the identical and parallel retest forms at the individual level. Individuals varied greatly in how they profited from retest experience, training, and the use of identical vs. parallel retest forms. The results suggest that even with carefully designed tasks, it is problematic to directly compare scores from initial tests and retests. Test administrators should emphasize learning potential instead of state level assessment, and inter-individual differences with regard to test experience should be taken into account when interpreting test results.

KW - Economics, empirical/statistics

KW - Figural matrix items

KW - Individual change

KW - Rational item construction

KW - Retest effects

KW - Training effects

UR - http://www.scopus.com/inward/record.url?scp=79957661124&partnerID=8YFLogxK

U2 - 10.1016/j.intell.2011.02.009

DO - 10.1016/j.intell.2011.02.009

M3 - Journal articles

VL - 39

SP - 233

EP - 243

JO - Intelligence

JF - Intelligence

SN - 0160-2896

IS - 4

ER -