With this study, we contribute to the research on the role of language in assessment. In this study, we employed an experimental study design (N = 1346 students) that systematically modified the linguistic complexity of test items in the field of mathematical word problems. Our results show that teachers expected a test question’s increased linguistic complexity to have a high impact on the item difficulty (d = 0.75). The effect of varying the linguistic complexity on a test item difficulty, based on student results, is much lower (d = 0.12). A statistically significant moderation effect indicates that students with low language proficiency are more hindered by linguistic complexity than students with high language proficiency.