Both in research and school practice, student feedback is increasingly used to obtain information about the quality of teaching. There are numerous empirical findings on the predictive validity of student questionnaires aggregated on the classroom level (e. g. Clausen, 2002), especially for the secondary school students. There is also evidence for factorial validity, on both primary and secondary school level (Fauth, Decristan, Rieser, Klieme & Büttner, 2014a; Lenske, 2016). However, further investigations based on mixed-method approaches have shown that it might be difficult to properly interpret the identified factors, since the associated items are often not answered as intended, at least on primary school level (Lenske, 2016). These studies at primary school level indicate that the factor structure alone is not sufficient as a validity indicator and can even be misleading for instructional development. It is unclear, though, whether the difficulties in understanding and answering such items disappear with the transition to secondary school. To examine this question in the context of a qualitative study, 14 secondary school students were asked to explain the reasons for their answers on a questionnaire regarding a specific lesson. The results of the content analysis showed that there are clear differences in item understanding depending on the teaching quality dimension. Similarly to the primary school level studies, the items regarding classroom management were interpretedcomparably well; items on differentiation and especially cognitive activation were not understood in theintended sense by most students.