The current study employs Mixture Distribution Rasch models to compare the psychometric properties of two rating scale variants (original rating scale with six response categories, N = 806 school students; a variant with four response categories, N = 905 school students) for five specific scales of the Physical Self-Description Questionnaire. In most cases, two-class solutions best fit the data, suggesting the presence of response sets. In particular, students with low educational levels appear to have problems using as many as six response categories correctly. The results suggest that for the Physical Self-Description Questionnaire, using a rating scale with six response categories can be problematic, because the test scores psychometric qualities may be substantially different across (latent) subgroups. Also discussed are implications for the use and development of assessment measures in general, highlighting the suitability of item response theory models for the investigation of test score properties and the need for heterogeneous samples.