In Germany, several Large Scale Assessments measure the mathematical competence of students at the end of secondary schooling, for instance the Programme for International Student Assessment 2012 (PISA 2012), the National Assessment Study 2012 (LV 2012) and the National Educational Panel Study in Ninth Grade (NEPS-K9). To date it remains unclear whether these measurements of mathematical competence are comparable and whether the assessment frameworks determine the same construct. Therefore, experts reviewed 226 mathematics items of the studies PISA 2012, LV 2012 and NEPS-K9 regarding their distribution among subdimensions, their formal and language demands as well as their linguistic complexity. As it turns out the tests are very similar in terms of the frameworks. However, the classification of the PISA 2012 items reveals that they are less difficult regarding the word level and the complexity of the sentence structure in comparison to the NEPS-K9 items. In contrast to the LV 2012, NEPS-K9 contains more multiple-choice items and less mathematical terms. Finally, the outcomes highlight the relevance of an elaborate analysis of dimensional and scales-based equivalence while taking into account the conceptual differences found in this study.