Indicators and instruments for determining and measuring school quality within inspection systems are usually based on frameworks for inspection. These frameworks rely heavily upon school and teacher effectiveness research. Thus, a central assumption is that the effectiveness and improvement-oriented school conditions, as measured within an inspection, are related to student achievement. It is unclear if this assumption really holds true, as empirical evidence is still lacking. This study uses data from a random sampling of schools (n = 37) and teachers (n = 1663) and the achievement data from students (n = 23,943) to validate the interpretations and uses of school inspection results regarding the factor school leadership. The study follows Kane’s argument-based approach for validation. Results reveal that principals of schools with recurrent high student achievement very often demonstrate instructional leadership. It is evident that these principals are also leading in a fundamentally different way from principals in schools with lower achievement in that they lead with the specific school context in mind. The study demonstrates that it is possible to make inferences from scores provided by school inspections and to generalize from them. However, this generalizability does not extend to making extrapolations or decisions based on these scores. The analyses make it clear that providing feedback solely on the strengths and weakness of a school is insufficient when it comes to stimulating school improvement through inspection.