Observer ratings of instructional quality: Do they fulfill what they promise?

Research output: Journal contributionsJournal articlesResearchpeer-review

Authors

Despite considerable interest in the topic of instructional quality in research as well as practice, little is known about the quality of its assessment. Using generalizability analysis as well as content analysis, the present study investigates how reliably and validly instructional quality is measured by observer ratings. Twelve trained raters judged 57 videotaped lesson sequences with regard to aspects of domain-independent instructional quality. Additionally, 3 of these sequences were judged by 390 untrained raters (i.e., student teachers and teachers). Depending on scale level and dimension, 16-44% of the variance in ratings could be attributed to instructional quality, whereas rater bias accounted for 12-40% of the variance. Although the trained raters referred more often to aspects considered essential for instructional quality, this was not reflected in the reliability of their ratings. The results indicate that observer ratings should be treated in a more differentiated manner in the future.

Original languageEnglish
JournalLearning and Instruction
Volume22
Issue number6
Pages (from-to)387-400
Number of pages14
ISSN0959-4752
DOIs
Publication statusPublished - 12.2012
Externally publishedYes

    Research areas

  • Generalizability theory, Instructional quality, Observer ratings, Reliability, Validity
  • Educational science

Recently viewed

Researchers

  1. Helge Peters

Publications

  1. Objekttheater begegnet digitalen Welten
  2. Measurement and modelling of NH3 emissions from field-applied biogas residues in North German energy crop rotations
  3. What Kind of Veto Player Is the Italian Senate?
  4. Materialität der Diagramme
  5. Loan managers’ trust and credit access for SMEs
  6. The Role of Trust in Natural Resource Management Conflicts
  7. Inter- und transdisziplinäre Forschungsprojekte als Ansatz für eine transformative Hochschulbildung
  8. Poetik der Assimilation
  9. Forecasting European interest rates in times of financial crisis – What insights do we get from international survey forecasts?
  10. "Sustainability State” in the Making?
  11. Typology of public outreach for biodiversity conservation projects in Spain
  12. Women outperform men in distinguishing between authentic and nonauthentic smiles
  13. Social Games
  14. Agriculture and everyday realities on small farms – An entrepreneurial challenge to farmers between the desire for autonomy and a secure existence. Two examples from east and south-east Poland
  15. Einkommenssituation Selbständiger in der Europäischen Union
  16. Expatriate training
  17. Environmental Performance and the Quality of Corporate Environmental Reports
  18. Zuwanderer in der Stadt
  19. KoLaS
  20. Soziale Arbeit im Kampf um Anerkennung
  21. „Mountainbiken“
  22. Systemische Schulentwicklung
  23. Posturale Instabilität bei Morbus Parkinson
  24. Sustelling
  25. Moral Sensitivity
  26. Fluctuations of carabid populations inhabiting an ancient woodland (Coleoptera, Carabidae)
  27. Why does repatriate career success vary?
  28. Mechanisms promoting tree species co-existence
  29. The impact of nuclear accidents on provisioning ecosystem services
  30. Einsam an der Spitze?
  31. Multisensory Design of Retail Environments
  32. Dating of event-layers (palaeotsunamis) on the Andaman Islands (Bay of Bengal) with Pb-210 and optically stimulated luminescence
  33. Mischverhältnisse
  34. Editorial
  35. Pathische Repräsentation
  36. Unterscheidung ist noch lange keine Diskriminierung
  37. Technology and Organization in Manufacturing
  38. MOOC: Zukunft der Bildung oder Robo-Uni?
  39. The reciprocal iso-inhibition volume concept
  40. Emotional and social development in childhood
  41. Die Regulierung von Carework im Privathaushalt
  42. Ernährung als Alltagsdilemma
  43. The reform of the German Works Constitution Act
  44. Die "wahre" Organisation erkennen