Hands in Focus: Sign Language Recognition Via Top-Down Attention

Research output: Contributions to collected editions/worksArticle in conference proceedingsResearchpeer-review

Authors

  • Noha Sarhan
  • Christian Wilms
  • Vanessa Closius
  • Ulf Brefeld
  • Simone Frintrop

In this paper, we propose a novel Sign Language Recognition (SLR) model that leverages the task-specific knowledge to incorporate Top-Down (TD) attention to focus the processing of the network on the most relevant parts of the input video sequence. For SLR, this includes information about the hands' shape, orientation and positions, and motion trajectory. Our model consists of three streams that process RGB, optical flow and TD attention data. For the TD attention, we generate pixel-precise attention maps focusing on both hands, thereby retaining valuable hand information, while eliminating distracting background information. Our proposed method outperforms state-of-the-art on a challenging large-scale dataset by over 2%, and achieves strong results with a much simpler architecture compared to other systems on the newly released AUTSL dataset [1].

Original languageEnglish
Title of host publication2023 IEEE International Conference on Image Processing, ICIP 2023 - Proceedings : Proceedings
Number of pages5
Place of PublicationPiscataway
PublisherIEEE Electromagnetic Compatibility Society
Publication date08.10.2023
Pages2555-2559
ISBN (print)978-1-7281-9836-1
ISBN (electronic)978-1-7281-9835-4
DOIs
Publication statusPublished - 08.10.2023
Event2023 IEEE International Conference on Image Processing - Kuala Lumpur Convention Centre, Kuala Lumpur, Malaysia
Duration: 08.10.202311.10.2023
Conference number: 30
https://2023.ieeeicip.org/

Bibliographical note

Publisher Copyright:
© 2023 IEEE.

    Research areas

  • Informatics - sign language recognition, top-down attention, deep learning

Recently viewed

Publications

  1. An Integrated Case Study of the Concepts and Applications of SAP ERP HCM
  2. Towards a heuristic for assessing adaptation knowledge: impacts, implications, decisions and actions
  3. "to expose, to show, to demonstrate, to inform, to offer. Artistic Practices around 1990"
  4. The representative turn in EU studies
  5. Magnesium recycling: State-of-the-Art developments, part II
  6. Conceptual and procedural mathematical knowledge of beginning mathematics majors and preservice teachers
  7. Development and validation of a questionnaire on parental health literacy in the context of promoting healthy lifestyles during childhood
  8. Applying FIDIC contracts in Jordan
  9. Microstructure, mechanical properties and fracture behaviors of large-scale sand-cast Mg-3Y-2Gd-1Nd-0.4Zr alloy
  10. Interplay of formative assessment and instructional quality—interactive effects on students’ mathematics achievement
  11. An Unusual Encounter with Oneself
  12. The emergence of selection rules and their encounter with group theory, 1913-1927
  13. Ideology as filter
  14. Biomedical Entity Linking with Triple-aware Pre-Training
  15. Is the market classification of risk always efficient?
  16. Existential insecurity and deference to authority
  17. Correlation between Isometric Maximum Strength and One Repetition Maximum in the Calf Muscle in Extended and Bended Knee Joint
  18. Contributing to sustainable development pathways in the South Pacific through transdisciplinary research
  19. Credit constraints and exports: A survey of empirical studies using firm level data
  20. State of the Art of Handling and Storage Systems on Container Terminals
  21. Fast Catch Bumerang
  22. The Bigger Picture of Corruption
  23. Non-native and native organisms moving into high elevation and high latitude ecosystems in an era of climate change
  24. Online to offline social networking