Hands in Focus: Sign Language Recognition Via Top-Down Attention

Research output: Contributions to collected editions/worksArticle in conference proceedingsResearchpeer-review

Authors

  • Noha Sarhan
  • Christian Wilms
  • Vanessa Closius
  • Ulf Brefeld
  • Simone Frintrop

In this paper, we propose a novel Sign Language Recognition (SLR) model that leverages the task-specific knowledge to incorporate Top-Down (TD) attention to focus the processing of the network on the most relevant parts of the input video sequence. For SLR, this includes information about the hands' shape, orientation and positions, and motion trajectory. Our model consists of three streams that process RGB, optical flow and TD attention data. For the TD attention, we generate pixel-precise attention maps focusing on both hands, thereby retaining valuable hand information, while eliminating distracting background information. Our proposed method outperforms state-of-the-art on a challenging large-scale dataset by over 2%, and achieves strong results with a much simpler architecture compared to other systems on the newly released AUTSL dataset [1].

Original languageEnglish
Title of host publication2023 IEEE International Conference on Image Processing, ICIP 2023 - Proceedings : Proceedings
Number of pages5
Place of PublicationPiscataway
PublisherIEEE Electromagnetic Compatibility Society
Publication date08.10.2023
Pages2555-2559
ISBN (print)978-1-7281-9836-1
ISBN (electronic)978-1-7281-9835-4
DOIs
Publication statusPublished - 08.10.2023
Event2023 IEEE International Conference on Image Processing - Kuala Lumpur Convention Centre, Kuala Lumpur, Malaysia
Duration: 08.10.202311.10.2023
Conference number: 30
https://2023.ieeeicip.org/

Bibliographical note

Publisher Copyright:
© 2023 IEEE.

    Research areas

  • Informatics - sign language recognition, top-down attention, deep learning

Recently viewed

Publications

  1. Biodiversity and ecosystem functioning relations in European forests depend on environmental context.
  2. Embracing scale-dependence to achieve a deeper understanding of biodiversity and its change across communities
  3. Combination of a reduced order state observer and an Extended Kalman Filter for Peltier cells
  4. Numerical Investigation of the Effect of Rolling on the Localized Stress and Strain Induction for Wire + Arc Additive Manufactured Structures
  5. Do we fail to exert self-control because we lack resources or motivation? Competing theories to explain a debated phenomenon
  6. What do people do when they use the internet?
  7. Machine Learning and Data Mining for Sports Analytics
  8. Collaborative modelling for active involvement of stakeholders in urban flood risk management
  9. Environmental Shareholder Value Matrix
  10. Online-Beratung für Eltern
  11. Navigating tensions in inclusive conservation
  12. Exploring the potential of using priority effects during ecological restoration to resist biological invasions in the neotropics
  13. Backward Extended Kalman Filter to Estimate and Adaptively Control a PMSM in Saturation Conditions
  14. Uncertainty, Pluralism, and the Knowledge-based Theory of the Firm
  15. ... address unknown?
  16. Toward supervised anomaly detection
  17. A comparison of the strength of biodiversity effects across multiple functions
  18. The Influence of Counterfactual Thinking about Uncontrolled Factors on Moral Judgment
  19. The 'need for speed'
  20. Perceptions of Organizational Downsizing
  21. Credit constraints, endogenous innovations, and price setting in international trade
  22. Toward a gecko-inspired, climbing soft robot
  23. CDS spreads, systemic risk and interconnectedness
  24. Foraging loads of stingless bees and utilisation of stored nectar for pollen harvesting
  25. A victim of regulatory arbitrage? Automatic exchange of information and the use of golden visas and corporate shells
  26. A theoretical multiscale analysis of electrical field for fuel cells stack structures
  27. Case study analysis of laser-assisted Low-Cost Automation assembly
  28. Effect of Bleaching on the Elution of Monomers from Modern Dental Composite Materials
  29. How context affects transdisciplinary research