Hands in Focus: Sign Language Recognition Via Top-Down Attention

Research output: Contributions to collected editions/worksArticle in conference proceedingsResearchpeer-review

Authors

  • Noha Sarhan
  • Christian Wilms
  • Vanessa Closius
  • Ulf Brefeld
  • Simone Frintrop

In this paper, we propose a novel Sign Language Recognition (SLR) model that leverages the task-specific knowledge to incorporate Top-Down (TD) attention to focus the processing of the network on the most relevant parts of the input video sequence. For SLR, this includes information about the hands' shape, orientation and positions, and motion trajectory. Our model consists of three streams that process RGB, optical flow and TD attention data. For the TD attention, we generate pixel-precise attention maps focusing on both hands, thereby retaining valuable hand information, while eliminating distracting background information. Our proposed method outperforms state-of-the-art on a challenging large-scale dataset by over 2%, and achieves strong results with a much simpler architecture compared to other systems on the newly released AUTSL dataset [1].

Original languageEnglish
Title of host publication2023 IEEE International Conference on Image Processing, ICIP 2023 - Proceedings : Proceedings
Number of pages5
Place of PublicationPiscataway
PublisherIEEE Electromagnetic Compatibility Society
Publication date08.10.2023
Pages2555-2559
ISBN (print)978-1-7281-9836-1
ISBN (electronic)978-1-7281-9835-4
DOIs
Publication statusPublished - 08.10.2023
Event2023 IEEE International Conference on Image Processing - Kuala Lumpur Convention Centre, Kuala Lumpur, Malaysia
Duration: 08.10.202311.10.2023
Conference number: 30
https://2023.ieeeicip.org/

Bibliographical note

Publisher Copyright:
© 2023 IEEE.

    Research areas

  • Informatics - sign language recognition, top-down attention, deep learning

Recently viewed

Publications

  1. Absolutely continuous random power series in reciprocals of Pisot numbers
  2. Visual Detection of Traffic Incident through Automatic Monitoring of Vehicle Activities
  3. The use of knowledge in inter-organisational knowledge-networks
  4. A Smart Sensing Architecture for Misalignment Measurements
  5. Towards a Heuristic for Scheduling Offshore Installation Processes
  6. A Model Based Feedforward Regulator Improving PI Control of an Ice-Clamping Device Activated by Thermoelectric Cooler
  7. Drawing as a Generative Activity and Drawing as a Prognostic Activity
  8. The Use of Anti-Windup Techniques in Didactic Level Systems
  9. Geometric Properties on the Perfect Decoupling Disturbance Control in Manufacturing Systems
  10. A sensorless control using a sliding-mode observer for an electromagnetic valve actuator in automotive applications
  11. Experimental analysis of measurement process for a QCM using the pulse coincidence method
  12. The Automated will
  13. Automatic generation of periodic representative volume elements for matrix-inclusion composites and their efficiency in multiscaling
  14. Concurrently Observed Actions Are Represented Not as Compound Actions but as Independent Actions
  15. The role of supervisor support for dealing with customer verbal aggression. Differences between ethnic minority and ethnic majority workers
  16. Intelligent software system for replacing a force sensor in the case of clearance measurement
  17. Magnesium recycling: State-of-the-Art developments, part II
  18. CaO dissolution during melting and solidification of a Mg-10 wt.% CaO alloy detected with in situ synchrotron radiation diffraction
  19. Feel the Music! Exploring the Cross-modal Correspondence between Music and Haptic Perceptions of Softness
  20. Self-Compassion as a Facet of Neuroticism? A Reply to the Comments of Neff, Tóth-Király, and Colosimo (2018)
  21. A switching model predictive control for overcoming a hysteresis effect in a hybrid actuator for camless internal combustion engines
  22. Wavelet characterizations for anisotropic Besov spaces
  23. Anonymized Firm Data under Test: Evidence from a Replication Study
  24. Simulation of stresses during casting of binary magnesium-aluminum alloys
  25. Privacy-Preserving Localization and Social Distance Monitoring with Low-Resolution Thermal Imaging and Deep Learning
  26. A microsystem for growth inhibition test of Enterococcus faecalis based on impedance measurement
  27. Using an adaptive memory strategy to improve a multistart heuristic for sequencing by hybridization
  28. High temperature deformation mechanisms and processing map for hot working of cast-homogenized Mg-3Sn-2Ca alloy
  29. Assessing pre-travel online destination experience values of destination websites
  30. Analysis of life cycle datasets for the material gold
  31. Using Long-Duration Static Stretch Training to Counteract Strength and Flexibility Deficits in Moderately Trained Participants
  32. Compression behavior of typical silicone rubbers for soft robotics applications at elevated temperatures
  33. Exploring intrinsic, instrumental and relational values for sustainable management of social-ecological systems
  34. Intra-firm Wage Compression and Cost Coverage of Training
  35. Using a Bivariate Polynomial in an EKF for State and Inductance Estimations in the Presence of Saturation Effects to Adaptively Control a PMSM
  36. Combination of a reduced order state observer and an Extended Kalman Filter for Peltier cells
  37. Improving Flood Forecasting in a Developing Country
  38. A transfer operator based numerical investigation of coherent structures in three-dimensional Southern ocean circulation
  39. Investigation of the Controllability of Inductive Power Transmission Systems based on Flexible Coils
  40. Confidence levels and likelihood terms in IPCC reports
  41. Optimal scheduling for Automated Guided Vehicles (AGV) in blocking job-shops
  42. Natural enemy diversity reduces temporal variability in wasp but not bee parasitism
  43. Unusual two‐bond 13C, 13C coupling constants in sulphones
  44. Elastomeric Prepregs for Soft Robotics Applications
  45. Predicting the future performance of soccer players
  46. Open Innovation in Schools
  47. Testing for a break in the persistence in yield spreads of EMU government bonds
  48. CubeQA—question answering on RDF data cubes