Neural relational inference for disaster multimedia retrieval

Research output: Journal contributionsJournal articlesResearchpeer-review

Authors

Events around the world are increasingly documented on social media, especially by the people experiencing them, as these platforms become more popular over time. As a consequence, social media turns into a valuable source of data for understanding those events. Due to their destructive potential, natural disasters are among events of particular interest to response operations and environmental monitoring agencies. However, this amount of information also makes it challenging to identify relevant content pertaining to those events. In this paper, we use a relational neural network model for identifying this type of content. The model is particularly suitable for unstructured text, that is, text with no particular arrangement of words, such as tags, which is commonplace in social media data. In addition, our method can be combined with a CNN for handling multimodal data where text and visual data are available. We perform experiments in three different scenarios, where different modalities are evaluated: visual, textual, and both. Our method achieves competitive performance in both modalities by themselves, while significantly outperforms the baseline on the multimodal scenario. We also demonstrate the behavior of the proposed method in different applications by performing additional experiments in the CUB-200-2011 multimodal dataset.

Original languageEnglish
JournalMultimedia Tools and Applications
Volume79
Issue number35-36
Pages (from-to)26735-26746
ISSN1380-7501
DOIs
Publication statusPublished - 09.2020
Externally publishedYes

    Research areas

  • Information retrieval, Machine learning, Multimodal, Natural language processing, Neural networks
  • Business informatics

Recently viewed

Activities

  1. HyperKult IX - Computer als Medium: Augmented Space 2000
  2. Transition or Collapse? Conceptualizing Dynamics of Change
  3. E-learning module on “Participation” in the context of IWRM – “Social Science” Part
  4. Management of ecosystem resilience as optimal investment in self-protection. A simple, but often non-convex problem
  5. 6th Workshop on Unintended Consequences
  6. Alternative Mouse – Alternative User? Towards a History of Assistive Media
  7. Administration 2.0
  8. kultur.informatik
  9. HyperKult 12
  10. Grenzflächen der Informatik 2004
  11. Environmental Sustainability and Supply Chain Management: A Framework of Cross-Functional Integration and Knowledge Transfer
  12. Strangely familiar. The spatial uncanny and the study of organizational space
  13. Work = work ≠ work: In defence of play
  14. Innovation als Kernaufgabe der modernen Universität
  15. Universität Lyon III (Jean Moulin)
  16. Curb Cuts and Computer. A Media-Archeological Perspective on Digital lnclusion
  17. Informatik und Kultur, „Grenzflächen der Informatik” 2003
  18. HyperKult 14
  19. IDGEC Synthesis Conference - 2006
  20. Alpen-Adria-Universität Klagenfurt
  21. The Convention on Biological Diversity, “Biopiracy”, and Justice. Reconstructing the “biopiracy” debate from the perspective of the concept of justice.
  22. South-East Asian Case Studies on EMA
  23. Higher Education for Sustainable Development: Developing and Assessing Students' Competencies For Dealing with Complexity and Uncertainty
  24. Sustainability Solutions Initiative - From Knowledge to Action
  25. How important are SDGs for teacher educators? – Engage educators in a global landscape: Mapping the terrain with a case study of a pilot professional development program from Ethiopia
  26. Tagung "Radioforschung/Audioforschung II
  27. Ziele
  28. Strangely familiar: The spatial uncanny and the study of organizational space

Publications

  1. A Framework for Applying Natural Language Processing in Digital Health Interventions
  2. Toward Automatically Labeling Situations in Soccer
  3. Studying properties of water data using manifold-aware anomaly detectors
  4. To use or not to use learning data
  5. Influence of data clouds fusion from 3D real-time vision system on robotic group dead reckoning in unknown terrain
  6. Endemic predators, invasive prey and native diversity
  7. Decision making in the context of environmental risks
  8. Performability analysis of an unreliable M/M/1-type queue
  9. Effects of tree diversity on canopy space occupation vary with tree size and canopy space definition in a mature broad-leaved forest
  10. Who’s afraid of the senses? Organization, management and the return of the sensorium
  11. Assessing the Bonding Interface Characteristics and Mechanical Properties of Bobbin Tool Friction Stir Welded Dissimilar Aluminum Alloy Joints
  12. Optimization of transport flow on two paths with respect to the passengers time costs
  13. Transformationsnarrative
  14. Structure and Organization of Product Development Projects
  15. Atomkraft international
  16. Consumer Online Search Behavior
  17. Assessing teachers' educational knowledge
  18. On the measuring accuracy of the “Vehrs-Hebel”, a scaling apparatus for nonverbal real-time assessment of perceived quantity
  19. A control strategy for electromagnetic near and far field calculation
  20. Art and Culture as an Urban Development Tool
  21. A(l)gora: the Mindscape
  22. “Processed Food on the Urban Data Highway. Food Delivery Services as In_Visible Infrastructure in the Production of Urbanity”