Harvesting information from captions for weakly supervised semantic segmentation

Research output: Contributions to collected editions/worksArticle in conference proceedingsResearchpeer-review

Authors

Since acquiring pixel-wise annotations for training convolutional neural networks for semantic image segmentation is time-consuming, weakly supervised approaches that only require class tags have been proposed. In this work, we propose another form of supervision, namely image captions as they can be found on the Internet. These captions have two advantages. They do not require additional curation as it is the case for the clean class tags used by current weakly supervised approaches and they provide textual context for the classes present in an image. To leverage such textual context, we deploy a multi-modal network that learns a joint embedding of the visual representation of the image and the textual representation of the caption. The network estimates text activation maps (TAMs) for class names as well as compound concepts, i.e. combinations of nouns and their attributes. The TAMs of compound concepts describing classes of interest substantially improve the quality of the estimated class activation maps which are then used to train a network for semantic segmentation. We evaluate our method on the COCO dataset where it achieves state of the art results for weakly supervised image segmentation.

Original languageEnglish
Title of host publication2019 International Conference on Computer Vision Workshops : ICCV 2019 : proceedings : 27 October-2 November 2019, Seoul, Korea
Number of pages10
Place of PublicationPiscataway
PublisherInstitute of Electrical and Electronics Engineers Inc.
Publication date10.2019
Pages4481-4490
Article number9022140
ISBN (print)978-1-7281-5024-6
ISBN (electronic)978-1-7281-5023-9
DOIs
Publication statusPublished - 10.2019
Externally publishedYes
Event17th IEEE/CVF International Conference on Computer Vision Workshop - ICCVW 2019 - Seoul, Korea, Republic of
Duration: 27.10.201928.10.2019
Conference number: 17
https://iccv2019.thecvf.com/

Bibliographical note

Publisher Copyright:
© 2019 IEEE.

    Research areas

  • Multimodal learning, Semantic segmentation, Weakly supervised learning, Weakly supervised semantic segmentation
  • Informatics

DOI

Recently viewed

Publications

  1. Applications of the Simultaneous Modular Approach in the Field of Material Flow Analysis
  2. An Optimization Approach for Crew Rostering in Public Bus Transit
  3. Vimentin promoter methylation analysis is a suitable complement of a gene mutation marker panel for the detection of preneoplastic and neoplastic colonic lesions
  4. Interrogating the Phenomenon of Suffering and Smiling by Nigerians: A Mixed Methods Study
  5. How selfish is a thirsty man? A pilot study on comparing sharing behavior with primary and secondary rewards
  6. Plastic deformation induced microstructure evolution through gradient enhanced crystal plasticity based on a non-convex Helmholtz energy
  7. Understanding and managing post-acquisition integration as change process
  8. Illegal Migration in Postfordism
  9. Grüner Umbau
  10. Playing with Information
  11. Editorial
  12. Videoportale: Broadcast Yourself? : Versprechen und Enttäuschung
  13. Introduction for Special Issue of Submissions from European Liberal Education Student Conference
  14. Gemeinsam lernen mit Selbstlernprogrammen
  15. Verstehen bildnerischer Narrationen in Eltern-Kind-Dialogen
  16. Alltag
  17. Das Nordfenster
  18. Das Radio
  19. Grain size evolution simulation in aluminium alloys AA 6082 and AA 7020 during hot forward extrusion process
  20. Nitrous oxide emission from biogras production systems on a coastal marsh soil
  21. Idol/Ikone
  22. Knowledge-networking capability in German SMEs
  23. Virtual Reality
  24. Diversität und Heterogenität
  25. John Stuart Mill zur Einführung
  26. Sozialwirtschaft
  27. Networking and Interaction between Regions and Higher Education Institutions
  28. Spät erkannt
  29. Review
  30. DRGs and the Professional Independence of Physicians
  31. Racheengel
  32. Kampfkunst
  33. Lotta Continua in Frankfurt, Türken-Terror in Köln.
  34. IGH
  35. The business case for sustainability in retrospect
  36. "i like reggae and Bob Marley is already dead"
  37. Samenbanken, Leihmütter, Retortenbabies