Transformer with Tree-order Encoding for Neural Program Generation

Publikation: Beiträge in SammelwerkenAufsätze in KonferenzbändenForschung

Standard

Transformer with Tree-order Encoding for Neural Program Generation. / Thellmann, Klaudia-Doris; Stadler, Bernhard; Usbeck, Ricardo et al.
Conference XXX. 2022.

Publikation: Beiträge in SammelwerkenAufsätze in KonferenzbändenForschung

Harvard

APA

Vancouver

Thellmann KD, Stadler B, Usbeck R, Lehmann J. Transformer with Tree-order Encoding for Neural Program Generation. in Conference XXX. 2022 doi: 10.48550/arXiv.2206.13354

Bibtex

@inbook{56b3c87266394178bbbbf7097e04d0af,
title = "Transformer with Tree-order Encoding for Neural Program Generation",
abstract = " While a considerable amount of semantic parsing approaches have employed RNN architectures for code generation tasks, there have been only few attempts to investigate the applicability of Transformers for this task. Including hierarchical information of the underlying programming language syntax has proven to be effective for code generation. Since the positional encoding of the Transformer can only represent positions in a flat sequence, we have extended the encoding scheme to allow the attention mechanism to also attend over hierarchical positions in the input. Furthermore, we have realized a decoder based on a restrictive grammar graph model to improve the generation accuracy and ensure the well-formedness of the generated code. While we did not surpass the state of the art, our findings suggest that employing a tree-based positional encoding in combination with a shared natural-language subword vocabulary improves generation performance over sequential positional encodings. ",
keywords = "cs.CL, cs.AI, 68T07, 68T50, I.2.7, Informatics",
author = "Klaudia-Doris Thellmann and Bernhard Stadler and Ricardo Usbeck and Jens Lehmann",
note = "This paper was authored in late 2020 and early 2021 for the most part",
year = "2022",
month = may,
day = "30",
doi = "10.48550/arXiv.2206.13354",
language = "English",
booktitle = "Conference XXX",

}

RIS

TY - CHAP

T1 - Transformer with Tree-order Encoding for Neural Program Generation

AU - Thellmann, Klaudia-Doris

AU - Stadler, Bernhard

AU - Usbeck, Ricardo

AU - Lehmann, Jens

N1 - This paper was authored in late 2020 and early 2021 for the most part

PY - 2022/5/30

Y1 - 2022/5/30

N2 - While a considerable amount of semantic parsing approaches have employed RNN architectures for code generation tasks, there have been only few attempts to investigate the applicability of Transformers for this task. Including hierarchical information of the underlying programming language syntax has proven to be effective for code generation. Since the positional encoding of the Transformer can only represent positions in a flat sequence, we have extended the encoding scheme to allow the attention mechanism to also attend over hierarchical positions in the input. Furthermore, we have realized a decoder based on a restrictive grammar graph model to improve the generation accuracy and ensure the well-formedness of the generated code. While we did not surpass the state of the art, our findings suggest that employing a tree-based positional encoding in combination with a shared natural-language subword vocabulary improves generation performance over sequential positional encodings.

AB - While a considerable amount of semantic parsing approaches have employed RNN architectures for code generation tasks, there have been only few attempts to investigate the applicability of Transformers for this task. Including hierarchical information of the underlying programming language syntax has proven to be effective for code generation. Since the positional encoding of the Transformer can only represent positions in a flat sequence, we have extended the encoding scheme to allow the attention mechanism to also attend over hierarchical positions in the input. Furthermore, we have realized a decoder based on a restrictive grammar graph model to improve the generation accuracy and ensure the well-formedness of the generated code. While we did not surpass the state of the art, our findings suggest that employing a tree-based positional encoding in combination with a shared natural-language subword vocabulary improves generation performance over sequential positional encodings.

KW - cs.CL

KW - cs.AI

KW - 68T07, 68T50

KW - I.2.7

KW - Informatics

U2 - 10.48550/arXiv.2206.13354

DO - 10.48550/arXiv.2206.13354

M3 - Article in conference proceedings

BT - Conference XXX

ER -

Zuletzt angesehen

Publikationen

  1. Explicit and Implicit Framing Effects on Product Attitudes When Using Country-of- Origin Cues
  2. Don’t underestimate the problems of user centredness in software development projectsthere are many!?
  3. The effects of different on-line adaptive response time limits on speed and amount of learning in computer assisted instruction and intelligent tutoring
  4. Control of a Three-Axis Robot with Super Twisting Sliding Mode Control
  5. A New Framework for Production Planning and Control to Support the Positioning in Fields of Tension Created by Opposing Logistic Objectives
  6. Detection time analysis of propulsion system fault effects in a hexacopter
  7. Reality-Based Tasks with Complex-Situations
  8. Introduction Mobile Digital Practices. Situating People, Things, and Data
  9. Mapping the intersection of planetary boundaries and environmentally extended input-output analysis: A systematic literature review
  10. The relationship between long-term memory ability and instructed second language learning
  11. Guest Editorial - ''Econometrics of Anonymized Micro Data''
  12. Experimental Evaluation of Data Fusion Techniques and Adaptive Control for Mobile Robot Localization
  13. Measuring cognitive load with subjective rating scales during problem solving
  14. Load mitigation and power tracking capability for wind turbines using linear matrix inequality-based control design
  15. Joint Item Response Models for Manual and Automatic Scores on Open-Ended Test Items
  16. Comparison of Odor Thresholds obtained by a Three Alternative Choice Procedure and by the Method of Limits
  17. Comparison of different FEM codes approach for extrusion process analysis
  18. Global Finite-Time Stabilization of Planar Linear Systems With Actuator Saturation
  19. Effectiveness of a guided multicomponent internet and mobile gratitude training program - A pragmatic randomized controlled trial
  20. A Note on Estimation of Empirical Models for Margins of Exports with Unknown Non-linear Functional Forms