Transformer with Tree-order Encoding for Neural Program Generation
Research output: Contributions to collected editions/works › Article in conference proceedings › Research
Standard
Conference XXX. 2022.
Research output: Contributions to collected editions/works › Article in conference proceedings › Research
Harvard
APA
Vancouver
Bibtex
}
RIS
TY - CHAP
T1 - Transformer with Tree-order Encoding for Neural Program Generation
AU - Thellmann, Klaudia-Doris
AU - Stadler, Bernhard
AU - Usbeck, Ricardo
AU - Lehmann, Jens
N1 - This paper was authored in late 2020 and early 2021 for the most part
PY - 2022/5/30
Y1 - 2022/5/30
N2 - While a considerable amount of semantic parsing approaches have employed RNN architectures for code generation tasks, there have been only few attempts to investigate the applicability of Transformers for this task. Including hierarchical information of the underlying programming language syntax has proven to be effective for code generation. Since the positional encoding of the Transformer can only represent positions in a flat sequence, we have extended the encoding scheme to allow the attention mechanism to also attend over hierarchical positions in the input. Furthermore, we have realized a decoder based on a restrictive grammar graph model to improve the generation accuracy and ensure the well-formedness of the generated code. While we did not surpass the state of the art, our findings suggest that employing a tree-based positional encoding in combination with a shared natural-language subword vocabulary improves generation performance over sequential positional encodings.
AB - While a considerable amount of semantic parsing approaches have employed RNN architectures for code generation tasks, there have been only few attempts to investigate the applicability of Transformers for this task. Including hierarchical information of the underlying programming language syntax has proven to be effective for code generation. Since the positional encoding of the Transformer can only represent positions in a flat sequence, we have extended the encoding scheme to allow the attention mechanism to also attend over hierarchical positions in the input. Furthermore, we have realized a decoder based on a restrictive grammar graph model to improve the generation accuracy and ensure the well-formedness of the generated code. While we did not surpass the state of the art, our findings suggest that employing a tree-based positional encoding in combination with a shared natural-language subword vocabulary improves generation performance over sequential positional encodings.
KW - cs.CL
KW - cs.AI
KW - 68T07, 68T50
KW - I.2.7
KW - Informatics
U2 - 10.48550/arXiv.2206.13354
DO - 10.48550/arXiv.2206.13354
M3 - Article in conference proceedings
BT - Conference XXX
ER -