We propose a novel approach for invoice recognition by integrating Large Language Model Embeddings as semantic features into the nodes of a Graph Attention Neural Network. Both the language model and the graph structure provide rich contextual information for our model to enhance the classification of OCR tokens from invoice documents. The experimental results demonstrate improvements in the classification performance on our datasets by over 3%, highlighting the effectiveness of our multiple attention mechanism. The approach is transferable to all kinds of service systems that process visually rich documents.