Supra-Laplacian Encoding for Transformer on Dynamic Graphs - Centre d'études et de recherche en informatique et communications
Communication Dans Un Congrès Année : 2024

Supra-Laplacian Encoding for Transformer on Dynamic Graphs

Encodage supra-laplacien pour les transformers sur graphes dynamiques

Résumé

Fully connected Graph Transformers (GT) have rapidly become prominent in the static graph community as an alternative to Message-Passing models, which suffer from a lack of expressivity, oversquashing, and under-reaching. However, in a dynamic context, by interconnecting all nodes at multiple snapshots with self-attention,GT loose both structural and temporal information. In this work, we introduce Supra-LAplacian encoding for spatio-temporal TransformErs (SLATE), a new spatio-temporal encoding to leverage the GT architecture while keeping spatio-temporal information. Specifically, we transform Discrete Time Dynamic Graphs into multi-layer graphs and take advantage of the spectral properties of their associated supra-Laplacian matrix. Our second contribution explicitly model nodes' pairwise relationships with a cross-attention mechanism, providing an accurate edge representation for dynamic link prediction. SLATE outperforms numerous state-ofthe-art methods based on Message-Passing Graph Neural Networks combined with recurrent models (e.g. , LSTM), and Dynamic Graph Transformers, on 9 datasets. Code is open-source and available at this link https://github.com/ykrmm/SLATE.
Fichier principal
Vignette du fichier
SLATENeurips24-1.pdf (4.36 Mo) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-04785441 , version 1 (15-11-2024)

Identifiants

Citer

Yannis Karmim, Marc Lafon, Raphaël Fournier-S'Niehotta, Nicolas Thome. Supra-Laplacian Encoding for Transformer on Dynamic Graphs. The Thirty-eighth Annual Conference on Neural Information Processing Systems, Dec 2024, Vancouver (CA), Canada. ⟨hal-04785441⟩
8 Consultations
0 Téléchargements

Altmetric

Partager

More