Give it Space! Explicit Disentangling of Positional and Semantic Representations in Encoders
Published in EMNLP, 2026
Preprint - explored how encoder-based models use absolute positional (AP) and relative positional (RP) information by explicitly disentangling positional and semantic representations. We find that the learned AP representations are low dimensional and used to encode document structure, while RP information is used as complementary to semantic matching.
Recommended citation: Lequeu, P. A., Barboule, C., & Piwowarski, B. (2026). Give it Space! Explicit Disentangling of Positional and Semantic Representations in Encoders. arXiv preprint arXiv:2605.30022.
Download Paper
