Signal Propagation in Transformers: Theoretical Perspectives and the Role of Rank Collapse

Sotiris Anagnostidis , Luca Biggio , Lorenzo Noci , Antonio Orvieto , *Sidak Pal Singh* , Aurelien Lucchi