Transformers are equivariant to permutations, as ConvNets are equivariant to translations.

Judge as a technical fact as of 2024-03-18. True: standard references or established understanding show that transformer self-attention layers without positional encodings are permutation-equivariant (permuting the input tokens permutes the outputs in the same way), analogous to convolution's translation equivariance. Mostly true: the property holds only with caveats the claim leaves out (for example, it breaks once positional encodings or causal masks are added), but the core statement is accepted as standard. Mixed: experts broadly treat the statement as only partly accurate. Mostly false: the property holds only for narrow components and not for transformers in any meaningful sense. False: transformers, even without positional encodings, are shown not to be permutation-equivariant.

Set before the research began.

1. E1 GROUP EQUIVARIANT STAND-ALONE SELF-ATTENTION FOR VISION arXiv (ICLR 2021 conference paper), by David W. Romero (Vrije Universiteit Amsterdam) and Jean-Baptiste Cordonnier (EPFL) · 2021

   “Note that the self-attention operation defined in Eq. 3 is equivariant to permutations of the input rows of X. That is, a permutation of the rows of X will produce the same output Y up to this permutation.”

2. E2 GROUP EQUIVARIANT STAND-ALONE SELF-ATTENTION FOR VISION arXiv (ICLR 2021 conference paper), by David W. Romero (Vrije Universiteit Amsterdam) and Jean-Baptiste Cordonnier (EPFL) · 2021

   “Proposition 4.1. The global self-attention formulation without positional encoding (Eqs. 3, 8) is permutation equivariant.”

3. E3 GROUP EQUIVARIANT STAND-ALONE SELF-ATTENTION FOR VISION arXiv (ICLR 2021 conference paper), by David W. Romero (Vrije Universiteit Amsterdam) and Jean-Baptiste Cordonnier (EPFL) · 2021

   “Though absolute positional encodings do disrupt permutations equivariance, they are unable to provide translation equivariance.”

4. E4 GROUP EQUIVARIANT STAND-ALONE SELF-ATTENTION FOR VISION arXiv (ICLR 2021 conference paper), by David W. Romero (Vrije Universiteit Amsterdam) and Jean-Baptiste Cordonnier (EPFL) · 2021

   “Translation equivariance is key to the success of CNNs. It describes the property that if a pattern is translated, its numerical descriptors are also translated, but not modified.”

An independent ICLR 2021 paper states that self-attention is permutation-equivariant: permuting the input rows permutes the output the same way (E1). It also gives this as a formal proposition for global self-attention without positional encoding (E2). The same paper describes CNN translation equivariance in parallel terms, which supports the analogy the claim draws (E4). However, it notes that absolute positional encodings break permutation equivariance (E3), and the claim leaves out that caveat. Under the criteria, this makes the claim mostly true rather than fully true.