Acknowledgement
This text represents a humble attempt to summarise and synthesise decades of
existing knowledge in deeplearning architectures,through thegeometric lens
ofinvariance andsymmetry. Wehope thatourperspectivewillmake iteasier
both for newcomers and practitioners to navigate the field,and for researchers
tosynthesisenovelarchitectures,asinstancesofourblueprint.Inaway,we
hope tohave presented "allyou needto build the architecturesthat are allyou
need"—a play on words inspired by Vaswani et al. (2017).
Theinitialpartofthetextwaswrittenduringthecoronaviruspandemic
(2020–2021).Asitoftenhappens,wehadthousandsofdoubtswhetherthe
whole picturemakes sense,and used opportunitiesprovidedby ourcolleagues
tohelpusbreakour"stagefright"andpresentearlyversionsofourwork,
whichsawthelightofdayinPetar’stalkatCambridge(courtesyofPietro
Liò)and Michael’stalks atOxford(courtesy ofXiaowenDong)and Imperial
College (hostedby Michael Huth and Daniel Rueckert). Petar was also able to
present ourwork atFriedrich-Alexander-UniversitätErlangen-Nürnberg—the
birthplaceof theErlangenProgram!—owing toakindinvitationfromAndreas
Maier. The feedback wereceived for these talks wasenormously invaluableto
keepingourspiritshigh,aswellaspolishingtheworkfurther.Last,butcer-
tainlynot least,wethankthe organisingcommittee ofICLR2021, whereour
work featured in a keynote talk, delivered by Michael.
Weshouldnotethatreconcilingsuchavastquantityofresearchisseldom
enabledbytheexpertiseofonlyfourpeople.Accordingly,wewouldliketo
giveduecreditto alloftheresearcherswho havecarefully studiedaspectsof
our textas itevolved,and providedus withcareful commentsand references:
Yoshua Bengio, Charles Blundell, Andreea Deac,Fabian Fuchs, Francesco Di
Giovanni, AleksaGordi
́
c, MarcoGori, Raia Hadsell, Will Hamilton, Maksym
Korablyov,MarcLackenby,AbbasMehrabian,ChristianMerkwirth,Razvan
Pascanu,SébastienRacaniere,BrunoRibeiro,AnnaScaife,JürgenSchmid-
huber,MarwinSegler,CsabaSzepesvári,CorentinTallec,NgânV
̃
u,Geordie
Williamson,PeterWirnsbergerandDavidWong.Theirexpertfeedbackwas
invaluable tosolidifyingourunification effortsand makingthemmoreuseful
to various niches. Though, of course, any irregularities within this text are our
responsibility alone.