Does the specific size of matrices affect the performance of matrix operations?

Question

I was reading DeepMind's paper on I2A's and realized that the sizes of the hidden layers in their model were all like 32, 64, 256, and so on: all powers of 2. I have found the same thing in other papers.

Is there any performance reason for it? Maybe related to data structure alignment?

More concretely, I would like to know if I should use this "special" sizes when training my own models.

Answer 1

While you can only be 100% certain when you ask the authors, most authors use this simply because you have to choose one value. The specific value doesn't matter too much, only the order of magnitude. Taking a power of 2 seems to be a natural choice.

You can also take a setup which uses a power of two and reduce the number by one. The computation time should be roughly equal, probably be a bit lower. If it is noticeably higher, there might be a performance benefit of using the choice of the author.

See also

• Datascience.SE: Why the number of neurons or convolutions chosen equal powers of two?
• Quora: Should I use powers of 2 when choosing the size of a batch size when training my Neural Network?

• nanaki: I also thought the specific value wasn't important, but this predilection for powers of 2 seemed strange =). +1 for the experiment suggestion and the links.
• Martin Thoma: Please also note that some people say they do it due to cache alignment. then you should ask them if they actually experimentally confirmed if that makes a difference. Sometimes people have heard something and either don't know enough to do it right or just assume that it worked add expected when nothing goes horribly wrong.