Matlab-like reshape in OpenCV

Question

I am translating Matlab code to OpenCV. The code involves lots of reshaping of large matrices. As the Matlab reshape differs from OpenCV you can't just use OpenCV reshape.

Here is why: Let's suppose we have a 2x4x3 matrix such as this:

a(:,:,1) =
 1 3 5 7
 2 4 6 8
a(:,:,2) =
 9 11 13 15
 10 12 14 16
a(:,:,3) =
 17 19 21 23
 18 20 22 24

When we perform reshape(a,4,3,2) the result is:

 ans(:,:,1) =
 
 1 5 9
 2 6 10
 3 7 11
 4 8 12
 
 ans(:,:,2) =
 
 13 17 21
 14 18 22
 15 19 23
 16 20 24

But if we perform a similar reshape in OpenCV we get:

 ans(:,:,1) =
 
 1 17 11
 5 21 15
 2 18 12
 6 22 16
 
 ans(:,:,2) =
 
 9 3 19
 13 7 23
 10 4 20
 14 8 24

This is due to the different ways of traversing the matrix in OpenCV and Matlab. Matlab first traverses columns then rows and last depth. OpenCV first traverses depth then rows and last columns.

I have written a function to do a Matlab-like reshape for 3D matrices. First I reshape the input matrix to a row matrix. For the matrix mentioned above, the result matrix will be [1 2 3 ... 24], then I reshape the row matrix to the desired channels. But as can you see, this code involves lots of creating auxiliary matrices and copying and transposing that will not be good for the performance.

Mat matlab_reshape(const Mat &m, int new_row, int new_col, int new_ch)
{
 int old_row, old_col, old_ch;
 old_row = m.size().height;
 old_col = m.size().width;
 old_ch = m.channels();
 Mat m1 ( 1, new_row*new_col*new_ch, m.depth() );
 vector <Mat> p(old_ch);
 split(m,p);
 for ( int i=0; i<p.size(); ++i ){
 Mat t ( p[i].size().height, p[i].size().width, m1.type() );
 t = p[i].t();
 Mat aux = m1.colRange(i*old_row*old_col,(i+1)*old_row*old_col).rowRange(0,1);
 t.reshape(0,1).copyTo(aux);
 }
 vector <Mat> r(new_ch);
 for ( int i=0; i<r.size(); ++i ){
 Mat aux = m1.colRange(i*new_row*new_col,(i+1)*new_row*new_col).rowRange(0,1);
 r[i] = aux.reshape(0, new_col);
 r[i] = r[i].t();
 }
 Mat result;
 merge(r, result);
 return result;
} 

Here is an example:

Mat_ <Vec3b> m(2,4);
m(0,0) = Vec3b(1,9,17);
m(0,1) = Vec3b(3,11,19);
m(0,2) = Vec3b(5.1,13,21);
m(0,3) = Vec3b(7,15,23);
m(1,0) = Vec3b(2,10,18);
m(1,1) = Vec3b(4,12,20);
m(1,2) = Vec3b(6,14,22);
m(1,3) = Vec3b(8,16,24);
cout << m << endl;
cout << m.reshape(2,4) << endl;
cout << matlab_reshape(m,4,3,2) << endl;

Performance comparison of reshaping matrices of [24700 x 10 x 1] size to matrices of [130 x 190 x 10] size:

My code:

0.006222
0.006528
0.006716
0.006733
0.006508

Matlab:

0.000039
0.000036
0.000042
0.000035
0.000031

• Loufylouf: Have you tried using valgrind with the callgrind tool to see what takes time in your code ? What are the compilation options you are using ?
• Loufylouf: And a last question, for now, is multi-threading considerable ?
• shuttle87: In r[i].t(); what is t()?
• Saeid: I used valgrind and it shows transpose and merge takes most of the time.
• Saeid: I am gonna use this code in a system that doesn't have access to a gpu so multi-processing won't help.

Answer 1

Your performance is never going to approach that of the library you're comparing to, unless you change the data layout.

I don't know Matlab, but I've done a small amount of work with OpenCV, so I can speculate how the reshape() function works.

Consider this 4✕2 array:

A B C D
a b c d

which can be represented like this in OpenCV (row-major order):

width=4, height=2,
content = { A, B, C, D, a, b, c, d }

To reshape to a 4✕2 array, all we do is change the width and height members, without touching the content.

But my guess is that Matlab indexes the content array in the opposite (column-major) order:

height=2, width=4
content = { A, a, B, b, C, c, D, d }

Reshaping this by swapping width and height yields

A C
a c
B D
b d

So the key to performance may well be to change order of dimensions in the translated code so that CV reshape does what we want - instead of declaring arrays as height✕width and indexing using (y, x), declare them as width✕height and index them as (x, y). Extending this from 2 dimensions to 3 shouldn't be a problem.

To avoid error-prone edits all over the code, I recommend writing a small wrapper around cv::Mat_ which can be addressed in Matlab order.

• Cris Luengo: That is indeed how MATLAB works. Replicating the exact same result in OpenCV requires two transpose operations, which means copying the data over twice, so it cannot be as fast as in MATLAB. I think this answer suggests not doing that, but instead working with the transposed images everywhere. This requires careful programming, because all indexing everywhere will need to have the indices transposed as well. But that is a really good solution.
• Toby Speight: Yes, that's exactly what I'm suggesting - except that my final paragraph says to write a wrapper that does the transpose for you. Even at my most careful, I would otherwise miss a transposition somewhere.
• Cris Luengo: But many OpenCV function calls will have to be updated anyway. You can't fix that with a wrapper that changes indexing. A filtering function that takes a filter size as width x height will have to be called with those two values reversed. A function that returns a list of coordinates will return (x,y), and these will have to be swapped. Etc. All functions in OpenCV assume that pixels on a row of the image are contiguous in memory, they won't be using the indexing operation of the wrapper object.
• Toby Speight: Ah, you're right - it's definitely not so simple as that.
• Toby Speight: I have colleagues who have had to write OpenCV code based on Matlab prototypes - perhaps I should ask around to see how they deal with this.