Dear list, I was just trying to shear an image to be plotted with matplotlib (to get some snazzy 3D effect) and realized that it's apparently not possible. Investigating further, I realized that the underlying agg library indeed supports shearing, as it simply uses an affine matrix for its transforms (very much like the rest of matplotlib), but it does not export that feature to matplotlib. So, I just quickly added some code to actually access the transformation matrix in the C++ code, so that one can use it from within python. The next step would be to hook that up to the usual code used in matplotlib. A patch is attached at the end of this post. Greetings Martin --------------- patch follows ----------------------- commit b7d0d23d90460ee790f1e94f387070a69be661c8 Author: Martin Teichmann <martin@renoir.(none)> Date: Wed Jul 6 14:04:44 2011 +0200 Make affine transformations work for images The current code only supports scaling and maybe rotation of images, but not all affine transformations, although it is already prepared to do so. This patch adds a method set_matrix to the C++ image handling code, so that one can set (and thus perform) arbitrary affine transformations. It also fixes a little bug which introduced weird side effects if images were resized more than once. diff --git a/src/_image.cpp b/src/_image.cpp index 3278b6c..7d40664 100644 --- a/src/_image.cpp +++ b/src/_image.cpp @@ -92,7 +92,6 @@ Image::apply_rotation(const Py::Tuple& args) agg::trans_affine M = agg::trans_affine_rotation(r * agg::pi / 180.0); srcMatrix *= M; - imageMatrix *= M; return Py::Object(); } @@ -156,7 +155,6 @@ Image::apply_scaling(const Py::Tuple& args) //printf("applying scaling %1.2f, %1.2f\n", sx, sy); agg::trans_affine M = agg::trans_affine_scaling(sx, sy); srcMatrix *= M; - imageMatrix *= M; return Py::Object(); } @@ -179,7 +177,6 @@ Image::apply_translation(const Py::Tuple& args) //printf("applying translation %1.2f, %1.2f\n", tx, ty); agg::trans_affine M = agg::trans_affine_translation(tx, ty); srcMatrix *= M; - imageMatrix *= M; return Py::Object(); } @@ -285,7 +282,6 @@ Image::reset_matrix(const Py::Tuple& args) args.verify_length(0); srcMatrix.reset(); - imageMatrix.reset(); return Py::Object(); } @@ -316,6 +312,31 @@ Image::get_matrix(const Py::Tuple& args) return ret; } +char Image::set_matrix__doc__[] = + "set_matrix(m11,m21,m12,m22,m13,m23)\n" + "\n" + "Set the affine transformation matrix\n" + " /m11,m12,m13\\\n" + " /m21,m22,m23|\n" + " \\ 0 , 0 , 1 /" + ; + +Py::Object +Image::set_matrix(const Py::Tuple& args) +{ + _VERBOSE("Image::set_matrix"); + + args.verify_length(6); + + double m[6]; + for (int i = 0;i < 6;i++) + { + m[i] = Py::Float(args[i]); + } + srcMatrix.load_from(m); + return Py::Object(); +} + char Image::resize__doc__[] = "resize(width, height, norm=1, radius=4.0)\n" "\n" @@ -376,8 +397,7 @@ Image::resize(const Py::Tuple& args, const Py::Dict& kwargs) ras.clip_box(0, 0, numcols, numrows); - //srcMatrix *= resizingMatrix; - //imageMatrix *= resizingMatrix; + imageMatrix = srcMatrix; imageMatrix.invert(); interpolator_type interpolator(imageMatrix); @@ -733,6 +753,7 @@ Image::init_type() add_varargs_method("get_size_out", &Image::get_size_out, Image::get_size_out__doc__); add_varargs_method("reset_matrix", &Image::reset_matrix, Image::reset_matrix__doc__); add_varargs_method("get_matrix", &Image::get_matrix, Image::get_matrix__doc__); + add_varargs_method("set_matrix", &Image::set_matrix, Image::set_matrix__doc__); add_keyword_method("resize", &Image::resize, Image::resize__doc__); add_varargs_method("set_interpolation", &Image::set_interpolation, Image::set_interpolation__doc__); add_varargs_method("set_resample", &Image::set_resample, Image::set_resample__doc__); diff --git a/src/_image.h b/src/_image.h index 8a3be54..89a923c 100644 --- a/src/_image.h +++ b/src/_image.h @@ -34,6 +34,7 @@ public: Py::Object buffer_rgba(const Py::Tuple& args); Py::Object reset_matrix(const Py::Tuple& args); Py::Object get_matrix(const Py::Tuple& args); + Py::Object set_matrix(const Py::Tuple& args); Py::Object resize(const Py::Tuple& args, const Py::Dict& kwargs); Py::Object get_aspect(const Py::Tuple& args); Py::Object get_size(const Py::Tuple& args); @@ -105,6 +106,7 @@ private: static char buffer_rgba__doc__[]; static char reset_matrix__doc__[]; static char get_matrix__doc__[]; + static char set_matrix__doc__[]; static char resize__doc__[]; static char get_aspect__doc__[]; static char get_size__doc__[];
Looks good. Does matplotlib still pass all regression tests with this change? (See here for information on running the regression tests: http://matplotlib.sourceforge.net/devel/coding_guide.html?highlight=nosetests#testing). Cheers, Mike On 07/06/2011 08:33 AM, Martin Teichmann wrote: > Dear list, > > I was just trying to shear an image to be plotted with > matplotlib (to get some snazzy 3D effect) and realized > that it's apparently not possible. Investigating further, > I realized that the underlying agg library indeed supports > shearing, as it simply uses an affine matrix for its > transforms (very much like the rest of matplotlib), > but it does not export that feature to matplotlib. > > So, I just quickly added some code to actually > access the transformation matrix in the C++ code, > so that one can use it from within python. The next > step would be to hook that up to the usual code used > in matplotlib. > > A patch is attached at the end of this post. > > Greetings > > Martin > > --------------- patch follows ----------------------- > > commit b7d0d23d90460ee790f1e94f387070a69be661c8 > Author: Martin Teichmann<martin@renoir.(none)> > Date: Wed Jul 6 14:04:44 2011 +0200 > > Make affine transformations work for images > > The current code only supports scaling and maybe rotation of > images, but not all affine transformations, although it is > already prepared to do so. > > This patch adds a method set_matrix to the C++ image handling code, > so that one can set (and thus perform) arbitrary affine transformations. > > It also fixes a little bug which introduced weird side effects > if images were resized more than once. > > diff --git a/src/_image.cpp b/src/_image.cpp > index 3278b6c..7d40664 100644 > --- a/src/_image.cpp > +++ b/src/_image.cpp > @@ -92,7 +92,6 @@ Image::apply_rotation(const Py::Tuple& args) > > agg::trans_affine M = agg::trans_affine_rotation(r * agg::pi / 180.0); > srcMatrix *= M; > - imageMatrix *= M; > return Py::Object(); > } > > @@ -156,7 +155,6 @@ Image::apply_scaling(const Py::Tuple& args) > //printf("applying scaling %1.2f, %1.2f\n", sx, sy); > agg::trans_affine M = agg::trans_affine_scaling(sx, sy); > srcMatrix *= M; > - imageMatrix *= M; > > return Py::Object(); > } > @@ -179,7 +177,6 @@ Image::apply_translation(const Py::Tuple& args) > //printf("applying translation %1.2f, %1.2f\n", tx, ty); > agg::trans_affine M = agg::trans_affine_translation(tx, ty); > srcMatrix *= M; > - imageMatrix *= M; > > return Py::Object(); > } > @@ -285,7 +282,6 @@ Image::reset_matrix(const Py::Tuple& args) > > args.verify_length(0); > srcMatrix.reset(); > - imageMatrix.reset(); > > return Py::Object(); > } > @@ -316,6 +312,31 @@ Image::get_matrix(const Py::Tuple& args) > return ret; > } > > +char Image::set_matrix__doc__[] = > + "set_matrix(m11,m21,m12,m22,m13,m23)\n" > + "\n" > + "Set the affine transformation matrix\n" > + " /m11,m12,m13\\\n" > + " /m21,m22,m23|\n" > + " \\ 0 , 0 , 1 /" > + ; > + > +Py::Object > +Image::set_matrix(const Py::Tuple& args) > +{ > + _VERBOSE("Image::set_matrix"); > + > + args.verify_length(6); > + > + double m[6]; > + for (int i = 0;i< 6;i++) > + { > + m[i] = Py::Float(args[i]); > + } > + srcMatrix.load_from(m); > + return Py::Object(); > +} > + > char Image::resize__doc__[] = > "resize(width, height, norm=1, radius=4.0)\n" > "\n" > @@ -376,8 +397,7 @@ Image::resize(const Py::Tuple& args, const Py::Dict& kwargs) > > ras.clip_box(0, 0, numcols, numrows); > > - //srcMatrix *= resizingMatrix; > - //imageMatrix *= resizingMatrix; > + imageMatrix = srcMatrix; > imageMatrix.invert(); > interpolator_type interpolator(imageMatrix); > > @@ -733,6 +753,7 @@ Image::init_type() > add_varargs_method("get_size_out",&Image::get_size_out, > Image::get_size_out__doc__); > add_varargs_method("reset_matrix",&Image::reset_matrix, > Image::reset_matrix__doc__); > add_varargs_method("get_matrix",&Image::get_matrix, > Image::get_matrix__doc__); > + add_varargs_method("set_matrix",&Image::set_matrix, > Image::set_matrix__doc__); > add_keyword_method("resize",&Image::resize, Image::resize__doc__); > add_varargs_method("set_interpolation", > &Image::set_interpolation, Image::set_interpolation__doc__); > add_varargs_method("set_resample",&Image::set_resample, > Image::set_resample__doc__); > diff --git a/src/_image.h b/src/_image.h > index 8a3be54..89a923c 100644 > --- a/src/_image.h > +++ b/src/_image.h > @@ -34,6 +34,7 @@ public: > Py::Object buffer_rgba(const Py::Tuple& args); > Py::Object reset_matrix(const Py::Tuple& args); > Py::Object get_matrix(const Py::Tuple& args); > + Py::Object set_matrix(const Py::Tuple& args); > Py::Object resize(const Py::Tuple& args, const Py::Dict& kwargs); > Py::Object get_aspect(const Py::Tuple& args); > Py::Object get_size(const Py::Tuple& args); > @@ -105,6 +106,7 @@ private: > static char buffer_rgba__doc__[]; > static char reset_matrix__doc__[]; > static char get_matrix__doc__[]; > + static char set_matrix__doc__[]; > static char resize__doc__[]; > static char get_aspect__doc__[]; > static char get_size__doc__[]; > > ------------------------------------------------------------------------------ > All of the data generated in your IT infrastructure is seriously valuable. > Why? It contains a definitive record of application performance, security > threats, fraudulent activity, and more. Splunk takes this data and makes > sense of it. IT sense. And common sense. > http://p.sf.net/sfu/splunk-d2d-c2 > _______________________________________________ > Matplotlib-devel mailing list > Mat...@li... > https://lists.sourceforge.net/lists/listinfo/matplotlib-devel
Dear List, dear Michael, > Looks good. Does matplotlib still pass all regression tests with this > change? It does pass all regression tests that were passed with the git version I started with. (There were 10 failures which are still there). In the meantime, I also wrote a class that already uses my extension. With it, you can plot rotated or sheared images with all backends. (There were some quirks to get sheard images on some backends, see examples/api/demo_affine_image.py, but it worked on some backends only). While writing it, I found some inconsistencies in matplotlib: - bounding boxes are not correctly transformed. BboxBase.tranformed only transformes the outer points of a bounding box, if you rotate it, this will give wrong results for several angles. I wrote a function that should be correct for all affine transformations: def transform_bbox(bbox, trans): x0, y0, x1, y1 = bbox.extents tx0, ty0 = trans.transform([x0, y0]) tx1, ty1 = trans.transform([x1, y1]) tx2, ty2 = trans.transform([x1, y0]) tx3, ty3 = trans.transform([x0, y1]) return Bbox.from_extents(min(tx0, tx1, tx2, tx3), min(ty0, ty1, ty2, ty3), max(tx0, tx1, tx2, tx3), max(ty0, ty1, ty2, ty3)) - The other inconsistency is that within matplotlib, extents are defined different: in imshow, the parameter extent expects the order (left, right, bottom, top), while BboxBase.extents is (left, bottom, right, top). This should be changed in the future, maybe the move to python 3 is a good time for that? But now to my code to draw images. It's a new class inheriting AxesImage, but is supposed to once replace AxesImage, as it is compatible. I'm re-writing _draw_unsampled_image, to actually draw a sampled image. Thats only because make_image, the method to be rewritten for a sampled image, is not flexible enough (the caller draws the image, but there is no way for make_image to tell where that image is to be put). In the future, the methods should be renamed (it's a private method, so that's no problem). class ShearImage(AxesImage): def _check_unsampled_image(self, _): return True def _draw_unsampled_image(self, renderer, gc): """ actually, draw sampled image. This method is more flexible than make_image """ mag = renderer.get_image_magnification() trans = Affine2D().scale(mag, mag) + self.get_transform() + \ self.axes.transData.get_affine() bbox = self.axes.bbox viewLim = transform_bbox(bbox, trans.inverted()) im, xmin, ymin, dxintv, dyintv, sx, sy = \ self._get_unsampled_image(self._A, self.get_extent(), viewLim) if im is None: return # I'm not if this check is required. -JJL im.set_interpolation(self._interpd[self._interpolation]) im.set_resample(self._resample) fc = self.axes.patch.get_facecolor() bg = mcolors.colorConverter.to_rgba(fc, 0) im.set_bg( *bg) # uncomment the following line to see the extent to which the image # is drawn # im.set_bg(0, 0, 0, 100) numrows, numcols = im.get_size() ex = self.get_extent() tex = Bbox.from_extents([ex[0], ex[2], ex[1], ex[3]]) tex = transform_bbox(tex, trans) if tex.xmin < bbox.xmin: left = bbox.xmin tx = 0 else: left = tex.xmin tx = tex.xmin - bbox.xmin if tex.ymin < bbox.ymin: bottom = bbox.ymin ty = 0 else: bottom = tex.ymin ty = tex.ymin - bbox.ymin trans = Affine2D().scale(dxintv / numcols, dyintv / numrows).translate(xmin, ymin) + \ trans + \ Affine2D().translate(-bbox.xmin - tx, -bbox.ymin - ty) im.set_matrix(*trans.get_matrix()[:2, :].T.ravel()) width = min(tex.xmax, bbox.xmax) - left height = min(tex.ymax, bbox.ymax) - bottom if width <= 0 or height <= 0: return im.resize(width * mag, height * mag, norm=self._filternorm, radius=self._filterrad) im._url = self.get_url() renderer.draw_image(gc, left, bottom, im) Last but not least, a little script to test the above. It shows a rotated image. You can scale and move the image nicely. If you uncomment the line mentioned above in ShearImage code, you can see where the image is actually drawn, and you will see that only the necessary parts are drawn if the image is smaller than the entire axes. The test script follows: from pylab import * ax = axes() im = ShearImage(ax) im.set_data(fromfunction(lambda x, y: sin(x + y ** 2), (100, 100))) im.set_extent(im.get_extent()) transform = Affine2D().rotate_deg(30) im.set_transform(transform) ax.images.append(im) show() Greetings Martin
On Tue, Jul 12, 2011 at 7:01 PM, Martin Teichmann <mar...@mb...> wrote: > (There were some quirks to get sheard images on some backends, > see examples/api/demo_affine_image.py, but it worked on some backends > only). I didn't have time to go through your code carefully, but my understanding is that you rely on Agg to get a transformed image (resampled image) and provide that image to backends, right? Note that when interpolation mode is "none", resampling should be prohibited. There is a good reason why we implemented the "none" interpolation mode even though this is not supported for some backends. Overriding this behavior (which I guess is the case of you current ShearImage implementation. Please correct me if I'm wrong) won't be acceptable. Please makes sure that resampling of images only happens when the interpolation mode is not "none". Regards, -JJ
I'm starting to get a better sense of the code now. One of the features of the current implementation is that images are resampled before going into the output of the vector backends, so that we can a) control file size and b) control the interpolation algorithm used. It looks like that separation is maintained with this approach, only difference is that images can be both prerotated as well as prescaled now. Is that correct? (I must admit, I've work on bits and pieces of the image code at the edges, but I don't understand it in great detail). It looks like overriding "draw_unsampled_image" is the wrong thing to do here, though. The purpose of that function is to draw an image in a vector backend without any resampling at all -- and here you seem to be adding that. It also only gets called when interpolation is "none" or "nearest", so your helpful changes currently don't work with other forms of interpolation. I think we need to either add a function "_draw_sampled_image", or modify make_image so it returns an offset with the new location to draw the image (because rotation may cause the bounds of the image to be shifted). This is great work, and is looking to be fairly significant. Would you be able to set up a personal git fork and branch for you work? It would make it easier for the rest of us to evaluate what you have done and kick the tires a bit. The outline of how to do this is here: http://matplotlib.sourceforge.net/devel/gitwash/index.html Cheers, Mike On 07/12/2011 06:01 AM, Martin Teichmann wrote: > Dear List, > dear Michael, > >> Looks good. Does matplotlib still pass all regression tests with this >> change? > It does pass all regression tests that were passed with the git version > I started with. (There were 10 failures which are still there). > > In the meantime, I also wrote a class that already uses my extension. > With it, you can plot rotated or sheared images with all backends. > (There were some quirks to get sheard images on some backends, > see examples/api/demo_affine_image.py, but it worked on some backends > only). > > While writing it, I found some inconsistencies in matplotlib: > > - bounding boxes are not correctly transformed. BboxBase.tranformed only > transformes the outer points of a bounding box, if you rotate it, this will give > wrong results for several angles. I wrote a function that should be correct for > all affine transformations: > > def transform_bbox(bbox, trans): > x0, y0, x1, y1 = bbox.extents > tx0, ty0 = trans.transform([x0, y0]) > tx1, ty1 = trans.transform([x1, y1]) > tx2, ty2 = trans.transform([x1, y0]) > tx3, ty3 = trans.transform([x0, y1]) > return Bbox.from_extents(min(tx0, tx1, tx2, tx3), min(ty0, ty1, ty2, ty3), > max(tx0, tx1, tx2, tx3), max(ty0, ty1, ty2, ty3)) > > - The other inconsistency is that within matplotlib, extents are > defined different: > in imshow, the parameter extent expects the order (left, right, bottom, top), > while BboxBase.extents is (left, bottom, right, top). This should be changed in > the future, maybe the move to python 3 is a good time for that? > > But now to my code to draw images. It's a new class inheriting AxesImage, > but is supposed to once replace AxesImage, as it is compatible. > > I'm re-writing _draw_unsampled_image, to actually draw a sampled image. > Thats only because make_image, the method to be rewritten for a sampled > image, is not flexible enough (the caller draws the image, but there is no way > for make_image to tell where that image is to be put). In the future, > the methods > should be renamed (it's a private method, so that's no problem). > > class ShearImage(AxesImage): > def _check_unsampled_image(self, _): > return True > > def _draw_unsampled_image(self, renderer, gc): > """ > actually, draw sampled image. This method is more flexible than > make_image > """ > mag = renderer.get_image_magnification() > trans = Affine2D().scale(mag, mag) + self.get_transform() + \ > self.axes.transData.get_affine() > bbox = self.axes.bbox > viewLim = transform_bbox(bbox, trans.inverted()) > > im, xmin, ymin, dxintv, dyintv, sx, sy = \ > self._get_unsampled_image(self._A, self.get_extent(), viewLim) > > if im is None: return # I'm not if this check is required. -JJL > im.set_interpolation(self._interpd[self._interpolation]) > im.set_resample(self._resample) > > fc = self.axes.patch.get_facecolor() > bg = mcolors.colorConverter.to_rgba(fc, 0) > im.set_bg( *bg) > # uncomment the following line to see the extent to which the image > # is drawn > # im.set_bg(0, 0, 0, 100) > numrows, numcols = im.get_size() > > ex = self.get_extent() > tex = Bbox.from_extents([ex[0], ex[2], ex[1], ex[3]]) > tex = transform_bbox(tex, trans) > if tex.xmin< bbox.xmin: > left = bbox.xmin > tx = 0 > else: > left = tex.xmin > tx = tex.xmin - bbox.xmin > if tex.ymin< bbox.ymin: > bottom = bbox.ymin > ty = 0 > else: > bottom = tex.ymin > ty = tex.ymin - bbox.ymin > trans = Affine2D().scale(dxintv / numcols, > dyintv / numrows).translate(xmin, ymin) + \ > trans + \ > Affine2D().translate(-bbox.xmin - tx, -bbox.ymin - ty) > im.set_matrix(*trans.get_matrix()[:2, :].T.ravel()) > > width = min(tex.xmax, bbox.xmax) - left > height = min(tex.ymax, bbox.ymax) - bottom > if width<= 0 or height<= 0: > return > im.resize(width * mag, height * mag, > norm=self._filternorm, radius=self._filterrad) > > im._url = self.get_url() > > renderer.draw_image(gc, left, bottom, im) > > Last but not least, a little script to test the above. It shows a rotated > image. You can scale and move the image nicely. If you uncomment the > line mentioned above in ShearImage code, you can see where the > image is actually drawn, and you will see that only the necessary parts > are drawn if the image is smaller than the entire axes. > The test script follows: > > from pylab import * > ax = axes() > im = ShearImage(ax) > im.set_data(fromfunction(lambda x, y: sin(x + y ** 2), (100, 100))) > im.set_extent(im.get_extent()) > transform = Affine2D().rotate_deg(30) > im.set_transform(transform) > ax.images.append(im) > show() > > Greetings > > Martin > > ------------------------------------------------------------------------------ > AppSumo Presents a FREE Video for the SourceForge Community by Eric > Ries, the creator of the Lean Startup Methodology on "Lean Startup > Secrets Revealed." This video shows you how to validate your ideas, > optimize your ideas and identify your business strategy. > http://p.sf.net/sfu/appsumosfdev2dev > _______________________________________________ > Matplotlib-devel mailing list > Mat...@li... > https://lists.sourceforge.net/lists/listinfo/matplotlib-devel
Hello List, Hello Michael, Hello Jae-Joon, Michale wrote: > It looks like overriding "draw_unsampled_image" is the wrong thing to do > here, though. JJ wrote: > Overriding this behavior (which I guess is the case of you current ShearImage > implementation. Please correct me if I'm wrong) won't be acceptable. I was (and am) aware of this problem, this is what I meant when I wrote "I should have overridden make_image". That's exactly the point: there are two ways to make the image: one using Agg and fancy interpolations, the other one to use the backend and have the interpolation "none". The former is done by make_image, the latter by _draw_unsampled_image. It would have been correct for me to override make_image, unfortunately this was not possible since make_image only makes the image, it is then drawn by _AxesImageBase.draw, which unfortunately is not flexible enough, as it does not allow for the image to be drawn elsewhere than the origin of the axes, which is not necessarily what I want. This is why I abused _draw_unsampled_image. I am aware that this is an abuse, but my code was only inteded as demonstration code. I don't know where to go from here. We could: - make make_image actually draw the image. This would be a break in the API. - introduce a new method (_draw_sampled_image seems like a good name) that does that. That indirectly changes the API since make_image won't be called anymore, overriding it doesn't make a difference anymore. - Rewrite matplotlib from scratch in Java (just kidding) I'm looking forward to your ideas. Greetings Martin -- Fachbereich Physik Freie Universität Berlin Arnimallee 14 14195 Berlin +49 30 6392 1234
On 07/14/2011 07:33 AM, Martin Teichmann wrote: > Hello List, > Hello Michael, Hello Jae-Joon, > > Michale wrote: >> It looks like overriding "draw_unsampled_image" is the wrong thing to do >> here, though. > JJ wrote: >> Overriding this behavior (which I guess is the case of you current ShearImage >> implementation. Please correct me if I'm wrong) won't be acceptable. > > I don't know where to go from here. We could: > > - make make_image actually draw the image. This would be a break in > the API. > - introduce a new method (_draw_sampled_image seems like a good name) > that does that. That indirectly changes the API since make_image won't > be called anymore, overriding it doesn't make a difference anymore. I think either of these is ok. make_image is not really in the public API anyway -- I know it doesn't have a preceding underscore, but I think this code may predate that convention. If you add _draw_sampled_image, I would remove make_image to eliminate confusion. > - Rewrite matplotlib from scratch in Java (just kidding) We generally, consider patches in first-come-first-serve order, so I would recommend taking whichever approach could be completed sooner ;) Cheers, Mike -- Michael Droettboom Science Software Branch Space Telescope Science Institute Baltimore, Maryland, USA