Would you like one of us to look through the code for possible improvements?

Sure, this is always appreciated!

It looks like there's a PR at mastodon/mastodon#30569 to upgrade libvips from 8.10.5 (December 2020) to the latest 8.15.2, which would then allow GIF resizing/thumbnailing without *magick.

We only support VIPS 8.13, because we want to block unsafe loaders.

This PR is to update our official container image from the stock Debian package (8.14 with too much dependencies) to 8.15 with only the dependencies we want.

we switched to ffmpeg to process animated GIFs, as vips do not support them properly at the moment.

libvips has had good GIF support since 8.13, I think. Though ffmpeg is a fine choice of course.

Not for animated GIFs I think? We need to handle those, and the only way we found was to resize every frame, then reconstruct the image. libvips/libvips#2668 is still open for resizing animated GIFs.

Not for animated GIFs I think?

It should work fine -- by default it'll just do frame zero, but pass n=-1 and it'll do all frames.

john@banana ~/pics $ /usr/bin/time -f %M:%e convert 3198.gif -resize 128x128 x2.gif
170232:10.48
john@banana ~/pics $ /usr/bin/time -f %M:%e vipsthumbnail 3198.gif[n=-1] -o x.gif --size 128
56576:2.21
$ /usr/bin/time -f %M:%e ffmpeg -hide_banner -v warning -i 3198.gif -filter_complex "[0:v] scale=128:-1:flags=lanczos,split [a][b]; [a] palettegen=reserve_transparent=on:transparency_color=ffffff [p]; [b][p] paletteuse" x3.gif
79372:0.59
john@banana ~/pics $ ls -l x*.gif
-rw-rw-r-- 1 john john 1062852 Jun 7 13:55 x2.gif
-rw-rw-r-- 1 john john 974588 Jun 7 13:59 x3.gif
-rw-r--r-- 1 john john 1193725 Jun 7 13:55 x.gif

Looking at the output gifs, libvips is probably the best quality, though ffmpeg is similar. imagemagick is noticeably worse. ffmpeg has the smallest output size, libvips has marginally the lowest memory use, ffmpeg is the fastest by a good distance.

libvips is reoptimising the palette, maybe ffmpeg is not and that explains the difference in CPU time? But I've not checked.

The n=-1 feature works for all animated images, so webp etc. too. Hopefully for animated PNG soon.

Hi @jcupitt, resize works on animated GIFs with n=-1, but not when the crop option is used.

Ah, yes, that's true, for crop you'd need to split to pages, crop, resize and rejoin, as you say.

We're currently using BINS = 10, scanning the vector to extract multiple matches is definitely possible, but it would also be pretty awkward, as we'd have to remember which colors we've already selected.

Hi @ClearlyClaire, nice to meet you!

Now I look at the code more carefully, you're already doing this -- you have getpoint() to get the pixel vector from the histogram image, then find_index to find the z (blue) which contains the maximum value. So I think your code is working.

I made a tiny test program:

#!/usr/bin/ruby
require "vips"
BINS = 16
def rgb_from_xyv(image, x, y, v)
 pixel = image.getpoint(x, y)
 z = pixel.find_index(v)
 r = (x + 0.5) * 256 / BINS
 g = (y + 0.5) * 256 / BINS
 b = (z + 0.5) * 256 / BINS
 [r, g, b]
end
def palette_from_image(image)
 histogram = image.hist_find_ndim(bins: BINS)
 _, colors = histogram.max(size: 10, out_array: true, x_array: true, y_array: true)
 colors['out_array'].map.with_index do |v, i|
 x = colors['x_array'][i]
 y = colors['y_array'][i]
 rgb_from_xyv(histogram, x, y, v)
 end.reverse
end
image = Vips::Image.new_from_file(ARGV[0], access: :sequential)
hist = palette_from_image(image)
puts("hist = #{hist}")

And I see:

$ ./mastodon.rb ~/pics/k2.jpg
hist = [[56.0, 56.0, 72.0], [8.0, 8.0, 8.0], [40.0, 40.0, 72.0], [248.0, 248.0, 248.0], [40.0, 40.0, 56.0], [56.0, 56.0, 56.0], [72.0, 56.0, 72.0], [56.0, 40.0, 72.0], [40.0, 40.0, 24.0], [232.0, 248.0, 248.0]]

ie. the dominant colour is [56.0, 56.0, 72.0], which seem right, from stepping though the process.

I tested with:

$ vips hist_find_ndim k2.jpg x.v --bins 16
$ vips max x.v --size 10 --out-array --x-array --y-array
36250 38784 41021 47482 54575 57936 60979 75699 95407 232563 
14 2 3 4 3 2 15 2 0 3 
15 2 2 3 3 2 15 2 0 3 
232563.000000

Then looked through the histogram image with vipsdisp (it can display these things at least semi-sensibly, and show all the values in the file).

image

https://github.com/jcupitt/vipsdisp

nip2 can be useful for developing libvips code, but it's a big, complicated thing and hard to get to grips with. I'm trying to make a simpler gtk4 version now, but it's not ready yet.

https://github.com/libvips/nip2

Hi!
Consider the following (highly artificial) image:
test

$ ./mastodon.rb test.png
hist = [[8.0, 8.0, 8.0], [248.0, 104.0, 8.0], [248.0, 104.0, 8.0], [248.0, 104.0, 8.0], [248.0, 104.0, 8.0], [104.0, 104.0, 8.0], [104.0, 104.0, 8.0], [104.0, 104.0, 8.0], [104.0, 104.0, 8.0], [8.0, 8.0, 24.0]]

You can see that out of the 9 most present colors, only 3 are represented, and 2 are repeated multiple times because they happen to have exactly the same frequency as other colors which only differ in the blue component.

Yeah, that is a bit cumbersome but I think this would work, after deduplication. Thanks!

Oh, I wasn't aware of bandunfold! This is a really nice simplification, thank you!

Yes, it's nice because it doesn't move any pixels around, just changes the way they are addressed. So it's a "free" operation (just a pointer copy).