Specialized heuristic for bucketing colors and filter pre-bucketing #197

usaradark commented 2025年12月09日 20:41:28 +01:00

This pull request serves to bucket colors more intuitively to how humans perceive the difference in colors.

Currently, colors are bucketed via DeltaE or its Improved variant. DeltaE describes the mathematical difference between a color's components with another color. While this is mathematically accurate, it isn't perceptually accurate or intuitive to how humans would bucket colors. This results in bucketing where neighboring similar-appearing colors of the same hue are not bucketed due to differing lightness or colorfuless.

This PR proposes a specialized heuristic for bucketing colors. The general idea is is that humans tend to bucket (group) colors based on hue first, then lightness, then colorfulness. The latter two components may be swapped depending on results. The actual implementation idea is to put less weight on hue differences when doing diff(), though when walking through the code, diff() might actually make more sense to be named similar() based on what it is calculating. I will elaborate more on this once I better understand what the actual logic and intentions are.

This pull request serves to bucket colors more intuitively to how humans perceive the difference in colors. Currently, colors are bucketed via `DeltaE` or its `Improved` variant. `DeltaE` describes the mathematical difference between a color's components with another color. While this is mathematically accurate, it isn't perceptually accurate or intuitive to how humans would bucket colors. This results in bucketing where neighboring similar-appearing colors of the same hue are not bucketed due to differing lightness or colorfuless. This PR proposes a specialized heuristic for bucketing colors. The general idea is is that humans tend to bucket (group) colors based on hue first, then lightness, then colorfulness. The latter two components may be swapped depending on results. The actual implementation idea is to put less weight on hue differences when doing `diff()`, though when walking through the code, `diff()` might actually make more sense to be named `similar()` based on what it is calculating. I will elaborate more on this once I better understand what the actual logic and intentions are.

usaradark commented 2025年12月09日 20:54:13 +01:00

An interesting problem I ran into is that post_read()'s filtering of colors post-bucketing results in bucketing issues.

If I understand the logic correctly, with no mix, the first new color that fits no buckets creates a new bucket. This bucket officially belongs to that new color. If this bucket gets dropped during the filter, that includes all of the colors binned into this bucket.

image

This image shows a bucketing exclusively by hue (for demonstration purposes). As you can see, there's a lot of purple. However, because the first color, which is what this bucket is, is very dark, this entire bucket and its colors will get dropped by the post_read() filter. I don't think this is quite what we want when filtering colors. I think we more-so want to filter the individual color, not the post-bucketed color (and all its bucketed colors).

If bucketing included lightness and colorfulness, this interaction would be much less exaggerated, but it's something to consider. It's possible that integrating color filtering during the bucketing step might yield better results, but it might be an issue if the user has a very dark or bright background, as the filter might not count the background properly. But regardless of if filtering before or after, a very dark/bright background is problematic.

An interesting problem I ran into is that `post_read()`'s filtering of colors post-bucketing results in bucketing issues. If I understand the logic correctly, with no `mix`, the first new color that fits no buckets creates a new bucket. This bucket officially belongs to that new color. If this bucket gets dropped during the filter, that includes all of the colors binned into this bucket.  This image shows a bucketing exclusively by hue (for demonstration purposes). As you can see, there's a lot of purple. However, because the first color, which is what this bucket is, is very dark, this entire bucket and its colors will get dropped by the `post_read()` filter. I don't think this is quite what we want when filtering colors. I think we more-so want to filter the individual color, not the post-bucketed color (and all its bucketed colors). If bucketing included lightness and colorfulness, this interaction would be much less exaggerated, but it's something to consider. It's possible that integrating color filtering during the bucketing step might yield better results, but it might be an issue if the user has a very dark or bright background, as the filter might not count the background properly. But regardless of if filtering before or after, a very dark/bright background is problematic.

usaradark commented 2025年12月10日 03:31:35 +01:00

Threshold: 20

Old
image

New
image

---

It isn't very obvious at a glance but essentially...

• colors are much more aggressively bucketed by hue
• deltas better map to thresholding value
  • colorfulness is a value that soft-caps around 40
  • hue delta only goes up to 180
  • those two components are now mapped similarly to lightness, having 1-100

However, because I'm using Cam16Jmh to separate hue and colorfulness, there are edge cases.

• delta hue has no affect towards 0 lightness (black for all hues)
• delta hue has no affect towards 0 colorfulness (grey for all hues)
• colorfulness has no affect towards 0 lightness (black for all colorfulness)

For deltas on Cam16Jab this is a non-issue because ab creates a coupling that accounts for these oddities(?). Regardless, we can't interface with Cam16Jab the same intuitive way as we can with Cam16Jmh.

To fix this, it's similar to what I had to do when I implemented salience in colorspace lch. I expect this to be an easier fix.

---

I originally had to do this fix for salience calculations for the lch colorspace. That was because I was working with components lch and not lab. This time for Cam16 I opted for salience calculations in Cam16Jab. Rather than finetune saliences for hue and chroma, I just grouped all color under ab, so it's more like a knob for "light" and "color".

...

But after some testing, it seems Jab does NOT fix this issue... which means I do have to apply these fixes regardless.

The good news is that this will improve overall salience calculations. And the more interesting bit is that it's possible that salience-sort and salience-bucketing will use the same calculations. More testing will need to be done.

Threshold: 20 Old  New  --- It isn't very obvious at a glance but essentially... - colors are much more aggressively bucketed by hue - deltas better map to thresholding value - colorfulness is a value that soft-caps around 40 - hue delta only goes up to 180 - those two components are now mapped similarly to lightness, having 1-100 However, because I'm using `Cam16Jmh` to separate hue and colorfulness, there are edge cases. - delta hue has no affect towards 0 lightness (black for all hues) - delta hue has no affect towards 0 colorfulness (grey for all hues) - colorfulness has no affect towards 0 lightness (black for all colorfulness) For deltas on `Cam16Jab` this is a non-issue because `ab` creates a coupling that accounts for these oddities(?). Regardless, we can't interface with `Cam16Jab` the same intuitive way as we can with `Cam16Jmh`. To fix this, it's similar to what I had to do when I implemented `salience` in colorspace `lch`. I expect this to be an easier fix. --- I originally had to do this fix for salience calculations for the `lch` colorspace. That was because I was working with components `lch` and not `lab`. This time for `Cam16` I opted for salience calculations in `Cam16Jab`. Rather than finetune saliences for hue and chroma, I just grouped all color under `ab`, so it's more like a knob for "light" and "color". ... But after some testing, it seems `Jab` does NOT fix this issue... which means I do have to apply these fixes regardless. The good news is that this will improve overall salience calculations. And the more interesting bit is that it's possible that salience-sort and salience-bucketing will use the same calculations. More testing will need to be done.

usaradark commented 2025年12月10日 05:16:24 +01:00

With new changes and fixes on component dependencies.

New new:
image

I individually tweaked each components j, m, and h such that at threshold 20, there is "good" contrast. Good contrast is a reasonable amount after JND where I don't have to put in the effort to distinguish colors.

My methodology on deciding bucketing heuristics is as follows:

• Threshold maps linearly to lightness as a baseline
• Colorfulness rarely goes above 40, so I mapped 0-100 to 0-40 on colorfulness
• Hue delta goes from 0-180, but i remapped to 0-100 for interfacing. I found this buckets colors a bit too aggressively so a weight adjustment was made until "good" contrast.

Combine, sum-weights to 3.0, and now we have what we have.

I'm going to revert my change to filtering for now until I get your feedback on that idea. From my initial results on my branch, the results are promising as it doesn't drop colors that shouldn't be dropped due to bucketing. It's (mostly) an easy fix.

With new changes and fixes on component dependencies. New new:  I individually tweaked each components `j`, `m`, and `h` such that at threshold `20`, there is "good" contrast. Good contrast is a reasonable amount after JND where I don't have to put in the effort to distinguish colors. My methodology on deciding bucketing heuristics is as follows: - Threshold maps linearly to lightness as a baseline - Colorfulness rarely goes above 40, so I mapped 0-100 to 0-40 on colorfulness - Hue delta goes from 0-180, but i remapped to 0-100 for interfacing. I found this buckets colors a bit too aggressively so a weight adjustment was made until "good" contrast. Combine, sum-weights to 3.0, and now we have what we have. I'm going to revert my change to filtering for now until I get your feedback on that idea. From my initial results on my branch, the results are promising as it doesn't drop colors that shouldn't be dropped due to bucketing. It's (mostly) an easy fix.

usaradark commented 2025年12月10日 05:50:00 +01:00

I see c4ac151b17 opted to merge diff into the Differences trait. This is a note for me to also do that...

I see c4ac151b17 opted to merge `diff` into the `Differences` trait. This is a note for me to also do that...

explosion-mental commented 2025年12月10日 07:04:53 +01:00

@usaradark wrote in #197 (comment):

With new changes and fixes on component dependencies.

New new: image

I individually tweaked each components j, m, and h such that at threshold 20, there is "good" contrast. Good contrast is a reasonable amount after JND where I don't have to put in the effort to distinguish colors.

My methodology on deciding bucketing heuristics is as follows:

* Threshold maps linearly to lightness as a baseline
* Colorfulness rarely goes above 40, so I mapped 0-100 to 0-40 on colorfulness
* Hue delta goes from 0-180, but i remapped to 0-100 for interfacing. I found this buckets colors a bit too aggressively so a weight adjustment was made until "good" contrast.

Combine, sum-weights to 3.0, and now we have what we have.

I'm going to revert my change to filtering for now until I get your feedback on that idea. From my initial results on my branch, the results are promising as it doesn't drop colors that shouldn't be dropped due to bucketing. It's (mostly) an easy fix.

Seems to lose some salience, think the first color that comes to mind w that image is pink.

@usaradark wrote in https://codeberg.org/explosion-mental/wallust/pulls/197#issuecomment-8835234: > With new changes and fixes on component dependencies. > > New new: [](/explosion-mental/wallust/attachments/8d96b1e5-4a43-40f1-b809-de82797ee741) > > I individually tweaked each components `j`, `m`, and `h` such that at threshold `20`, there is "good" contrast. Good contrast is a reasonable amount after JND where I don't have to put in the effort to distinguish colors. > > My methodology on deciding bucketing heuristics is as follows: > > * Threshold maps linearly to lightness as a baseline > > * Colorfulness rarely goes above 40, so I mapped 0-100 to 0-40 on colorfulness > > * Hue delta goes from 0-180, but i remapped to 0-100 for interfacing. I found this buckets colors a bit too aggressively so a weight adjustment was made until "good" contrast. > > > Combine, sum-weights to 3.0, and now we have what we have. > > I'm going to revert my change to filtering for now until I get your feedback on that idea. From my initial results on my branch, the results are promising as it doesn't drop colors that shouldn't be dropped due to bucketing. It's (mostly) an easy fix. Seems to lose some salience, think the first color that comes to mind w that image is pink.

usaradark commented 2025年12月10日 07:10:56 +01:00

It's more or less done. I had to bring back pre-histo color filtering, as it seems necessary or else entire buckets worth of useful color within dropped buckets are gone.

Threshold 20:
2025年12月09日T21:49:28,708166192-08:00

Some other todo's on my end which will probably be on a different PR:

• add earlier fixes of jmh from bucketing to salience
• figure out more stuff tomorrow

It's more or less done. I had to bring back pre-histo color filtering, as it seems necessary or else entire buckets worth of useful color within dropped buckets are gone. Threshold 20:  Some other todo's on my end which will probably be on a different PR: - add earlier fixes of jmh from bucketing to salience - figure out more stuff tomorrow

usaradark commented 2025年12月10日 07:15:13 +01:00

@explosion-mental wrote in #197 (comment):

Seems to lose some salience, think the first color that comes to mind w that image is pink.

I'm not too sure what you mean by that statement.

Pink is the is not only the most salient color in the image, which is sorted last (working as intended), but it's also rather significant to the image (not just a small highlight of it, but we currently don't do anything about "significance" or "prominancy"). That said, there isn't a lot of shades of that hot pink, and even if there were, it gets bucketed with the somewhat sane threshold of 20.

If we wanted more shades of pink, here is an example with threshold 10.
image

Or if that's too many colors, here's 15.
image

edit addition:
Note that pink is actually three properties: moderately-high lightness, moderate colorfulness, and a red hue. It's a specific color, so asking for "more pink colors" is like asking for "more vanilla flavors"; like what other "vanilla" do you want? Vanilla is a very specific flavor, and it's a similar idea for this hot pink you that is significant to the image.

If a good chunk of the image has a very salient color, that doesn't immediately mean there will be more buckets of it, especially if there there is flat shading, as we can see in this demo.

edit 2 - hue sorted might be easier to interpret

thresh 10
image

thresh 15
image

@explosion-mental wrote in https://codeberg.org/explosion-mental/wallust/pulls/197#issuecomment-8835531: > Seems to lose some salience, think the first color that comes to mind w that image is pink. I'm not too sure what you mean by that statement. Pink is the is not only the most salient color in the image, which is sorted last (working as intended), but it's also rather significant to the image (not just a small highlight of it, but we currently don't do anything about "significance" or "prominancy"). That said, there isn't a lot of shades of that hot pink, and even if there were, it gets bucketed with the somewhat sane threshold of 20. If we wanted more shades of pink, here is an example with threshold 10.  Or if that's too many colors, here's 15.  edit addition: Note that pink is actually three properties: moderately-high lightness, moderate colorfulness, and a red hue. It's a specific color, so asking for "more pink colors" is like asking for "more vanilla flavors"; like what other "vanilla" do you want? Vanilla is a very specific flavor, and it's a similar idea for this hot pink you that is significant to the image. If a good chunk of the image has a very salient color, that doesn't immediately mean there will be more buckets of it, especially if there there is flat shading, as we can see in this demo. edit 2 - hue sorted might be easier to interpret thresh 10  thresh 15 

usaradark commented 2025年12月10日 17:06:16 +01:00

Seems to lose some salience, think the first color that comes to mind w that image is pink.

I think I see what you mean now. Needed some sleep to let that sink in.

We need a way to decide what color we will use as our background. By "pink" coming into your mind, I think you meant that "pink" would be the chosen background, meaning the background would be tinted pink, hence "pink theme". We currently don't have any logic for that (used to). We nuked it when we (or I) nuked histoscore sorting.

On master, for Salience, it grabs the max count histogram bucket and reserves that as the background.

The background is then reserved as the FIRST element in histo. But doing this is prone to error, as for every operation on histo after that reservation just remove the background, operate on histo, then re-insert the background. There were times when I accidentally transformed the background and wasted a few hours wondering why colors weren't working correctly.

But now with the new histo infrastructure, and that generally we have a lot more colors to work with (>16, preferring more colors rather than <=16), we can play more with how we choose the background.

Here's what I'm thinking.

Naive approach

• Grab the histo bucket of max count, set as background

Smarter approach that makes sense as a background

• Backgrounds are supposed to be low in salience
• Max count histo bucket might be highly salient against a dark background. In my demo image, there's actually 2/3 prominent colors, the hot pink, the muted teal, and dim blue.
• Of the top prominent colors, choose the color that has the least salience against a generic dark background (or light, if light).
• Use that prominent, low salience color as the background.

This method prefers unimportant colors towards background. With some additions, we can also allow the user to reasonably "cycle" their background color. They might actually be able to choose if they want pink, teal, OR blue as their base background. The implementation details for that can be decided some other time if we want to do that.

Alternative smarter approach... but might be harder

• Current color filter on the histogram removes blacks, whites, and greys.
• If we relax it just a bit, we can get less salient colors, which use those colors as the background, maybe factor in some "prominency".
• Problem is now our histo has some rather low-salient colors. An additional filter might be needed to remove these colors.
• Another problem, illustrations commonly use black colors for outlines. So we would need to not relax our initial filter too much.
• I don't like this method. There's too many moving parts. Doesn't work reliably for a majority of image types.

---

I'm not too sure how you'd like to proceed with deciding the background color, but more importantly where in the code. Once you figure that one out you can assign me to implement the background choosing details unless you have something very specific in mind or would like to do it yourself.

> Seems to lose some salience, think the first color that comes to mind w that image is pink. I think I see what you mean now. Needed some sleep to let that sink in. We need a way to decide what color we will use as our background. By "pink" coming into your mind, I think you meant that "pink" would be the chosen background, meaning the background would be tinted pink, hence "pink theme". We currently don't have any logic for that (used to). We nuked it when we (or I) nuked histoscore sorting. On master, for `Salience`, it grabs the max count histogram bucket and reserves that as the background. https://codeberg.org/explosion-mental/wallust/src/commit/bb408366ee3b1dc6b2841c9063d29e4f7bbd5e12/src/colorspaces/salience.rs#L133-L147 The `background` is then reserved as the FIRST element in `histo`. But doing this is prone to error, as for every operation on `histo` after that reservation just remove the background, operate on `histo`, then re-insert the background. There were times when I accidentally transformed the `background` and wasted a few hours wondering why colors weren't working correctly. But now with the new `histo` infrastructure, and that generally we have a lot more colors to work with (>16, preferring more colors rather than <=16), we can play more with how we choose the background. Here's what I'm thinking. ### Naive approach - Grab the `histo` bucket of max count, set as background ### Smarter approach that makes sense as a background - Backgrounds are supposed to be low in salience - Max count `histo` bucket might be highly salient against a dark background. In my demo image, there's actually 2/3 prominent colors, the hot pink, the muted teal, and dim blue. - Of the top prominent colors, choose the color that has the least salience against a generic dark background (or light, if light). - Use that prominent, low salience color as the background. This method prefers unimportant colors towards background. With some additions, we can also allow the user to reasonably "cycle" their background color. They might actually be able to choose if they want pink, teal, OR blue as their base background. The implementation details for that can be decided some other time if we want to do that. ### Alternative smarter approach... but might be harder - Current color filter on the histogram removes blacks, whites, and greys. - If we relax it just a bit, we can get less salient colors, which use those colors as the background, maybe factor in some "prominency". - Problem is now our `histo` has some rather low-salient colors. An additional filter might be needed to remove these colors. - Another problem, illustrations commonly use black colors for outlines. So we would need to not relax our initial filter too much. - I don't like this method. There's too many moving parts. Doesn't work reliably for a majority of image types. --- I'm not too sure how you'd like to proceed with deciding the background color, but more importantly where in the code. Once you figure that one out you can assign me to implement the background choosing details unless you have something very specific in mind or would like to do it yourself.

usaradark commented 2025年12月11日 04:56:02 +01:00

Massive logic error... doing some fixes... also applying fix of bucketing on to salience...

Massive logic error... doing some fixes... also applying fix of bucketing on to salience...

usaradark commented 2025年12月11日 05:52:51 +01:00

You got the match for improved vs non-improved reversed, lol. I fixed it.

You got the match for improved vs non-improved reversed, lol. I fixed it.

usaradark commented 2025年12月11日 08:21:41 +01:00

Update
2025年12月10日T23:10:44,992672661-08:00

Calculations for component deltas on bucketing were done incorrectly. Fixed and now bucketing seems to be less conflicting. Maybe hues could still be merged more aggressively, but the user could also just use a higher threshold. But now we have the hue + varying lightness and varying colorfulness. Previously it seemed harder to get something this reliable.

Salience uses similar calculations for bucketing. For now it uses very similar constants, as it actually seems to work decently well. However, later I do plan to make that tool to do a proper testing on sorting for salience to get those weights down, or just an entirely different method. What's here works good enough.

Ready to merge. I don't really want to touch anymore as I'm already slowly getting out of scope for this PR.

Update  Calculations for component deltas on bucketing were done incorrectly. Fixed and now bucketing seems to be less conflicting. Maybe hues could still be merged more aggressively, but the user could also just use a higher threshold. But now we have the hue + varying lightness and varying colorfulness. Previously it seemed harder to get something this reliable. Salience uses similar calculations for bucketing. For now it uses very similar constants, as it actually seems to work decently well. However, later I do plan to make that tool to do a proper testing on sorting for salience to get those weights down, or just an entirely different method. What's here works good enough. Ready to merge. I don't really want to touch anymore as I'm already slowly getting out of scope for this PR.

explosion-mental commented 2026年02月08日 09:39:26 +01:00

@usaradark just wondering, instead of true/false, is there a way to get a number? A qualification of how "good" or how much room there is for salience.
This would help with 5d3b7dfa95 handling better a multithreaded approach. Some of the questions also come down to the threshold. what exactly is a "good threshold"? I think we can leave it as "the best threshold is the one that lets the colors have the greatest salience in between them", or similar.

@usaradark just wondering, instead of true/false, is there a way to get a number? A qualification of how "good" or how much room there is for salience. This would help with https://codeberg.org/explosion-mental/wallust/commit/5d3b7dfa95cd9e29392d706cc16221f420f1e465 handling better a multithreaded approach. Some of the questions also come down to the threshold. what exactly is a "good threshold"? I think we can leave it as "the best threshold is the one that lets the colors have the greatest salience in between them", or similar.

usaradark commented 2026年02月25日 20:50:43 +01:00

@explosion-mental Did you figure this one out yet? To get just the number you would call the underlying delta_e functions.

Or if you're working with a Spec that implements Difference, then you would call the proxy methods.

I remember having this issue with the prior implementation of iirc col_diff method, where it returned a bool instead of a number.

As for the multithreading...

"the best threshold is the one that lets the colors have the greatest salience in between them", or similar.

Did you phrase this correctly? It sounds like you want black and white, lol.

I'm thinking something of somewhere between "somewhat perceptible" to "immediately perceptible" between colors. This should result in the most distinct colors. If you want I can try to figure out this value. But to actually determine it I would need to actually progress and do tests for #199. I can just get the component weights and find a threshold that we can work with. Do you want me to do that?

@explosion-mental Did you figure this one out yet? To get just the number you would call the underlying `delta_e` functions. https://codeberg.org/explosion-mental/wallust/src/commit/2cd542161fa396377e86cb96378bf6c8b86d73ae/src/histogram/diff.rs#L103-L223 Or if you're working with a `Spec` that implements `Difference`, then you would call the proxy methods. I remember having this issue with the prior implementation of iirc `col_diff` method, where it returned a bool instead of a number. As for the multithreading... > "the best threshold is the one that lets the colors have the greatest salience in between them", or similar. Did you phrase this correctly? It sounds like you want black and white, lol. I'm thinking something of somewhere between "somewhat perceptible" to "immediately perceptible" between colors. This should result in the most distinct colors. If you want I can try to figure out this value. But to actually determine it I would need to actually progress and do tests for #199. I can just get the component weights and find a threshold that we can work with. Do you want me to do that?