I'd like to merge this but I need to be careful with changing API behavior. Thoughts?

No solution for the API but I just want to +1 the fix, I've been bitten by this recently.

I was loading each time the font using dc.LoadFontface(string, float64) when I needed to render in a given size (at this time performances were not in the scope).
Later, for performances improvements, I've added a font cache: caching the font with
gg.LoadFontFace(string, float64) and using the font with dc.SetFontFace(font.Face). I've been surprised that the rendering result was not quite the same.

Yes this is fixing/unifying the SetFontFace method. I'm also trying to do the same thing as @remeh did with different results. Thus we have a fork at the moment with this PR included and it seems to work fine.

However one thing I don't understand is what the hell are these numbers in LoadFontface supposed to mean? What's " * 72 / 96"? 72 of what? and 96 of what?

I had a similar issue with font sizing in the past. Working with a different library in a different language but the root cause is the same I think. The issue was that the font description almost never has correct values for width and height of the font. What I had to do is to use glyphs to get a real text size just before "rendering" it. This is significantly slower because I had to basically render the whole text and get a real height after that but it works every time. You can remove these magic numbers doing that completely at the cost of speed tho.

iirc, 72 is Pixels per Inch and 96 is the assumed DPI for any monitor.

Though i only know this because of working with C# Winforms & WPF. Winforms assumes everything is in PPI (72px) and WPF uses proper N * (PPI / DPI)

This hit me, because i was hosting a winforms control inside WPF and the fontsizes didn't match.