homepage

You're right, the result over a set would be unpredictable.
The point of the itertools module is to be able to a) cope with massive amounts of data and b) be a set of tools instead of complete solutions for all problems.
Because of both of the points above, groupby() doesn't load all the data into memory or attempt to sort the data by itself. Furthermore, there's no way for groupby() to know whether the iterable it's passed is going to yield sorted or unsorted data.
It's your responsibility to know whether the iterable you're passing is already sorted or not, and sort it first, if it's possible and there's a need to do so.