Fast way to Hash Numpy objects for Caching

Question

Implementing a system where, when it comes to the heavy mathematical lifting, I want to do as little as possible.

I'm aware that there are issues with memoisation with numpy objects, and as such implemented a lazy-key cache to avoid the whole "Premature optimisation" argument.

def magic(numpyarg,intarg):
 key = str(numpyarg)+str(intarg)
 try:
 ret = self._cache[key]
 return ret
 except:
 pass
 ... here be dragons ...
 self._cache[key]=value
 return value

but since string conversion takes quite a while...

t=timeit.Timer("str(a)","import numpy;a=numpy.random.rand(10,10)")
t.timeit(number=100000)/100000 = 0.00132s/call

What do people suggest as being 'the better way' to do it?

• tacaswell: possible duplicate of How to hash a large object (dataset) in Python?
• paperskilltrees: Note that str(a) only shows a part of the array, as would be later pointed out in this comment: stackoverflow.com/questions/16589791/…

Answer 1

Borrowed from this answer... so really I guess this is a duplicate:

>>> import hashlib
>>> import numpy
>>> a = numpy.random.rand(10, 100)
>>> b = a.view(numpy.uint8)
>>> hashlib.sha1(b).hexdigest()
'15c61fba5c969e5ed12cee619551881be908f11b'
>>> t=timeit.Timer("hashlib.sha1(a.view(numpy.uint8)).hexdigest()", 
 "import hashlib;import numpy;a=numpy.random.rand(10,10)") 
>>> t.timeit(number=10000)/10000
2.5790500640869139e-05

• jorgeca: Nice! For multidimensional arrays this gives a different hash (for the "same" array) depending on whether it's fortran or c contiguous. If that's an issue, calling np.ascontiguousarray should solve it.
• Cong Ma: Not sure why a known slow hash function sha1 is chosen. SHA-1 is OK for minimising hash collision but poor at speed. For speed you'll need something like murmurhash or xxhash (the latter claims to be even faster).
• senderle: @CongMa, thanks for the extra info. There are lots of options! But as you'll notice, this is already two orders of magnitude faster. And speed is never the only concern. It's probably worth using a well-understood hash if the alternative is only a few millionths of a second faster.

Answer 2

There is a package for this called joblib. Found from this question.

from joblib import Memory
location = './cachedir'
memory = Memory(location)
# Create caching version of magic
magic_cached = memory.cache(magic)
result = magic_cached(...)
# Or (for one-time use)
result = memory.eval(magic, ...)

• Alex Fortin: It would be better to have a quote from those links copied over in your answer, in case these websites go offline.

Answer 3

For small numpy arrays also this might be suitable:

tuple(map(float, a))

if a is the numpy array.

• Maksym Ganenko: Oh yes, tuple is hashable in comparison with list!