How to store and retrieve a dictionary with redis

Question

# I have the dictionary my_dict
my_dict = {
 'var1' : 5
 'var2' : 9
}
r = redis.StrictRedis()

How would I store my_dict and retrieve it with redis. For example, the following code does not work.

#Code that doesn't work
r.set('this_dict', my_dict) # to store my_dict in this_dict
r.get('this_dict') # to retrieve my_dict

• mirekphd: Redis is used as a data lake here. If you don't need individual key:value pairs at retrieval time time, you can use Parquet in MinIO / S3, will be faster and more scalable (to petabytes rather than gigabytes).

Answer 1

You can do it by hmset (multiple keys can be set using hmset).

hmset("RedisKey", dictionaryToSet)

import redis
conn = redis.Redis('localhost')
user = {"Name":"Pradeep", "Company":"SCTL", "Address":"Mumbai", "Location":"RCP"}
conn.hmset("pythonDict", user)
conn.hgetall("pythonDict")
{'Company': 'SCTL', 'Address': 'Mumbai', 'Location': 'RCP', 'Name': 'Pradeep'}

• andilabs: if it is nested data structure rather than simply dict, e.g containing some arrays etc. serialzie your data with json.dumps() write as string and after retrive from redis user json.loads() for deserializing it back to python data structure
• ryechus: json.dumps() and json.loads() will only work if you are fine with your dictionary keys always being strings. If you aren't then you might consider using pickle.
• Tommy: json is not compatible with bytes so json serilization is not a global solution, e.g., if you have a dict with a bytes value this will not work.
• hlongmore: By way of note, the documentation for hmset does not tell you this, but it raises a DataError if you try to store an empty dict.
• mirekphd: Be warned that the underlying Redis command HMSET has been deprecated (moved to a different function) in version 4.0.0 (July 2017) but redis-py still executes it here in hmset().

Answer 2

you can pickle your dict and save as string.

import pickle
import redis
r = redis.StrictRedis('localhost')
mydict = {1:2,2:3,3:4}
p_mydict = pickle.dumps(mydict)
r.set('mydict',p_mydict)
read_dict = r.get('mydict')
yourdict = pickle.loads(read_dict)

• Tommy: This is true, but depending on the rate of reads and writes, this may add serious overhead. Pickling is a slow operation
• Paradoxis: Please note that if this is used with user input your server is prone to remote code exection, pickle.loads is should only be used on 100% trusted data
• Akhil Mathew: pickle can be dangerous if mishandled. Use msgpack for better serialization of data before storing it into Redis.
• thanos.a: Pickling has also the important down-part that you cannot debug the stored data in Redis as they are binary.
• Donn Lee: In python3 + redis 3.x, i had to use pickle.dumps(mydict, protocol=0) and pickle.loads(str.encode(read_dict))

Answer 3

As the basic answer has already give by other people, I would like to add some to it.

Following are the commands in REDIS to perform basic operations with HashMap/Dictionary/Mapping type values.

1. HGET => Returns value for single key passed
2. HSET => set/updates value for the single key
3. HMGET => Returns value for single/multiple keys passed
4. HMSET => set/updates values for the multiple key
5. HGETALL => Returns all the (key, value) pairs in the mapping.

Following are their respective methods in redis-py library :-

1. HGET => hget
2. HSET => hset
3. HMGET => hmget
4. HMSET => hmset
5. HGETALL => hgetall

All of the above setter methods creates the mapping, if it doesn't exists. All of the above getter methods doesn't raise error/exceptions, if mapping/key in mapping doesn't exists.

Example:
=======
In [98]: import redis
In [99]: conn = redis.Redis('localhost')
In [100]: user = {"Name":"Pradeep", "Company":"SCTL", "Address":"Mumbai", "Location":"RCP"}
In [101]: con.hmset("pythonDict", {"Location": "Ahmedabad"})
Out[101]: True
In [102]: con.hgetall("pythonDict")
Out[102]:
{b'Address': b'Mumbai',
 b'Company': b'SCTL',
 b'Last Name': b'Rajpurohit',
 b'Location': b'Ahmedabad',
 b'Name': b'Mangu Singh'}
In [103]: con.hmset("pythonDict", {"Location": "Ahmedabad", "Company": ["A/C Pri
 ...: sm", "ECW", "Musikaar"]})
Out[103]: True
In [104]: con.hgetall("pythonDict")
Out[104]:
{b'Address': b'Mumbai',
 b'Company': b"['A/C Prism', 'ECW', 'Musikaar']",
 b'Last Name': b'Rajpurohit',
 b'Location': b'Ahmedabad',
 b'Name': b'Mangu Singh'}
In [105]: con.hget("pythonDict", "Name")
Out[105]: b'Mangu Singh'
In [106]: con.hmget("pythonDict", "Name", "Location")
Out[106]: [b'Mangu Singh', b'Ahmedabad']

I hope, it makes things more clear.

• ak3191: how you can make the key dynamic

Answer 4

HMSET is deprecated per the Redis docs. You can now use HSET with a dictionary as follows:

import redis
r = redis.Redis('localhost')
 
key = "hashexample" 
entry = { 
 "version":"1.2.3", 
 "tag":"main", 
 "status":"CREATED", 
 "timeout":"30"
}
r.hset(key, mapping=entry)

---

Caution: very unintuitively, hset won't accept a dictionary (raising an error suggesting it does not accept dictionaries, see [1]) if it is simply passed to the 2nd positional (unnamed) argument. You need to pass the dictionary to a named argument mapping=.

[1] *** redis.exceptions.DataError: Invalid input of type: 'dict'. Convert to a bytes, string, int or float first.

• NealWalters: Thanks! I'm trying to find the doc where all this is spelled out. Do you know where it is. For example, what are the two "Nones" for.
• Saransh Singh: @NealWalters: See the line on the HMSET command page - redis.io/commands/hmset for deprecation warning.
• Jashwant: @Tad Guski, how to you expire hashexample?

Answer 5

If you want to store a python dict in redis, it is better to store it as json string.

import json
import redis
r = redis.StrictRedis(host='localhost', port=6379, db=0)
mydict = { 'var1' : 5, 'var2' : 9, 'var3': [1, 5, 9] }
rval = json.dumps(mydict)
r.set('key1', rval)

While retrieving de-serialize it using json.loads

data = r.get('key1')
result = json.loads(data)
arr = result['var3']

What about types (eg.bytes) that are not serialized by json functions ?

You can write encoder/decoder functions for types that cannot be serialized by json functions. eg. writing base64/ascii encoder/decoder function for byte array.

• Tommy: I downvoted this because some dicts cannot be serialized to JSON, for example, dicts with a bytes value.
• Saji Xavier: You can write an encoder/decoder function (according to the requirement, eg. base64/ascii encoding) for the types that cannot be encoded/decoded by default.
• ZettaCircl: Disagreeing about " ... latter operation is O(N)." N being the number of fields you have link to the key. Doing N SET/GET or 1 HGET/HSET is the same complexity. See : redis.io/commands/hmset Time-wise, HGET/HSET are atomic transaction, and so are performed faster by REDIS. You are just moving the complexity from Redis to Python Code.
• thanos.a: The import json is missing.
• Mark Setchell: @thanos.a I agree - added it.

Answer 6

Another way: you can use RedisWorks library.

pip install redisworks

>>> from redisworks import Root
>>> root = Root()
>>> root.something = {1:"a", "b": {2: 2}} # saves it as Hash type in Redis
...
>>> print(root.something) # loads it from Redis
{'b': {2: 2}, 1: 'a'}
>>> root.something['b'][2]
2

It converts python types to Redis types and vice-versa.

>>> root.sides = [10, [1, 2]] # saves it as list in Redis.
>>> print(root.sides) # loads it from Redis
[10, [1, 2]]
>>> type(root.sides[1])
<class 'list'>

Disclaimer: I wrote the library. Here is the code: https://github.com/seperman/redisworks

• hlongmore: By way of note, RedisWorks uses hmset under the hood if you set a variable to a dict, and thus if you do root.something = {} you will get a DataError, because hmset doesn't allow empty dictionaries. I mention this because the documentation for redis doesn't tell you this.
• Seperman: Interesting. Yes it does use hmset. I will look into this. @hlongmore
• ZettaCircl: But still, can it support bytes in dictionary ?
• disooqi: BTW, Redis.hmset() is deprecated. you should use Redis.hset() instead.
• Seperman: Yeah I need to fix it when I get a chance... PR's are very welcome too!

Answer 7

One might consider using MessagePack which is endorsed by redis.

import msgpack
data = {
 'one': 'one',
 'two': 2,
 'three': [1, 2, 3]
}
await redis.set('my-key', msgpack.packb(data))
val = await redis.get('my-key')
print(msgpack.unpackb(val))
# {'one': 'one', 'two': 2, 'three': [1, 2, 3]}

Using msgpack-python and aioredis

Answer 8

The redis SET command stores a string, not arbitrary data. You could try using the redis HSET command to store the dict as a redis hash with something like

for k,v in my_dict.iteritems():
 r.hset('my_dict', k, v)

but the redis datatypes and python datatypes don't quite line up. Python dicts can be arbitrarily nested, but a redis hash is going to require that your value is a string. Another approach you can take is to convert your python data to string and store that in redis, something like

r.set('this_dict', str(my_dict))

and then when you get the string out you will need to parse it to recreate the python object.

• Narcisse Doudieu Siewe: he can convert his data to json and store the result in redis

Answer 9

An other way you can approach the matter:

import redis
conn = redis.Redis('localhost')
v={'class':'user','grants': 0, 'nome': 'Roberto', 'cognome': 'Brunialti'}
y=str(v)
print(y['nome']) #<=== this return an error as y is actually a string
conn.set('test',y)
z=eval(conn.get('test'))
print(z['nome']) #<=== this really works!

I did not test it for efficiency/speed.

Answer 10

DeprecationWarning: Redis.hmset() is deprecated. Use Redis.hset() instead.

Since HMSET is deprecated you can use HSET:

import redis
r = redis.Redis(host='localhost', port=6379, decode_responses=True)
r.hset('user:23', mapping={'id': 23, 'name': 'ip'})
r.hgetall('user:23')

Answer 11

If you don't know exactly how to organize data in Redis, I did some performance tests, including the results parsing. The dictonary I used (d) had 437.084 keys (md5 format), and the values of this form:

{"path": "G:\tests2687円.3575.json",
 "info": {"f": "foo", "b": "bar"},
 "score": 2.5}

First Test (inserting data into a redis key-value mapping):

conn.hmset('my_dict', d) # 437.084 keys added in 8.98s
conn.info()['used_memory_human'] # 166.94 Mb
for key in d:
 json.loads(conn.hget('my_dict', key).decode('utf-8').replace("'", '"'))
 # 41.1 s
import ast
for key in d:
 ast.literal_eval(conn.hget('my_dict', key).decode('utf-8'))
 # 1min 3s
conn.delete('my_dict') # 526 ms

Second Test (inserting data directly into Redis keys):

for key in d:
 conn.hmset(key, d[key]) # 437.084 keys added in 1min 20s
conn.info()['used_memory_human'] # 326.22 Mb
for key in d:
 json.loads(conn.hgetall(key)[b'info'].decode('utf-8').replace("'", '"'))
 # 1min 11s
for key in d:
 conn.delete(key)
 # 37.3s

As you can see, in the second test, only 'info' values have to be parsed, because the hgetall(key) already returns a dict, but not a nested one.

And of course, the best example of using Redis as python's dicts, is the First Test

Answer 12

Try rejson-py which is relatively new since 2017. Look at this introduction.

from rejson import Client, Path
rj = Client(host='localhost', port=6379)
# Set the key `obj` to some object
obj = {
 'answer': 42,
 'arr': [None, True, 3.14],
 'truth': {
 'coord': 'out there'
 }
}
rj.jsonset('obj', Path.rootPath(), obj)
# Get something
print 'Is there anybody... {}?'.format(
 rj.jsonget('obj', Path('.truth.coord'))
)
# Delete something (or perhaps nothing), append something and pop it
rj.jsondel('obj', Path('.arr[0]'))
rj.jsonarrappend('obj', Path('.arr'), 'something')
print '{} popped!'.format(rj.jsonarrpop('obj', Path('.arr')))
# Update something else
rj.jsonset('obj', Path('.answer'), 2.17)

Answer 13

Try the JSON format for a python dict, It is the standard according to the Redis documentation and fairly simple.

import redis
data = {
 'dog': {
 'scientific-name' : 'Canis familiaris'
 }
}
r = redis.Redis()
r.json().set('doc', '$', data)
doc = r.json().get('doc', '$')
dog = r.json().get('doc', '$.dog')
scientific_name = r.json().get('doc', '$..scientific-name')

Answer 14

Lots of good answers but this worked for me.

• store dictionary
• get dictionary
• nested hash instead of mapping the dict as key to field and value to value like other answers above. (see example 1)
• get all field/values and go from there as normally you would in a project where you want to dump a dict to a redis hash where the dict is a nested hash. (see example 2)

note: these commands were done in the python repl

1. if you want

{'field1': 'Hello', 'field2': 'World'}

enter image description here

Use

r = redis.Redis(host="localhost", port=6379, db=0, decode_responses=True)
pdict = {'field1': 'Hello', 'field2': 'World'}
r.hmset("queues_test", pdict)

Also refer to other answers, particularly Saji Xavier's since its simple and works.

If you want a nested hash like

{'queue1': '{"field1": "Hello", "field2": "World"}'

enter image description here

then

# to set
import json
r = redis.Redis(host="localhost", port=6379, db=0, decode_responses=True)
pdict = {'field1': 'Hello', 'field2': 'World'}
pdict_string = json.dumps(pdict)
r.hset("queues_data", "queue1", pdict_string)
# to get a single field value
r.hget("queues_data", "queue1")
# '{"field1": "Hello", "field2": "World"}'
# to get all fields
data = r.hgetall("queues_data")
# {'queue1': '{"field1": "Hello", "field2": "World"}'
queue1 = data['queue1']
queue1
# '{"field1": "Hello", "field2": "World"}'
result = json.loads(queue1)
result
# {'field1': 'Hello', 'field2': 'World'}
result['field1']
# 'Hello'

Then if you just need the keys/values

list(data.keys())
# ['queue1', 'queue2']
list(data.values())
# ['{"field1": "Hello", "field2": "World"}', '{"field1": "Hello", "field2": "World"}']

Then if you want get the dict back for all values in one line use

lvalues = list(data.values()) 
# ['{"field1": "Hello", "field2": "World"}', '{"field1": "Hello", "field2": "World"}']
[json.loads(x) for x in lvalues]
# [{'field1': 'Hello', 'field2': 'World'}, {'field1': 'Hello', 'field2': 'World'}]

Answer 15

1. convert the dictionary to string (str)
2. Deserialize the dictionary (eval)

import redis
r = redis.Redis(port=6310, decode_responses=True)
mydict = str({ 'var1' : 5, 'var2' : 9, 'var3': [1, 5, 9] })
r.set('key1', mydict)
value = eval(r.get('key1'))

Hope it help. This is a python code