Error in json.loads() for data that has base64 decoding applied

Question

I am trying to use json.loads() in python

I get the error:

the JSON object must be str, not 'bytes'

a = {'h': '123', 'w': '12345', 'data': "b'eyJod2lkIjpwomfcwpvepovnepovqrepniLLKJAMSNDMSNDMAWEFMOEDAad='"}
a.update(json.loads(base64.b64decode(a['data'])))

Here the 'data' portion of a was being loaded in as a json dump with b64encoding.

'data':base64.b64encode(json.dumps(test).encode()); where test = some string eg('epovqrepniLLKJAMSNDMSNDMAWEFMOEDAad=')

I have tried using:

a.update(json.loads(base64.b64decode(a['data']).decode('utf-8')))

Giving me a 'utf-8' codec can't decode bytes in position: invalid continuation byte

I have also tried using decodebytes instead of b64decode to no avail.

I'd really appreciate any help!

• min2bro: You don't need to json.loads(data=a) as it is already a python dict, you just need to output this dict directly. And outputing json string from a dict is json.dumps()'s job
• lenz: The data value looks pretty corrupted. Do you know what "b'eyJ... was before it was encoded with JSON and B64? By removing the outer quotes and applying b64decode to the byte string b'eyJ...', I get something that looks like JSON in the beginning (b'{"hwid":), but the remainder is clearly not valid JSON.
• LemonSnippet: @lenz you are correct, these were dummy values. The actual encoded data was valid. I need to be able to remove the outer quotes in the dict. I am however, unable to use json,loads on the dict even though I used b64decode
• lenz: Please show some actual (possibly abbreviated) values, so one can test a recovery strategy.
• LemonSnippet: @lenz Apologies. There was sensitive data and I couldn't put it here.

Answer 1

Thank you all for your help.

After lots of searching on Stackoverflow coupled with testing on my local machine I was able to drill it down to this.

The object (a['data']) that was being passed in had some values that were not utf-8 decodable.

It was in the form of b'xxxsknoen'

I ended up deleting the b and the quotes in the front and end and then converting it to an str.

var = base64.b64decode(str(a['data'])[2:-1]).decode('utf-8')
a.update(json.loads(var))