JSON Decoding String

Question

I am trying to decode a JSON string using

json.loads(request.POST.get('d'))

where d is a POST parameter containing a JSON string.

I get the following error in the stacktrace:

ValueError: Unterminated string starting at: line 1 column 22 (char 22)

This is the JSON string:

{"data":{"40":{"html":"<span style=\"color:#ffffff;\">test</span>","background":"transparent"},"41":{"html":"","background":"transparent"},"42":{"html":"","background":"transparent"}},"action":"save"}

However it works if I dont apply the span tag in data->40->html

{"data":{"40":{"html":"test","background":"transparent"},"41":{"html":"","background":"transparent"},"42":{"html":"","background":"transparent"}},"action":"save"}

What is the problem here?

• Keith: Where did you get the json string from? is it exactly what you get out of request.POST.get('d')) ?
• Michael Bates: The string comes from a Javascript array. The 'html' element of the array comes from a CKEditor instance.
• Keith: I mean the string that you posted here. How did you get it? Because what you posted works fine, so I suspect that what comes out of the POST may be a little different.
• Michael Bates: And yes, the string is exactly what comes out of request.POST.get('d')
• Michael Bates: What are you doing to emulate the issue?

Answer 1

I suppose there is something with backslashes in the source string.

When I parse

"""{"data":{"40":{"html":"<span style=\"color:#ffffff;\">test</span>","background":"transparent"},"41":{"html":"","background":"transparent"},"42":{"html":"","background":"transparent"}},"action":"save"}""" 

with json.loads(), it fails with a similar error.

However, when I disable escape sequences (r'' string literal), it works:

r"""{"data":{"40":{"html":"<span style=\"color:#ffffff;\">test</span>","background":"transparent"},"41":{"html":"","background":"transparent"},"42":{"html":"","background":"transparent"}},"action":"save"}"""

Obviously, '\"' in your string is being escaped and results in '"' when you construct the string, probably in JS(?). Haven't seen the code that builds it, but try adding an extra backslash: '\\"'

UPDATE: You may replace r'\' with r'\\' in a string. But it is better to understand how does the string look to begin with. When you inserted the string body into your message, where did you get it from?

• Michael Bates: The string comes from a html wysiwyg textarea
• Ellioh: Check what happens to backslash symbols if they are entered by user. Do they become double backslashes now, without the replacement?
• Michael Bates: No they don't, but I don't think this is the issue. It appears request.POST.get('d') is being cutoff where there is a semicolon in the string.
• Ellioh: Then try to replace a backslash with a pair of backslashes, and look at the result. Probably it will satisfy you.
• Michael Bates: Backslashes are being escaped into double backslashes, I just checked. As I said the problem seems to be semicolons. If I send {"data":"te;st"} all thats in request.POST.get('d') is: {"data":"te

Answer 2

How do you know that is the string you're getting? It works for me:

>>> ss = r'{"data":{"40":{"html":"<span style=\"color:#ffffff;\">test</span>","background":"transparent"},"41":{"html":"","background":"transparent"},"42":{"html":"","background":"transparent"}},"action":"save"}'
>>> json.loads(ss)
{u'action': u'save', u'data': {u'42': {u'html': u'', u'background': u'transparent'}, u'40': {u'html': u'<span style="color:#ffffff;">test</span>', u'background': u'transparent'}, u'41': {u'html': u'', u'background': u'transparent'}}}

Notice that I used a raw string for ss because otherwise \" will just be replaced by " in the string resulting in '"<span style="color:#ffffff;">test</span>"' which doesn't work for obvious reasons.

• Michael Bates: Ah ok, that sounds like the problem. How can I convert a variable into a raw string?
• mgilson: To make a string "raw" you just prefix it with r: r"this is a raw string". "this is not a raw string". This doesn't work with "variables" if that's what your asking though: foo = "bar"; rbar #doesn't make bar a raw string
• Ellioh: Have you printed request.POST.get('d') before inserting it into your message? It is necessary to understand what exactly it contains. Probably it is enough to replace r'\' with r'\\' in it.
• Michael Bates: The html comes from a user input textarea so the problem with replacing r'\' with r'\\' is that if a user explicitly types '\' it will get replaced with '\\'..

Answer 3

This worked for us:

json.loads(request.POST.get('d').encode('string-escape'))