Process escape sequences in a string in Python

Question

Sometimes when I get input from a file or the user, I get a string with escape sequences in it. I would like to process the escape sequences in the same way that Python processes escape sequences in string literals.

For example, let's say myString is defined as:

>>> myString = "spam\\neggs"
>>> print(myString)
spam\neggs

I want a function (I'll call it process) that does this:

>>> print(process(myString))
spam
eggs

It's important that the function can process all of the escape sequences in Python (listed in a table in the link above).

Does Python have a function to do this?

• Nas Banov: hmmm, how exactly would you expect a string containing 'spam'+"eggs"+'''some'''+"""more""" to be processed?
• dln385: @Nas Banov That's a good test. That string contains no escape sequences, so it should be exactly the same after processing. myString = "'spam'+\"eggs\"+'''some'''+\"\"\"more\"\"\"", print(bytes(myString, "utf-8").decode("unicode_escape")) seems to work.
• Apalala: Most answers to this question have serious problems. There seems to be no standard way to honor escape sequences in Python without breaking unicode. The answer posted by @rspeer is the one that I adopted for Grako as it so far handles all known cases.
• Glen Whitney: I disagree with Apalala; using unicode_escape (on a properly latin1-encoded input) is completely reliable, and as the issue that Hack5 links to in his comment to user19087's answer shows, is the method recommended by the python developers.
• SuperStormer: Related: how do I .decode('string-escape') in Python3?

Answer 1

The correct thing to do is use the 'string-escape' code to decode the string.

>>> myString = "spam\\neggs"
>>> decoded_string = bytes(myString, "utf-8").decode("unicode_escape") # python3 
>>> decoded_string = myString.decode('string_escape') # python2
>>> print(decoded_string)
spam
eggs

Don't use the AST or eval. Using the string codecs is much safer.

• Nas Banov: hands down, the best solution! btw, by docs it should be "string_escape" (with underscore) but for some reason accepts anything in the pattern 'string escape', 'string@escape" and whatnot... basically 'string\W+escape'
• dln385: @Nas Banov The documentation does make a small mention about that: Notice that spelling alternatives that only differ in case or use a hyphen instead of an underscore are also valid aliases; therefore, e.g. 'utf-8' is a valid alias for the 'utf_8' codec.
• Apalala: This solution is not good enough because it doesn't handle the case in which there are legit unicode characters in the original string. If you try: >>> print("juancarlo\\tañez".encode('utf-8').decode('unicode_escape')) You get: juancarlo añez
• Christian Aichinger: Agreed with @Apalala: this is not good enough. Check out rseeper's answer below for a complete solution that works in Python2 and 3!
• metatoaster: Since latin1 is assumed by unicode_escape, redo the encode/decode bit, e.g. s.encode('utf-8').decode('unicode_escape').encode('latin1').decode('utf8')

Answer 2

unicode_escape doesn't work in general

It turns out that the string_escape or unicode_escape solution does not work in general -- particularly, it doesn't work in the presence of actual Unicode.

If you can be sure that every non-ASCII character will be escaped (and remember, anything beyond the first 128 characters is non-ASCII), unicode_escape will do the right thing for you. But if there are any literal non-ASCII characters already in your string, things will go wrong.

unicode_escape is fundamentally designed to convert bytes into Unicode text. But in many places -- for example, Python source code -- the source data is already Unicode text.

The only way this can work correctly is if you encode the text into bytes first. UTF-8 is the sensible encoding for all text, so that should work, right?

The following examples are in Python 3, so that the string literals are cleaner, but the same problem exists with slightly different manifestations on both Python 2 and 3.

>>> s = 'naïve \\t test'
>>> print(s.encode('utf-8').decode('unicode_escape'))
naà ̄ve test

Well, that's wrong.

The new recommended way to use codecs that decode text into text is to call codecs.decode directly. Does that help?

>>> import codecs
>>> print(codecs.decode(s, 'unicode_escape'))
naà ̄ve test

Not at all. (Also, the above is a UnicodeError on Python 2.)

The unicode_escape codec, despite its name, turns out to assume that all non-ASCII bytes are in the Latin-1 (ISO-8859-1) encoding. So you would have to do it like this:

>>> print(s.encode('latin-1').decode('unicode_escape'))
naïve test

But that's terrible. This limits you to the 256 Latin-1 characters, as if Unicode had never been invented at all!

>>> print('Ernő \\t Rubik'.encode('latin-1').decode('unicode_escape'))
UnicodeEncodeError: 'latin-1' codec can't encode character '\u0151'
in position 3: ordinal not in range(256)

Adding a regular expression to solve the problem

(Surprisingly, we do not now have two problems.)

What we need to do is only apply the unicode_escape decoder to things that we are certain to be ASCII text. In particular, we can make sure only to apply it to valid Python escape sequences, which are guaranteed to be ASCII text.

The plan is, we'll find escape sequences using a regular expression, and use a function as the argument to re.sub to replace them with their unescaped value.

import re
import codecs
ESCAPE_SEQUENCE_RE = re.compile(r'''
 ( \\U........ # 8-digit hex escapes
 | \\u.... # 4-digit hex escapes
 | \\x.. # 2-digit hex escapes
 | \\[0-7]{1,3} # Octal escapes
 | \\N\{[^}]+\} # Unicode characters by name
 | \\[\\'"abfnrtv] # Single-character escapes
 )''', re.UNICODE | re.VERBOSE)
def decode_escapes(s):
 def decode_match(match):
 try:
 return codecs.decode(match.group(0), 'unicode-escape')
 except UnicodeDecodeError:
 # In case we matched the wrong thing after a double-backslash
 return match.group(0)
 return ESCAPE_SEQUENCE_RE.sub(decode_match, s)

And with that:

>>> print(decode_escapes('Ernő \\t Rubik'))
Ernő Rubik

• v.oddou: we need more encompassing types of answers like that. thanks.
• AncientSwordRage: Does this work with os.sep at all? I'm trying to do this: patt = '^' + self.prefix + os.sep ; name = sub(decode_escapes(patt), '', name) and it's not working. Semicolon is there in place of a new line.
• OpenAI stole this from rspeer: @Pureferret I'm not really sure what you're asking, but you probably shouldn't run this on strings where the backslash has a different meaning, such as Windows file paths. (Is that what your os.sep is?) If you have backslashed escape sequences in your Windows directory names, the situation is pretty much unrecoverable.
• AncientSwordRage: The escape sequence doesn't have escapes in them, but I'm getting a 'bogus escape string ' error
• OpenAI stole this from rspeer: @MarkIngram Yes, I'm using Python 3. I don't understand the relevance of the example you posted, which is doing something unrelated to my code. My code doesn't use bytestrings at any step.

Answer 3

The actually correct and convenient answer for python 3:

>>> import codecs
>>> myString = "spam\\neggs"
>>> print(codecs.escape_decode(bytes(myString, "utf-8"))[0].decode("utf-8"))
spam
eggs
>>> myString = "naïve \\t test"
>>> print(codecs.escape_decode(bytes(myString, "utf-8"))[0].decode("utf-8"))
naïve test

Details regarding codecs.escape_decode:

• codecs.escape_decode is a bytes-to-bytes decoder
• codecs.escape_decode decodes ascii escape sequences, such as: b"\\n" -> b"\n", b"\\xce" -> b"\xce".
• codecs.escape_decode does not care or need to know about the byte object's encoding, but the encoding of the escaped bytes should match the encoding of the rest of the object.

Background:

• @rspeer is correct: unicode_escape is the incorrect solution for python3. This is because unicode_escape decodes escaped bytes, then decodes bytes to unicode string, but receives no information regarding which codec to use for the second operation.
• @Jerub is correct: avoid the AST or eval.
• I first discovered codecs.escape_decode from this answer to "how do I .decode('string-escape') in Python3?". As that answer states, that function is currently not documented for python 3.

• jwd: This is the real answer (: Too bad it relies upon a poorly-documented function.
• OpenAI stole this from rspeer: This is the answer for situations where the escape sequences you have are \x escapes of UTF-8 bytes. But because it decodes bytes to bytes, it doesn't -- and can't -- decode any escapes of non-ASCII Unicode characters, such as \u escapes.
• Hack5: Just an FYI, this function is technically not public. see bugs.python.org/issue30588
• Glen Whitney: Moreover, in the link that Hack5 provides, the python maintainers make it clear that escape_decode may be removed without warning in any future version, and that the "unicode_escape" codec is the recommended way to go about this.
• holdenweb: Interestingly, six years later escape-decoderemains undocumented!

Answer 4

The ast.literal_eval function comes close, but it will expect the string to be properly quoted first.

Of course Python's interpretation of backslash escapes depends on how the string is quoted ("" vs r"" vs u"", triple quotes, etc) so you may want to wrap the user input in suitable quotes and pass to literal_eval. Wrapping it in quotes will also prevent literal_eval from returning a number, tuple, dictionary, etc.

Things still might get tricky if the user types unquoted quotes of the type you intend to wrap around the string.

• dln385: I see. This seems to be potentially dangerous as you say: myString = "\"\ndoBadStuff()\n\"", print(ast.literal_eval('"' + myString + '"')) seems to try to run code. How is ast.literal_eval any different/safer than eval?
• Greg Hewgill: @dln385: literal_eval never executes code. From the documentation, "This can be used for safely evaluating strings containing Python expressions from untrusted sources without the need to parse the values oneself."

Answer 5

The (currently) accepted answer by Jerub is correct for python2, but incorrect and may produce garbled results (as Apalala points out in a comment to that solution), for python3. That's because the unicode_escape codec requires its source to be coded in latin-1, not utf-8, as per the official python docs. Hence, in python3 use:

>>> myString="špåm\\nëðþ\\x73"
>>> print(myString)
špåm\nëðþ\x73
>>> decoded_string = myString.encode('latin-1','backslashreplace').decode('unicode_escape')
>>> print(decoded_string)
špåm
ëðþs

This method also avoids the extra unnecessary roundtrip between strings and bytes in metatoaster's comments to Jerub's solution (but hats off to metatoaster for recognizing the bug in that solution).

• Glen Whitney: When I posted this, I did not realize there was a duplicate question for which this exact answer had already been given: stackoverflow.com/a/57192592/5583443
• Karl Knechtel: The important thing here is not just that latin-1 is used, but that non-latin-1 characters are turned into escape sequences via the 'backslashreplace' error handling. This just happens to give the exact format that the .decode step is trying to replace. So this works with, for example, myString='日本\u8a9e', correctly giving 日本語. However, it doesn't handle the truly nasty cases described in my answer.
• Karl Knechtel: (On the other hand, it certainly can be argued that input with a single trailing backslash should fail...)
• Mark Ransom: Is it really always latin-1, or does it depend on the default encoding for your particular version of Python? Is this true even on Linux for example?
• Glen Whitney: Well, in the table in the python docs that I link to above, in the 'unicode_escape' entry, it states "Decode from Latin-1 source code." So that seems pretty clear/definitive to me...

Answer 6

Quote the string properly so that it looks like the equivalent Python string literal, and then use ast.literal_eval. This is safe, but much trickier to get right than you might expect.

It's easy enough to add a " to the beginning and end of the string, but we also need to make sure that any " inside the string are properly escaped. If we want fully Python-compliant translation, we need to account for the deprecated behaviour of invalid escape sequences.

It works out that we need to add one backslash to:

• any sequence of an even number of backslashes followed by a double-quote (so that we escape a quote if needed, but don't escape a backslash and un-escape the quote if it was already escaped); as well as

• a sequence of an odd number of backslashes at the end of the input (because otherwise a backslash would escape our enclosing double-quote).

Here is an acid-test input showing a bunch of difficult cases:

>>> text = r'''\\ \ \" \\" \\\" \'你好'\n\u062a\xff\N{LATIN SMALL LETTER A}"''' + '\\'
>>> text
'\\\\ \\ \\" \\\\" \\\\\\" \\\'你好\'\\n\\u062a\\xff\\N{LATIN SMALL LETTER A}"\\'
>>> print(text)
\\ \ \" \\" \\\" \'你好'\n\u062a\xff\N{LATIN SMALL LETTER A}"\

I was eventually able to work out a regex that handles all these cases properly, allowing literal_eval to be used:

>>> def parse_escapes(text):
... fixed_escapes = re.sub(r'(?<!\\)(\\\\)*("|\\$)', r'\\1円2円', text)
... return ast.literal_eval(f'"{fixed_escapes}"')
... 

Testing the results:

>>> parse_escapes(text)
'\\ \\ " \\" \\" \'你好\'\nتÿa"\\'
>>> print(parse_escapes(text))
\ \ " \" \" '你好'
تÿa"\

This should correctly handle everything - strings containing both single and double quotes, every weird situation with backslashes, and non-ASCII characters in the input. (I admit it's a bit difficult to verify the results by eye!)

Answer 7

This is a bad way of doing it, but it worked for me when trying to interpret escaped octals passed in a string argument.

input_string = eval('b"' + sys.argv[1] + '"')

It's worth mentioning that there is a difference between eval and ast.literal_eval (eval being way more unsafe). See Using python's eval() vs. ast.literal_eval()?

• Karl Knechtel: Just to make sure the warning is up front: Please do not use eval for input that could ever possibly come from outside the program. It allows the user supplying that input to run arbitrary code on your computer. It is not at all trivial to sandbox.

Answer 8

This should be the accepted answer for python 3.

def unescape_escape_sequences(s):
 return s.encode('utf-8').decode('unicode_escape').encode('latin1').decode('utf8')

Here is why this works:

1. First we unescape the escape sequences while leaving utf-8 characters unchanged as bytes (s.encode('utf-8').decode('unicode_escape')). Unfortunately, this produces a string (i.e. character symbols) under the assumption that the bytes should be interpreted as if they are 'latin1' encoded (rather than utf-8/unicode encoding we started with) (see 'unicode_escape' in the docs).
2. Next we convert the wrongly latin1-decoded string back to bytes using the 'latin1' codec (.encode('latin1')). The resulting bytes are the same as when we started, except for the removed escape characters.
3. Finally, we re-convert from bytes to character symbols (i.e. a string), this time with the assumption that the bytes are encoded as utf-8 (as they were intended all along) (.decode('utf8')).

This handles the edge cases where other answers fail, AND relies only on documented functions in the python 3 standard library.

Particularly, this handles the edge case where the input string includes (1) non-latin1 unicode characters and (2) escape sequences to unescape.

unescape_escape_sequences("naïve \\t test") 
# 'naïve \t test'
unescape_escape_sequences(r'Δ\nΔ')
# 'Δ\nΔ'
unescape_escape_sequences("juancarlo\\tañez")
# 'juancarlo\tañez'
unescape_escape_sequences('Ernő \\t Rubik')
# 'Ernő \t Rubik'
unescape_escape_sequences("naïve \\t test")
# 'naïve \t test'
unescape_escape_sequences("spam\\neggs")
# 'spam\neggs'

These other answers from this post fail in (all but the last of) the above cases:

s = 'Ernő \\t Rubik'
bytes(s, "utf-8").decode("unicode_escape") # incorrect output
s.encode('latin-1').decode('unicode_escape') # error

Finally, this answer avoids using codecs.escape_decode() which is not documented in python 3.

Be aware you may run into trouble if your input string is not unicode/utf-8.

Answer 9

Below code should work for \n is required to be displayed on the string.

import string
our_str = 'The String is \\n, \\n and \\n!'
new_str = string.replace(our_str, '/\\n', '/\n', 1)
print(new_str)

• ShadowRanger: This doesn't work as written (the forward slashes make the replace do nothing), uses wildly outdated APIs (the string module functions of this sort are deprecated as of Python 2.0, replaced by the str methods, and gone completely in Python 3), and only handles the specific case of replacing a single newline, not general escape processing.