Remove non-printable characters from string in Python 3

Question

My aim is to print bytes string lines in Python 3, but truncate each line to a maximum width so that it fits the current terminal window.

My first attempt was only print(output[:max_width]), but this did not work because Python counts e.g. tabs \t as one character, but the terminal displays them as multiple characters. Also, it would evaluate carriage-returns etc.

Therefore now I have this snippet of code, line is a bytes string:

output = line.decode(codec, "replace")
if max_width:
 output = "".join(c for c in output if c.isprintable())
 print(output[:max_width])
else:
 print(output)

However, I guess it's pretty slow to refactor each string line this way just to filter out non-printable characters like \t and \r (and whatever characters I might have forgotten).

Please note that codec is specified by the user. It might be "ascii", utf-8, utf-16 or any other valid built-in codec.

Can you please suggest me how to improve this code's performance?

• Phrancis: I found this on Stack Overflow which may fit your needs: Stripping non printable characters from a string in python
• Byte Commander: I should add that the encoding codec may vary. It could be ascii as well as utf-16 or some iso codec. Not sure which methods are suitable for that, but manually crafting a set of forbidden characters is probably difficult.
• Peilonrayz: It may be worth adding more context to this question, the code is sparse, and I don't really understand why you are doing this. Currently my answer would be: "it seems good, but why are you not doing: output = ''.join(c for c in output if c.isprintable()); print(output[:max_width] if max_width else output)?"
• Byte Commander: @JoeWallis It's part of a script to inspect files for characters that don't match a specific encoding. For example I set the encoding to "ascii" and the script will print all lines of the file that are not translatable to ASCII. And those lines can be very long, especially if it is a binary file that was not meant to be read line-wise. So I want to give the user the ability to trim the preview lines to a specific width. But your way of shortening the if/else/print is not applicable, because I don't want to remove special characters if I'm not trimming the line length.

Answer 1

Something that may help performance wise could be itertools.islice. This will allow you to call str.isprintable() max_width amount of times, as this is a binary file that may not have many \ns it can save a lot of effort.

output = line.decode(codec, "replace")
if max_width:
 print("".join(itertools.islice((c for c in output if c.isprintable()), max_width)))
else:
 print(output)

This on it's own may not help on files that have a lot of \ns. The bottle neck in these file would most likely be the overhead incurred by print. And so it's much faster to build a string to display once. In these cases you would want to use something like:
(Untested code)

def read_data(path):
 with open(path) as f:
 for line in f:
 output = line.decode(codec, "replace")
 if max_width:
 yield "".join(itertools.islice(
 (c for c in output if c.isprintable()),
 max_width))
 else:
 yield output
print('\n'.join(read_data(...)))

However the above is not good on machines with limited memory or extremely large files. In these cases you would want to use a buffer and print the buffer when a threshold has been reached.

---

After PEP 3138 your method to remove non-printables seems to be the correct way.