Decode a string: Python

Question

I built a string with a tuple like this:

t = tuple(data)
querysring="INSERT INTO %s VALUES %s "%(table,t)

When I print the string the result is:

INSERT INTO AGENT VALUES ('Bock', 'Fran\\xc3\\xa7ois Bock', 'Individual', 'fb****@mail.com')

But I want something like this:

 INSERT INTO AGENT VALUES ('Bock', 'François Bock', 'Individual', 'fb****@mail.com')

It is possible to decode the string ? I'm using Python2.x but I can use Python3.x

I try this:

querysring=u"INSERT INTO %s VALUES %s "%(table,t)
print(ftfy.fix_text(querysring))

But it's not working

• roganjosh: I'm not sure how this question differs significantly from your previous question stackoverflow.com/questions/43629059/encoding-in-tuple-python. Also, you shouldn't be using Python string formatting to generate your query, you should have parameterised queries, which may itself fix your issue.
• Fedour: Yes but how can I make it ? I don't know the numbers values there is in my tuple(data)
• Kadir: With python2 put # -*- coding: utf-8 -*- into your first line.

Answer 1

I think your issue is superficial and related to how print displays lists and list items differently. The printed output of a list is in ascii even if the items within the list are correctly encoded in utf-8. First, using chardet library:

from chardet.universaldetector import UniversalDetector
a = ['Bock', 'François Bock']
detector = UniversalDetector()
detector.feed(str(a))
detector.close()
print "Encoding for the str(list): ", detector.result
detector = UniversalDetector()
detector.feed(a[1])
detector.close()
print "Encoding for list[1]: ", detector.result
print "The whole list: ", a
print "Item in list: ", a[1]

Aside from the off-putting printouts, it's possible to still write to the database with the correct encoding with a parameterized query. The last part of the below code writes to a file to confirm that the data encoding is preserved:

import sqlite3
conn = sqlite3.connect(":memory:")
conn.text_factory = str
c = conn.cursor()
c.execute("CREATE TABLE IF NOT EXISTS testing(test1 TEXT, test2 TEXT)")
conn.commit()
my_tuple = 'Bock', 'François Bock'
table = 'testing'
placeholders = ', '.join('?' for item in my_tuple)
query = "INSERT INTO {} VALUES ({})".format(table, placeholders)
c.execute(query, my_tuple)
c.execute("SELECT * FROM testing")
all_data = c.fetchone()
# Check the printouts
print all_data
print all_data[1]
# For good measure, write them to a file
with open('check_output.txt', 'w') as outfile:
 outfile.write(', '.join(item for item in all_data))

• Fedour: Thank you for you help and your explanation. I use psycopg2, so the parameterized query is a bit different but it's really help me !
• roganjosh: @FrancoisBock you're welcome, is it %s instead of ?? Either way, don't use string formatting, that leaves you open to SQL injection.
• Fedour: Yes that's it " '%s' "