Python encoding issue with pandas read_sql

Question

So I am trying to encode two strings to utf-8 so I can use them with pandas.read_sql:

selectedTable = "ACC__AccountCodes"
baseSql = "SELECT * FROM FileMaker_Fields WHERE TableName="

Now when I encode these two things:

baseSql.encode('utf-8')
selectedTable.encode('utf-8')
sqlString = "{}{}".format(baseSql, selectedTable)

My output looks like this:

b'SELECT * FROM FileMaker_Fields WHERE TableName='b'A\x00C\x00C\x00_\x00_\x00A\x00c\x00c\x00o\x00u\x00n\x00t\x00C\x00o\x00d\x00e\x00s\x00''

So when I run it with encoding set to 'latin1' I get the error:

pandas.io.sql.DatabaseError: Execution failed on sql 'SELECT * FROM FileMaker_Fields WHERE TableName=ACC__AccountCodes': ('HY000', '[HY000] [\x00F\x00i\x00l\x00e\x00M\x00a\x00k\x00e\x00r\x00]\x00[\x00F\x00i\x00l\x00e\x00M\x00a\x00k\x00e\x00r\x00]\x00 \x00F\x00Q\x00L\x000\x000\x000\x007\x00/\x00(\x001\x00:\x004\x007\x00)\x00:\x00 \x00T\x00h\x00e\x00 \x00c\x00o\x00l\x00u\x00m\x00n\x00 \x00n\x00a\x00m\x00e\x00d\x00 \x00"\x00A\x00"\x00 \x00d\x00o\x00e\x00s\x00 \x00n\x00o\x00t\x00 \x00e\x00x\x00i\x00s\x00t\x00.....

I can't seem to find anything addressing this. Everything I have tried leads me back to this which causes a sql error for invalid syntax. I am using pyodbc which expects utf-8 encoding as the input. Thoughts?

• Gord Thompson: You shouldn't need explicit encoding. I recently set up pyodbc with FileMaker ODBC on Windows and the default Unicode encoding for pyodbc (which is UTF-16LE, BTW) worked fine.
• L. Norman: Ok, thanks I will give it another shot. Thanks!

Answer 1

You don't have to encode them as utf-8; Try passing them as normal strings to pandas read_sql function, it should work fine, if not, then you have a problem somewhere else... but encoding is not what you want here.

Pyodbc accepts unicode strings in the query as normal, so that is not your problem at all.

I suggest also reading the Unicode section on pyodbc documentation that contains specific unicode configuration for some databases, although in your case I don't see that being a problem at all, because it is related to the database driver encoding and not your sql query, which should be a normal unicode string every time.

• L. Norman: Edited my question. I am using pyodbc which expects utf-8 encoding as the input.
• nosklo: @L.Norman Your assumptions are wrong. I'm also using pyodbc and it works fine with normal unicode strings here. Show your code and the error you get when you use normal unicode strings, instead.
• nosklo: Why are you setting the encoding in first place?
• L. Norman: I believe my database output has utf characters in it. Edited my post