UnicodeDecodeError: 'utf-8' codec can't decode byte (python)

Question

I'm playing around, trying to pull a bitcoin block template using the getblocktemplate method. I got a response but when I try to decode it from bytes I'm getting an error. Check it out:

s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
HOST = "184.70.15.166" # choose a node from blockchain.info
PORT = 8333
s.connect((HOST, PORT))
s.send(msg)
s.recv(1024)
x = random.randrange(320)
print(x)
data = json.dumps({'version': '2.0', 'id': 'x', 'method': 'getblocktemplate'}).encode('utf-8').strip()
s.send(data)
resp = s.recv(2048).decode('utf8')
print (resp)
# prints id number -
315
# prints response (in bytes) -
b'\xf9\xbe\xb4\xd9verack\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00]\xf6\xe0\xe2\xf9\xbe\xb4\xd9alert\x00\x00\x00\x00\x00\x00\x00\xa8\x00\x00\x00\x1b\xf9\xaa\xea`\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xff\xff\xff\x7f\x00\x00\x00\x00\xff\xff\xff\x7f\xfe\xff\xff\x7f\x01\xff\xff\xff\x7f\x00\x00\x00\x00\xff\xff\xff\x7f\x00\xff\xff\xff\x7f\x00/URGENT: Alert key compromised, upgrade required\x00F0D\x02 e?\xeb\xd6A\x0fG\x0fk\xae\x11\xca\xd1\x9cHA;\xec\xb1\xac,\x17\xf9\x08\xfd\x0f\xd5;\xdc:\xbdR\x02 m\x0e\x9c\x96\xfe\x88\xd4\xa0\xf0\x1e\xd9\xde\xda\xe2\xb6\xf9\xe0\r\xa9L\xad\x0f\xec\xaa\xe6n\xcfh\x9b\xf7\x1bP\xf9\xbe\xb4\xd9ping\x00\x00\x00\x00\x00\x00\x00\x00\x08\x00\x00\x00-8\xc9\x03\xd7\xbc\x0f\xc9\xc2\x1d36'
# try to decode from bytes and get this error -
Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
 File "/usr/lib/python3/dist-packages/spyderlib/widgets/externalshell/sitecustomize.py", line 699, in runfile
 execfile(filename, namespace)
 File "/usr/lib/python3/dist-packages/spyderlib/widgets/externalshell/sitecustomize.py", line 88, in execfile
 exec(compile(open(filename, 'rb').read(), filename, 'exec'), namespace)
 File "/home/myfolder/Desktop/bitcoin/bitcoin 2017/connection.py", line 66, in <module>
 print (resp.decode('utf8'))
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf9 in position 0: invalid start byte

What do you think? I've been looking all over the web and can't find a fix that works.

• Neo X: It seems that the data received by s.recv is not properly utf-8 encoded, you may want to double check the server's api.
• Engine: The getblocktemplate method is pretty standard w/ the json-rpc. I wonder if I'm making a mistake with the data formats somewhere. All I really want to do is pull the latest block template using he getblocktemplate method. This is driving me nuts.

Answer 1

An immediate solution is using binascii like this way:

For example, your data is:

a = b'\xf9\xbe\xb4\xd9verack\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00]\xf6\xe0\xe2\xf9\xbe\xb4\xd9alert\x00\x00\x00\x00\x00\x00\x00\xa8\x00\x00\x00\x1b\xf9\xaa\xea`\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xff\xff\xff\x7f\x00\x00\x00\x00\xff\xff\xff\x7f\xfe\xff\xff\x7f\x01\xff\xff\xff\x7f\x00\x00\x00\x00\xff\xff\xff\x7f\x00\xff\xff\xff\x7f\x00/URGENT: Alert key compromised, upgrade required\x00F0D\x02 e?\xeb\xd6A\x0fG\x0fk\xae\x11\xca\xd1\x9cHA;\xec\xb1\xac,\x17\xf9\x08\xfd\x0f\xd5;\xdc:\xbdR\x02 m\x0e\x9c\x96\xfe\x88\xd4\xa0\xf0\x1e\xd9\xde\xda\xe2\xb6\xf9\xe0\r\xa9L\xad\x0f\xec\xaa\xe6n\xcfh\x9b\xf7\x1bP\xf9\xbe\xb4\xd9ping\x00\x00\x00\x00\x00\x00\x00\x00\x08\x00\x00\x00-8\xc9\x03\xd7\xbc\x0f\xc9\xc2\x1d36'

You can do like this:

import binascii
# binascii.b2a_uu takes at most 45 bytes at once
# this why i'm splitting the data into chunks of 45 bytes at most
b = [a[k:k+45] for k in range(0, len(a), 45)]
# You can also use:
# binascii.b2a_uu(k).decode('UTF8')
final = "".join(binascii.b2a_uu(k).decode() for k in b)
print(final)

Output:

M^;ZTV79E<F%C:P !=]N#B^;ZTV6%L97)T *@ ;
M^:KJ8 $ /___W\ ____?_[__W\!____?P #___]_
M /___W\ +U521T5.5#H@06QE<G0@:V5Y(&-O;7!R;VUI<V5D+"!U<&=R861E
M(')E<75I<F5D 8ドルP1 (@93_KUD$/1P]KKA'*T9Q(03OLL:PL%_D(_0_5.]PZ
MO5("(&T.G);^B-2@\![9WMKBMOG@#:E,K0_LJN9NSVB;]QM0^;ZTV7!I;F< 
7 @ M.,D#U[P/R<(=,S8 

• Engine: Thanks but it still doesn't seem to quite be producing a comprehensible result. I wonder if I'm making a mistake with the data formats somewhere.
• Chiheb Nexus: If you have an idea about the decoded output data please share it. Otherwise, the bitcoin package built by Vitalik uses binascii as well.
• Engine: I believe your answer is applicable. I think I was making errors in outputting the data type. The response needed to be a json object and I needed to handle the json load and a problem with a byte that couldn't be encoded to utf-8. Learning as I go.

Answer 2

I found a python rpc library for bitcoin called

python-bitcoinrpc

It solved my problem:

rpc_connection = AuthServiceProxy('http://%s:%[email protected]:8332'%('rpc_username','rpc_password'))
block_template = rpc_connection.getblocktemplate()
print(block_template)

Friendly API and works fantastically.