pythonic way to split bytes

Question

I have incoming data from an absolute rotary encoder. It is 24 bits wide. 12 bits are the number of turns and 12 bits are the angle.

I'm reading the data with an Arduino and sending it to the computer over a serial port. The most natural thing to do would be to combine all of the bits and send it as 3 bytes.

Splitting up the 3 bytes into 12 bits/12 bits at the computer in Python 3 feels unpythonic to me. Here is what I have so far:

import struct
# fake message, H is header last 3 bytes are data
msg = b'H\x90\xfc\xf9' 
header, bytes3 = struct.unpack('>c3s', msg)
print('bytes3 ', bytes3)
val = int.from_bytes(bytes3, 'big')
print('val ', val)
print('bin(val) ', bin(val))
upper12 = val >> 12
print('bin(upper12)', bin(upper12))
lower12 = val & 0b000000000000111111111111
print('bin(lower12)', bin(lower12))

which gives console output of

bytes3 b'\x90\xfc\xf9'
val 9501945
bin(val) 0b100100001111110011111001
bin(upper12) 0b100100001111
bin(lower12) 0b110011111001

This code seems to work fine, but bitshifting (val >> 12) and bitwise anding (val & 0b000...) are a little wonky. Also it feels funny to specify Big Endian twice: first in the struct.unpack format string and second in int.from_bytes.

Is there a more elegant way to achieve this?

Update: timing

I'm still on the fence on which technique is most Pythonic. But I did time 3 techniques:

1. original technique above
2. technique by AlexV with overlapping unsigned 16 bit integers
3. change comms protocol to send a padded 4 byte unsigned integer that struct can convert directly to integer

import struct
import time
import random
import timeit
def gen_random_msg(nbyte = 3):
 # make a n byte message with header 'H' prepended
 data = random.randint(0, 2**12-1)
 data = data.to_bytes(nbyte, 'big')
 msg = b'H' + data
 return msg
def original_recipe():
 msg = gen_random_msg(3)
 header, bytes3 = struct.unpack('>c3s', msg)
 val = int.from_bytes(bytes3, 'big')
 upper12 = val >> 12
 lower12 = val & 4095 # 4095 = 2**12-1
def overlap16bits():
 msg = gen_random_msg(3)
 header, val = struct.unpack('>cH', msg[0:-1])
 upper12 = val >> 4
 lower12 = struct.unpack('>H', msg[2:])[0] & 0xfff
def fourbyte():
 msg = gen_random_msg(4)
 header, val = struct.unpack('>cI', msg)
 upper12 = val >> 12
 lower12 = val & 4095 # 4095 = 2**12-1
niter = int(1e6)
t0 = time.time()
for i in range(niter): gen_random_msg(3)
t1 = time.time()
for i in range(niter): gen_random_msg(4)
t2 = time.time()
for i in range(niter): original_recipe()
t3 = time.time()
for i in range(niter): overlap16bits()
t4 = time.time()
for i in range(niter): fourbyte()
t5 = time.time()
gentime3 = t1-t0
gentime4 = t2-t1
original_time = t3-t2
overlap_time = t4-t3
fourbyte_time = t5-t4
print('gentime3: ', gentime3, '. gentime4: ', gentime4)
print ('original recipe: ', original_time - gentime3)
print ('overlap 16 bits: ', overlap_time - gentime3)
print ('four bytes: ', fourbyte_time - gentime4)

this has console output:

gentime3: 3.478888988494873 . gentime4: 3.4476888179779053
original recipe: 1.3416340351104736
overlap 16 bits: 1.435237169265747
four bytes: 0.7956202030181885

It takes more time to generate 1M dummy msg than it does to process the bytes. The fastest technique was to change the comms spec and pad the 3 bytes to make a 4 byte unsigned integer. The speed up was good (approx 2x) for "fourbytes", but requires changing the communications specification. At this point I think it comes down to personal preference as to which algorithm is the best.

Answer 1

You could unpack them twice into (overlapping) unsigned shorts of 16bit and shift/mask them accordingly.

upper12 = struct.unpack(">H", msg[1:-1])[0] >> 4
lower12 = struct.unpack(">H", msg[2:])[0] & 0xFFF

Telling Python to interpret them as short integers helps you to get rid of int.from_bytes.

Answer 2

why even unpack? This is a little faster.

def no_unpack(msg):
 header = msg[0]
 upper12 = ((msg[1]<<8) | msg[2]) >> 4
 lower12 = ((msg[2]<<8) | msg[3]) & 0xfff