Binary file reading and plotting data

Question

This code reads binary file. Read data is converted into signed values and gets plotted if the user wants it.

Data looks as below:

00000000: 0800 01dc 0500 02a4 0000 88ff f918 a9ff ................ 00000010: fa53 88ff 3200 0161 2d00 029a 2e00 feff .S..2..a-....... 00000020: f933 f6ff fa14 efff ebff f9bf eaff fa0e .3.............. 00000030: eaff 2500 0132 1700 0232 1f00 eeff f9ef ..%..2...2...... 00000040: eeff fa73 e6ff f3ff f9a6 efff fa9c f5ff ...s............ 00000050: ecff f92c ebff fa69 eaff ffff 0188 0100 ...,...i........ 00000060: fabd f6ff 0500 01d6 0300 0211 0100 1400 ................ 00000070: 0177 1400 0205 1400 0f00 017b 0e00 02e5 .w.........{.... 00000080: 0400 ffff f949 ffff fab3 fbff 0000 01f9 .....I.......... 00000090: 0100 fa3f f3ff ffff f94c fcff fa2e f7ff ...?.....L...... 000000a0: 0200 01ad 0200 027a 0100 1b00 015c 1500 .......z.....\.. 000000b0: 026a 1400 1200 0183 0d00 02cf 0e00 1200 .j.............. 000000c0: 01f0 0a00 02c1 0d00 ffff f977 f9ff fa48 ...........w...H 000000d0: faff 1000 01eb 0400 02cb 0000 1400 0178 ...............x

Code:

import matplotlib.pyplot as plt
import numpy as np
import matplotlib.animation as animation
import struct
from optparse import OptionParser
from collections import OrderedDict
import sys
import binascii
import time
import math
def twos_complement(hexstr, bits):
 value = int(hexstr,16)
 if value & (1 << (bits-1)):
 value -= 1 << bits
 return value
def reverse_digits(s):
 if len(s) != 4:
 print("wrong length passed %d!!"% (len(s)))
 return None
 return s[2:] + s[0:2]
params_information = OrderedDict([("division", (5, 5)), ("fast_velocity", (30, -30)), ("test_velocity", (60, -60)), ("test_position", (22, -22)), ("another_test_velocity", (28, -28))])
class TESTParams():
 def __init__(self, file_name):
 self.file_name = file_name
 self.all_params = {}
 self.param_size = 20
 def get_param(self, param):
 return self.all_params[param]
 def plot_param(self, param):
 # Create a figure of size 8x6 inches, 80 dots per inch
 plt.figure(figsize=(8, 6), dpi=80)
 # Create a new subplot from a grid of 1x1
 plt.subplot(1, 1, 1)
 plt.xlabel(str(param))
 plt.ylabel("limits")
 x = np.linspace(params_information[param][0], params_information[param][1], num=len(self.all_params[param]))
 plt.plot(x, self.all_params[param])
 plt.show()
 def get_all_params(self):
 """
 gets division/fast/test/another_test velocity and position from file
 file format is binary and it has data in below format repeated till EOF
 struct __attribute__((__packed__)) test_data
 {
 uint16_t division;
 int16_t fast_velocity; 
 int16_t test_velocity; 
 int16_t test_position; 
 int16_t another_test_velocity;
 };
 Below code does converts raw characters read from file into signed 16 bit numbers
 before adding to the dictionary.
 """
 with open(self.file_name) as file:
 data = file.read()
 hex_data = binascii.hexlify(data)
 for i in range(0, len(hex_data), self.param_size):
 test_data = hex_data[i:i+self.param_size]
 j = 0
 division = 0
 for param in params_information:
 if not param in self.all_params:
 self.all_params[param] = []
 if param == "division":
 division = float(twos_complement(reverse_digits(test_data[j:j+4]), 16))
 if division == 0:
 print("division is 0!!")
 return
 self.all_params[param].append(float(division))
 else:
 self.all_params[param].append(float(twos_complement(reverse_digits(test_data[j:j+4]), 16)/division))
 j += 4
def main():
 parser = OptionParser(usage="usage: %prog [options] filename -p -v",
 version="%prog 1.0")
 parser.add_option("-f", "--file", dest='filename',
 help="file name to get test data from")
 parser.add_option("-p", "--print_all_params", default=False, dest='print_all_params',
 help="print all params division/fast/test/another_test velocities and position", action='store_true')
 parser.add_option("-o", "--plot_param", default=False, dest='plot_param',
 help="plot param, pass - division , fast_velocity, test_velocity , test_position another_test_velocity")
 (options, args) = parser.parse_args()
 if not options.filename:
 parser.error('Filename not given')
 p = TESTParams(options.filename)
 p.get_all_params()
 if options.verify and p.verify_all_params():
 pass
 if options.print_all_params:
 for i, j, k, l, m in zip(p.get_param("division"), p.get_param("fast_velocity"),
 p.get_param("test_velocity"), p.get_param("test_position"),
 p.get_param("another_test_velocity")):
 print("division [%f] fast_velocity[%f] test_velocity[%f] test_position[%f] another_test_velocity[%f]"% (i, j, k, l, m))
 if options.plot_param in params_information:
 p.plot_param(options.plot_param)
if __name__ == '__main__':
 main()

• Gloweye: How does the data file look? Could you include a small sample?
• noman pouigt: @Gloweye: done.

Answer 1

Error reporting

Errors in Python code are better represented by exceptions than by printing. Exceptions are also usually more sensible than returning special values, because the exception does not need to be checked manually. The interpreter will tell you quite clearly if you forgot to think about one ...

So you might change your code from

print("wrong length passed %d!!" % (len(s)))
return None

to

raise ValueError("Wrong length passed! Found {} expected 4".format(len(s))

Similarly, raise ArithmeticError("Division by 0!") could be used in get_all_params().

Interpreting binary data

Quick note: since you are working with binary data here, it won't hurt to explicitely tell Python to open the file in binary mode with open(self.file_name, "rb") as file_: (also note that I changed file to file_ since file is/was a builtin in Python 2).

The current pipeline (bytes → hex → reverse hex → two's complement → int) is presumably more complicated than necessary. If you happen to be able to use Python 3, int.from_bytes(...) is IMHO the easiest option to integrate with your current code. struct.unpack(...) (also available in Python 2) is maybe even better, since it can not just read one integer at a time, but the whole struct! Using something like struct.unpack("<Hhhhh, <slice-binary-data-from-file-here) (please verify byte-order before using!) could make a large chunk of your code obsolete. Both can handle different byte-orders and are also able to interpret the signed integers.

Command-line arguments

optparse is deprecated/will not see updates in favor of argparse both in Python 2 and in Python 3. Since argparse is based on optparse, they are very similar.