Car computer in Python / GPS tracking VERSION 2

Question

This is an updated question with new code, taking into account your comments from last time: Car computer in python / GPS tracking

I have an old car that I use for long distance driving and have coded an onboard computer based on a Raspberry Pi 3 and a few other modules. I use a FONA 808 from Adafruit as the cellular modem (via serial), the Sparkfun NEO-M9N as a GPS sensor (i2c), an OLED display (i2c) and a small temperature sensor via 1-wire. Here is a picture of the computer in action, just so you have a better picture of it:

car computer

Notable improvements compared to version 1 in the old questions are (in a nutshell):

• config.ini file
• communication with remote server either per DB direct insert or per POST to a PHP script
• cleaned up use of globals, imports
• using the DRY principle as much as possible, creating new helper functions
• shortening of code where possible
• testing function for GPS module

Without further ado, here the code, split up in a few files:

car_computer.py

import os
from threading import Thread
import glob
import serial
import board
import subprocess
import smbus
import time
from time import sleep
import datetime
import configparser
import re
import urllib
from urllib import request
import random
from luma.core.interface.serial import i2c
from luma.core.render import canvas
from luma.oled.device import sh1106
from PIL import ImageFont, Image, ImageDraw
import pymysql
import csv
import json
import hashlib 
import pynmea2
from haversine import haversine, Unit
config = configparser.ConfigParser()
config.read('/home/pi/Desktop/car_computer.ini')
################## start display ##################
device = sh1106(i2c(port=1, address=config['display']['i2c_port']), rotate=0)
device.clear()
pending_redraw = False
output = Image.new("1", (128,64))
add_to_image = ImageDraw.Draw(output)
def setup_font(font_filename, size):
 return ImageFont.truetype(os.path.join(config['general']['folder'],config['general']['folder_fonts'],font_filename), size)
fa_solid = setup_font('fa-solid-900.ttf', 12)
fa_solid_largest = setup_font('fa-solid-900.ttf', 40)
text_largest = setup_font('digital-7.ttf', 58)
text_medium = setup_font('digital-7.ttf', 24)
text_small = setup_font('digital-7.ttf', 18)
icons = { #to look up the icons on FontAwesome.com, remove quote marks and \u from the search query 
 "save": "\uf56f","cloud": "\uf0c2","check": "\uf058","upload": "\uf382","no_conn": "\uf127","location": "\uf124","question": "\uf128","altitude": "\uf077","distance": "\uf1b9","temperature": "\uf2c9" }
def wipe(zone):
 add_to_image.rectangle(tuple(int(v) for v in re.findall("[0-9]+", config['display'][zone])), fill="black", outline ="black")
def icon(zone,name):
 add_to_image.text(tuple(int(v) for v in re.findall("[0-9]+", config['display'][zone])), icons[name], font=fa_solid, fill="white")
def text(zone,text,fontsize=text_medium):
 add_to_image.text(tuple(int(v) for v in re.findall("[0-9]+", config['display'][zone])), text, font=fontsize, fill="white")
################## upload data from GPS folder via FONA to MySQL ##################
def fix_nulls(s):
 return (line.replace('0円', '') for line in s)
def upload_data():
 global pending_redraw
 while True:
 sleep(5)
 current_dir = os.path.join(config['general']['folder'],config['general']['folder_data'])
 archive_dir = os.path.join(config['general']['folder'],config['general']['folder_data_archive'])
 path, dirs, files = next(os.walk(current_dir))
 file_count = len(files)
 if file_count < 2:
 print("Not enough GPS.csv files found so it's probably in use now or doesn't exist")
 return
 list_of_files = glob.glob(current_dir+"/*.csv")
 oldest_file = min(list_of_files, key=os.path.getctime)
 oldest_file_name = os.path.basename(oldest_file)
 
 try:
 openPPPD()
 
 if config['db']['mode'] == "db":
 print("mode = db")
 db = pymysql.connect(config['db']['db_host'],config['db']['db_user'],config['db']['db_pw'],config['db']['db_name'])
 cursor = db.cursor()
 csv_data = csv.reader(fix_nulls(open(oldest_file)))
 next(csv_data)
 for row in csv_data:
 if row:
 statement = 'INSERT INTO '+config['db']['db_table']+' (gps_time, gps_lat, gps_long, gps_speed) VALUES (%s, %s, %s, %s)'
 cursor.execute(statement,row)
 print("Committing to db")
 db.commit()
 cursor.close()
 
 if config['db']['mode'] == "server":
 print("mode = server")
 csv_data = csv.reader(fix_nulls(open(oldest_file)))
 next(csv_data)
 row_nb = 1
 row_data = {}
 rows_encoded_nb = 1
 rows_encoded = {}
 for row in csv_data:
 if row:
 row_data[row_nb] = {'gps_time': int(row[0]), 'gps_lat' : round(float(row[1]), 5), 'gps_long' : round(float(row[2]), 5), 'gps_speed' : round(float(row[3]), 1)}
 if row_nb % int(config['db']['server_batchsize']) == 0:
 rows_encoded[rows_encoded_nb] = row_data
 rows_encoded_nb +=1
 row_data = {}
 row_nb +=1
 rows_encoded[rows_encoded_nb] = row_data
 row_data = {}
 
 for i in rows_encoded : 
 checksum = hashlib.md5(str(rows_encoded[i]).encode())
 checksum = checksum.hexdigest()
 
 req = request.Request(config['db']['server_addr'], method="POST")
 req.add_header('Content-Type', 'application/json')
 data = {
 "hash": checksum,
 "ID": config['db']['server_ID'],
 "pw": config['db']['server_pw'],
 "data": rows_encoded[i]
 }
 data = json.dumps(data)
 data = data.encode()
 r = request.urlopen(req, data=data)
 print(r.read())
 
 #sleep(1)
 closePPPD()
 print("Successfully committed to db")
 wipe('GPRS_ZONE')
 icon('GPRS_START',"check")
 pending_redraw = True
 os.rename(current_dir+"/"+oldest_file_name, archive_dir+"/archive_"+oldest_file_name) 
 sleep(60)
 wipe('GPRS_ZONE')
 
 except Exception as e:
 print("Database error:", e)
 wipe('GPRS_ZONE')
 icon('GPRS_START',"no_conn")
 pending_redraw = True
 closePPPD()
 sleep(60)
 wipe('GPRS_ZONE')
 pending_redraw = True
 return
 
 sleep(300)
 
################## config and start GPS ##################
BUS = None
reading_nr = 1
reading_nr_upload = 1
reading_nr_upload_nbrowsinlog = 0
total_km = 0
prev_lat = 0
prev_long = 0
def connectBus():
 global BUS
 BUS = smbus.SMBus(1)
def debug_gps():
 sleep(1)
 time = datetime.datetime.now()
 gga1 = pynmea2.GGA('GN', 'GGA', (time.strftime("%H%M%S"), '1929.045', 'S', '02410.516', 'E', '1', '04', '2.6', '69.00', 'M', '-33.9', 'M', '', '0000'))
 gga2 = pynmea2.GGA('GN', 'GGA', (time.strftime("%H%M%S"), '1929.075', 'S', '02410.506', 'E', '1', '04', '2.6', '73.00', 'M', '-33.9', 'M', '', '0000'))
 rmc1 = pynmea2.RMC('GN', 'RMC', (time.strftime("%H%M%S"), 'A', '1929.055', 'S', '02411.516', 'E', '28', '076.2', time.strftime("%d%m%y"), 'A'))
 rmc2 = pynmea2.RMC('GN', 'RMC', (time.strftime("%H%M%S"), 'A', '1929.045', 'S', '02411.506', 'E', '29', '076.2', time.strftime("%d%m%y"), 'A'))
 nmea = [gga1,rmc1,gga2,rmc2]
 return str(random.choice(nmea))
def parseResponse(gpsLine):
 global pending_redraw
 gpsChars = ''.join(chr(c) for c in gpsLine)
 
 ##### uncomment only for testing when the GPS chip has no reception #####
 gpsChars = debug_gps()
 #print(gpsChars)
 
 if "GGA" in gpsChars:
 if ",1," not in gpsChars:
 print("GGA?")
 wipe('STATUS_ICON_ZONE')
 wipe('STATUS_ZONE')
 icon('STATUS_ICON_START', "location")
 icon('STATUS_START', "question")
 pending_redraw = True
 sleep(1)
 return False
 try:
 nmea = pynmea2.parse(gpsChars, check=True)
 if "0.0" in str(nmea.latitude) or "0.0" in str(nmea.longitude):
 return False
 
 #show that we have a location and delete whatever was there
 wipe('STATUS_ICON_ZONE')
 icon('STATUS_ICON_START', "location")
 wipe('STATUS_ZONE')
 
 ## update altitude
 icon('ALTI_ICON_START', "altitude")
 wipe('ALTI_ZONE')
 text('ALTI_START', str('%.0f'%(nmea.altitude)))
 
 ## update total distance
 global reading_nr
 global total_km
 global prev_lat
 global prev_long
 dist = 0
 if reading_nr != 1:
 dist = haversine(((float(prev_lat)), (float(prev_long))), ((float(nmea.latitude)), (float(nmea.longitude))))
 total_km = total_km+dist
 icon('DIST_ICON_START', "distance")
 wipe('DIST_ZONE')
 text('DIST_START', "%0.1f" % total_km)
 prev_lat = nmea.latitude
 prev_long = nmea.longitude
 
 pending_redraw = True
 reading_nr +=1
 
 except Exception as e:
 print("GGA parse error:", e)
 wipe('STATUS_ZONE')
 pending_redraw = True
 pass
 
 if "RMC" in gpsChars:
 if ",A," not in gpsChars: # 1 for GGA, A for RMC
 print("RMC?")
 wipe('STATUS_ICON_ZONE')
 wipe('STATUS_ZONE')
 icon('STATUS_ICON_START', "location")
 icon('STATUS_START', "question")
 pending_redraw = True
 sleep(1)
 return False
 try:
 nmea = pynmea2.parse(gpsChars, check=True)
 if "0.0" in str(nmea.latitude) or "0.0" in str(nmea.longitude):
 return False
 
 #show that we have a location and delete whatever was there
 wipe('STATUS_ICON_ZONE')
 icon('STATUS_ICON_START', "location")
 wipe('STATUS_ZONE')
 
 ## update speed
 wipe('SPEED_ZONE')
 text('SPEED_START', str('%.0f'%(nmea.spd_over_grnd*1.852)), fontsize=text_largest)
 
 ## log every log_frequency nth GPS coordinate in CSV file
 global reading_nr_upload
 global reading_nr_upload_nbrowsinlog
 if reading_nr_upload % int(config['gps']['log_frequency']) == 0:
 t = datetime.datetime.combine(nmea.datestamp, nmea.timestamp).strftime("%s")
 d = datetime.datetime.combine(nmea.datestamp, nmea.timestamp).strftime("%Y%m%d%H")
 filename = os.path.join(config['general']['folder'],config['general']['folder_data'],'gps_' + d + '.csv')
 with open(filename, 'a', newline='') as csvfile:
 gps_writer = csv.writer(csvfile, delimiter=',', quotechar='|', quoting=csv.QUOTE_MINIMAL)
 gps_writer.writerow([t, nmea.latitude, nmea.longitude, nmea.spd_over_grnd*1.852])
 reading_nr_upload_nbrowsinlog +=1
 #print("Added to log. Total in Log from this session is", reading_nr_upload_nbrowsinlog)
 wipe('STATUS_ZONE')
 icon('STATUS_START',"save")
 reading_nr_upload +=1
 pending_redraw = True
 
 except Exception as e:
 print("RMC parse error:", e)
 wipe('STATUS_ZONE')
 pending_redraw = True
 pass
def readGPS():
 c = None
 response = []
 try:
 while True: # Newline, or bad char.
 global BUS
 c = BUS.read_byte(int(config['gps']['i2c_port'], 16))
 if c == 255:
 return False
 elif c == 10:
 break
 else:
 response.append(c)
 parseResponse(response)
 except IOError:
 time.sleep(0.5)
 connectBus()
connectBus()
def updateGPS():
 while True:
 readGPS()
################## config external thermometer ##################
def update_temp_ext(temp_signature='t=', update_interval=config['temp_ext']['update_interval']):
 global pending_redraw
 icon('TEMP_ICON_START', "temperature")
 base_dir = config['temp_ext']['w1_folder']
 device_folder = glob.glob(base_dir + '28*')[0]
 device_file = device_folder + '/w1_slave'
 while True:
 f = open(device_file, 'r')
 lines = f.readlines()
 f.close()
 equals_pos = lines[1].find(temp_signature)
 if equals_pos != -1:
 temp_string = lines[1][equals_pos+2:]
 temp_c = round(float(temp_string) / 1000.0)
 wipe('TEMP_ZONE')
 text('TEMP_START', str(temp_c))
 pending_redraw = True
 time.sleep(int(update_interval))
 
################## update display ##################
def update_display():
 sleep(0.5)
 while True:
 global pending_redraw
 if pending_redraw:
 device.display(output)
 pending_redraw = False
 time.sleep(0.2)
 
################## start cellular connection ################## 
def openPPPD():
 subprocess.call("sudo pon fona", shell=True)
 print("FONA on")
 wipe('GPRS_ZONE')
 icon('GPRS_START', "cloud")
 global pending_redraw
 pending_redraw = True
 sleep(20)
 try:
 urllib.request.urlopen(config['db']['ping'])
 print("Connection is on")
 wipe('GPRS_ZONE')
 icon('GPRS_START', "upload")
 pending_redraw = True
 return True
 except:
 print("Connection error")
 wipe('GPRS_ZONE')
 icon('GPRS_START', "no_conn")
 pending_redraw = True
 return False
# Stop PPPD
def closePPPD():
 print("turning off PPPD")
 subprocess.call("sudo poff fona", shell=True)
 print("turned off")
 return True
 
################## threading and program execution ##################
if __name__ == '__main__':
 temp_ext_thread = Thread(target = update_temp_ext)
 display_thread = Thread(target=update_display)
 gps_thread = Thread(target = updateGPS)
 data_thread = Thread(target = upload_data)
 
 display_thread.start() 
 gps_thread.start()
 data_thread.start()
 temp_ext_thread.start()
 
 display_thread.join()

car_computer.ini

[general]
folder = /home/pi/Desktop
folder_fonts = fonts
folder_data = data/gps
folder_data_archive = data/gps/archive
[db]
mode = server
ping = https://XXX
db_host = XXX
db_user = XXX
db_pw = XXX
db_name = XXX
db_table = gps_data
server_addr = https://www.XXX.com/gps_logger.php
server_batchsize = 50
server_ID = XXX
server_pw = XXX
[display]
i2c_port = 0x3c
### coordinates always: padding-left, padding-top. the first pair of zone is mostly = start (except to offset small icons)
# temp_ext
TEMP_ZONE = [(14,44), (36,64)]
TEMP_START = (14,44)
TEMP_ICON_ZONE = [(0,48), (15,64)]
TEMP_ICON_START = (3,48)
# alti
ALTI_ZONE = [(14,22), (69,40)]
ALTI_START = (14,22)
ALTI_ICON_ZONE = [(0,24), (15,40)]
ALTI_ICON_START = (0,26)
# distance
DIST_ZONE = [(14,0), (69,21)]
DIST_START = (14,0)
DIST_ICON_ZONE = [(0,4), (15,21)]
DIST_ICON_START = (0,4)
# speed
SPEED_ZONE = [(66,0), (128,45)]
SPEED_START = (66,0)
# GPRS status
GPRS_ZONE = [(114,46), (128,64)]
GPRS_START = (114,50)
# GPS status, incl. GPS startup icon
STATUS_ICON_ZONE = [(70,50), (88,64)]
STATUS_ICON_START = (70,50)
STATUS_ZONE = [(86,46), (113,64)]
STATUS_START_TEXT = (86,46)
STATUS_START = (86,50)
[gps]
i2c_port = 0x42
log_frequency = 5
[temp_ext]
update_interval = 30
w1_folder = /sys/bus/w1/devices/

gps_logger.php

<?php 
$mysqli = new mysqli($hostname_db, $username_db, $password_db,$database_db);
$json = file_get_contents('php://input');
if($data = json_decode($json)) {
 $sql = "SELECT * FROM gps_computers WHERE gc_ID = ? AND gc_pw = ?";
 $statement = $mysqli->prepare($sql);
 $statement->bind_param('is', $data->ID, $data->pw);
 if(!$statement->execute()) { die("Query error: ".$statement->error); } 
 $auth = $statement->get_result();
 $totalRows_auth = $auth->num_rows;
 if($totalRows_auth == 1) {
 echo "";
 } else {
 echo "no auth match found";
 die;
 }
 $i = 0;
 foreach ($data->data as $key => $record) {
 $sql = "SELECT * FROM gps_data WHERE gps_time = ? AND gps_computerID = ?";
 $statement = $mysqli->prepare($sql);
 $statement->bind_param('ii', $record->gps_time, $data->ID);
 if(!$statement->execute()) { die("Query error: ".$statement->error); } 
 $doublecheck = $statement->get_result();
 $totalRows_doublecheck = $doublecheck->num_rows;
 if($totalRows_doublecheck != 1) {
 $i++;
 $sql = "INSERT INTO `gps_data` (`gps_time`, `gps_lat`, `gps_long`, `gps_speed`, `gps_computerID`) VALUES (?, ?, ?, ?, ?);";
 $statement = $mysqli->prepare($sql);
 $statement->bind_param('isssi', $record->gps_time, $record->gps_lat, $record->gps_long, $record->gps_speed, $data->ID);
 if(!$statement->execute()) { die("Query error: ".$statement->error); } 
 }
 }
 echo $i." records inserted";
} else {
 echo "error reading json";
}
?>

I have to say I'm quite proud of how it's turned out! Happy to edit the question if you want more info to better help :)

• Reinderien: How is the PHP hosted - in an instance of Apache (or something else)? Why not just use more Python, for uniformity?
• Damien Bourdonneau: It's on a shared host running on Apache. I guess I could do more python, didnt really think of it as I've started programming using php and it always was my go-to. Other than consistency, would you see any advantages in using python?

Answer 1

First, I need to say that this is a really cool project; quite fun.

You ask:

Other than consistency, would you see any advantages in using python?

Bad code can be written in any language, but it's really easy to write bad code in PHP. At the risk of starting a language-religion flame war, there is not a single application I could in good conscience recommend to a new PHP build for which other languages aren't a better fit. Fun fact: according to Wikipedia,

72% of PHP websites use discontinued versions of PHP.

As such, many of the greyed-out entries in PHP Sadness still have significant impact. So less "use Python instead" and more "use something that isn't PHP". Anyway, on to less controversial topics:

Global code

Move lines like

device = sh1106(i2c(port=1, address=config['display']['i2c_port']), rotate=0)
device.clear()
pending_redraw = False
output = Image.new("1", (128,64))
add_to_image = ImageDraw.Draw(output)
fa_solid = setup_font('fa-solid-900.ttf', 12)
fa_solid_largest = setup_font('fa-solid-900.ttf', 40)
text_largest = setup_font('digital-7.ttf', 58)
text_medium = setup_font('digital-7.ttf', 24)
text_small = setup_font('digital-7.ttf', 18)
icons = { #to look up the icons on FontAwesome.com, remove quote marks and \u from the search query 
 "save": "\uf56f","cloud": "\uf0c2","check": "\uf058","upload": "\uf382","no_conn": "\uf127","location": "\uf124","question": "\uf128","altitude": "\uf077","distance": "\uf1b9","temperature": "\uf2c9" }

out of global scope. Among a long list of other reasons, this harms testability and pollutes your namespace. Also, since this is an embedded environment, I'd be worried about memory usage being impacted by long-lived references that needn't be.

Consider making classes, and/or moving code into free functions, and/or moving those classes and functions into more narrow-purpose Python module files.

Style

Get a linter. My favourite is PyCharm. It will help you learn the PEP8 standard, including (significantly) untangling your function definitions by separating them with two empty new lines.

Pathlib

These:

 current_dir = os.path.join(config['general']['folder'],config['general']['folder_data'])
 archive_dir = os.path.join(config['general']['folder'],config['general']['folder_data_archive'])

will be simplified by the use of pathlib. Store a Path for your config['general']['folder'] so that you don't need to re-traverse that configuration object. Drop your os.path calls.

Application location

/home/pi/

is a bad home for an application. You should take this opportunity to learn how standard Unix applications store their data. Configuration in /etc, application script in /usr/bin, data files in /usr/share, etc.

pi is a login user, and your device has internet connectivity. I'm unclear on how your application is started and whether you intend on running it as a service, but whatever the case, you would benefit from making a jail that has this application run as a heavily permissions-restricted, dedicated-purpose user.

Method complexity

Break up upload_data into multiple subroutines, given its length and complexity.

Connection management

Surround your unprotected connect/close:

 db = pymysql.connect(config['db']['db_host'],config['db']['db_user'],config['db']['db_pw'],config['db']['db_name'])
 cursor = db.cursor()
 cursor.close()

in a try/finally. I would stop short of using a with statement since that library has surprising behaviour on __exit__.

Data vs. config

These are good examples of config entries:

mode = server
ping = https://XXX
db_host = XXX
db_user = XXX
db_pw = XXX
db_name = XXX

These are not:

TEMP_ZONE = [(14,44), (36,64)]
TEMP_START = (14,44)
TEMP_ICON_ZONE = [(0,48), (15,64)]
TEMP_ICON_START = (3,48)

So far as I can tell, those are UI coordinates. They aren't meaningful for a user to configure and should instead live in the application, perhaps in a dedicated UI constants file.

Answer 2

There is a potential race condition in update_display()

def update_display():
 sleep(0.5)
 while True:
 global pending_redraw
 if pending_redraw: <-- test flag
 device.display(output) possible race condition in between
 pending_redraw = False <-- clear flag
 time.sleep(0.2)

If one of the other threads sets pending_redraw to True after this thread checks if pending_redraw: but before it sets pending_redraw = False (e.g., during the call to device.display(output)), the True will be lost.

configparser

configparser.ConfigParser supports value interpolation. This let's you write the directory configurations like so:

[general]
folder = /home/pi/Desktop
folder_fonts = %(folder)/fonts
folder_data = %(folder)/data/gps
folder_data_archive = %(folder)/data/gps/archive

And then access the directories without needing os.path.join():

dir = config['general']['folder_data']
archive_dir = config['general']['folder_data_archive']

ConfigParser include methods such as getint(), getfloat() and getboolean:

c = BUS.read_byte(config['gps'].getint('i2c_port'), 16))

And you can define your own such methods:

def getcoord(string):
 return tuple(int(v) for v in re.findall("[0-9]+", string))
def getpath(string):
 return pathlib.Path(string)
config = configparser.ConfigParser(
 converters={'coord':getcoord, 'path':getpath, 'zone':getcoord}
 )

then other code can be simpler:

def wipe(zone):
 add_to_image.rectangle(config['display'].getzone(zone),
 fill="black", outline ="black")

or

def setup_font(font_filename, size):
 return ImageFont.truetype(
 config['general'].getpath('folder_fonts') / font_filename,
 size)

upload_data()

config is loaded once at the beginning of the application. Yet upload_data reloads current_dir and archive_dir inside a while True: loop. Move them before the loop.

The code uses os.walk() to get a list of files and then uses glob.glob() to get the list again a few lines later. Get the list just once. Also, the file names include a date component, so you can just grab the max() filename

file_paths = list(current_dir.glob("*.csv")) #<-- assuming use `getpath()` 
if len(file_paths) < 2:
 print("Not enough GPS.csv files found so it's probably in use now or doesn't exist")
else:
 oldest_file = max(file_paths)
 oldest_file_name = oldest_file.name

OpenPPPD() could be a context manager used in a with statement so that the cellphone connection gets closed automatically. Look at @contextlib.contextmanager

Catching bare exceptions using except Exception as e: is generally a bad idea. Specify the exception or a tuple of exceptions that you expect could occur. Note that os.rename() might have raised an exception, in which case ClosePPPD() would have already been called when the exception handler runs.

• Damien Bourdonneau: Thanks for the input, that’s really helpful! I had heard of the rave condition thing but I’m not sure of how to address it. Is it something I just have to live with?
• RootTwo: @DamienBourdonneau, in this case the race condition probably doesn't matter. One of the other threads might set pending_redraw = True just before update_display clears it. So a redraw may get missed. But something will change (GPS, temp, ?) and that thread will cause a display update soon anyway. I just point it out, because using threads can have some gotchas.