Reading JSON from Arduino via Serial Missing Parts of Message

Question

I have a project (Digital Dashboard for a Motorhome) in Pi that reads JSON data from an Arduino board and sends it via Serial. The Pi gets the JSON data, makes some adjustments and then sends it to an webapp (fronted of the digital dashboard). I'm getting a lot of the data correct but as you will see below on the log I'm also getting randomly missing parts of the JSON string, which makes the gauges not super "realtime".

Why could this be happening?

Raspberry Pi Python code:

@webiopi.macro
def getSerialArduinoData():
 data = ""
 
 # empty input buffer before starting processing
 while (serial.available() > 0):
 serial.readString()
 webiopi.sleep(1)
 serial.writeString("d") #send command "d" to request data from Arduino
 if (serial.available() > 0):
 data = serial.readString() # read available data as string
 print("")
 print(data)
 print()
 
 return data

Arduino Code:

void loop() {
 if (Serial.available() > 0) {
 input = Serial.read();
 switch (input) {
 case 'd':
 // call all data here
 // Get the current millis.
 unsigned long currentMillis = millis();
 //Initiate Json Document
 JsonDocument docJson;
 //Add Time to Json
 docJson["timeMillis"] = currentMillis;
 // Read all data
 readAirTanks(docJson);
 readFuelLevel(docJson);
 readEngineOilPressure(docJson);
 // Only read external Temperature and Humidity if module is connected.
 if (adafruitSensorAHTX0_OK) {
 docJson["externalTempHumidity"] = true;
 readExternalTempHumidity(docJson);
 } else {
 docJson["externalTempHumidity"] = false;
 }
 // Only read internal Temperature, Humidity and Pressure if module is connected.
 if (adafruitSensorMS8607_OK) {
 docJson["internalTempHumidityPressure"] = true;
 readInternalTempHumidityPressure(docJson);
 } else {
 docJson["internalTempHumidityPressure"] = false;
 }
 // Only read J1708 data if module is connected to vehicle. 
 if (Vehicle_OK) {
 docJson["ecmData"] = true;
 readJ1708Data(docJson);
 } else {
 docJson["ecmData"] = false;
 }
 // Send Json over Serial.
 serializeJson(docJson, Serial);
 // Add a newline character to mark the end of the JSON object
 Serial.println();
 break;
 default:
 break;
 }
 }
}

Pi log sample:

{"timeMillis":289391,"primaryAirTank":120,"secondaryAirTank":120,"fuelLevel":49,"engineOilPressure":56,"externalTempHumidity":true,"externalTemperature":86,"externalHumidity":62,"internalTempHumidityPressure":true,"internalTemperature":71,"internalHumidity":62,"internalPressure":993,"ecmData":true,"vehicleSpeed":0,"engineLoad":1,"coolantTemperature":134,"batteryVoltage":24.91321,"rpm":999,"intakeManifoldAirTemp":96}
{"timeMillis":290478,"primaryAirTank":120,"secondaryAirTank":120,"fuelLevel":51,"engineOilPressure":57,"externalTempHumidity":true,"externalTemperature":86,"externalHumidity":62,"internalTempHumidityPressure":true,"internalTemperature":71,"internalHumidity":63,"internalPressure":993,"ecmData":true,"vehicleSpeed":0,"engineLoad":1,"coolantTemperature":134,"batteryVoltage":25.17189,"rpm":1001,
{"timeMillis":291527,"primaryAirTank":120,"secondaryAirTank":120,"fuelLevel":50,"engineOilPressure":57,"externalTempHumidity":true,"externalTemperature":86,"externalHumidity":62,"internalTempHumidityPressure":true,"internalTemperature":71,"internalHumidity":63,"internalPressure":993,"ecmData":true,"vehicleSpeed":0,"engineLoad":1,"coolantTemperature":134,"batteryVoltage":25.4047,"rpm":1002,"intakeManifoldAirTemp":96}
{"timeMillis":292570,"primaryAirTank":120,"secondaryAirTank":120,"fuelLevel":50,"engineOilPressure":57,"externalTempHumidity":true,"externalTemperature":86,"externalHumidity":63,"internalTempHumidityPressure":true,"internalTemperature":71,"internalHumidity":63,"internalPressure":993,"ecmData":true,"vehicleSpeed":0,"engineLoad":1,"coolantTemperature":134,"batteryVoltage":25.61423,"rpm":999,"intakeMani

• Edgar Bonet: On the Raspberry Pi, type sudo apt install picocom && picocom -b 9600 /dev/ttyACM0 (with 9600 and ttyACM0 replaced with the appropriate port and baud rate). Then hit d. If you see the full JSON strings, then your problem is on the Python side. Exit picocom with Ctrl-A Ctrl-Q.
• Honeybadger22: Thank you for the reply! I will try some of these code and report back.
• Juraj: please don't edit the solution into the question. I rolled it back. use the Answer box to add the solution of the problem
• Honeybadger22: Thank you @Juraj I made the correction and posted as an answer.

Answer 1

For anyone else looking for this in the future the change I had to make was in the Python side, using pyserial instead of the version provided by the webiopi.

This is the final code:

import serial
# Configure the serial port (replace with your port and baud rate)
ser = serial.Serial('/dev/ttyACM0', 9600, timeout=1)
@webiopi.macro
def getSerialArduinoData():
 json = ""
 webiopi.sleep(1)
 ser.write(b'd') #send command "d" to request data from Arduino
 data_bytes = ser.read_until(b'\n') # Read until the newline character
 json = data_bytes.decode('utf-8') # Decode the bytes to a string
 json = json.strip() # Remove the delimiter from the end
 return json

• the busybee: Did you check why serial.readString() fails? I doubt it has something to do with the interface implementation as such, but more with some property like timeout. Are you aware that serial transmission with 9600 bps takes some time?