Firmata: strings sent from the computer become damaged after Arduino

Question

I have a Python program that acts as a Firmata client. This program connects to the Arduino at a user-specified COM-port (with Firmata onboard, the sketch can be seen below), sends the user-requested string message to the Arduino, and awaits Arduino's reply. If the reply is present, the program prints it out as a list of ASCII symbol codes.

I expect the program and the sketch to work this way (block scheme is below), but the strings coming from the Arduino are damaged. They don't contain all symbols which were sent, but they contain only the first symbols in some random order, with a 0円 symbol present after each character in the string. For example,

String sent to the Arduino	Received as ASCII symbol codes	Received as a string
A	(65, 0)	A0円
AB	(65, 0) where is the B?	A0円
ABC	(65, 0, 67, 0) the B lacks presence	A0円C0円
ABCD	(65, 0, 67, 0) the received data length remains the same, but B and D aren't present	A0円C0円
ABCD(10)	(65, 0, 67, 0, 40, 1, 48, 1) WHAT IS THIS?	A0円C0円(10円1円
DHTt(4)	(68, 0, 84, 0, 40, 0, 41, 1)	D0円T0円(0円)1円

Program's screenshot:

Screenshot

Awaited work algorithm block scheme: Block scheme

Python program code:

import pyfirmata, time
class CustomArduino(pyfirmata.Arduino):
 def __init__(self, port: str):
 pyfirmata.Arduino.__init__(self, port)
 self.iterator = pyfirmata.util.Iterator(self)
 self.iterator.start()
 self.add_cmd_handler(0x71, self.receiveString)
 def receiveString(self, *args, **kwargs):
 print(f'Received: {args}')
 def sendString(self, data):
 self.send_sysex(0x71, bytearray(data, 'utf-8'))
 def send_sysex(self, sysex_cmd, data=[]):
 """
 Sends a SysEx msg.
 :arg sysex_cmd: A sysex command byte
 : arg data: a bytearray of 7-bit bytes of arbitrary data
 """
 msg = bytearray([pyfirmata.pyfirmata.START_SYSEX, sysex_cmd])
 msg.extend(data)
 msg.append(pyfirmata.pyfirmata.END_SYSEX)
 self.sp.write(msg)
print('STRING TESTER\nPlease input your COM port: ', end = '')
myCom = str(input())
device = CustomArduino(myCom)
while True: 
 device.sendString(input('Send: '))
 time.sleep(1)

Arduino sketch:

#include <Firmata.h>
void stringCallback(char *myString)
{
 //The function accepts one charset.
 Firmata.sendString(myString); //send the *same* charset back
}
void sysexCallback(byte command, byte argc, byte *argv)
{
 Firmata.sendSysex(command, argc, argv);
}
void setup()
{
 Firmata.setFirmwareVersion(FIRMATA_FIRMWARE_MAJOR_VERSION, FIRMATA_FIRMWARE_MINOR_VERSION);
 Firmata.attach(STRING_DATA, stringCallback); //when there is a STRING incoming to the Arduino, call 'stringCallback'
 Firmata.attach(START_SYSEX, sysexCallback);
 Firmata.begin(57600);
 pinMode(13, OUTPUT);
}
void loop()
{
 while (Firmata.available()) {
 Firmata.processInput();
 }
}

How I can deal with this?

• PMF: Firmata doesn't normally do something with a string (the commands are binary), so what do you want the arduino to do, actually? What you see (and why there are so many 0's in the reply) is that Firmata encodes texts, both on sending and on receiving.
• Starter: @PMF, the 0円s insertion wouldn't be a problem if the string returned by the Arduino contained all the characters present in the string. However, alongside with 0円s insertion, not every single character present in the sent string exists in the received string.
• Starter: @PMF, edited my question to make it answer your comment.

Answer 1

You have to understand the Firmata protocol to write your own client library. The spec is here: https://github.com/firmata/protocol.

In this particular case, you need to know that data that is sent (the string in this case) can never use bit 7 of the data, since these values have special meanings. Therefore, Firmata sends every byte as two bytes, first bits 0-6 and second bit 7 as bit 0 of the next byte. Therefore, the conversion to UTF-8 is not sufficient. You need to do a two-byte-per-8-bits encoding. The same applies for the reply.

• Starter: The pyfirmata client library is alright, considering that any library is going to be bad in case of its misuse, just as in my situation. In my case, the problem was on the Python program's side

Answer 2

Alternatively, if you want a system without Firmata's baggage of ancient MIDI constraints, like its bizarre 7-bit limit, you might consider DaqPort https://www.daqarta.com/dw_rraa.htm

• Starter: your option is too hard to study in.

Answer 3

Instead of using the UTF-8 encoding,

def receiveString(self, *args, **kwargs):
 print(f'Received: {args}')
def sendString(self, data):
 self.send_sysex(0x71, bytearray(data, 'utf-8'))

it is necessary to use the pyfirmata special bytearray manipulation methods: two_byte_iter_to_str and str_to_two_byte_iter:

def receiveString(self, *args, **kwargs):
 print(f'Received: {pyfirmata.util.two_byte_iter_to_str(args)}')
def sendString(self, data):
 self.send_sysex(0x71, pyfirmata.util.str_to_two_byte_iter(data))