So many missed bytes with serial comm

Question

I have a setup with serial comm from Raspberry Pi to Arduino. It's communicating at baud 9600 on both devices, and from Raspberry Pi 3b+ GPIO-14,UART0,TX0 to Arduino Nano RX0. There's a level shifter in between setup as recommended by AdaFruit tutorials. I'm sending data from Raspberry Pi as such:

#loop here
m = []
m.append(255)
m.append(i) // index value between 0-150
m.append(p[0][i]) // value between 0-254
m.append(p[1][i]) // value between 0-254
m.append(p[2][i]) // value between 0-254
#print statement here
'''
Data: [255, 94, 0, 0, 0, 255, 95, 254, 0, 0, 255, 96, 0, 254, 0]
...
'''

And receiving data on Arduino as such:

void serialEvent() {
 while (Serial.available()) {
 // get the new byte:
 byte inChar = (byte)Serial.read();
 sprintf(printString, "Received: %d", inChar);
 Serial.println(printString);
 }
}
/*
Received: 255
Received: 94
Received: 0
Received: 255
Received: 95
Received: 254
Received: 0
Received: 255
Received: 96
Received: 0
Received: 254
Received: 255
Received: 94
Received: 0
Received: 0
Received: 0
Received: 255
Received: 254
Received: 255
...
*/

Any idea why there are so many missed bytes and how to fix this?

EDIT: @juraj There is a ground between both. I've measured with a portable oscilloscope. I can post the picture of the setup tomorrow.

@Edgar Bonet The loop function is empty, and looks as follows:

void loop() {
}

I also don't have any data being sent back to the Raspberry Pi. I have the Serial TX wire out from Arduino completely disconnected

• the busybee: It does not matter if you wire TX or not. Serial.println() takes the same time in any case. Did you try to shorten the string, like just the hex representation of the received bytes?
• Edgar Bonet: Re: "I have the Serial TX wire out from Arduino completely disconnected": this is irrelevant. The Arduino doesn't care, and doesn't even know the wire is disconnected. It is still sending the data out. This takes time: about 14 times the time it took to receive the data.
• errolflynn: @EdgarBonet Could you please clarify why it is sending out the data? Is it because of the serial print statement?
• Edgar Bonet: Yes, Serial.println(...) means "please, send out this data".

Answer 1

Without the full codes, we can only guess. My guess would be the serial receive buffer on the Arduino is overflowing.

One reason this could happen is if serialEvent() is not called frequently enough. This function is called by the Arduino core on every loop iteration, just after loop() returns. If your loop() takes too long (e.g. you are using delay() or blocking waits), the receive buffer can overflow before serialEvent() has a chance to empty it.

Another reason is if the Raspberry Pi is sending too much data, too fast. For every byte the Arduino receives, it must send back between 13 and 15 bytes. If too much data comes too fast, the transmit buffer will eventually get full, and then Serial.println() becomes really slow, because it has to wait for the bytes to actually get out on the wire. This in turn slows down serialEvent() to the point where it can make the receive buffer overflow.

Edit: If this (the Raspberry sending data too fast) is the cause of the problem, the obvious solution is to have it send data more slowly, as:

repeat:
 send a message
 delay for some suitable time

• Nick Gammon: I totally agree with Edgar Bonet. You are sending 13 to 15 bytes per input byte. This takes time in the hardware, regardless of whether you have anything connected to it or not. So you expect it to misread data after 1/15 of the size of the output buffer, which I think is 64 bytes by default.
• errolflynn: @NickGammon Thanks, that sounds reasonable to me. Do either of you have a suggestion for receiving data and debugging correctly? Should I have both TX and RX hooked up and do some sort of acknowledgment?
• Nick Gammon: I have a post here about debugging using SPI (and also I2C) if you have those pins free. That is much faster than using serial comms.
• Nick Gammon: @errolflynn Also reduce the length of your debugging messages, that will help. For example, instead of "Received:" send "R".