LoRa stops working after several hours. Arduino reset button ineffective

Question

I am designing a master-slave LoRa link between two Arduino Pro Minis. The goal is to control a car heater a few hundred meters from my home. The master sends a command to the slave as needed. The slave acknowledges the command.

If no ACK is received in 5 sec the command is resent. In addition to this the master queries the slave every minute to verify the link.

This library is used to control the DRF1278F

I presume that a receive callback or a call to LoRa.parsePacket() returns either nothing or a complete packet, so I do not ever have to handle partial packets.

Chinese Arduino Pro Mini 3.3V 8MHz boards are used.
I am using a module DRF1278F module which contains an RFIC SX1278

Connections

Arduino --> LoRa module
----------------------------------
 SS 10 --> NSEL 10k pullup to Vcc
 SCK 13 --> SCK 
MISO 12 --> SDO 100k pulldown to GND 
MOSI 11 --> SDI 
 INT 3 --> IO0 
 RST 6 --> RST 

The device is built on a piece of PCB acting as a ground plane, all GND connections are done to this plane.
It is constructed with short direct wiring. Slave device Problem.

After several hours of operation either the master or the slave sort of hangs up. No radio operations any more. There is activity on the SPI but very different to normal operation. I have tested for weeks and not yet survived for one night.

Pushing the reset button does not restore operations. Removing 3.3V power always restore normal operation, even a very brief short circuit of 3.3V. There is no need to rest in power down, so probably not an overheat issue.

It can be either the master or the slave end that hangs up. Power cycling only the hung up unit will restore communications. My simple protocol will then resyncronize itself and continue running.

The arduino watch dog timer is activated, however it does not trigger when the hangup occurs as the arduino program loop still runs.

Things tested.

The 3.3 V supply has less than 50mV p-p ripple as checked with an oscilloscope set to infinite persistence for 8 hours, while a hangup occurred.

The interface can be run either polled or interrupt driven, no difference. No String objects are used.

Using different Vcc like 3.0V and 3.6V makes no difference.
Extra Vcc decoupling made no improvement.
Lowering SPI speed from 8MHz to 4MHz made no difference.
Removed Arduino LED on pin 13.

My problem symptoms are similar to this old question

What can I do to resolve this issue?

Slave code:

/*
Relay controller, remote end of link
*/
#include <SPI.h> 
#include <LoRa.h>
#include <avr/wdt.h>
//#define TTT 
#define csPin 10
#define resetPin 6
#define irqPin 3 // must be a hardware interrupt pin
#define myLED 8
#define GNDRET 7
#define powerPin 9
unsigned long LEDtime;
const unsigned long LEDonTime = 500;
char outgoing[40];
byte msgCount = 0;
unsigned long lastLoopTime = 0;
unsigned long heaterStarted = 0l;
const unsigned long heaterTime = 60 * 60 * 1000L; // One hour in ms
unsigned interval = 2000;
int temp = 2310;
byte localAddress = 0x11, destination = 0xAA;
unsigned long ttt;
volatile char incoming[40];
volatile int recipient;
volatile byte sender, incomingMsgId,
incomingLength, gotMsg, ovrn;
void setup()
{
 Serial.begin(9600);
 while (!Serial);
 pinMode(myLED, OUTPUT);
 digitalWrite(myLED, LOW);
 pinMode(GNDRET, OUTPUT);
 digitalWrite(GNDRET, LOW);
 pinMode(powerPin, OUTPUT);
 digitalWrite(powerPin, LOW);
 Serial.println("LoRa Switch 0.1");
 LoRa.setPins(csPin, resetPin, irqPin);
/* 
 LoRa.setTxPower(10);
 LoRa.setSpreadingFactor(12);
 LoRa.setCodingRate4(8);
 LoRa.setSPIFrequency(4E6);
 LoRa.enableCrc();
*/
 wdt_enable(WDTO_1S);
 if (!LoRa.begin(433E6))
 {
 Serial.println("LoRa init failed. Check your connections.");
 {
 // blinking in panic
 LEDtime = millis();
 while (millis() - LEDtime < 75)
 {
 digitalWrite(myLED, HIGH);
 }
 while (millis() - LEDtime < 150)
 {
 digitalWrite(myLED, LOW);
 }
 }
 }
 LoRa.onReceive(onReceive); // Assign callback
 LoRa.receive();
 Serial.println("LoRa init succeeded.");
}
void loop()
{
 if (millis() - lastLoopTime > 1000)
 {
 // Once per second loop
 lastLoopTime = millis();
 }
// onReceive(LoRa.parsePacket()); // Polling
 if (gotMsg) // Flag from CallBack routine
 {
 digitalWrite(myLED, HIGH);
 LEDtime = millis();
#ifdef TTT
 Serial.print("Received from: 0x");
 Serial.println(sender, HEX);
 Serial.print("Sent to: 0x");
 Serial.println(recipient, HEX);
 Serial.print("Message ID: ");
 Serial.println(incomingMsgId);
 Serial.print("Message length: ");
 Serial.println(incomingLength);
 Serial.print("Message: <<");
 for (int i = 0; incoming[i] != 0; i++)
 Serial.write(incoming[i]);
 Serial.println(">>");
 Serial.print("RSSI: ");
 Serial.println(LoRa.packetRssi());
 Serial.print("Snr: ");
 Serial.println(LoRa.packetSnr());
 Serial.println();
#endif
 // Command processing
 if (incomingLength == 4)
 {
 // Start heater, it will either timeout or be stopped by a message
 if ((byte)incoming[0] == 0x55 && (byte)incoming[1] == 0xAA)
 {
 Serial.println("Set on");
 digitalWrite(powerPin, HIGH);
 heaterStarted = millis();
 }
 if ((byte)incoming[0] == 0xCC && (byte)incoming[1] == 0x99) // Stop heater
 {
 Serial.println("Set off");
 digitalWrite(powerPin, LOW);
 heaterStarted = 0L;
 }
 // Build ack answer
 if (heaterStarted > 0L)
 {
 // Compute session time in minutes
 ttt = millis() - heaterStarted;
 ttt /= (1000L * 60L); // minutes since start
 }
 else
 ttt = 0L;
 outgoing[0] = (heaterStarted > 0L) ? 'T' : 'F'; // Heater state
 snprintf(outgoing + 1, 30, "%3lu", ttt);
 incoming[0] = 0; // We are done, clear flags
 gotMsg = 0;
 sendAck(outgoing);
 }
 LoRa.receive();
 }
 if (LEDtime) // LED timeout
 {
 if (millis() - LEDtime > LEDonTime)
 {
 digitalWrite(myLED, LOW);
 LEDtime = 0;
 }
 }
 if (heaterStarted) // Heater timeout
 {
 if (millis() - heaterStarted > heaterTime)
 {
 digitalWrite(powerPin, LOW);
 heaterStarted = 0L;
 }
 }
 wdt_reset();
 // END of loop
}
void sendMessage(char *outgoing)
{
 LoRa.beginPacket(); // start packet
 LoRa.write(destination); // add destination address
 LoRa.write(localAddress); // add sender address
 LoRa.write(msgCount++); // add message ID
 LoRa.write(bsize(outgoing)); // add payload length
 LoRa.print(outgoing); // add payload
 LoRa.endPacket(); // finish packet and send it
}
void sendAck(char *outgoing)
{
 Serial.println("Send ACK");
 LoRa.beginPacket();
 LoRa.write(destination);
 LoRa.write(localAddress);
 LoRa.write(incomingMsgId); // report ID
 LoRa.write(bsize(outgoing)); // add payload length
 LoRa.print(outgoing); // add payload
 LoRa.endPacket();
}
byte bsize(char *ooo)
{
 byte i = 0;
 while (ooo[i++]);
 return i - 1;
}
byte bsize(volatile char *oo)
{
 return bsize((volatile char *)oo);
}
byte insptr;
void onReceive(int packetSize) // has to be a non blocking function
{
 if (packetSize == 0) return; // if there's no packet, return
 if (gotMsg) ovrn = true; // gotMsg should be reset by consumer by now
 // read packet header bytes:
 recipient = LoRa.read(); // recipient address
 sender = LoRa.read(); // sender address
 incomingMsgId = LoRa.read(); // incoming msg ID
 incomingLength = LoRa.read(); // incoming msg length
 insptr = 0;
 while (LoRa.available())
 {
 if (insptr < 38)
 incoming[insptr++] = (char)LoRa.read();
 else
 LoRa.read(); // Flush
 }
 incoming[insptr] = 0; // End of C string
 gotMsg = true;
}
/*
void setTxPower(int level, int outputPin = PA_OUTPUT_PA_BOOST_PIN);
void setFrequency(long frequency);
void setSpreadingFactor(int sf);
void setSignalBandwidth(long sbw);
void setCodingRate4(int denominator);
void setPreambleLength(long length);
void setSyncWord(int sw);
void enableCrc();
void disableCrc();
*/

Master code:

/*
Relay controller, Controller and GUI base
*/
#include <SPI.h> 
#include <LoRa.h>
#include <avr/wdt.h>
#define TTT 
#define csPin 10
#define resetPin 6
#define irqPin 3 // must be a hardware interrupt pin
#define ackLED 9
#define LEDGND 8
#define remoteLED 7
unsigned long LEDtime;
const unsigned long LEDonTime = 500;
char outgoing[40];
byte msgSequ = 0, mincnt, ack_TO;
unsigned long lastLoopTime = 0;
unsigned interval = 2000;
byte localAddress = 0xAA, destination = 0x11;
unsigned long ttt;
volatile char incoming[40];
volatile int recipient;
volatile byte sender, incomingMsgId, incomingLength, isACKed, gotMsg, ovrn;
char cc;
void setup()
{
 Serial.begin(9600);
 while (!Serial);
 pinMode(ackLED, OUTPUT);
 digitalWrite(ackLED, LOW);
 pinMode(remoteLED, OUTPUT);
 digitalWrite(remoteLED, LOW);
 pinMode(LEDGND, OUTPUT);
 digitalWrite(LEDGND, LOW);
 isACKed = 1;
 Serial.println("LoRa Master 0.1");
 LoRa.setPins(csPin, resetPin, irqPin);
/* no specials
 LoRa.setTxPower(10);
 LoRa.setSPIFrequency(4E6);
 LoRa.setSpreadingFactor(12);
 LoRa.setCodingRate4(8);
 LoRa.enableCrc();
*/
 wdt_enable(WDTO_1S);
 if (!LoRa.begin(433E6))
 {
 Serial.println("LoRa init failed. Check your connections.");
 {
 // blinking in panic
 LEDtime = millis();
 while (millis() - LEDtime < 75)
 {
 digitalWrite(ackLED, HIGH);
 }
 while (millis() - LEDtime < 150)
 {
 digitalWrite(ackLED, LOW);
 }
 }
 }
 LoRa.onReceive(onReceive); // Assign callback
 LoRa.receive();
 Serial.println("LoRa init succeeded.");
}
void loop()
{
 digitalWrite(ackLED, isACKed == 0);
 if (millis() - lastLoopTime > 1000)
 {
 // Once per second loop
 lastLoopTime = millis();
 if (isACKed == 0)
 {
 if (ack_TO++ > 5)
 {
 sendMessage(outgoing); // Resend
 LoRa.receive();
 ack_TO = 0;
 Serial.println("Resend...");
 }
 }
 if (mincnt++ > 60)
 {
 // Once per minute loop
 mincnt = 0;
 if (isACKed) // Free to ask
 {
 // Heartbeat
 cmdAsk();
 isACKed = 0;
 }
 }
 }
 if ((cc = Serial.read()) > 0) // Test version, cmd 
 {
 if (cc == '1')
 {
 cmdOn();
 isACKed = 0;
 }
 else if (cc == '0')
 {
 cmdOff();
 isACKed = 0;
 }
 else if (cc == 'Q')
 {
 cmdAsk();
 isACKed = 0;
 }
 }
// onReceive(LoRa.parsePacket()); // Polling 
 if (gotMsg) // Flag from onReceive routine
 {
#ifdef TTT
 Serial.print("Received from: 0x");
 Serial.println(sender, HEX);
 Serial.print("Sent to: 0x");
 Serial.println(recipient, HEX);
 Serial.print("Message ID: ");
 Serial.println(incomingMsgId);
 Serial.print("Message length: ");
 Serial.println(incomingLength);
 Serial.print("Message: <<");
 for (int i = 0; incoming[i] != 0; i++)
 Serial.write(incoming[i]);
 Serial.println(">>");
 Serial.print("RSSI: ");
 Serial.println(LoRa.packetRssi());
 Serial.print("Snr: ");
 Serial.println(LoRa.packetSnr());
 Serial.println();
#endif
 // Command processing
 if (incomingLength == 4)
 if (incomingMsgId == msgSequ - 1) // Ack is on last sent
 {
 if (incoming[0] == 'T') digitalWrite(remoteLED, HIGH);
 if (incoming[0] == 'F') digitalWrite(remoteLED, LOW);
 isACKed = 1;
 }
 incoming[0] = 0; // Message is processed, clear flags
 gotMsg = 0;
 }
 if (LEDtime) // LED timeout
 {
 if (millis() - LEDtime > LEDonTime)
 {
 LEDtime = 0;
 }
 }
 wdt_reset();
} // END of loop
void cmdOn()
{
 outgoing[0] = 0x55; // Heater on
 outgoing[1] = 0xAA;
 outgoing[2] = 'X';
 outgoing[3] = 'Y';
 sendMessage(outgoing);
 LoRa.receive();
}
void cmdOff()
{
 outgoing[0] = 0xCC; // Heater off
 outgoing[1] = 0x99;
 outgoing[2] = 'A';
 outgoing[3] = 'B';
 sendMessage(outgoing);
 LoRa.receive();
}
void cmdAsk()
{
 outgoing[0] = 0x0F;
 outgoing[1] = 0xF0;
 outgoing[2] = '7';
 outgoing[3] = '7';
 sendMessage(outgoing);
 LoRa.receive();
}
void sendMessage(char *outgoing)
{
 LoRa.beginPacket(); // start packet
 LoRa.write(destination); // add destination address
 LoRa.write(localAddress); // add sender address
 LoRa.write(msgSequ++); // add message ID
 LoRa.write(bsize(outgoing)); // add payload length
 LoRa.print(outgoing); // add payload
 LoRa.endPacket(); // finish packet and send it
}
byte bsize(char *ooo)
{
 byte i = 0;
 while (ooo[i++]);
 return i - 1;
}
byte bsize(volatile char *oo)
{
 return bsize((volatile char *)oo);
}
unsigned char insptr;
void onReceive(int packetSize) // is a non blocking function
{
 if (packetSize == 0) return; // if there's no packet, return
 if (gotMsg) ovrn = true; // gotMsg should be reset by consumer by now
 // read packet header bytes:
 recipient = LoRa.read(); // recipient address
 sender = LoRa.read(); // sender address
 incomingMsgId = LoRa.read(); // incoming msg ID
 incomingLength = LoRa.read(); // incoming msg length
 insptr = 0;
 while (LoRa.available())
 {
 if (insptr < 38)
 incoming[insptr++] = (char)LoRa.read();
 else
 LoRa.read(); // Flush
 }
 incoming[insptr] = 0;
 gotMsg = true;
}
/*
void setTxPower(int level, int outputPin = PA_OUTPUT_PA_BOOST_PIN);
void setFrequency(long frequency);
void setSpreadingFactor(int sf);
void setSignalBandwidth(long sbw);
void setCodingRate4(int denominator);
void setPreambleLength(long length);
void setSyncWord(int sw);
void enableCrc();
void disableCrc();
RSSI always a value between -30 ( very close) and - 120
SNR 0 is noise floor... max = 10
*/

• Nick Gammon: If pressing Reset on the Arduino doesn't help it seems likely to me that the Lora part is the problem. Pressing Reset wouldn't reset that (necessarily).
• jsotola: does it always hang up after the same amount of time?
• Nick Gammon: Removing 3.3V power always restore normal operation - to which end? Sending or receiving?
• Nick Gammon: between two Arduino Pro Micros ... Chinese Arduino Pro Mini 3.3V 8MHz boards are used - which is it? Mini or Micro?
• jsotola: maybe it has something to do with a sequence counter of some kind reaching an overflow condition

Answer 1

I feel I want to report the probable solution to this problem.
I hope an answer to my own question is appropriate.

After much testing I found an interaction between the boot loader and the watchdog. This post describes it well: Watchdog timer blog post
So, replacing the boot loader seems to remove the hangup, so far ... Why the watchdog triggers in the first place remains to be found out but presently it does little or no harm.

A side note, the reset button seems to reset the ProMini if pushed reasonably soon after the hangup. If pushed hours later it has no effect.

• Wirewrap: Has been running continiously for over a year now wihout any hangup.