Cannot change fuses on ATmega32U4

Question

I have a custom board based on Arduino Leonardo (ATmega32u4). As usual I used the Arduino IDE to burn the bootloader with my Atmel ICE ISP programmer.

This time it failed complaining about some errors writing the fuse bits. So I tried to manually set them with avrdude but it turns out it can't.

I first dumped out the current values:

avrdude: Device signature = 0x1e9587 (probably m32u4)
avrdude: safemode: lfuse reads as 5E
avrdude: safemode: hfuse reads as D8
avrdude: safemode: efuse reads as F3
avrdude> d lock
>>> d lock 
0000 ff 

Then I tried to change, say, lfuse to 0xFF:

avrdude "-C/home/mark/.arduino15/packages/arduino/tools/avrdude/6.3.0-arduino17/etc/avrdude.conf" -v -patmega32u4 -catmelice_isp -Pusb -Ulfuse:w:0xff:m

here the output:

avrdude: reading input file "0xff"
avrdude: writing lfuse (1 bytes):
Writing | ################################################## | 100% 0.09s
avrdude: 1 bytes of lfuse written
avrdude: verifying lfuse memory against 0xff:
avrdude: load data lfuse data from input file 0xff:
avrdude: input file 0xff contains 1 bytes
avrdude: reading on-chip lfuse data:
Reading | ################################################## | 100% 0.00s
avrdude: verifying ...
avrdude: verification error, first mismatch at byte 0x0000
 0x5e != 0xff
avrdude: verification error; content mismatch
avrdude: safemode: lfuse reads as 5E
avrdude: safemode: hfuse reads as D8
avrdude: safemode: efuse reads as F3
avrdude: safemode: lfuse changed! Was ff, and is now 5e
Would you like this fuse to be changed back? [y/n] y

But it failed:

avrdude: safemode: Wrote lfuse to ff, read as 5e. 9 attempts left
avrdude: safemode: Wrote lfuse to ff, read as 5e. 8 attempts left
avrdude: safemode: Wrote lfuse to ff, read as 5e. 7 attempts left
avrdude: safemode: Wrote lfuse to ff, read as 5e. 6 attempts left
avrdude: safemode: Wrote lfuse to ff, read as 5e. 5 attempts left
avrdude: safemode: Wrote lfuse to ff, read as 5e. 4 attempts left
avrdude: safemode: Wrote lfuse to ff, read as 5e. 3 attempts left
avrdude: safemode: Wrote lfuse to ff, read as 5e. 2 attempts left
avrdude: safemode: Wrote lfuse to ff, read as 5e. 1 attempts left
avrdude: safemode: Wrote lfuse to ff, read as 5e. 0 attempts left
avrdude: and COULD NOT be changed
avrdude: safemode: Fuses not recovered, sorry

I can program the MCU with a firmware (using the ISP programmer) and it works, so it's not damaged. What can I do in order to fix this writing issue on the fuse bits?