I am facing a weird issue while using xtrabackup with xbstream. The weird part is that it was working fine just few days ago and apart from a reboot there were no changes on the machine. I have at least 100 machines with this exact same script, same OS, same packages and xtrabackup binary. Everything is same on all the machines. It works on all the machines except one.
OS Details
uname -a
5.10.227-219.884.amzn2.x86_64 #1 SMP Tue Oct 22 16:38:23 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
NAME="Amazon Linux"
VERSION="2"
ID="amzn"
ID_LIKE="centos rhel fedora"
VERSION_ID="2"
PRETTY_NAME="Amazon Linux 2"
MySQL Version and xtrabackup version
/usr/bin/xtrabackup --version
/usr/bin/xtrabackup version 2.4.29 based on MySQL server 5.7.44 Linux (x86_64) (revision id: 2e6c0951)
mysql-community-server-5.7.44-1.el7.x86_64
xtrabackup command which causes error
/usr/bin/xtrabackup --defaults-file=/etc/my.cnf --backup --slave-info --login-path=backup_user --tmpdir=/data/tmp --parallel=4 --use-memory=1024MB --stream=xbstream | /usr/bin/lz4 - /S3-Mountpoint/backup/app_`date +%F`.lz4
The error I receive is
xtrabackup: Error writing file 'UNOPENED' (Errcode: 32 - Broken pipe)
xb_stream_write_data() failed.
xtrabackup: Error: write to logfile failed
xtrabackup: Error writing file 'UNOPENED' (Errcode: 32 - Broken pipe)
xtrabackup: Error: xtrabackup_copy_logfile() failed.
So while checking what could be wrong, I checked with local machine path and it too doesn't work, fails with same error
/usr/bin/xtrabackup --defaults-file=/etc/my.cnf --backup --slave-info --login-path=backup_user --tmpdir=/data/tmp --parallel=4 --use-memory=1024MB --stream=xbstream | /usr/bin/lz4 - /local_machine_path/app_`date +%F`.lz4
I tried the same with tar and it does work
/usr/bin/xtrabackup --defaults-file=/etc/my.cnf --backup --slave-info --login-path=backup_user --tmpdir=/data/tmp --use-memory=1024MB --stream=tar | /usr/bin/gzip > /S3-Mountpoint/backup/app_`date +%F`.tar.gz
Again I tried it without xbsteram on local machine and it works
/usr/bin/xtrabackup --defaults-file=/etc/my.cnf --backup --slave-info --login-path=bakcup_user --tmpdir=/data/tmp --parallel=4 --use-memory=1024MB --target-dir=/local_machine_path/app_`date +%F`
So my conclusion is that something is wrong with xbstream, I am not sure though. Could this be a bug? Or am I missing something? Please help out.