Shell script to archive files to AWS S3

Question

This is a simple script that moves files to S3 if they are> 14 days old. It ignores files that are < 14 days old. If the file is successfully synced to AWS S3 then we can remove it from the server. If not, we leave the file where it is. The echo commands in the script are for debugging. I am aware that this could be a one liner, so interested to see what people suggest.

#!/bin/bash
fortnight=$(python -c "from datetime import datetime, timedelta; print (datetime.now()-timedelta(days=14)).strftime('%Y%m%d')")
for f in /var/mail/catchall-????????.gz
do
 if [[ "$f" < "/var/mail/catchall-$fortnight.gz" ]] ; then
 mv "$f" ./archive
 echo "$f"
 echo "$(basename $f)"
 cd ./archive
 if s3cmd sync "$(basename $f)" s3://vpcgwmail ; then
 echo "S3 sync successful, removing file"
 rm "$(basename $f)"
 else
 echo "s3 sync not succesful" 
 fi
 fi
done

Answer 1

It's a fine script, but you have a bug: executing cd ./archive in a loop. Perhaps you did not notice it because you tested with only a single matching file. If there are multiple matching files, then cd ./archive will not work, unless you have multiple nested archive/archive/archive/... directories, which would be silly.

One solution is to wrap the cd command and the rest of the loop body in a (...) subshell:

if [[ "$f" < "/var/mail/catchall-$fortnight.gz" ]] ; then
 mv "$f" ./archive
 echo "$f"
 echo "$(basename $f)"
 (
 cd ./archive
 if s3cmd sync "$(basename $f)" s3://vpcgwmail ; then
 echo "S3 sync successful, removing file"
 rm "$(basename $f)"
 else
 echo "s3 sync not succesful" 
 fi
 )
fi

Another solution is to not cd ./archive at all:

if [[ "$f" < "/var/mail/catchall-$fortnight.gz" ]] ; then
 mv "$f" ./archive
 echo "$f"
 echo "$(basename $f)"
 if s3cmd sync ./archive/"$(basename $f)" s3://vpcgwmail ; then
 echo "S3 sync successful, removing file"
 rm ./archive/"$(basename $f)"
 else
 echo "s3 sync not succesful" 
 fi
fi

Whichever alternative is fine.

The indentation was also a bit off, from about the middle of the for-loop body.

• kafka: Excellent thanks for the feedback. The potential bug is a good point. Although the script will run daily, and hence will only process one file at a time (in theory!), it shouldn't encounter it. However, better safe than sorry!