Compare script using awk and bash

Question

I try to create compare script using awk and another command and running successful.

But I think the script I create it's to long.

Is there anyone can shorten my script below ?

After plan A shorten the code than, plan B is :

 1. I want eliminated a lot temp file (.txt), only need `lengkap.txt`
 2. Put command in variable if can

It's my pleasure if anyone can help me

Below is the code that I create based searching and trying.

#!/bin/bash
### Path Folder who will be compare ###
path1=/home/rio/apps1
path2=/home/rio/apps2
### Find all filename and convert to MD5 ###
find $path1 -type f | xargs md5sum > checksums.md5
find $path2 -type f | xargs md5sum > checksums2.md5
### Compare to find different folder ###
awk 'NR==FNR{c[1ドル]++;next};c[1ドル] == 0' checksums.md5 checksums2.md5 > hasil1.txt
awk 'NR==FNR{c[1ドル]++;next};c[1ドル] == 0' checksums2.md5 checksums.md5 > hasil2.txt
### Merge result of compare ###
awk '{print 0ドル}' hasil1.txt hasil2.txt > perbedaan.txt
### Filter Just Filename Difference ###
cat perbedaan.txt | awk '{print 2ドル}' > hasilperbedaan.txt
### File about result compare (just filename) ###
cekhasil=/home/rio/hasilperbedaan.txt
### Check if File result compare empty or not ###
if [ -s "$cekhasil" ]
then 
 echo " file exists and is not empty "
 ### Find All filename and date, after that put as we want ###
 find $path1 -type f -ls | awk '{print 11ドル" "8ドル" "9ドル" "10ドル }' > filedate1.txt
 find $path2 -type f -ls | awk '{print 11ドル" "8ドル" "9ドル" "10ドル }' > filedate2.txt
 ### Compare to get the date of filename ###
 awk 'A[1ドル]++' hasilperbedaan.txt filedate1.txt > pre_hasil1.txt
 awk 'A[1ドル]++' hasilperbedaan.txt filedate2.txt > pre_hasil2.txt
 ### Merge result of compare with date ###
 awk '{print 0ドル}' pre_hasil1.txt pre_hasil2.txt > lengkap.txt
else
 echo " file does not exist, or is empty "
fi

• Ellen Spertus: Welcome to Code Review Stack Exchange! You might find useful the advice on choosing a good title for a question. You can edit yours. codereview.stackexchange.com/help/how-to-ask
• pacmaninbw: Please tell us what the code is supposed to do.
• hjpotter92: one thing that seems fishy about the script is that it writes a lot of files in cwd, which are all not cleaned after the job is done.
• Ario: @pacmaninbw it's for compare folder with different server but same folder (example : apps).
• Ario: @hjpotter92, yes, I think so too. That's why need advice to shorten the script.

Answer 1

Well, it took time to figure out what your program is doing. So I did shorten it in two phases.

First phase: I removed all duplicate temporary files, and used a pipe when a temp file was used once.

STEP1: you just make a checksum of all files in $path1 and $path2

### Find all filename and convert to MD5 ###
find $path1 -type f | xargs md5sum > checksums.md5
find $path2 -type f | xargs md5sum > checksums2.md5

You don't need 2 temp files, as path is included in filename. So you can replace this with a single find. I use sort to be able to use uniq afterwards:

find "$path1" "$path2" -type f | xargs md5sum | sort > cksum.md5

STEP2: you find unique checksums in both files, and get the corresponding filenames.

### Compare to find different folder ###
awk 'NR==FNR{c[1ドル]++;next};c[1ドル] == 0' checksums.md5 checksums2.md5 > hasil1.txt
awk 'NR==FNR{c[1ドル]++;next};c[1ドル] == 0' checksums2.md5 checksums.md5 > hasil2.txt
### Merge result of compare ###
awk '{print 0ドル}' hasil1.txt hasil2.txt > perbedaan.txt
### Filter Just Filename Difference ###
cat perbedaan.txt | awk '{print 2ドル}' > hasilperbedaan.txt

As we have a checksum-sorted file, we just filter with uniq, and get filenames.

Note: As we use a MD5 checksum (128 bits), length is 128bits / 8bits * 2hex=32

uniq -u -w32 cksum.md5 | awk '{print 2ドル}' > "$cekhasil"

If you prefer awk to uniq, just compare previous record's 1ドル with current's.

STEP 3: you match found entries with a new search (find) in the two source paths.

if [ -s "$cekhasil" ]
then
 echo " file exists and is not empty "
 ### Find All filename and date, after that put as we want ###
 find $path1 -type f -ls | awk '{print 11ドル" "8ドル" "9ドル" "10ドル }' > filedate1.txt
 find $path2 -type f -ls | awk '{print 11ドル" "8ドル" "9ドル" "10ドル }' > filedate2.txt
 ### Compare to get the date of filename ###
 awk 'A[1ドル]++' hasilperbedaan.txt filedate1.txt > pre_hasil1.txt
 awk 'A[1ドル]++' hasilperbedaan.txt filedate2.txt > pre_hasil2.txt
 ### Merge result of compare with date ###
 awk '{print 0ドル}' pre_hasil1.txt pre_hasil2.txt > lengkap.txt
else
 echo " file does not exist, or is empty "
fi

Here, it is better to avoid new full find, and just loop over filenames.

if [ -s "$cekhasil" ]
then
 echo " file exists and is not empty "
 while read -r fn ; do
 ls -dils "$fn" | awk '{print 11ドル" "8ドル" "9ドル" "10ドル }'
 done < "$cekhasil" > lengkap2.txt
else
 echo " file does not exist, or is empty "
fi

Alltogether, the script becomes:

#!/bin/bash
path1=/home/br/dev/tools/bash
path2=/home/br/dev/tools/bash2
outfile=hasilperbedaan.txt
find "$path1" "$path2" -type f | xargs md5sum | sort > cksum.md5
uniq -u -w32 cksum.md5 | awk '{print 2ドル}' > "$outfile"
if [ -s "$outfile" ]; then
 echo " file exists and is not empty "
 while read -r fn ; do
 ls -dils "$fn" | awk '{print 11ドル" "8ドル" "9ドル" "10ドル }'
 done < "$output" > lengkap2.txt
else
 echo " file does not exist, or is empty "
fi

At this point, we can notice the 2 remaining temp files are used only once. We just get rid of them, and just pipe all together to get the output file.

 #!/bin/bash
 
 path1=/home/br/dev/tools/bash
 path2=/home/br/dev/tools/bash2
 outputfile="lengkap-v2.txt"
 
 find "$path1" "$path2" -type f | xargs md5sum | sort |
 uniq -u -w32 | awk '{print 2ドル}' |
 while read -r fn ; do
 ls -dils "$fn" | awk '{print 11ドル" "8ドル" "9ドル" "10ドル }'
 done > "$outputfile"
 [[ -s "$outputfile" ]] && echo " file exists and is not empty " ||
 echo " file does not exist, or is empty "

Notes:

• I don't understand your second question: "Put command in variable if can", so I skip that one.
• You should never use the output of find (except with -print0 options) or ls as input of any command. But it is out of scope for this answer.