@@ -5,6 +5,8 @@ Handy list of oft-used Linux commands that I will never remember. Not intended t
55- [ Shell 101] ( #shell-101 )
66 - [ List directory contents with size] ( #list-directory-contents-with-size )
77 - [ Count number of items in directory] ( #count-number-of-items-in-directory )
8+ - [ Find all files larger than a given size] ( #find-all-files-larger-than-a-given-size )
9+ - [ Run command on every file] ( #run-command-on-every-file )
810 - [ Monitor GPU usage] ( #monitor-gpu-usage )
911 - [ File Permissions] ( #file-permissions )
1012 - [ Symbolic Links] ( #symbolic-links )
@@ -24,6 +26,8 @@ Handy list of oft-used Linux commands that I will never remember. Not intended t
2426- [ Data Wrangling] ( #data-wrangling )
2527 - [ grep] ( #grep )
2628 - [ awk and sed] ( #awk-and-sed )
29+ - [awk to print all columns from nth to last](#awk-to-print-all-columns-from-nth-to-last)
30+ - [sed for removing and adding characters](#sed-for-removing-and-adding-characters)
2731- [ Machine Learning Toolchain] ( #machine-learning-toolchain )
2832 - [ Install Tensorflow GPU] ( #install-tensorflow-gpu )
2933## Shell 101
@@ -40,6 +44,21 @@ du -sch * .[!.]* | sort -rh
4044ls < directory path> -1 | wc -l
4145```
4246
47+ ### Find all files larger than a given size
48+ 49+ ``` bash
50+ # find all files "-type f" in the current directory "."
51+ # of size greater than 20MB and list the files using "ls -lh"
52+ # print the file size "5ドル" and the whole file name "0,2ドル"
53+ find . -type f -size +20000k -exec ls -lh {} \; | awk ' {print 5ドル ":"; 1ドル=2ドル=3ドル=4ドル=5ドル=6ドル=7ドル=8ドル=""; print substr(0,2ドル)}' | sed -r ' s .{9} '
54+ ```
55+ 56+ ### Run command on every file
57+ 58+ ``` bash
59+ for i in ./* txt; do echo " File $i " ; done
60+ ```
61+ 4362### Monitor GPU usage
4463
4564``` bash
@@ -133,6 +152,11 @@ rsync -avzPn ~/local/directory username@remote_host:/home/username/destination_d
133152# using ssh key
134153rsync -avzPn -e " ssh -i ~/ec2_keyfile.pem"
135154
155+ # read file names from a list stored in another file - https://serverfault.com/a/212441
156+ # works better when source directory is set to root "/" and file names are stored with absolute path
157+ # source directory gets appended to the file paths inside the listfile
158+ rsync -avzP /source/directory --files-from=/full/path/to/listfile /destination/directory
159+ 136160# with xargs for parallel execution - https://stackoverflow.com/a/25532027
137161# This will list all folders in ~/data, pipe them to xargs,
138162# which will read them one-by-one and and run 4 rsync processes at a time.
@@ -322,7 +346,37 @@ More `grep` funda [here](https://www.grymoire.com/Unix/Grep.html)
322346
323347### awk and sed
324348
325- TODO: Refer [ here] ( https://missing.csail.mit.edu/2020/data-wrangling/ ) now
349+ #### awk to print all columns from nth to last
350+ 351+ ``` bash
352+ # print 5th column, set columns 1 to 8 as empty, then print all columns
353+ # https://stackoverflow.com/a/2961994
354+ < command output with list of formatted strings> | awk ' {print 5ドル ":"; 1ドル=2ドル=3ドル=4ドル=5ドル=6ドル=7ドル=8ドル=""; print substr(0,2ドル)}'
355+ ```
356+ 357+ #### sed for removing and adding characters
358+ 359+ ``` bash
360+ # add in the beginning of every line in file - https://stackoverflow.com/q/2099471
361+ sed -i -e ' s/^/\/home\/users\/anurag\//'
362+ 363+ # remove last character of every line in file
364+ sed -i ' s/.$//'
365+ 366+ # add :8080 at the end of every line in portsfile
367+ sed -i ' s/$/:8080/'
368+ 369+ # REPLACE last character of each line in file with p3
370+ sed -i ' s/.$/p3/'
371+ 372+ # remove last character of every line in file
373+ sed -i ' s/.$//'
374+ 375+ # remove first two characters of every line in file
376+ sed -r ' s .{2} '
377+ ```
378+ 379+ For more tips, refer [ here] ( https://missing.csail.mit.edu/2020/data-wrangling/ ) .
326380
327381## Machine Learning Toolchain
328382
0 commit comments