Technical Stuff

In machine learning(ML), clean data for training is the key for a better model.
Must remove noise(unwanted data like special characters) from input data to get clean clean data.
.To do this one of the first step is to make sure data format is consistent in entire input file.
One of the best way to check this is below command. This command should return
Only one value, if not your data file is not properly formatted.

cat file_filename | awk -F’,’ ‘{print NF}’ | sort -u

Let’s see in detail about this command. This command contains below sub parts

cat: is the command to display contents of a text file
awk: this is very power full text processing tool.
sort:this command is to sort and u is for unique and removing duplicates
NF: is number of fields. This will be used In AWK
Pipe(|): this is also very power full command line utilitiy which combines more than one command.

Let’s take a sample file which contains name, age and city of some customers separated by comma, which means each line should contain only three fields separated by comma. But if you see in the below data, in the second line it contains four fields separated by comma.This is very very common scenario in Machine learning training data. And checking this format in a large file not easy. Using above command we can easily verify it.

Chandra, 35,Singapore
Sekhar,Chandra, 26,Guntur
Sekhar,35,Singapore

Applying above command on this text will return the results 3 and 4 as shown below as input data is not consistent. This is due to second line contains one extra comma in it.


After removing extra comma from the second line, that command returns only one value which is 3 as shown below. This return value may vary basing on number of fields in each line, but each line should contain same number of fields which makes this command returns only one value.



cat command will display the contents of the file and redirect to AWK because of pipe and awk will split each line with comma as -F is a field separator and NF will return all the fields after split each line. In this case we have three lines in that first and third lines has three and second line has 4 after split using comma. Sort will show the sorted result with duplicates and -u will show only unique values.

This command will work efficiently on very large files as well. I tried on data files whose file size is around one million lines and it returned results in seconds.


Happy Learning!!!

Recently I learned useful tips from my manager for 'grep' which helps a lot when troubleshooting the issues with logs. In this post I will explain about below

• How to search multiple strings in a file at the same time.
• How to display found string in the color.
• How to exclude particular string in the search result
• How to display total no.of lines in the grep result

Search a string in a file using grep: Check this link for basic usage of grep command.

#include<stdio.h>
#include<unistd.h>
#include<stdlib.h>
int main()
{
 printf("Hello");
 exit(0);
}

programs$ ./a.out
Helloprograms$

#include<stdio.h>
#include<unistd.h>
#include<stdlib.h>
int main()
{
 printf("Hello");
 _exit(0);
}

#include<stdio.h>
#include<unistd.h>
#include<stdlib.h>
int main()
{
 printf("Hello\n");
 _exit(0);
}

programs$ ./a.out
Hello
programs$

We got the output Hello, this is due to we are forcefully flushing the data using '\n'. Infact printf() function wont print or flush the data until buffer completes or end of the character is \n. printf internally maintains some buffer.

int main()
{
 printf("Hello");
}

programs$ gdb a.out
GNU gdb 6.3.50-20050815 (Apple version gdb-1824) (Wed Feb 6 22:51:23 UTC 2013)
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "x86_64-apple-darwin"...Reading symbols for shared libraries .. done
(gdb) br main
Breakpoint 1 at 0x100000f14: file _exit.c, line 5.
(gdb) r
Starting program: /Users/kh1279/Desktop/practice/Blog/programs/a.out 
Reading symbols for shared libraries +............................. done
Breakpoint 1, main () at _exit.c:5
5		printf("Hello");
(gdb) 
(gdb) n
6	}
(gdb) 
0x00007fff933ab7e1 in start ()
(gdb) 
Single stepping until exit from function start, 
which has no line number information.
0x00007fff933ab808 in dyld_stub_exit ()
(gdb) 
Single stepping until exit from function dyld_stub_exit, 
which has no line number information.
0x00007fff8b4a4f74 in exit ()
(gdb) 
Single stepping until exit from function exit, 
which has no line number information.
Hello0x00007fff8b4eb576 in dyld_stub___exit ()
(gdb) 
Single stepping until exit from function dyld_stub___exit, 
which has no line number information.
0x00007fff8efa3ae0 in _exit ()
(gdb) 
Single stepping until exit from function _exit, 
which has no line number information.
Program exited with code 0377.
(gdb) 
The program is not being run.
(gdb)