How to extract only values greater than a threshold

Question

I am new to bash scripting. Could someone help me with the following? I have a log file with output as shown below.

I'm trying to grep for lines of an output with logDurationMillis>=950ms

logAlias:Overall,logDurationMillis:382,logTimeStart:2019年07月24日_15:30:06.075,logTimeStop:2019年07月24日_15:30:06.107
logAlias:Overall,logDurationMillis:388,logTimeStart:2019年07月24日_15:30:06.406,logTimeStop:2019年07月24日_15:30:06.444
logAlias:Overall,logDurationMillis:545,logTimeStart:2019年07月24日_15:30:06.583,logTimeStop:2019年07月24日_15:30:06.638
logAlias:Overall,logDurationMillis:961,logTimeStart:2019年07月24日_15:30:06.599,logTimeStop:2019年07月24日_15:30:06.660
logAlias:Overall,logDurationMillis:640,logTimeStart:2019年07月24日_15:30:07.197,logTimeStop:2019年07月24日_15:30:07.237
logAlias:Overall,logDurationMillis:934,logTimeStart:2019年07月24日_15:30:07.474,logTimeStop:2019年07月24日_15:30:07.508
logAlias:Overall,logDurationMillis:336,logTimeStart:2019年07月24日_15:30:07.546,logTimeStop:2019年07月24日_15:30:07.582

The values are always in the second comma-delimited column.

Answer 1

With awk:

1. if you know "logDurationMillis" is the second item:

   awk -F'[:,]' -v limit=950 '4ドル >= limit' file
   

2. otherwise

   awk -F'[:,]' -v limit=950 '{
    for (i=1; i<NF; i+=2) 
    if ($i == "logDurationMillis" && $(i+1) >= limit) 
    print
   }' file
   

• jesse_b: I did not know you could do that with the delimiter! Awesome
• Aslam534: logDurationMillis is the second field so i tried the 2nd script you have provided, It worked.. Thank you so much
• Olivier Dulac: @Jesse_b: awk (even regular awk) can accept regexps as delimiter, and also extended ones (ex: -F 'this|that[ _]*also' will separate fields with "this" and with "thatalso" or "that also" or "that_also" (and allow even multiple occurences of space or _, as there is a '*' in the regexp). It can be really handy. Here, in this answer, it separates fields with either 1 "," or 1 ":", so it will separate fields at each occurence of "," or ":" . (if the line starts with a separator, 1ドル is empty and 2ドル is the next field)
• Olivier Dulac: @glennjackman: Could you please edit that first one? to put 4ドル as you clearly intended to. (I am not sure if it is correct etiquette to edit it myself as it "changes" the answer...)
• glenn jackman: I did update it. It would be absolutely appropriate for you to edit my answer. The OP changed the input data after I answered, and a simple change like that does not alter the intent of the answer.

Answer 2

Assuming your file is named logFile, this command will do it:

egrep ',logDurationMillis:(9[5-9][0-9]|[1-9][0-9]{3,}),' logFile

• Ed Morton: Using a regexp for a numeric comparison results in hard to understand/maintain code. To compare numbers, just use awk with a numeric comparison - there's zero reason to do this with regexps in grep.

Answer 3

another awk one:

awk -F':|,' '4ドル > 950' file

prints:

logAlias:Overall,logDurationMillis:961,logTimeStart:2019年07月24日_15:30:06.599,logTimeStop:2019年07月24日_15:30:06.660

---

Update (due to OPs Question:)

you can reconstruct the fields like this:

awk -F':|,' '4ドル > 950 {print 3ドル ":" 4ドル "," 5ドル ":" 6ドル ":" 7ドル ":" 8ドル }' file

prints:

logDurationMillis:961,logTimeStart:2019年07月24日_15:30:06.599

(there is probably a simpler way that concatinates the fields 3-8, but you also need to reconstruct the different field separators)

• G-Man Says 'Reinstate Monica': (1) This is basically the same as glenn jackman’s answer. (2) The question says ">=950", but your answer says > 950.
• ilkkachu: That double pipe || looks odd. The intervening part would match the empty string too, but what use is that? Why not just :|,?
• G-Man Says 'Reinstate Monica': @ilkkachu: Good catch; I missed that. I suspect that nath meant :|, — as I said, equivalent to glenn’s [:,] — but reflexively typed || for "OR". But, AFAICT in six to eight minutes of testing, this answer works. Can you demonstrate that the extra | actually causes it to fail?
• Ed Morton: @G-Man a | immediately following | is undefined behavior per POSIX, see pubs.opengroup.org/onlinepubs/9699919799/basedefs/…: | ... A <vertical-line> appearing ... immediately following a <vertical-line> ... produces undefined results.. So that code could do anything in any given awk.
• Aslam534: Thanks Nath, 'awk -F':||,' '4ドル > 950' file' works :) Actual log file is like logAlias:Overall,logDurationMillis:382,logTimeStart:2019年07月24日_15:30:06.075,logTimeStop:2019年07月24日_15:30:06.107, UniqueId.... question :How do i get only following fields in result logDurationMillis:961,logTimeStart:2019年07月24日_15:30:06.075,UniqueId