Advanced Grepping

Shell & Grep: Where we count certain strings in a log file by hour.

for var in `seq 10 23`;
do
echo 'Feb 28 '$var;
grep 'Feb 28 '$var /log/cli/Hanson/2015-02-28.log | grep 'Duplicate order cancel' | wc -l;
done

How can we see: Number of Errors in several files. 

Output as: Number Filename

 grep -RIci "ERROR" . | awk -v FS=":" -v OFS="\t" '$2>0 { print $2, $1 }' | sort -hr
 

Image slicing and dicing

Handy code to chop up an image file:
Takes a file called convert_cake_layers.png
Chops it up by the sizes 300 X 154 and outputs the files cl_0.png -> cl_9.png

#!/bin/bash
seq 0 9 | awk "{ print \"convert cake_layers.png -crop 300x154+0+\" \$0*154 \" cl_\" \$0 \".png\" }" | bash

Moving the cursor on the command line

Bash shortcuts source


Ctrl + a – go to the start of the command line
Ctrl + e – go to the end of the command line
Ctrl + k – delete from cursor to the end of the command line
Ctrl + u – delete from cursor to the start of the command line
Ctrl + w – delete from cursor to start of word (i.e. delete backwards one word)
Ctrl + y – paste word or text that was cut using one of the deletion shortcuts (such as the one above) after the cursor
Ctrl + xx – move between start of command line and current cursor position (and back again)

Ctrl + r – search the history backwards


and dont forget good old ^^. ^x^y = run last command but replace 'x' with 'y'

Replace command


$ ls myfile
$ ^myfile^myotherfile


^X^Y - will replace X with Y.

Hence - the second command will do: ls myotherfile

Searching in files

Search for files containing the string "lookfor"

find . -type f -exec grep -i "lookfor" {} \; -print


SSH to bounce jobs

Code to SSH to another box, kill the old process and bring up a new one.

SSH, source the login file and cd to the right directory:

ssh -f user@somebox " . ./.bashrc; cd /apps/myapp/;


Kills the old process:

cat ../pid | xargs kill;


Calls startMyApp and pipes stdout, stderr to files. Creates a file called pid containing the process ID of this process:

nohup ./startMyApp 1>../stdout 2>../stderr & echo $! > ../pid


Putting it all together:

ssh -f user@somebox " . ./.bashrc; cd /apps/myapp/; cat ../pid | xargs kill; sleep 10; nohup ./startMyApp 1>../stdout 2>../stderr & echo $! > ../pid' > startMyAppRemote.log


Bash if statement + RegEx + String matching

After going slowly crazy trying to remember how an if statement is formed in shell with its lovely and entirely useless help messages.

 I've decided to post this: #!/bin/sh FULLHOSTNAME=`hostname -f` if [ "$FULLHOSTNAME" = "loneqflocfd0025.uk.db.com" ]; then echo "hi" fi 

  Notes: You need a space after '[' and before ']' You need a ';' ...and while I'm here - here is how to do a regex if:

  if [[ "los" =~ lo.* ]]; then 

  ...String matching with variables When comparing strings " is very different to ' ' will not evaluate variables " will evaluate them 

Hence: fish=haddock echo 'Hello $fish'=== Hello $fish echo "Hello $fish" === Hello haddock 

 

Using dates and comparing current time with something that may have taken less than 2 seconds:


  start=`date +%s`

  # Do something


  now=`date +%s`

   if [ $start -gt $((now - 2)) ];
   then
      printf "Fast fail detected. I know about:\n" 


   fi




ssh in a loop

Runs the command 'ls' on several machines and stores the results in output.txt


while read host; do
echo "running on $host"
ssh -n $host "ls " >> output.txt;
done < boxes.txt


where boxes.txt is a file containing a list of machines to ssh to

advanced reading

Alternative:

for loop in {281..284}
do
echo "running on frread@nygeqgd0$loop.us.db.com"
ssh -n frread@nygeqgd0$loop.us.db.com "ls " >> output.txt;
done


2nd Alternative:

for loop in `seq 281 284`

more bash loops

Finding a single file in multiple jar files

this script copied from:
http://www.devdaily.com/blog/post/java/shell-script-search-search-multiple-jar-files-class-pattern


#!/bin/sh

LOOK_FOR="codehaus/xfire/spring"

for i in `find . -name "*jar"`
do
   echo "Looking in $i ..."
   jar tvf $i | grep $LOOK_FOR > /dev/null
   if [ $? == 0 ]
   then
     echo "==> Found \"$LOOK_FOR\" in $i"
   fi
done


Removing spaces from filenames

newname=`echo "$name" | tr -d " "`

from:
http://ubuntuforums.org/showthread.php?t=368708

linux command line colors

Run this command to dump the colors to a text file:
dircolors --print-database >> .ls_colors

Edit the text file.

Add this line to your .profile file:
eval $(dircolors -b ~/.ls_colors)


http://articles.techrepublic.com.com/5100-10878_11-5749641.html

Summing all the numbers in a file

This works on a file of the form

a=1
bbbb=2
cc=4
and will output '7'

 awk '{  split($1, ar, "="); sum += ar[2]; } END { print sum; }' < ab2.txt

....and while we are here in Awk:
$0 = the whole line
$1..n = the line after a split by space
END must be in capitals

.. Awk in a shell script using strings not files:
This takes the string s1 and splits it (by space) and stores the first 2 columns.
(to split by something other than space use -F) see here

col1=`echo $s1 | awk  '{print $1}'`
col2=`echo $s1 | awk  '{print $2}'`

Awk with ls
Split by underscore, only show unique names
List files -> Keep part of the filename before '_' -> only keeps unique names

ls | awk '{ split($1, ar, "_"); print ar[1] }' | uniq

Awk for http logs. Find all 400 errors in this todays log:

head -100 2014-02-13.log | awk -v FS='\t' '{ print $6,$0 }' | grep ^40* 

Alternate way of finding 500 errors: 

cat /log/nginx_access/current | awk '$9 == "500" { print $0 }' | less

Not technically awk - Shell script that greps & sorts logs by each hour:

start=$(date --date '5 apr 2014 0:00' +%s)
stop=$(date --date '5 apr 2014 23:00' +%s)
#These are expressed in seconds. You can use a for loop on their values, increasing them by 1 hour (3600 seconds):
for t in $(seq ${start} 3600 ${stop})
do
        d=$(date --date @${t} +'%d/%b/%Y:%H')
        d_pretty=$(date --date @${t} +'%d-%b-%Y-%H')
        #echo $d
        grep $d 2014-04-05.log | cut -f 8 | sort | uniq -c| sort -rn | head -20 > out_$d_pretty.txt
done
 

Grep can count occurrences in a file:

 grep -c RuntimeError 2015-10-*