HOW TO : Use grep and awk to find count of unique entries

I have use grep extensively before to analyze data in log files before. A good example is this post about using grep and sort to find the unique hits to a website. Here is another way to do it using grep and awk.

Say the log file you are analyzing is in the format below and you need to get the unique number of BundleIDs

[code]2013-02-25 12:00:06,684 ERROR [com.blahblah.sme.command.request.CustomCommand] Unable to execute AssignServiceCommand, request = ‘<AssignServiceToRequest><MemberId>123456</MemberId><OrderBundle><BundleId>5080</BundleId></OrderBundle></AssignServiceToRequest>'[/code]

you can use grep and awk to find the number of times a unique bundleID appears by running

[code]grep -i bundleID LOG_FILE_NAME | awk ‘{ split ($11,a,">"); print a[6]}’ | sort | uniq -c | sort -rn [/code]

breaking down the commands

grep -i : tells grep to only show the lines from the file (LOG_FILE_NAME) containing the text bundleID and makes the search case insensitive

awk ‘{ split ($11,a,”>”); print a[6]}’ : tells awk to grab the input from grep and take the 11th item (by default awk separates content with a space) and split the string into an array (a) using > as a delimiter. And finally print out the value of the array a’s sixth member

sort : sorts the output from awk into ascending order

uniq -c : takes the output from sort and counts uniq items

sort -qn : takes the output from uniq and does a reverse order sort

The output looked like this

[code]
173 5080</BundleId
12 5090</BundleId
8 2833</BundleId
1 2412</BundleId
1 2038</BundleId
1 1978</BundleId
1 1924</BundleId
[/code]

HOW TO : Configure tcpdump to rotate capture files based on size

quick note for self. If you are capturing traffic using tcpdump, you can rotate the capture files based on size

[code]sudo tcpdump -i INTERFACE_TO_CAPTURE_TRAFFIC_ON -C 10 -s0 -W NO_OF_FILES_TO_ROTATE_THROUGH -w /PATH_TO_CAPTURE_FILE [/code]

explanation of the options used

-i : specify the interface you want to capture the traffic on. If  not specified, tcpdump will listen on the lowest numbered interface. i.e. eth0

-C : specify the size of the file multiplied by 1000000 bytes. In this example, the file created would be 10000000 bytes. Or ~9.8MB

-s : specify the packet length to capture. 0 (zero) tells tcpdump to capture the entire packet

-W : specify the number of files to rotate through once the files size specified in -C is reached. The files keep rotating throughout the capture

-w : Specify the path to the capture file. tcpdump appends an integer to the end of the file based on the number of files it has to rotate through.

HOW TO : Use curl to check the impact of DNS changes

Ran into an interesting scenario at work today. We had to check the impact of a DNS change on a certain hostname. Normally, you would edit your host file entry to reflect the DNS change and do your testing. Here is another way you can do it using cURL. In this particular example, I am checking the SSL certificate details of the hostname .

[code]curl –insecure –trace-ascii debug.txt https://HOSTNAME:PORT –resolve HOSTNAME:PORT:IP_ADDRESS [/code]

That’s a pretty convoluted command :). Let’s try to break it down

[code]–insecure [/code]

: tells cURL to ignore certificate warnings. This is helpful if you are using self signed certs

[code]–trace-ascii [/code]

: tells cURL to save the SSL connection details (in debug mode) to a file called debug.txt

[code]–resolve [/code]

: tells cURL to use the options mentioned after it to resolve the hostname, rather than using DNS. The format for resolve is <host:port:address>

NOTE: You need to have version 7.21.3 or higher of cURL to use this option

Here’s a real world example. Say, I want to see how the IP address 72.30.38.140 would reacts if www.google.com requests are routed to it

[code]

[email protected]:~$ curl –insecure –trace-ascii debug.txt https://www.google.com –resolve www.google.com:443:72.30.38.140
The document has moved <A HREF="http://www.google.com/?s=https">here</A>.<P>
<!– ir2.fp.sp2.yahoo.com uncompressed/chunked Mon Nov 12 22:44:41 UTC 2012 –>
[email protected]:~$ more debug.txt
== Info: Added www.google.com:443:72.30.38.140 to DNS cache
== Info: About to connect() to www.google.com port 443 (#0)
== Info: Trying 72.30.38.140… == Info: connected
== Info: Connected to www.google.com (72.30.38.140) port 443 (#0)
== Info: successfully set certificate verify locations:
== Info: CAfile: none
CApath: /etc/ssl/certs
== Info: SSLv3, TLS handshake, Client hello (1):
=> Send SSL data, 223 bytes (0xdf)
0000: ……P.|v..1..kA…….=J.xr.=ft.3.|…Z…..9.8………5…..
0040: …………….3.2…..E.D…../…A………………………
0080: …….W………www.google.com………..4.2……………….
00c0: ………………………….
== Info: SSLv3, TLS handshake, Server hello (2):
<= Recv SSL data, 42 bytes (0x2a)
0000: …&..P.{.I"L….3x..N…9…./<n….A..5.
== Info: SSLv3, TLS handshake, CERT (11):
<= Recv SSL data, 1272 bytes (0x4f8)
0000: ……….0…0..S……….0…*.H……..0N1.0…U….US1.0…
0040: U….Equifax1-0+..U…$Equifax Secure Certificate Authority0…1
0080: 00401230014Z..150703045000Z0..1)0′..U… 2g8aO5wI1bKJ2ZD588UsLvD
00c0: e3gTbg8DU1.0…U….US1.0…U….California1.0…U….Sunnyvale1
0100: .0…U….Yahoo Inc.1.0…U….www.yahoo.com0.."0…*.H……..
0140: …..0……….5.p./……..O…k.C…9E+.J..H.s….Bm.T.E.-..<
0180: ^…m…r.v<\…&Qq..l………. @'(q.m..ZJ.*kt…!.AWU…….M.
01c0: …n…O….0.._…H….4……>.m..K…….Z…:.Df%.lR.!…(!.
0200: .FV.dQ…f.V….P,.J9.c..dM.s>C=….Y..#…47#2…..cP.{….g.rU
0240: .d…P……………..0…0…U………..0…U………….t5.
0280:……U..0:..U…3010/.-.+.)http://crl.geotrust.com/crls/secure
02c0: ca.crl0..[..U…..R0..N..www.yahoo.com..yahoo.com..us.yahoo.com.
0300: .kr.yahoo.com..uk.yahoo.com..ie.yahoo.com..fr.yahoo.com..in.yaho
0340: o.com..ca.yahoo.com..br.yahoo.com..de.yahoo.com..es.yahoo.com..m
0380: x.yahoo.com..it.yahoo.com..sg.yahoo.com..id.yahoo.com..ph.yahoo.
03c0: com..qc.yahoo.com..tw.yahoo.com..hk.yahoo.com..cn.yahoo.com..au.
0400: yahoo.com..ar.yahoo.com..vn.yahoo.com0…U.#..0…H.h.+….G.# .
0440: O3….0…U.%..0…+………+…….0…*.H……………2..0.
0480: S.’.y….GD.Q…=…K+..q..kv…….<h…….ZLE.h$..M2^.C..IT..
04c0: ".5j….Vc7.4……1.Wu.[.a>+………9..{.a:………
== Info: SSLv3, TLS handshake, Server finished (14):
<= Recv SSL data, 4 bytes (0x4)
0000: ….
== Info: SSLv3, TLS handshake, Client key exchange (16):
=> Send SSL data, 262 bytes (0x106)
0000: …….R…..b.,.&.. s.Ob;.E_.EnSw../D…’…..(aB<<……F..]..
0040: o………~…*..r?.C..%..22…J.bu&.x(j|…….>A5..OF.G…C.$.
0080: .9u9n.z…K…..u…..~:W.{Sii.{2..6……..<…..i…8y$y…..6
00c0: …1.(M…fx….#k..r….47..t.q…..A.?.0. .D…..~…G+.,….~
0100: ..=.#y
== Info: SSLv3, TLS change cipher, Client hello (1):
=> Send SSL data, 1 bytes (0x1)
0000: .
== Info: SSLv3, TLS handshake, Finished (20):
=> Send SSL data, 16 bytes (0x10)
0000: ….!9)…6…+.
== Info: SSLv3, TLS change cipher, Client hello (1):
<= Recv SSL data, 1 bytes (0x1)
0000: .
== Info: SSLv3, TLS handshake, Finished (20):
<= Recv SSL data, 16 bytes (0x10)
0000: …..(qN..l.]…
== Info: SSL connection using AES256-SHA
== Info: Server certificate:
== Info: subject: serialNumber=2g8aO5wI1bKJ2ZD588UsLvDe3gTbg8DU; C=US; ST=California; L=Sunnyvale; O=Yahoo Inc.; CN=www.yahoo.com
== Info: start date: 2010-04-01 23:00:14 GMT
== Info: expire date: 2015-07-03 04:50:00 GMT
== Info: subjectAltName does not match www.google.com
=> Send header, 167 bytes (0xa7)
0000: GET / HTTP/1.1
0010: User-Agent: curl/7.21.6 (x86_64-pc-linux-gnu) libcurl/7.21.6 Ope
0050: nSSL/1.0.0e zlib/1.2.3.4 libidn/1.22 librtmp/2.3
0082: Host: www.google.com
0098: Accept: */*
00a5:
<= Recv header, 32 bytes (0x20)
0000: HTTP/1.1 301 Moved Permanently
<= Recv header, 37 bytes (0x25)
0000: Date: Mon, 12 Nov 2012 22:44:41 GMT
<= Recv header, 42 bytes (0x2a)
0000: Location: http://www.google.com/?s=https
<= Recv header, 23 bytes (0x17)
0000: Vary: Accept-Encoding
<= Recv header, 19 bytes (0x13)
0000: Connection: close
<= Recv header, 28 bytes (0x1c)
0000: Transfer-Encoding: chunked
<= Recv header, 40 bytes (0x28)
0000: Content-Type: text/html; charset=utf-8
<= Recv header, 24 bytes (0x18)
0000: Cache-Control: private
<= Recv header, 2 bytes (0x2)
0000:
<= Recv data, 173 bytes (0xad)
0000: 000009d
0009: The document has moved <A HREF="http://www.google.com/?s=https">
0049: here</A>.<P>.<!– ir2.fp.sp2.yahoo.com uncompressed/chunked Mon
0089: Nov 12 22:44:41 UTC 2012 –>.
00a8: 0
00ab:
== Info: Closing connection #0
== Info: SSLv3, TLS alert, Client hello (1):
=> Send SSL data, 2 bytes (0x2)
0000: ..

[/code]

HOW TO : Compare two directories in Linux

Quick post on using diff to compare two directories in Linux. This will show the list of files and subdirectories that are different in either directories

[code]diff /PATH_TO_FIRST_DIRECTORY /PATH_TO_SECOND_DIRECTORY -r –brief  [/code]

Options used

  • r : Searched recursively through the directory
  • –brief : Only shows the names of the files that differ. If you want details of the content that differs, remove this option

HOW TO : grep for response codes in apache logs

If you want to grep for certain http response codes in a apache log file

  • Look for all access requests with a 200 response code[code] grep -i "[: ]200[: ]" HTTP_ACCESS_LOG [/code]
  • Look for all access requests that do NOT have a 200 response code[code] grep -i -v "[: ]200[: ]" HTTP_ACCESS_LOG [/code]

Details of the options

  • [code]"[: ]"[/code]

    tells grep to look for space or tab before the specified string, which in this case is 200.

HOW TO : Find files, search for content in them, replace the content

The title pretty much says it all :). Here is a quick  one liner, using multiple tools, to look for files in a directory, search for certain content in them and replace them with other content

[code]find -type f | xargs grep -l ORIGINAL_CONTENT | xargs perl -p -i -e ‘s/ORIGINAL_CONTENT/NEW_CONTENT/g’ [/code]

You can theoretically take out the grep (second command) and directly pipe the find output to perl and get the same outcome.

Going over list of the options used

find

  • “-type f” lists all objects of type file in the directory (and sub directories)

grep

  • “-l” lists the names of the files (with relative path) which have the text ORIGINAL_CONTENT in them

perl

  • “-p” forces perl to loop through requests. In this case files
  • “-e” tells perl that the next argument is a perl statement
  • “-i” tells perls to edit the file in place (i.e. no need for an output file)

HOW TO : Use screen to multitask

Want your terminal to look like this? 

In addition to the coolness, factor it also helps you do (monitor) multiple things at  a time. In this screenshot

  • I am checking the resource utilization on my web server
  • Tailing the web server logs to look for errors
  • have a small console for me to run any commands

You can achieve this by using the nifty screen utility. Screen allows you to multiplex between multiple consoles.  So you can open one terminal and have multiple consoles on it. The commands for using screen are a bit hard to get used to. Here are the shortcuts I used to achieve the screen above

  1. Install the screen package
  2. Create a new screen session by running[code] screen [/code]
  3. Add a new screen console by executing[code] ctrl + a [/code]

    [code]c[/code]

  4. Split the screen by executing[code]ctrl + a [/code]

    [code] Shift + s [/code]

  5. Name the different consoled by executing[code]ctrl + a[/code]

    [code]Shift + a[/code]

As you might have figured out by now, “ctrl + a” puts you into screen command mode. You can get a list of all available options by executing

[code]ctrl +a[/code]

 

[code]?[/code]

Here’s a quick reference guide that has more details http://aperiodic.net/screen/quick_reference 

Have fun multitasking 🙂

HOW TO : grep options to display before and after lines of matching content

For my own notes.. if you are using grep to parse through the contents of a file and want to see the preceding or proceeding content than the line that matched your query, you can use the following options

preceding content [code]grep -B NUMBER_OF_LINES_TO_DISPLAY query filename[/code]

for example, if I was searching for kudithipudi in a file names access.log and want to see 2 lines prior to the match, I would use [code]grep -B 2 kudithipudi access.log[/code]

proceeding content[code]grep -A NUMBER_OF_LINES_TO_DISPLAY query filename[/code]

for example, if I was searching for kudithipudi in a file names access.log and want to see 2 lines after the match, I would use [code]grep -A 2 kudithipudi access.log[/code]

preceding and proceeding content[code]grep -C NUMBER_OF_LINES_TO_DISPLAY query filename[/code]

for example, if I was searching for kudithipudi in a file names access.log and want to see 2 lines before and after the match, I would use [code]grep -C 2 kudithipudi access.log[/code]

HOW TO : Use templates in puppet to pass hostnames

puppet, is a configuration management framework that can be used to perform several different things to validate/configure your infrastructure. We have been using puppet for sometime at my work and have just started moving into some of the advanced uses of the tool.

One of the features offered by puppet is the capability to use templates to configure different servers.

For example, say you want to configure an application on server ABCD, XYZ and 123. And the configuration file for all these servers is the same, other than the hostname of the server. The configuration file has to reside in /opt/application/config.conf . The config.xml file looks like this

[code]

db.name=blah
db.user=blahblah
db.hostname=XYZ
log.level=ERROR
log.location=/var/log/application

[/code]

Here is how you can do it in puppet.

Define a module which uses a template and then configure the template to put the host specific entry in the template. Let’s name our module test_config

  • Create the module
    • cd $PUPPET_HOME/modules
    • mkdir test_config/{files,manifests,templates}
  • Create the template
    • cd templates
    • vi config.conf.template and add the following to the file[code]db.name=blah
      db.user=blahblah
      db.hostname=<%= fqdn %>
      log.level=ERROR
      log.location=/var/log/application [/code]
      • note : see how I replaced the hostname XYZ, which was specific to one server with <%= fqdn %>. This is one of the “facts” provided by puppet. you can get a list of all the facts by running facter on any of the puppet clients.
  • Configure the module to use the template. In this case, we want the module to place the file config.conf in /opt/application
    • cd manifests
    • vi init.pp and add the following to the file[code]class test_config {
      file { "/opt/application/config.conf":
      ensure => present,
      owner => appuser,
      group => appuser,
      mode => 755,
      content => template("test_config/config.conf.template"),
      }
      }[/code]
      • note : There are several other options you can use for the class file.. I just gave an example of some of the common ones. Like setting the owner, group and the rights.
  • Finally configure the clients to use the module. In the individual node config files, include the module you just created. Here is how the config for node ABCD would look like[code]node ABCD {
    include test_config
    }[/code]

The next time the puppet client runs on host ABCD, it would create the file /opt/application/config.conf with the right hostname in the config file.