Kudithipudi.Org

I have use grep extensively before to analyze data in log files before. A good example is this post about using grep and sort to find the unique hits to a website. Here is another way to do it using grep and awk.

Say the log file you are analyzing is in the format below and you need to get the unique number of BundleIDs

2013-02-25 12:00:06,684 ERROR [com.blahblah.sme.command.request.CustomCommand] Unable to execute AssignServiceCommand, request = '<AssignServiceToRequest><MemberId>123456</MemberId><OrderBundle><BundleId>5080</BundleId></OrderBundle></AssignServiceToRequest>'

you can use grep and awk to find the number of times a unique bundleID appears by running

grep -i bundleID LOG_FILE_NAME | awk '{ split ($11,a,">"); print a[6]}' | sort | uniq -c | sort -rn 

breaking down the commands

grep -i : tells grep to only show the lines from the file (LOG_FILE_NAME) containing the text bundleID and makes the search case insensitive

awk ‘{ split ($11,a,”>”); print a[6]}’ : tells awk to grab the input from grep and take the 11th item (by default awk separates content with a space) and split the string into an array (a) using > as a delimiter. And finally print out the value of the array a’s sixth member

sort : sorts the output from awk into ascending order

uniq -c : takes the output from sort and counts uniq items

sort -qn : takes the output from uniq and does a reverse order sort

The output looked like this

173 5080</BundleId
12 5090</BundleId
8 2833</BundleId
1 2412</BundleId
1 2038</BundleId
1 1978</BundleId
1 1924</BundleId

Use the following option to check out how network (NFS) shares are performing on your sever

iostat -n 2

The -n option tells iostat to just show the network shares

2 tells iostat to refresh the stats every 2 seconds

Want your terminal to look like this? 

In addition to the coolness, factor it also helps you do (monitor) multiple things at  a time. In this screenshot

• I am checking the resource utilization on my web server
• Tailing the web server logs to look for errors
• have a small console for me to run any commands

You can achieve this by using the nifty screen utility. Screen allows you to multiplex between multiple consoles.  So you can open one terminal and have multiple consoles on it. The commands for using screen are a bit hard to get used to. Here are the shortcuts I used to achieve the screen above

1. Install the screen package
2. Create a new screen session by running

    screen 

3. Add a new screen console by executing

    ctrl + a 

   c

4. Split the screen by executing

   ctrl + a 

    Shift + s 

5. Name the different consoled by executing

   ctrl + a

   Shift + a

As you might have figured out by now, “ctrl + a” puts you into screen command mode. You can get a list of all available options by executing

ctrl +a

?

Here’s a quick reference guide that has more details http://aperiodic.net/screen/quick_reference 

Have fun multitasking

For my own notes.. if you are using grep to parse through the contents of a file and want to see the preceding or proceeding content than the line that matched your query, you can use the following options

preceding content

grep -B NUMBER_OF_LINES_TO_DISPLAY query filename

for example, if I was searching for kudithipudi in a file names access.log and want to see 2 lines prior to the match, I would use

grep -B 2 kudithipudi access.log

proceeding content

grep -A NUMBER_OF_LINES_TO_DISPLAY query filename

for example, if I was searching for kudithipudi in a file names access.log and want to see 2 lines after the match, I would use

grep -A 2 kudithipudi access.log

preceding and proceeding content

grep -C NUMBER_OF_LINES_TO_DISPLAY query filename

for example, if I was searching for kudithipudi in a file names access.log and want to see 2 lines before and after the match, I would use

grep -C 2 kudithipudi access.log

puppet, is a configuration management framework that can be used to perform several different things to validate/configure your infrastructure. We have been using puppet for sometime at my work and have just started moving into some of the advanced uses of the tool.

One of the features offered by puppet is the capability to use templates to configure different servers.

For example, say you want to configure an application on server ABCD, XYZ and 123. And the configuration file for all these servers is the same, other than the hostname of the server. The configuration file has to reside in /opt/application/config.conf . The config.xml file looks like this

db.name=blah
db.user=blahblah
db.hostname=XYZ
log.level=ERROR
log.location=/var/log/application

Here is how you can do it in puppet.

Define a module which uses a template and then configure the template to put the host specific entry in the template. Let’s name our module test_config

• Create the module

• cd $PUPPET_HOME/modules
• mkdir test_config/{files,manifests,templates}
• Create the template
• cd templates
• vi config.conf.template and add the following to the file

  db.name=blah
  db.user=blahblah
  db.hostname=<%= fqdn %>
  log.level=ERROR
  log.location=/var/log/application 

• note : see how I replaced the hostname XYZ, which was specific to one server with <%= fqdn %>. This is one of the “facts” provided by puppet. you can get a list of all the facts by running facter on any of the puppet clients.
• Configure the module to use the template. In this case, we want the module to place the file config.conf in /opt/application
• cd manifests
• vi init.pp and add the following to the file

  class test_config {
  file { "/opt/application/config.conf":
  ensure => present,
  owner => appuser,
  group => appuser,
  mode => 755,
  content => template("test_config/config.conf.template"),
  }
  }

• note : There are several other options you can use for the class file.. I just gave an example of some of the common ones. Like setting the owner, group and the rights.
• Finally configure the clients to use the module. In the individual node config files, include the module you just created. Here is how the config for node ABCD would look like

  node ABCD {
  include test_config
  }

The next time the puppet client runs on host ABCD, it would create the file /opt/application/config.conf with the right hostname in the config file.