Kudithipudi.Org

April 29, 2012

HOW TO : grep options to display before and after lines of matching content

Filed under: HOWTO,Linux,Technology — Vinay @ 8:59 am

For my own notes.. if you are using grep to parse through the contents of a file and want to see the preceding or proceeding content than the line that matched your query, you can use the following options

preceding content 

grep -B NUMBER_OF_LINES_TO_DISPLAY query filename

for example, if I was searching for kudithipudi in a file names access.log and want to see 2 lines prior to the match, I would use 

grep -B 2 kudithipudi access.log

proceeding content 

grep -A NUMBER_OF_LINES_TO_DISPLAY query filename

for example, if I was searching for kudithipudi in a file names access.log and want to see 2 lines after the match, I would use 

grep -A 2 kudithipudi access.log

preceding and proceeding content 

grep -C NUMBER_OF_LINES_TO_DISPLAY query filename

for example, if I was searching for kudithipudi in a file names access.log and want to see 2 lines before and after the match, I would use 

grep -C 2 kudithipudi access.log

April 26, 2012

HOW TO : Use templates in puppet to pass hostnames

Filed under: HOWTO,Linux,Technology — Vinay @ 10:00 pm

puppet, is a configuration management framework that can be used to perform several different things to validate/configure your infrastructure. We have been using puppet for sometime at my work and have just started moving into some of the advanced uses of the tool.

One of the features offered by puppet is the capability to use templates to configure different servers.

For example, say you want to configure an application on server ABCD, XYZ and 123. And the configuration file for all these servers is the same, other than the hostname of the server. The configuration file has to reside in /opt/application/config.conf . The config.xml file looks like this


db.name=blah
db.user=blahblah
db.hostname=XYZ
log.level=ERROR
log.location=/var/log/application

Here is how you can do it in puppet.

Define a module which uses a template and then configure the template to put the host specific entry in the template. Let’s name our module test_config

  • Create the module
    • cd $PUPPET_HOME/modules
    • mkdir test_config/{files,manifests,templates}
  • Create the template
    • cd templates
    • vi config.conf.template and add the following to the file 
      db.name=blah
db.user=blahblah
db.hostname=<%= fqdn %>
log.level=ERROR
log.location=/var/log/application
      • note : see how I replaced the hostname XYZ, which was specific to one server with <%= fqdn %>. This is one of the “facts” provided by puppet. you can get a list of all the facts by running facter on any of the puppet clients.
  • Configure the module to use the template. In this case, we want the module to place the file config.conf in /opt/application
    • cd manifests
    • vi init.pp and add the following to the file 
      class test_config {
file { "/opt/application/config.conf":
ensure => present,
owner => appuser,
group => appuser,
mode => 755,
content => template("test_config/config.conf.template"),
}
}
      • note : There are several other options you can use for the class file.. I just gave an example of some of the common ones. Like setting the owner, group and the rights.
  • Finally configure the clients to use the module. In the individual node config files, include the module you just created. Here is how the config for node ABCD would look like 
    node ABCD {
include test_config
}

The next time the puppet client runs on host ABCD, it would create the file /opt/application/config.conf with the right hostname in the config file.

April 17, 2012

HOW TO : Configure Jboss to use hugepages in RHEL/CentOS

Filed under: HOWTO,Linux,Technology,Web — Vinay @ 5:23 pm

Most of us worry about paging to disk (swap), but if you are running a transaction intensive application the paging that happens in RAM also starts to impact the application performance. This happens due to the size of the “block” that is used to store data in memory. Hugepages allows you to store the data in bigger blocks, hence reducing the need to page while interacting with the data.

Here is how you can enable hugepages and configure jboss (actually any Java app) to use hugepages on a RHEL/CentoOS system.

OS CONFIGURATION

  1. Check if your system is capable of supporting hugepages by running 
    grep HUGETLB /boot/config-`uname -r`

    If you see the response as below, you should be good 

    CONFIG_HUGETLBFS=y
CONFIG_HUGETLB_PAGE=y
  • Next check if huge pages are already being used by running 
    cat /proc/sys/vm/nr_hugepages
  1. If the response is anything other than 0, that means hugepages have already been configured.
  • Find the block size for hugepages by running 
    cat /proc/meminfo | grep -i hugepagesize
  • Calculate the amount of memory you want to dedicate to hugepages. (note: memory allocated to hugepages cannot be used by other processes in the system, unless they are configured to use it)
  1. For example, I want to dedicate 3GB of RAM for hugepages. So the number of hugepages would be 
    (3*1024*1024)/2048
  • Configure the number of hugepages on the system by editing the /etc/sysctl.conf and adding the option 
    vm.nr_hugepages = 1536

    (note: I put in 1536 since that was the value I got from the above example)

  • Restart the server and check if hugepages has been enabled by running 
    cat /proc/meminfo | grep -i huge
  1. You should see something like this 
    AnonHugePages:    839680 kB
HugePages_Total:    1500
HugePages_Free:     1500
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB

JBOSS CONFIGURATION

  1. At this point your system is configured with hugepages and any application that is configured to use them can leverage them.  In this example, we want to configure Jboss to utilize these hugepages
  2. Add the groupid of the user that Jboss is running under to the /etc/sysctl.conf file. In my case, the jboss user group had a GID of 505, so I added this line to /etc/sysctl.conf 
    vm.hugetlb_shm_group = 505
  3. Next allocate the memory to the user by editing /etc/security/limits.conf and allocating the memory. Again, in my case, I added the following to /etc/security/limits.conf 
    # Allocate memory for Jboss user to take advantage of hugepages
jboss   soft    memlock 1500
jboss   hard    memlock 1500
  4. Finally add the following to the Jboss startup parameters. I edited the $JBOSS_HOME/bin/run.sh file. (note: the startup file can be different based on your config) with the option 
     -XX:+UseLargePages
  5. Restart Jboss and you are good to go

note : A lot articles that I read online say that hugepages are effective when you are allocating large amounts of RAM to the application. The use case of just using 3GB above was just that.. a use case.

While I cannot personally vouch for it, a lot of users have noted that they saw >2 fold increase in performance.

April 16, 2012

HOW TO : Install RPM packages without checking gpg key

Filed under: HOWTO,Linux — Vinay @ 5:07 pm

This is on RHEL and CentOS distros. If you want to install packages without checking the GPG key (hope you know why you are doing this!!), here is the command line option

sudo yum install package_to_install --nogpgcheck

April 11, 2012

HOW TO : Check number of processes (including threads) being run/executed by a user in Linux

Filed under: HOWTO,Linux — Vinay @ 6:50 pm

Quick how to for finding out the list of processes, including threads spawned by these processes

 ps -eLf | grep USERNAME

Explanation of the options

  • e : Select all processes
  • L : Show threads
  • f : Extra full format

April 5, 2012

HOW TO : Log all commands issued in shell to syslog

Filed under: HOWTO,Linux — Vinay @ 6:41 pm

Inspired from this blog post by Vaidas Jablonskis.  This tip has been tested on Redhat and Centos distributions.

If you ever wanted to log all the commands issued by users on a server, you can edit the default profile configuration to enable this

  • Edit /etc/bashrc file and add the following at the end of the file 
    PROMPT_COMMAND='history -a >(logger -t "$USER[$$] $SSH_CONNECTION")'
  • Log out and log back into your session
  • Now all your commands are logged in the default log file (/var/log/messages)

April 2, 2012

Project Uptime : Progress Report 5 : Getting ready for Reddit and Hacker News

Filed under: Databases,HOWTO,Linux,Technology,Web — Vinay @ 9:03 pm

A very timely post on Hacker News by Ewan Leith about configuring a low end server to take ~11million hits/per month gave me some more ideas on optimizing the performance of this website. Ewan used a combination of nginx and varnish to get the server to respond to such traffic.

From my earlier post, you might recall, that I planned on checking out nginx as the web server, but then ended up using Apache. My earlier stack looked like this Based on the recommendations from Ewan’s article, I decided to add Varnish to the picture. So here is how the stack looks currently

And boy, did the performance improve or what. Here are some before and after performance charts based on a test run from blitz.io. The test lasted for 60 seconds and was for 250 simultaneous connections.

BEFORE

  • Screenshot of Response times and hit rates. Note that the server essentially stopped responding 25 minutes into the test.
  • Screenshot of the analysis summary. 84% error rate!!

AFTER

  • Screenshot of response times and hit rates
  • Screenshot of summary of Analysis. 99.98% success rate!!

 

What a difference!!.. The server in fact stopped responding after the first test and had to be hard rebooted.  So how did I achieve it? By mostly copying the ideas from Ewan :) . The final configuration for serving the web pages looks like this on the server end

Varnish (listens on TCP 80) –> Apache (listens on TCP 8080)

NOTE : All the configuration guides (as with the previous entries of the posts in this series) are specific to Ubuntu.

  1. Configure Apache to listen on port 8080
    1. Stop Apache 
       sudo service apache2 stop
    2. Edit the following files to change the default port from 80 to 8080
      1. /etc/apache2/ports.conf
        1. Change 
          NameVirtualHost *:80
Listen 80
        2. to 
          NameVirtualHost *:8080
Listen 8080
      2. /etc/apache2/sites-available/default.conf (NOTE: This is the default sample site that comes with the package. You can create a new one for your site.  If you do so, you need to edit your site specific conf file)
        1. Change 
           <VirtualHost *:80>
        2. To 
          <VirtualHost *:8080>
    3. Restart apache and ensure that it is listening on port 8080 by using this trick.
  2. Install Varnish and configure it to listen on port 80
    1. Add the Varnish repository to the system and install the package 
      sudo curl http://repo.varnish-cache.org/debian/GPG-key.txt | apt-key add -
sudo echo "deb http://repo.varnish-cache.org/ubuntu/ lucid varnish-3.0" >> /etc/apt/sources.list
sudo apt-get update
sudo apt-get install varnish
    2. Configure Varnish to listen on port 80 and use 64Mb of RAM for caching. (NOTE: Varnish uses port 8080 to get to the backend, in this case Apache, by default. So there is no need to configure it specifically).
      1. Edit the file /etc/default/varnish
        1. Change 
          DAEMON_OPTS="-a :6081 \
-T localhost:6082 \
-f /etc/varnish/default.vcl \
-S /etc/varnish/secret \
-s malloc,256m"
        2. To 
           DAEMON_OPTS="-a :80 \
-T localhost:6082 \
-f /etc/varnish/default.vcl \
-S /etc/varnish/secret \
-s malloc,64m"
    3. Restart Varnish 
      sudo service varnish restart

      and you are ready to rock and roll.

There are some issues with this setup in terms of logging. Unlike your typical web server logs, where every request is logged, I noticed that not all the requests were being logged. I guess, that is because varnish is serving the content from cache. I have to figure out how to get that working. But that is for another post :) .

March 30, 2012

HOW TO : List files that don’t contain a string using find and grep

Filed under: HOWTO,Linux — Vinay @ 3:32 pm

If you run into a situation, where you need to search through a bunch of files and print the names of the files that don’t contain a particular string, here is how you do it in Linux

find -name PATTERN_FOR_FILE_NAMES | xargs grep -L STRING_YOU_ARE_SEARCHING_FOR

The -L option for grep does this (according to the manual)

Suppress normal output; instead print the name of each input file from which no output would normally have been printed.  The scanning will stop on the first match.

HOW TO : Capture all traffic to and from a host using tcpdump

Filed under: HOWTO,Linux,Networking — Vinay @ 10:59 am

Quick one liner for capturing traffic destined to and arriving from a host (IP address) using tcpdump and writing it to a file for analyzing later on

tcpdump -s0 host x.x.x.x -w destination.pcap

March 29, 2012

Project Uptime : Progress Report – 4

Filed under: HOWTO,Linux,security,Technology — Vinay @ 12:26 am

Continuing to lock down the server as part of project uptime a bit more.. I highly recommend enabling and using iptables on every Linux server. I want to restrict inbound traffic to the server to only SSH (tcp port 22) and HTTP(S) (tcp port 80/443). Here’s the process

Check the current rules on the server

sudo iptables -L

Add rules to allow SSH, HTTP and HTTPS traffic and all traffic from the loopback interface

sudo iptables -I INPUT -i lo -j ACCEPT
sudo iptables -A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
sudo iptables -A INPUT -p tcp --dport ssh -j ACCEPT
sudo iptables -A INPUT -p tcp --dport http -j ACCEPT
sudo iptables -A INPUT -p tcp --dport https -j ACCEPT

Drop any traffic that doesn’t match the above mentioned criteria

sudo iptables -A INPUT -j DROP

save the config and create script for the rules to survive reboots by running

sudo su -
iptables-save > /etc/firewall.rules

now create a simple script that will load these rules during startup. Ubuntu provides a pretty neat way to do this. You can write a simple script and place it in /etc/network/if-pre-up.d and the system will execute this before bringing up the interfaces. You can get pretty fancy with this, but here is a simple scrip that I use

samurai@samurai:/etc/network/if-pre-up.d$ cat startfirewall
#!/bin/bash

# Import iptables rules if the rules file exists

if [ -f /etc/firewall.rules ]; then
iptables-restore </etc/firewall.rules
fi

exit 0

Now you can reboot the server and check if your firewall rules are still in effect by running

sudo iptables -L
Older Posts »

Powered by WordPress