March 2012

Stay Hungry.. Stay Foolish

Just read a really nice article on the Harvard Business Review by Walter Isaacson on leadership lessons of Steve Jobs at http://hbr.org/2012/04/the-real-leadership-lessons-of-steve-jobs/ar/pr .  There were a lot of points that Walter makes in the article, but the one that resonated a lot with me was the need to “Stay Hungy.. Stay Foolish“. This is similar to having a “hacker mentality”, which I believe is one of the key ingredient for a person to be successful (professionally and personally).

Being a hacker means..

  • being inquisitive about how things work.
  • always being a student
  • being able to challenge “this is how we always did things”
  • never stop perfecting
  • remembering that one can (and used to) survive on ramen  noodles 🙂

Project Uptime : Progress Report – 2

After I installed the new kernel as mentioned in this update, I was still seeing the server booting up with the older version. I tried to have the kernel install option (apt-get install linux-image-VERSION) to overwrite the grub config, but the server wouldn’t boot up after that.

So after a lot of head scratching and googling, I found the solution. Rather than have the install program automatically update the grub config file, you have to manually edit it.

So install the latest kernel using the commands as mentioned in my previous post (https://kudithipudi.org/2012/03/07/project-uptime-progress-report-1/) and when prompted about the having an existing grub menu and if you want to overwrite it, just say no. Then do the following

  • Check the new kernel image filename by running [code] ll /boot/vmlinuz* [/code]
  • Edit the grub boot menu by editing the file /boot/grub/menu.1st and add a section for the new kernel image.
  • Edit the default boot option to the new kernel image (NOTE.. the sequence starts from 0)
  • Reboot the server and enjoy the new kernel

Here’s my before and after comparisons for the server I am building

BEFORE  (this is after the line ## End Default Options ##)

[code]

title Ubuntu 11.10, kernel 3.0.0-12-virtual
root (hd0)
kernel /boot/vmlinuz-3.0.0-12-virtual root=/dev/xvda1 console=hvc0 ro quiet splash
initrd /boot/initrd.img-3.0.0-12-virtual

title Ubuntu 11.10, kernel 3.0.0-12-virtual (recovery mode)
root (hd0)
kernel /boot/vmlinuz-3.0.0-12-virtual root=/dev/xvda1 console=hvc0 ro single
initrd /boot/initrd.img-3.0.0-12-virtual

title Ubuntu 11.10, kernel 3.0.0-16-virtual
root (hd0)
kernel /boot/vmlinuz-3.0.0-16-virtual root=/dev/xvda1 console=hvc0 ro quiet splash
initrd /boot/initrd.img-3.0.0-16-virtual
title Chainload into GRUB 2
root (hd0)
kernel /boot/grub/core.img

title Ubuntu 11.10, memtest86+
root (hd0)

[/code]

AFTER

[code]

title Ubuntu 11.10, kernel 3.0.0-12-virtual
root (hd0)
kernel /boot/vmlinuz-3.0.0-12-virtual root=/dev/xvda1 console=hvc0 ro quiet splash
initrd /boot/initrd.img-3.0.0-12-virtual

title Ubuntu 11.10, kernel 3.0.0-12-virtual (recovery mode)
root (hd0)
kernel /boot/vmlinuz-3.0.0-12-virtual root=/dev/xvda1 console=hvc0 ro single
initrd /boot/initrd.img-3.0.0-12-virtual

title Ubuntu 11.10, kernel 3.0.0-16-virtual
root (hd0)
kernel /boot/vmlinuz-3.0.0-16-virtual root=/dev/xvda1 console=hvc0 ro quiet splash
initrd /boot/initrd.img-3.0.0-16-virtual
title Chainload into GRUB 2
root (hd0)
kernel /boot/grub/core.img

title Ubuntu 11.10, memtest86+
root (hd0)

[/code]

BTW.. Grub is the boot manager (and a lot more) in Linux.

Love kids..

We spent the weekend with Sri and family and I got a chance to capture some nice shots of Sindhu and Virat. Kids are such a pleasure to take pictures of.. innocent, dont_care_how_I_look..giggly 🙂

Sindhu gazing at Virat

Virat loving the attention
Do you want my balloons?

Are you looking at me?

One happy family.. aren’t they beautiful??

HOW TO : Increasing number of processes that can be run by a user in Linux

By default, most of the Linux distros limit the number of processes that a user can spawn. This is put in place to limit (un)intended cases when a process might just fork off processes without a limit and bring down a server.

For RHEL (and CentOS), the default is 1024 processes per user. In some cases, you do need to increase the number of processes that a particular user can spawn. For example if you are running a database or an application server, you definitely want to tweak this number because these apps tend to create a lot of threads.

As a side note, if you run into this limitation on a machine running jboss, you typically see an error with the following string in your server logs [code]java.lang.OutOfMemoryError: unable to create new native thread.[/code]

. Looking at the error, one would think it is related to memory issues :).

OK.. back to the subject at hand. Here is the process for identifying your limits and then tweaking them as required in RHEL or CentOS.

  • Check the current limits on the number of processes a user can run by executing [code]ulimit -u[/code]
  • Edit the /etc/security/limits.conf file and add the required limits. You can get all the possible options by running man limits.conf. For example, if I wanted all the users to have a soft limit of 2000 and a hard limit of 4000, my limits.conf file wold look like this [code]# Increase the number of threads per process
    *       soft    nproc   200
    *       hard    nproc   4000 [/code]
  • Edit the /etc/security/limits.d/90-nproc.conf file and update it to have the same soft limits. By default, it has 1024 as the limit. So an updated file with my new limits as in the example above would look like this [code]
    # Default limit for number of user’s processes to prevent
    # accidental fork bombs.
    # See rhbz #432903 for reasoning.

    *          soft    nproc     2000[/code]

  • Restart the server. The updated settings won’t take affect until this is done
  • Check if you have the new limits by running [code]ulimit -u[/code]

You can also check the limits of a particular user by finding a process ID being executed by that user and running [code]sudo cat /proc/PROCESS_ID/limits [/code]

HOW TO : Configure Jboss for writing web access logs

One of the capabilities of Jboss is that it can serve HTTP traffic. By default Jboss does not log any of the HTTP traffic in it’s log files. Here is a quick howto on enabling this logging. This post is specific to Jboss 4.x (ancient!!) and I will post another one soon on how do it in version 5.x and newer.

Edit the server.xml file located in $JBOSS_HOME/servers/$PROFILE/deploy/jboss-web.deployer and replace the commented out access logger section as such

FROM

[code]<!–
<Valve className="org.apache.catalina.valves.AccessLogValve"
prefix="localhost_access_log." suffix=".log"
pattern="common" directory="${jboss.server.log.dir}"
resolveHosts="false" />
–> [/code]

TO

[code]<Valve className="org.apache.catalina.valves.AccessLogValve"
prefix="localhost_access_log." suffix=".log"
pattern="common" directory="${jboss.server.log.dir}"
resolveHosts="false" /> [/code]

This will start creating a file with the format localhost_access_log.CURRENT_DATE.log in the $JBOSS_HOME/server/$PROFILE/log folder

But it isn’t fun if you just leave the default logging right :). The pattern formats of common and combined are similar to the standard apache logging options. But if you wanted to have certain content and format in the log files, you have a lot of options. Jboss community has documented all the data that is exposed through this valve at http://docs.jboss.org/jbossweb/latest/api/org/apache/catalina/valves/AccessLogValve.html

So say, I want to log the referrer header, user agent and the value of a cookie called JSESSONID and log all this data into a file called jboss_web_access_log, I setup the options as such

[code]<Valve className="org.apache.catalina.valves.AccessLogValve"
prefix="jboss_web_access_log." suffix=".log"
pattern="%h %p %l %u %t %r %s %b ‘%{Referer}i’ ‘%{User-Agent}i’ ‘%{JSESSIONID}c’"
directory="${jboss.server.log.dir}"
resolveHosts="false" /> [/code]