Instead of posting individual posts every day to track my progress (or lack of) in the 60 day challenge, I created this page (https://kudithipudi.org/2012-60-day-resolution/) for the daily entries..
Wish me luck :).
admin
Viva la resistance
I have a confession to make.. I like Big Macs and Krispy Kreme Donuts :). And they have contributed heavily to the increase in my .. hmm.. how do I say this.. mid section :). Plus, it doesn’t help that there is a Krispy Kreme factory and a McDonald’s right on my way to work. And add on to the fact that I haven’t been running for the last year or so, I am proud to say that I have joined the >65% of Americans that are obese.
On my way to work yesterday, I was thinking about what shape (physically) I would be in when Virat grows up. I am sure he doesn’t want to have a dad that can’t play some hoops with him :).
So here’s my 2 month resolution. I am starting with a couple of months because there is a good chance that it might become a habit and then go from there :).
- Exercise for 30 minutes a day (7 days a week)
- Eat dessert only once a week
- No krispy kreme
- No Big Mac
- No fries
- No pop
For every pledge I break, I am going to leave work at 5:00 PM for a week. Believe me when I say that is a tough punishment :). You see.. I love what I do :).
Viva La Resistance!!!
HOW TO : Log all commands issued in shell to syslog
Inspired from this blog post by Vaidas Jablonskis. This tip has been tested on Redhat and Centos distributions.
If you ever wanted to log all the commands issued by users on a server, you can edit the default profile configuration to enable this
- Edit /etc/bashrc file and add the following at the end of the file[code]PROMPT_COMMAND=’history -a >(logger -t "$USER[$$] $SSH_CONNECTION")’ [/code]
- Log out and log back into your session
- Now all your commands are logged in the default log file (/var/log/messages)
HOW TO : Configure Jboss to send log messages to syslog
Jboss uses the log4j framework for providing logging services. log4j is a very flexible framework and can do a lot of things. One of the features provided by log4j is to send log messages to multiple destinations. Here is a quick how to on configuring Jboss to send log messages using the syslog protocol to a syslog server. This is pretty useful, when you are trying to consolidate logs from multiple sources into a central location.
First, some background about how log4j is configured in Jboss
The log4j configuration in Jboss is managed by the file jboss-log4j.xml located at $JBOSS_HOME/server/$JBOSS_PROFILE/conf.
There are three parts to this configuration file
- Appenders
- An appender is a way to define a particular logging method. By default, Jboss provides a bunch of appenders in this config file, but only the FILE and CONSOLE appenders are enabled. The FILE appender writes the log messages to a log file and rotates them based on the criteria in the appender. The CONSOLE appender just sends messages to the console. This will come into picture, when you are not running Jboss as a service. In addition, there are appenders for syslog, snmp, email that are commented out.
- Categories
- A category is where you define the class you want to log messages for and which appender it should use. If you don’t specify an appender or the threshold for the logging level, logging for this class will be done at the default log levels and by the appender specified by the default (root) category.
- Default (root) Category
- As mentioned above, this is the catch all for classes that are not specified specifically in the categories section.
So pictorially, it would look like this 
Getting back to the reason for this post, here is how you would enable the syslog appender and then configure a category to use this appender. For this example, we will use a class names org.kudithipudi
- Enable the syslog appender by un-commenting the following section in the jboss-log4j.xml file[code] <!– Syslog events –>
<appender name="SYSLOG">
<errorHandler/>
<param name="Threshold" value="ERROR"/>
<param name="Facility" value="LOCAL7"/>
<param name="FacilityPrinting" value="true"/>
<param name="SyslogHost" value="localhost"/>
<layout>
<param name="ConversionPattern" value="[%d{ABSOLUTE},%c{1}] %m%n"/>
</layout>
</appender>
[/code] - Add a new category to use this appender [code] <category name="org.kudithipudi">
<priority value="INFO" />
<appender-ref ref="SYSLOG"/>
</category> [/code] - Restart Jboss and you should see messages from Jboss being sent to the syslog server
Couple of notes..
- Even though we are specifying the threshold of INFO in the category, because we specified a threshold of ERROR in the appender, only message of ERROR type will be sent to the syslog server. This is actually pretty useful when you want to specify two appenders to a category and log them at different levels. You can set another appender to INFO level and add it to this category. And in essence, the appender will log everything of INFO and higher, while the syslog appender will only process ERROR messages.
- The destination for the syslog messages is the SysLogHost parameter. In this example, I just used localhost.
Never assume.
When troubleshooting performance issues..never take anything for granted..yes, even if something was not touched or restarted, chances are something touching it has been and might have affected it.
This goes esp for the network (IP and fiber) which don’t change as often as the rest of the environment.
Project Uptime : Progress Report 5 : Getting ready for Reddit and Hacker News
A very timely post on Hacker News by Ewan Leith about configuring a low end server to take ~11million hits/per month gave me some more ideas on optimizing the performance of this website. Ewan used a combination of nginx and varnish to get the server to respond to such traffic.
From my earlier post, you might recall, that I planned on checking out nginx as the web server, but then ended up using Apache. My earlier stack looked like this 
Based on the recommendations from Ewan’s article, I decided to add Varnish to the picture. So here is how the stack looks currently
And boy, did the performance improve or what. Here are some before and after performance charts based on a test run from blitz.io. The test lasted for 60 seconds and was for 250 simultaneous connections.
BEFORE
- Screenshot of Response times and hit rates. Note that the server essentially stopped responding 25 minutes into the test.
- Screenshot of the analysis summary. 84% error rate!!
AFTER
- Screenshot of response times and hit rates
- Screenshot of summary of Analysis. 99.98% success rate!!
What a difference!!.. The server in fact stopped responding after the first test and had to be hard rebooted. So how did I achieve it? By mostly copying the ideas from Ewan :). The final configuration for serving the web pages looks like this on the server end
Varnish (listens on TCP 80) –> Apache (listens on TCP 8080)
NOTE : All the configuration guides (as with the previous entries of the posts in this series) are specific to Ubuntu.
- Configure Apache to listen on port 8080
- Stop Apache [code] sudo service apache2 stop [/code]
- Edit the following files to change the default port from 80 to 8080
- /etc/apache2/ports.conf
- Change [code]NameVirtualHost *:80
Listen 80
[/code] - to [code]NameVirtualHost *:8080
Listen 8080
[/code] - /etc/apache2/sites-available/default.conf (NOTE: This is the default sample site that comes with the package. You can create a new one for your site. If you do so, you need to edit your site specific conf file)
- Change [code] <VirtualHost *:80> [/code]
- To [code]<VirtualHost *:8080> [/code]
- Restart apache and ensure that it is listening on port 8080 by using this trick.
- Install Varnish and configure it to listen on port 80
- Add the Varnish repository to the system and install the package[code]sudo curl http://repo.varnish-cache.org/debian/GPG-key.txt | apt-key add –
sudo echo "deb http://repo.varnish-cache.org/ubuntu/ lucid varnish-3.0" >> /etc/apt/sources.list
sudo apt-get update
sudo apt-get install varnish
[/code] - Configure Varnish to listen on port 80 and use 64Mb of RAM for caching. (NOTE: Varnish uses port 8080 to get to the backend, in this case Apache, by default. So there is no need to configure it specifically).
- Edit the file /etc/default/varnish
- Change [code]DAEMON_OPTS="-a :6081 \
-T localhost:6082 \
-f /etc/varnish/default.vcl \
-S /etc/varnish/secret \
-s malloc,256m"
[/code] - To [code] DAEMON_OPTS="-a :80 \
-T localhost:6082 \
-f /etc/varnish/default.vcl \
-S /etc/varnish/secret \
-s malloc,64m"
[/code] - Restart Varnish [code]sudo service varnish restart[/code]
and you are ready to rock and roll.
There are some issues with this setup in terms of logging. Unlike your typical web server logs, where every request is logged, I noticed that not all the requests were being logged. I guess, that is because varnish is serving the content from cache. I have to figure out how to get that working. But that is for another post :).
31 Days and 29 posts
In early March 2012, I decided to write at least one blog post per day for the whole month. How did I do? 29 posts in 31 days. I should acknowledge that I cheated a bit :), by blogging two posts in a day, but scheduling them to be published in different days. 
My learning from the month long exercise?
- There is truth to the adage “practice makes one perfect” :). The more I wrote, the quicker I was in getting the posts completed. I used to take a couple of weeks to a month in completing a post, but now, I can crank one out in a few minutes.
- I stuck to the “perfect is the enemy of good” principal. Even though I knew that some of the posts were not as good as I wanted them to be, I kept posting them and then editing them later on.
- More content = more traffic. Even if you don’t write earth shattering articles, there is just more content for the search engines to index you on. I saw an uptick in the traffic to the site in March.
Let’s see, how long I can keep it up.
And no.. this is not an April fools joke :).
HOW TO : Perform OCR on PDF files for free
I had to convert a scanned PDF file into an editable document recently. You can do this using OCR and there is a ton of software out there, that does this. There are even web based services that do this. But each of them had limitations (either had to buy the software or limit in the number of pages that can be scanned). I didn’t want to buy the license, since this is not something I would be doing regularly and the document I had to convert was 61 pages, so none of the online services allowed me to do it. I remembered reading that Google Docs, added this (OCR) capability a while ago and since I have a Google Apps account, I decided to give it a try.
Google also has a limit of 2 pages per OCR conversion. So after some brainstorming, I came up with this quick hack to use Google Docs for converting large PDF files into editable content.
- Split the PDF file into two page documents using PDFsam (Open Source PDF Split and Merge Tool).
- Log into your Google Docs interface at http://docs.google.com . All you need is a Google Account to use this feature
- Create a folder (collection) to organize your files. This is not required, but it will make searching for the files a lot easier
- Check the settings to convert PDF files to editable
- Upload the PDF files you created in step 1.
- As you upload the files, Google creates an editable document with the text from the PDF files. You can then create a new document and copy/paste the content from all the smaller files.
I think someone with more programming chops than me can improve this by using the Google API to do the copy/paste from the smaller docs into the final document :).
HOW TO : List files that don't contain a string using find and grep
If you run into a situation, where you need to search through a bunch of files and print the names of the files that don’t contain a particular string, here is how you do it in Linux
[code]find -name PATTERN_FOR_FILE_NAMES | xargs grep -L STRING_YOU_ARE_SEARCHING_FOR [/code]
The -L option for grep does this (according to the manual)
Suppress normal output; instead print the name of each input file from which no output would normally have been printed. The scanning will stop on the first match.
HOW TO : Capture all traffic to and from a host using tcpdump
Quick one liner for capturing traffic destined to and arriving from a host (IP address) using tcpdump and writing it to a file for analyzing later on
[code]tcpdump -s0 host x.x.x.x -w destination.pcap [/code]