Quick how to on capturing HTTP headers using tcpdump on a web server (running Linux).
tcpdump -s 1024 -C 1024000 -w /tmp/httpcapture dst port 80
Cache servers depend on cache control headers provided by the web server. Essentially, the web server (based on the configuration) specify’s what content is cache-able and for how long. (Note: Some of the cache servers might ignore this and have a default cache period for specific content. But that is not for another post 
)
Here is a quick and dirty way to configure Apache 2.x server to enable cache control settings on all content in a directory
ExpiresActive On
<Directory "/var/www/html/static">
Options FollowSymLinks MultiViews
Order allow,deny
Allow from all
ExpiresDefault "modification plus 1 hour"
</Directory>
This configuration tells apache to enable cache headers for all content in the /var/www/html/static folder. The cache expiration is set to expire 1 hour from the modification time of the content.
I attended a webinar hosted by Deepak Singh from Amazon’s Web Service group on analytics in the cloud. He made a very compelling case for utilizing the cloud to build out your analytics infrastructure. Esp with the growing data sizes that we deal with now, I think it makes absolute sense. You can utilize different software stacks and grow (and shrink) your hardware stack as required. Great stuff..
But there is a catch. Most of the data generated by current organizations is “inside” their perimeters. Whether it is the OLAP database collecting all your data or that application that spews gigabytes of logs, most of the data is housed in your infrastructure. So if you want to use the cloud to perform analytics on this data, you have to first transfer this data to the cloud. And therein lies the problem. As Deepak mentioned in the webinar, human beings have to yet conquer the limitations of physics . You have to have a pretty big pipe to the Internet to just transfer this data.
Amazon has come up with various means to help with this issue. They are creating copies of publicly available data sets within their cloud so that customers don’t have to transfer them. They are also working with companies to keep private data sets in the cloud for other customers to use. So similar to how you would be able to spin up a Redhat AMI, by paying some license fee to Redhat, I believe they are looking at providing customers access to this private data sets by paying some fee to the company providing this data set. It is a win-win-win situation for Amazon, the company providing the private data set and Amazon’s web services customers. They also support a one time import of data from physical disk or tape.
Coming back to the title of this post . I think this field is still in it’s infancy. Once companies start migrating their infrastructure to the cloud (And yes, it will happen. It is only a matter of time .), it will be a lot easier to leverage the cloud to perform your analytics. All your data will be in the cloud and you start leveraging the hardware and software stacks in the cloud.
LinkedIn (professional networking site) is providing a way to map your networks to see where you have your strongest connections. Here is a map of my networks. You can click on the image to get to the live map.
My strongest connections so far are at
I wish they came up with a map showing the location of my network too. That way, I can find out if I can get a job in New Zealand through my network .
This HOW TO is more for my notes. We had a request at work, where we had to parse some log files and create a graph from the data in the log files.
The log files looked like this
0m0.107s 0m0.022s 0m0.015s 2011-01-05_02_22 0m0.102s 0m0.024s 0m0.014s 2011-01-05_02_23
I wrote the following perl script to get the log file to look as such
| 0m0.107s| 0m0.022s| 0m0.015s| 2011-01-05 | 02:22 | 0m0.102s| 0m0.024s| 0m0.014s| 2011-01-05 | 02:23
perl script
#!/usr/bin/perl
# Modules to load
# use strict;
use warnings;
# Variables
my $inputFile = 'input.txt';
my $version = 0.1;
my $logFile = 'parsed_input.csv';
# Sub Functions
sub Log($$$);
sub Trim($);
# Clear the screen
system $^O eq 'MSWin32' ? 'cls' : 'clear';
# Open the output log file
open(LOGFILE,"> $logFile") || die "Couldn't open $logFile, exiting $!\n";
# Open the input file
open(INPUTFILE,"< $inputFile") || die "Couldn't open $inputFile, exiting $!\n";
# Process the input file, one line at a time
while (defined ($line = <INPUTFILE>)) {
chomp $line;
# Check for blank line
if ($line =~ /^$/)
{
# Start a new line in the output
print LOGFILE "\n";
}
else
{
# Split the date and time
if ($line =~ /2011/)
{
@date = split (/_/,$line);
print LOGFILE "| $date[0] | $date[1]:$date[2]";
}
else
{
# Write the value to the output
print LOGFILE "| $line";
}
}
}
I then took the parsed log files and imported them into the cloud based reporting engine provided by Zoho at http://reports.zoho.com
The final result are these reports
Did I say, I love technology?
I don’t have a Facebook or twitter account and that surprises a lot of my friends since I am such a geek . And the reason I keep (kept) giving was that I want to be in control of my destiny. In this case, destiny being content. While Facebook and Twitter provide you with a easy way to connect with friends/relatives/stalkers etc, I believe it gives a lot of leeway on control over the content for the companies running these applications. I have all the means and ways to communicate with my friends and advertise what I need to world. How I do that? That is a blog post that I have been “drafting” for the last couple of months ..Hope to publish it soon. And looks like the wider audience is finally waking up to it.
Check out this article on ReadWriteWeb regd how the tech leaders are calling for a boycott of Facebook and advocating for an open social networking protocol (http://www.readwriteweb.com/archives/more_web_industry_leaders_quit_facebook_call_for_o.php).
Another article on the same website, speaks about a study by the Advanced Institute of Science and Technology in Korea, which shows that Twitter is really not a social networking site, but more of a medium to broadcast your content (http://www.readwriteweb.com/archives/study_twitter_isnt_very_social.php). Doesn’t really support the argument I made earlier that Twitter is not going to make it..but it certainly supports the notion that once the hype is gone the influence of twitter as a medium will decrease.
I am sure a lot of people shop on express.com , but I probably get the credits for being the first blogger to post that express.com has not been responding to DNS queries since ~7:00 PM CST (4/26). Looks like Qwest is hosting DNS for Express. The name servers (most probably global load balancers) are not responding to DNS requests.
Here’s what I get, when I queried for www.express.com
I feel for the poor ops team scrambling around to bring up the service . Another reason, you want diversity in your DNS hosting.
Instructions for configuring the mime type mappings in Jboss. Mime types essentially tells the application processing the content (typically a browser), what the content is. More information here (http://en.wikipedia.org/wiki/Internet_media_type).
Not the building .. In fact, it just opened up officially today. But it looks like the IT team of the Burj did not anticipate the traffic spike to it’e website http://www.burjdubai.com/, when the building opens. The site has been down since early CST.
All the free publicity the site is getting from the media is wasted because the site is down (OK.. I am exagerrating things a bit ). Â If only the IT team at the Burj thought about this and deployed the site on a CDN, they could have averted this downtime. Using a CDN to power your site is becoming more of a norm than a luxury now a days. And with all the options in the market you have for a CDN, there is no excuse for any IT team to not implement this for a customer facing website.
We run multiple applications in Jboss at my work and one of the applications used to take an inordinate time to come up. A typical application would take < 1 minute to get deployed and this particular application for some reason was taking ~7-8 minutes. We initially thought it was a bug in the code and gave hell to our development team .. But on closer investigation, we found out that a feature we enabled in the Jboss server settings which allows content to be hosted on network storage was causing the issue.
I blogged the feature in Jboss to follow sym links here (http://kudithipudi.org/2008/07/25/howto-configure-jboss-to-follow-symbolic-links/). So essentially when Jboss was started, it was checking all the content in these network path to check for applications to deploy. And traversing a network share with 1000s of directories isn’t fun ..
We fixed it by making a simple edit to the start up script. Here’s the psuedo code for the script
And now the application starts in less than a minute .
I guess there might be other elegant ways to do this. i.e. Configure Jboss to only deploy certain applications, but this did the trick for us .
Powered by WordPress