HOWTO

HOW TO : Use Python to look for credit card numbers

Simple script in python to look for credit card numbers in a file.

[code]

#Importing modules
import re
import os

# Define variables
inputFile = ‘test.txt’
searchPattern = ‘((\D(6011|5[1-5]\d{2}|4\d{3}|3\d{3})\d{11,12}\D)|(^(6011|5[1-5]\d{2}|4\d{3}|3\d{3})\d{11,12}\D))’

tempinputFile = open(inputFile)
tempLine = tempinputFile.readline()

while tempLine:
print ("LINE: " + tempLine)
foundContent = re.search(searchPattern,tempLine, re.IGNORECASE)
if foundContent:
print("FOUND: " + foundContent.group())
tempLine = tempinputFile.readline()

tempinputFile.close() [/code]

The script started out as a simple check for any 16 digit numbers that had a non numeric character on either end. But I tweaked it a little bit to look for credit card like numbers using the regex from http://www.regular-expressions.info/creditcard.html. Finally I added an option to match credit card like numbers if the numbers start at the beginning of the line (i.e there is no non-numeric number before the credit card number)

HOW TO : Use word boundaries in Regular Expressions

If you are every looking to match strings as a whole word instead of literal strings using a regular express, Word Boundaries are your friend

For example if you are looking to match a string of 10 numbers in a log file, you can use

[code]grep -i ‘\b[0-9]\{10,\}\b’ –color -H -n FILE_NAME_TO_SEARCH [/code]

The \b is the option that tells grep to look for a word character.

More information about the option is available at http://www.regular-expressions.info/wordboundaries.html

HOW TO : Use screen to multitask

Want your terminal to look like this? 

In addition to the coolness, factor it also helps you do (monitor) multiple things at  a time. In this screenshot

  • I am checking the resource utilization on my web server
  • Tailing the web server logs to look for errors
  • have a small console for me to run any commands

You can achieve this by using the nifty screen utility. Screen allows you to multiplex between multiple consoles.  So you can open one terminal and have multiple consoles on it. The commands for using screen are a bit hard to get used to. Here are the shortcuts I used to achieve the screen above

  1. Install the screen package
  2. Create a new screen session by running[code] screen [/code]
  3. Add a new screen console by executing[code] ctrl + a [/code]

    [code]c[/code]

  4. Split the screen by executing[code]ctrl + a [/code]

    [code] Shift + s [/code]

  5. Name the different consoled by executing[code]ctrl + a[/code]

    [code]Shift + a[/code]

As you might have figured out by now, “ctrl + a” puts you into screen command mode. You can get a list of all available options by executing

[code]ctrl +a[/code]

 

[code]?[/code]

Here’s a quick reference guide that has more details http://aperiodic.net/screen/quick_reference 

Have fun multitasking 🙂

Project PaaS : Day 2 on Google App Engine

It looks like I was able to accomplish writing the application that I wanted to on the App Engine in 2 days!!  at least in it’s basic form.  After some help from Google, I updated the application I created yesterday (http://samurai-apps.appspot.com/) to display the User Agent string being sent by the client.

The code has been updated to github at https://github.com/kudithipudi/google-app-engine/

Lessons from day 2?

  • Python doesn’t like tabs :). Always use spaces to ident. I was using Notepad++ as the editor and it automatically puts tabs when you hit enter. Why? Looks like that is the best practice according to this style guide (http://www.python.org/dev/peps/pep-0008/)
  • The “Logs” console in the SDK toolkit should be your best friend. It let’s you know if there is any error in your code and what line it believes the error is at.

Next, I will try to pretty it up a bit.

Isn’t it amazing that I was able to create a simple app in a matter of 2 days and host it on an “infinitely” scalable  platform without even taking our my credit card.

HOW TO : Configure Jboss to not show backend server name when proxying https (ssl) traffic

Phew.. that was a long title :).  Was running into an issue with the setup shown in the picture below

When we try to access the web site using https, the html content being served back was showing the app server name as the reference, rather than the web site.

So in this example, let’s say the web address was kudithipudi.org and the app server was app-server-kudithipudi, the HTML content was showing https://app-server-kudithipudi:8080 as the source.

Here’s how, we fixed it.

Edit the server.xml file found in $JBOSS_HOME/server/$JBOSS_PROFILE/deploy/jboss-web.deployer and update the HTTPS connector to use the web address (kudithipudi.org) as the proxyName.

BEFORE

[code]<Connector port="8443" protocol="HTTP/1.1" SSLEnabled="true"
maxThreads="250" scheme="https" secure="true"
clientAuth="false" sslProtocol="TLS"
keystoreFile="/opt/jboss/jboss-as/server/kudithipudi/conf/ssl/kudithipudi.keystore"
keystorePass="xxxxxx" />
[/code]

AFTER

[code]<Connector port="8443" protocol="HTTP/1.1" SSLEnabled="true"
maxThreads="250" scheme="https" secure="true"
clientAuth="false" sslProtocol="TLS"
proxyName="kudithipudi.org" proxyPort="443"
keystoreFile="/opt/jboss/jboss-as/server/kudithipudi/conf/ssl/kudithipudi.keystore"
keystorePass="xxxxxx" />

[/code]

HOW TO : grep options to display before and after lines of matching content

For my own notes.. if you are using grep to parse through the contents of a file and want to see the preceding or proceeding content than the line that matched your query, you can use the following options

preceding content [code]grep -B NUMBER_OF_LINES_TO_DISPLAY query filename[/code]

for example, if I was searching for kudithipudi in a file names access.log and want to see 2 lines prior to the match, I would use [code]grep -B 2 kudithipudi access.log[/code]

proceeding content[code]grep -A NUMBER_OF_LINES_TO_DISPLAY query filename[/code]

for example, if I was searching for kudithipudi in a file names access.log and want to see 2 lines after the match, I would use [code]grep -A 2 kudithipudi access.log[/code]

preceding and proceeding content[code]grep -C NUMBER_OF_LINES_TO_DISPLAY query filename[/code]

for example, if I was searching for kudithipudi in a file names access.log and want to see 2 lines before and after the match, I would use [code]grep -C 2 kudithipudi access.log[/code]

HOW TO : Use templates in puppet to pass hostnames

puppet, is a configuration management framework that can be used to perform several different things to validate/configure your infrastructure. We have been using puppet for sometime at my work and have just started moving into some of the advanced uses of the tool.

One of the features offered by puppet is the capability to use templates to configure different servers.

For example, say you want to configure an application on server ABCD, XYZ and 123. And the configuration file for all these servers is the same, other than the hostname of the server. The configuration file has to reside in /opt/application/config.conf . The config.xml file looks like this

[code]

db.name=blah
db.user=blahblah
db.hostname=XYZ
log.level=ERROR
log.location=/var/log/application

[/code]

Here is how you can do it in puppet.

Define a module which uses a template and then configure the template to put the host specific entry in the template. Let’s name our module test_config

  • Create the module
    • cd $PUPPET_HOME/modules
    • mkdir test_config/{files,manifests,templates}
  • Create the template
    • cd templates
    • vi config.conf.template and add the following to the file[code]db.name=blah
      db.user=blahblah
      db.hostname=<%= fqdn %>
      log.level=ERROR
      log.location=/var/log/application [/code]
      • note : see how I replaced the hostname XYZ, which was specific to one server with <%= fqdn %>. This is one of the “facts” provided by puppet. you can get a list of all the facts by running facter on any of the puppet clients.
  • Configure the module to use the template. In this case, we want the module to place the file config.conf in /opt/application
    • cd manifests
    • vi init.pp and add the following to the file[code]class test_config {
      file { "/opt/application/config.conf":
      ensure => present,
      owner => appuser,
      group => appuser,
      mode => 755,
      content => template("test_config/config.conf.template"),
      }
      }[/code]
      • note : There are several other options you can use for the class file.. I just gave an example of some of the common ones. Like setting the owner, group and the rights.
  • Finally configure the clients to use the module. In the individual node config files, include the module you just created. Here is how the config for node ABCD would look like[code]node ABCD {
    include test_config
    }[/code]

The next time the puppet client runs on host ABCD, it would create the file /opt/application/config.conf with the right hostname in the config file.

HOW TO : Configure Jboss to append log files instead of overwriting them

If you use the default logging options for Jboss, it has a nasty habit of overwriting log files on a restart. So, if you were in the middle of troubleshooting an issue and had to restart Jboss, you will end up loosing all the historic data. You can change this default behavior by changing one option in the log4j config file

  • Edit the $JBOSS_HOME/server/$JBOSS_PROFILE/conf/jboss-log4j.xml and replace [code]<param name="Append" value="false"/>[/code]

    with [code]<param name="Append" value="true"/>[/code]

  • You don’t even have to restart Jboss for this new setting to take place, since Jboss reads the log4j config every 60 seconds and updates the logging parameters accordingly.

Project Uptime : Recap

Final post on Project Uptime. Before I go into the details of how well I fared against the original goals, here is a screenshot of the uptime of the site for the last two weeks.. Hope I can keep that number for the rest of this year :). Recap of the original goals and their status

GOAL : Move to a fresh VM with the latest kernel

  • STATUS : This was successfully completed. Details of the install are here and here.

GOAL : Upgrade to the latest version of Apache.

  • STATUS : Although I didn’t install 2.4 version of Apache as planned, I was able to get similar (or better) performance by installing varnish as a caching engine. Details of thee apache/varnish install are here.

GOAL : Upgrade to latest version of MySQL and tune it for memory usage

  • STATUS : This was probably the easiest part of the project. Details of the install are here. I didn’t necessarily tweak it for low memory usage though.

GOAL : Configure cloudflare to serve a static version of front page, in case the server goes down. Design the static page to point people to my other digital presences (Google+, LinkedIn, Flickr etc)

  • STATUS : Cloudflar doesn’t have the capability to direct static pages if the origin server is down. At least not for the free tier. I also didn’t design a simple page to host my digital presence. Will try to find a good template for it down the road.

All in all.. not bad 🙂