The code has been updated to github at https://github.com/kudithipudi/google-app-engine/

Lessons from day 2?

• Python doesn’t like tabs . Always use spaces to ident. I was using Notepad++ as the editor and it automatically puts tabs when you hit enter. Why? Looks like that is the best practice according to this style guide (http://www.python.org/dev/peps/pep-0008/)
• The “Logs” console in the SDK toolkit should be your best friend. It let’s you know if there is any error in your code and what line it believes the error is at.

Next, I will try to pretty it up a bit.

Isn’t it amazing that I was able to create a simple app in a matter of 2 days and host it on an “infinitely” scalable  platform without even taking our my credit card.

When we try to access the web site using https, the html content being served back was showing the app server name as the reference, rather than the web site.

So in this example, let’s say the web address was kudithipudi.org and the app server was app-server-kudithipudi, the HTML content was showing https://app-server-kudithipudi:8080 as the source.

Here’s how, we fixed it.

Edit the server.xml file found in $JBOSS_HOME/server/$JBOSS_PROFILE/deploy/jboss-web.deployer and update the HTTPS connector to use the web address (kudithipudi.org) as the proxyName.

BEFORE

<Connector port="8443" protocol="HTTP/1.1" SSLEnabled="true"
maxThreads="250" scheme="https" secure="true"
clientAuth="false" sslProtocol="TLS"
keystoreFile="/opt/jboss/jboss-as/server/kudithipudi/conf/ssl/kudithipudi.keystore"
keystorePass="xxxxxx" />

AFTER

<Connector port="8443" protocol="HTTP/1.1" SSLEnabled="true"
maxThreads="250" scheme="https" secure="true"
clientAuth="false" sslProtocol="TLS"
proxyName="kudithipudi.org" proxyPort="443"
keystoreFile="/opt/jboss/jboss-as/server/kudithipudi/conf/ssl/kudithipudi.keystore"
keystorePass="xxxxxx" />

preceding content

grep -B NUMBER_OF_LINES_TO_DISPLAY query filename

for example, if I was searching for kudithipudi in a file names access.log and want to see 2 lines prior to the match, I would use

grep -B 2 kudithipudi access.log

proceeding content

grep -A NUMBER_OF_LINES_TO_DISPLAY query filename

for example, if I was searching for kudithipudi in a file names access.log and want to see 2 lines after the match, I would use

grep -A 2 kudithipudi access.log

preceding and proceeding content

grep -C NUMBER_OF_LINES_TO_DISPLAY query filename

for example, if I was searching for kudithipudi in a file names access.log and want to see 2 lines before and after the match, I would use

grep -C 2 kudithipudi access.log

One of the features offered by puppet is the capability to use templates to configure different servers.

For example, say you want to configure an application on server ABCD, XYZ and 123. And the configuration file for all these servers is the same, other than the hostname of the server. The configuration file has to reside in /opt/application/config.conf . The config.xml file looks like this

db.name=blah
db.user=blahblah
db.hostname=XYZ
log.level=ERROR
log.location=/var/log/application

Here is how you can do it in puppet.

Define a module which uses a template and then configure the template to put the host specific entry in the template. Let’s name our module test_config

• Create the module

• cd $PUPPET_HOME/modules
• mkdir test_config/{files,manifests,templates}
• Create the template
• cd templates
• vi config.conf.template and add the following to the file

  db.name=blah
  db.user=blahblah
  db.hostname=<%= fqdn %>
  log.level=ERROR
  log.location=/var/log/application 

• note : see how I replaced the hostname XYZ, which was specific to one server with <%= fqdn %>. This is one of the “facts” provided by puppet. you can get a list of all the facts by running facter on any of the puppet clients.
• Configure the module to use the template. In this case, we want the module to place the file config.conf in /opt/application
• cd manifests
• vi init.pp and add the following to the file

  class test_config {
  file { "/opt/application/config.conf":
  ensure => present,
  owner => appuser,
  group => appuser,
  mode => 755,
  content => template("test_config/config.conf.template"),
  }
  }

• note : There are several other options you can use for the class file.. I just gave an example of some of the common ones. Like setting the owner, group and the rights.
• Finally configure the clients to use the module. In the individual node config files, include the module you just created. Here is how the config for node ABCD would look like

  node ABCD {
  include test_config
  }

The next time the puppet client runs on host ABCD, it would create the file /opt/application/config.conf with the right hostname in the config file.

• Edit the $JBOSS_HOME/server/$JBOSS_PROFILE/conf/jboss-log4j.xml and replace

  <param name="Append" value="false"/>

  with

  <param name="Append" value="true"/>

• You don’t even have to restart Jboss for this new setting to take place, since Jboss reads the log4j config every 60 seconds and updates the logging parameters accordingly.

GOAL : Move to a fresh VM with the latest kernel

• STATUS : This was successfully completed. Details of the install are here and here.

GOAL : Upgrade to the latest version of Apache.

• STATUS : Although I didn’t install 2.4 version of Apache as planned, I was able to get similar (or better) performance by installing varnish as a caching engine. Details of thee apache/varnish install are here.

GOAL : Upgrade to latest version of MySQL and tune it for memory usage

• STATUS : This was probably the easiest part of the project. Details of the install are here. I didn’t necessarily tweak it for low memory usage though.

GOAL : Configure cloudflare to serve a static version of front page, in case the server goes down. Design the static page to point people to my other digital presences (Google+, LinkedIn, Flickr etc)

• STATUS : Cloudflar doesn’t have the capability to direct static pages if the origin server is down. At least not for the free tier. I also didn’t design a simple page to host my digital presence. Will try to find a good template for it down the road.

All in all.. not bad

1. Configure Apache to have the wordpress folder as the default directory. I did this by changing the DocumentRoot option in the vhost
2. Changed the permissions on the wordpress directories (so that wordpress can make rewrite rule changes on the fly)

sudo chmod -v 664 $WORDPRESS_DIRECTORY/.htaccess

sudo chmod 755 $WORDPRESS_DIRECTORY/wp-content 

Here is how you can enable hugepages and configure jboss (actually any Java app) to use hugepages on a RHEL/CentoOS system.

OS CONFIGURATION

1. Check if your system is capable of supporting hugepages by running

   grep HUGETLB /boot/config-`uname -r`

   If you see the response as below, you should be good

   CONFIG_HUGETLBFS=y
   CONFIG_HUGETLB_PAGE=y
   

• Next check if huge pages are already being used by running

  cat /proc/sys/vm/nr_hugepages 

1. If the response is anything other than 0, that means hugepages have already been configured.

• Find the block size for hugepages by running

  cat /proc/meminfo | grep -i hugepagesize 

• Calculate the amount of memory you want to dedicate to hugepages. (note: memory allocated to hugepages cannot be used by other processes in the system, unless they are configured to use it)

1. For example, I want to dedicate 3GB of RAM for hugepages. So the number of hugepages would be

   (3*1024*1024)/2048

• Configure the number of hugepages on the system by editing the /etc/sysctl.conf and adding the option

  vm.nr_hugepages = 1536

  (note: I put in 1536 since that was the value I got from the above example)

• Restart the server and check if hugepages has been enabled by running

  cat /proc/meminfo | grep -i huge 

1. You should see something like this

   AnonHugePages:    839680 kB
   HugePages_Total:    1500
   HugePages_Free:     1500
   HugePages_Rsvd:        0
   HugePages_Surp:        0
   Hugepagesize:       2048 kB
   

JBOSS CONFIGURATION

1. At this point your system is configured with hugepages and any application that is configured to use them can leverage them.  In this example, we want to configure Jboss to utilize these hugepages
2. Add the groupid of the user that Jboss is running under to the /etc/sysctl.conf file. In my case, the jboss user group had a GID of 505, so I added this line to /etc/sysctl.conf

   vm.hugetlb_shm_group = 505 

3. Next allocate the memory to the user by editing /etc/security/limits.conf and allocating the memory. Again, in my case, I added the following to /etc/security/limits.conf

   # Allocate memory for Jboss user to take advantage of hugepages
   jboss   soft    memlock 1500
   jboss   hard    memlock 1500
   

4. Finally add the following to the Jboss startup parameters. I edited the $JBOSS_HOME/bin/run.sh file. (note: the startup file can be different based on your config) with the option

    -XX:+UseLargePages

5. Restart Jboss and you are good to go

note : A lot articles that I read online say that hugepages are effective when you are allocating large amounts of RAM to the application. The use case of just using 3GB above was just that.. a use case.

While I cannot personally vouch for it, a lot of users have noted that they saw >2 fold increase in performance.

sudo yum install package_to_install --nogpgcheck 

1. Workstation 1
1. create a directory under your dropbox root, that you want to use as your git home directory. Say DROPBOX/git
2. Install Git for Windows, or whatever git client you want to use
3. Change the home path on the git client by executing

   HOME='PATH_TO_DROPBOX/DROPBOX/git' 

4. Check if the home path has been changed by executing

   echo $HOME

5. Create your SSH keys and configure your public key on the git server
2. Workstation 2
1. Repeat and rinse step 1 – 4 specified for workstation 1. You don’t need to create the SSH keys since the other clients will recognize the keys that dropbox would have synced up.