Web

HOW TO : Use curl to check the impact of DNS changes

Ran into an interesting scenario at work today. We had to check the impact of a DNS change on a certain hostname. Normally, you would edit your host file entry to reflect the DNS change and do your testing. Here is another way you can do it using cURL. In this particular example, I am checking the SSL certificate details of the hostname .

[code]curl –insecure –trace-ascii debug.txt https://HOSTNAME:PORT –resolve HOSTNAME:PORT:IP_ADDRESS [/code]

That’s a pretty convoluted command :). Let’s try to break it down

[code]–insecure [/code]

: tells cURL to ignore certificate warnings. This is helpful if you are using self signed certs

[code]–trace-ascii [/code]

: tells cURL to save the SSL connection details (in debug mode) to a file called debug.txt

[code]–resolve [/code]

: tells cURL to use the options mentioned after it to resolve the hostname, rather than using DNS. The format for resolve is <host:port:address>

NOTE: You need to have version 7.21.3 or higher of cURL to use this option

Here’s a real world example. Say, I want to see how the IP address 72.30.38.140 would reacts if www.google.com requests are routed to it

[code]

samurai@samurai:~$ curl –insecure –trace-ascii debug.txt https://www.google.com –resolve www.google.com:443:72.30.38.140
The document has moved <A HREF="http://www.google.com/?s=https">here</A>.<P>
<!– ir2.fp.sp2.yahoo.com uncompressed/chunked Mon Nov 12 22:44:41 UTC 2012 –>
samurai@samurai:~$ more debug.txt
== Info: Added www.google.com:443:72.30.38.140 to DNS cache
== Info: About to connect() to www.google.com port 443 (#0)
== Info: Trying 72.30.38.140… == Info: connected
== Info: Connected to www.google.com (72.30.38.140) port 443 (#0)
== Info: successfully set certificate verify locations:
== Info: CAfile: none
CApath: /etc/ssl/certs
== Info: SSLv3, TLS handshake, Client hello (1):
=> Send SSL data, 223 bytes (0xdf)
0000: ……P.|v..1..kA…….=J.xr.=ft.3.|…Z…..9.8………5…..
0040: …………….3.2…..E.D…../…A………………………
0080: …….W………www.google.com………..4.2……………….
00c0: ………………………….
== Info: SSLv3, TLS handshake, Server hello (2):
<= Recv SSL data, 42 bytes (0x2a)
0000: …&..P.{.I"L….3x..N…9…./<n….A..5.
== Info: SSLv3, TLS handshake, CERT (11):
<= Recv SSL data, 1272 bytes (0x4f8)
0000: ……….0…0..S……….0…*.H……..0N1.0…U….US1.0…
0040: U….Equifax1-0+..U…$Equifax Secure Certificate Authority0…1
0080: 00401230014Z..150703045000Z0..1)0′..U… 2g8aO5wI1bKJ2ZD588UsLvD
00c0: e3gTbg8DU1.0…U….US1.0…U….California1.0…U….Sunnyvale1
0100: .0…U….Yahoo Inc.1.0…U….www.yahoo.com0.."0…*.H……..
0140: …..0……….5.p./……..O…k.C…9E+.J..H.s….Bm.T.E.-..<
0180: ^…m…r.v<\…&Qq..l………. @'(q.m..ZJ.*kt…!.AWU…….M.
01c0: …n…O….0.._…H….4……>.m..K…….Z…:.Df%.lR.!…(!.
0200: .FV.dQ…f.V….P,.J9.c..dM.s>C=….Y..#…47#2…..cP.{….g.rU
0240: .d…P……………..0…0…U………..0…U………….t5.
0280:……U..0:..U…3010/.-.+.)http://crl.geotrust.com/crls/secure
02c0: ca.crl0..[..U…..R0..N..www.yahoo.com..yahoo.com..us.yahoo.com.
0300: .kr.yahoo.com..uk.yahoo.com..ie.yahoo.com..fr.yahoo.com..in.yaho
0340: o.com..ca.yahoo.com..br.yahoo.com..de.yahoo.com..es.yahoo.com..m
0380: x.yahoo.com..it.yahoo.com..sg.yahoo.com..id.yahoo.com..ph.yahoo.
03c0: com..qc.yahoo.com..tw.yahoo.com..hk.yahoo.com..cn.yahoo.com..au.
0400: yahoo.com..ar.yahoo.com..vn.yahoo.com0…U.#..0…H.h.+….G.# .
0440: O3….0…U.%..0…+………+…….0…*.H……………2..0.
0480: S.’.y….GD.Q…=…K+..q..kv…….<h…….ZLE.h$..M2^.C..IT..
04c0: ".5j….Vc7.4……1.Wu.[.a>+………9..{.a:………
== Info: SSLv3, TLS handshake, Server finished (14):
<= Recv SSL data, 4 bytes (0x4)
0000: ….
== Info: SSLv3, TLS handshake, Client key exchange (16):
=> Send SSL data, 262 bytes (0x106)
0000: …….R…..b.,.&.. s.Ob;.E_.EnSw../D…’…..(aB<<……F..]..
0040: o………~…*..r?.C..%..22…J.bu&.x(j|…….>A5..OF.G…C.$.
0080: .9u9n.z…K…..u…..~:W.{Sii.{2..6……..<…..i…8y$y…..6
00c0: …1.(M…fx….#k..r….47..t.q…..A.?.0. .D…..~…G+.,….~
0100: ..=.#y
== Info: SSLv3, TLS change cipher, Client hello (1):
=> Send SSL data, 1 bytes (0x1)
0000: .
== Info: SSLv3, TLS handshake, Finished (20):
=> Send SSL data, 16 bytes (0x10)
0000: ….!9)…6…+.
== Info: SSLv3, TLS change cipher, Client hello (1):
<= Recv SSL data, 1 bytes (0x1)
0000: .
== Info: SSLv3, TLS handshake, Finished (20):
<= Recv SSL data, 16 bytes (0x10)
0000: …..(qN..l.]…
== Info: SSL connection using AES256-SHA
== Info: Server certificate:
== Info: subject: serialNumber=2g8aO5wI1bKJ2ZD588UsLvDe3gTbg8DU; C=US; ST=California; L=Sunnyvale; O=Yahoo Inc.; CN=www.yahoo.com
== Info: start date: 2010-04-01 23:00:14 GMT
== Info: expire date: 2015-07-03 04:50:00 GMT
== Info: subjectAltName does not match www.google.com
=> Send header, 167 bytes (0xa7)
0000: GET / HTTP/1.1
0010: User-Agent: curl/7.21.6 (x86_64-pc-linux-gnu) libcurl/7.21.6 Ope
0050: nSSL/1.0.0e zlib/1.2.3.4 libidn/1.22 librtmp/2.3
0082: Host: www.google.com
0098: Accept: */*
00a5:
<= Recv header, 32 bytes (0x20)
0000: HTTP/1.1 301 Moved Permanently
<= Recv header, 37 bytes (0x25)
0000: Date: Mon, 12 Nov 2012 22:44:41 GMT
<= Recv header, 42 bytes (0x2a)
0000: Location: http://www.google.com/?s=https
<= Recv header, 23 bytes (0x17)
0000: Vary: Accept-Encoding
<= Recv header, 19 bytes (0x13)
0000: Connection: close
<= Recv header, 28 bytes (0x1c)
0000: Transfer-Encoding: chunked
<= Recv header, 40 bytes (0x28)
0000: Content-Type: text/html; charset=utf-8
<= Recv header, 24 bytes (0x18)
0000: Cache-Control: private
<= Recv header, 2 bytes (0x2)
0000:
<= Recv data, 173 bytes (0xad)
0000: 000009d
0009: The document has moved <A HREF="http://www.google.com/?s=https">
0049: here</A>.<P>.<!– ir2.fp.sp2.yahoo.com uncompressed/chunked Mon
0089: Nov 12 22:44:41 UTC 2012 –>.
00a8: 0
00ab:
== Info: Closing connection #0
== Info: SSLv3, TLS alert, Client hello (1):
=> Send SSL data, 2 bytes (0x2)
0000: ..

[/code]

November 12, 2012 by HOWTO Linux security Technology Uncategorized Web 0

Interesting (infrastructure) tidbits about Microsoft Azure

I attended a session organized by aditi regarding Microsoft Azure and Windows 8, called “Go Cloud 8” today. One of the speakers in the event was Deepak Rao, Microsoft’ Director of Cloud Computing. He shared some interesting numbers about the infrastructure running Microsoft Azure

  • 8 carrier grade data centers around the world. “Carrier” grade because of the sheer size of them.
  • The data center in Chicago houses more than 350,000 servers and is supported by only 30 FTEs (which makes me think about the number of contractors they have there 🙂 )
  • 1 in 4 x86 servers produced were bought by Microsoft. Not sure if it was in 2011 or 2012!!

Deepak also gave an real world example of how one of their customers used Azure.

BPro Inc provides software to counties and states for helping report election results. They run their backend on the Azure platform. During normal periods, they run ~10 instances of compute nodes. But during the election day (11/6) this week, BPro spun up 8600 compute nodes in less than 15 minutes at 4:00 PM EST, to help support the load created by the demand for election results and than again shutdown all of them at around 1:00 AM EST when the demand decreased. Using the “list” pricing of $0.12/hr/compute node, that massive increase in capacity cost them ~$8K!!.

That is pretty impressive and I usually don’t use the work impressive in the same sentence as Microsoft 🙂

November 8, 2012 by Technology Uncategorized Web Windows 1

HOW TO : Download SSL certificate using openssl and importing it into a keystore

Following up on my earlier post about using keytool to import and export certificates into a keystore. Here is some more information on using openssl to download the certificate from a remote server and then using keytool to import it into the keystore.

keytool needs the certificate to be in X509 format, so we will use sed to format the certificate.

[code]echo -n | openssl s_client -connect HOST:PORTNUMBER | sed -ne ‘/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p’ > /tmp/$SERVERNAME.cert [/code]

breaking down the command

[code]echo -n[/code]

send an end of line signal to openssl. This allows openssl (or rather the server it is trying to connect to) to disconnect the session

[code]openssl s_client -connect HOST:PORTNUMBER[/code]

asks openssl to act as a client and connect to the HOST on the specificed PORTNUMBER

[code]sed -ne ‘/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p’ [/code]

asks sed to take the input from openssl and only output the content between BEGIN CERTIFICATE and END CERTIFICATE.

NOTE: If you get an error like “SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert unexpected message”, it means the server doesn’t support SSL negotation. Using the command option -no_tls1 helps work around this error. This option will tell openssl to disable TLS1 negotiation.

October 15, 2012 by HOWTO Technology Uncategorized Web 0

HOW TO : grep for response codes in apache logs

If you want to grep for certain http response codes in a apache log file

  • Look for all access requests with a 200 response code[code] grep -i "[: ]200[: ]" HTTP_ACCESS_LOG [/code]
  • Look for all access requests that do NOT have a 200 response code[code] grep -i -v "[: ]200[: ]" HTTP_ACCESS_LOG [/code]

Details of the options

  • [code]"[: ]"[/code]

    tells grep to look for space or tab before the specified string, which in this case is 200.

September 5, 2012 by HOWTO Linux Uncategorized Web 0

Another day.. Another Hack

The net is up in arms about a new release from team Ghostshell of compromise data. Details of the leak can be found at http://www.theregister.co.uk/2012/08/28/team_ghostshell_megahack/ and the source of the data is at http://pastebin.com/BuabHTvr .

I thought I would put my nascent python skills to use and write a simple script to parse through the release and download all the data. Hoping to analyze it later on. It is pretty basic, but does the job of parsing the release and downloading the content. You can get the script at https://github.com/kudithipudi/Misc-Scripts/blob/master/parseHellfire.py

Watch out for an analysis of the content soon :).

August 28, 2012 by Programming security Uncategorized Web 0

2012 May Project

Continuing on my project/month theme.. here is what I want to accomplish in May 2012

  • Understand Google App Engine and Windows Azure Platforms
  • Write a simple application and deploy it both the platforms
  • The application that I am envisioning will display the “user agent” string of the client trying to access the application. I know there are tons of sites that already do this.. but I think this is an useful tool to have in your bag of tricks :). It is simple enough that I think I can program it in a month.

Why am I doing this? I understand the IaaS area pretty well, but am not well versed in the PaaS arena. Hoping this adventure will teach me some new things. And yes, I do plan on documenting my journey :).

Wish me luck 🙂

May 1, 2012 by Technology Uncategorized Web 6

HOW TO : Configure Jboss to not show backend server name when proxying https (ssl) traffic

Phew.. that was a long title :).  Was running into an issue with the setup shown in the picture below

When we try to access the web site using https, the html content being served back was showing the app server name as the reference, rather than the web site.

So in this example, let’s say the web address was kudithipudi.org and the app server was app-server-kudithipudi, the HTML content was showing https://app-server-kudithipudi:8080 as the source.

Here’s how, we fixed it.

Edit the server.xml file found in $JBOSS_HOME/server/$JBOSS_PROFILE/deploy/jboss-web.deployer and update the HTTPS connector to use the web address (kudithipudi.org) as the proxyName.

BEFORE

[code]<Connector port="8443" protocol="HTTP/1.1" SSLEnabled="true"
maxThreads="250" scheme="https" secure="true"
clientAuth="false" sslProtocol="TLS"
keystoreFile="/opt/jboss/jboss-as/server/kudithipudi/conf/ssl/kudithipudi.keystore"
keystorePass="xxxxxx" />
[/code]

AFTER

[code]<Connector port="8443" protocol="HTTP/1.1" SSLEnabled="true"
maxThreads="250" scheme="https" secure="true"
clientAuth="false" sslProtocol="TLS"
proxyName="kudithipudi.org" proxyPort="443"
keystoreFile="/opt/jboss/jboss-as/server/kudithipudi/conf/ssl/kudithipudi.keystore"
keystorePass="xxxxxx" />

[/code]

April 30, 2012 by HOWTO Technology Uncategorized Web 0

Project Uptime : Progress Report 7 : Putting the finishing touches

We finally come to one of the last posts of Project Uptime. Now that all the components have been setup, I finally copied the wordpress directory from my old server to the new one. The only changes, I had to make after copying the files were

  1. Configure Apache to have the wordpress folder as the default directory. I did this by changing the DocumentRoot option in the vhost
  2. Changed the permissions on the wordpress directories (so that wordpress can make rewrite rule changes on the fly)

[code]sudo chmod -v 664 $WORDPRESS_DIRECTORY/.htaccess

sudo chmod 755 $WORDPRESS_DIRECTORY/wp-content [/code]

April 18, 2012 by HOWTO Technology Uncategorized Web 0

HOW TO : Configure Jboss to use hugepages in RHEL/CentOS

Most of us worry about paging to disk (swap), but if you are running a transaction intensive application the paging that happens in RAM also starts to impact the application performance. This happens due to the size of the “block” that is used to store data in memory. Hugepages allows you to store the data in bigger blocks, hence reducing the need to page while interacting with the data.

Here is how you can enable hugepages and configure jboss (actually any Java app) to use hugepages on a RHEL/CentoOS system.

OS CONFIGURATION

  1. Check if your system is capable of supporting hugepages by running[code]grep HUGETLB /boot/config-`uname -r`[/code]

    If you see the response as below, you should be good[code]CONFIG_HUGETLBFS=y
    CONFIG_HUGETLB_PAGE=y
    [/code]

  • Next check if huge pages are already being used by running[code]cat /proc/sys/vm/nr_hugepages [/code]
  1. If the response is anything other than 0, that means hugepages have already been configured.
  • Find the block size for hugepages by running[code]cat /proc/meminfo | grep -i hugepagesize [/code]
  • Calculate the amount of memory you want to dedicate to hugepages. (note: memory allocated to hugepages cannot be used by other processes in the system, unless they are configured to use it)
  1. For example, I want to dedicate 3GB of RAM for hugepages. So the number of hugepages would be[code](3*1024*1024)/2048[/code]
  • Configure the number of hugepages on the system by editing the /etc/sysctl.conf and adding the option[code]vm.nr_hugepages = 1536[/code]

    (note: I put in 1536 since that was the value I got from the above example)

  • Restart the server and check if hugepages has been enabled by running[code]cat /proc/meminfo | grep -i huge [/code]
  1. You should see something like this[code]AnonHugePages:    839680 kB
    HugePages_Total:    1500
    HugePages_Free:     1500
    HugePages_Rsvd:        0
    HugePages_Surp:        0
    Hugepagesize:       2048 kB
    [/code]

JBOSS CONFIGURATION

  1. At this point your system is configured with hugepages and any application that is configured to use them can leverage them.  In this example, we want to configure Jboss to utilize these hugepages
  2. Add the groupid of the user that Jboss is running under to the /etc/sysctl.conf file. In my case, the jboss user group had a GID of 505, so I added this line to /etc/sysctl.conf[code]vm.hugetlb_shm_group = 505 [/code]
  3. Next allocate the memory to the user by editing /etc/security/limits.conf and allocating the memory. Again, in my case, I added the following to /etc/security/limits.conf[code]# Allocate memory for Jboss user to take advantage of hugepages
    jboss   soft    memlock 1500
    jboss   hard    memlock 1500
    [/code]
  4. Finally add the following to the Jboss startup parameters. I edited the $JBOSS_HOME/bin/run.sh file. (note: the startup file can be different based on your config) with the option[code] -XX:+UseLargePages[/code]
  5. Restart Jboss and you are good to go

note : A lot articles that I read online say that hugepages are effective when you are allocating large amounts of RAM to the application. The use case of just using 3GB above was just that.. a use case.

While I cannot personally vouch for it, a lot of users have noted that they saw >2 fold increase in performance.

April 17, 2012 by HOWTO Linux Technology Uncategorized Web 6

Project Uptime : Progress Report 6 : Tweaking Varnish

The server has held up pretty well, since the installation of varnish. Based on this wiki post, I added the following to /etc/varnish/default.vcl

[code]
<pre>
# Drop any cookies sent to WordPress.
sub vcl_recv {
if (!(req.url ~ "wp-(login|admin)")) {
unset req.http.cookie;
}
}

# Drop any cookies WordPress tries to send back to the client.
sub vcl_fetch {
if (!(req.url ~ "wp-(login|admin)")) {
unset beresp.http.set-cookie;
}
}
[/code]

I think the comments are pretty self explanatory.

April 14, 2012 by Technology Uncategorized Web 0