quick note for self
hdparm -tT /dev/sdx
sdx : actual device you want to test
HOWTO
HOW TO : Use awk to print values larger than certain number
Quick how to on using awk to filter results if a certain value (column) is larger than a set value.
For example, if you have a file (servers.txt) with lines in this format
a_datacenter, servers 20 error, servers xyz b_datacenter, servers 21 c_datacenter, servers 50
and you want to show only the lines that have server value larger than 20, you can do this in awk by running
grep datacenter servers.txt | awk '$3 > 20 {print ;}' | more
breaking down the commands
grep – parsing down the output to just show the lines containing datacenter
awk – $3 > 20 : Get the third variable (awk seperates text using spaces by default) and check if it is greater than 20
print – print the entire line
HOW TO : Search for a record in MongoDB based on length
Quick entry for my own records.
MongoDB is one of the popular open source document database that is part of the nosql movement. One of the applications we deployed at work uses MongoDB as an internal storage engine. We ran into an issue where MongoDB was trying to replicate data to MySQL and the replication stopped because of a size mismatch for an object between MongoDB and MySQL. Essentially MongoDB was trying to insert a record into MySQL that was larger than the defined length.
Here is the query we used to find the culprit objects. We used the awesome Robomongo client to connect to the MongoDB instance.
[code]db.some_table_to_search.find({$where:"this.some_column_to_search.length > 40"})[/code]
Breaking down the command
db -> Specifies the database you are trying to search
some_table_to_search -> Specifie the table you are trying to search
some_column_to_search -> Specified the particular column you are trying to search.
In this specific example, we were looking for entries longer than 40 characters for this column.
If you come from the traditional RDBMS world, here is a link from MongoDB comparing terminology between RDBMS and MongoDB.
HOW TO : Convert PFX/P12 crypto objects into a java keystore
We needed to add a certificate that is currently in PKCS#12 format currently into a java keystore at work recently. The typical step would be due to create an empty keystore and then import the certificate from the PKCS#12 store using the following command
[code]keytool -importkeystore -srckeystore sourceFile.p12 -srcstoretype PKCS12 -destkeystore destinationFile.jks[/code]
Note: PKCS#12 files can have extensions “.p12” or “.pfx”
The command executed without any issues, but we received the following error when we started the application server using this newly created keystore
[code]java.io.IOException: Error initializing server socket factory SSL context: Cannot recover key [/code]
It didn’t make sense, because we were able to view the certificate in the keystore and were using the right password in the configuration files.
After a lot of searching and head scratching, the team came up with the following solution
- Export the public key and private key from the PKCS#12 store using openssl.
- Import these keys into the java keystore (default format of JKS)
The commands used were
[code]
openssl pkcs12 -in sourcePKCS12File.p12 -nocerts -out privateKey.pem
openssl pkcs12 -in sourcePKCS12File.p12 -nokeys -out publicCert.pem
openssl pkcs12 -export -out intermittentPKCS12File.p12 -inkey privateKey.pem -in publicCert.pem
keytool -importkeystore -srckeystore intermittantPKCS12File.p12 -srcstoretype PKCS12 -destkeystore finalKeyStore.jks
[/code]
HOW TO : Use grep and awk to find count of unique entries
I have use grep extensively before to analyze data in log files before. A good example is this post about using grep and sort to find the unique hits to a website. Here is another way to do it using grep and awk.
Say the log file you are analyzing is in the format below and you need to get the unique number of BundleIDs
[code]2013-02-25 12:00:06,684 ERROR [com.blahblah.sme.command.request.CustomCommand] Unable to execute AssignServiceCommand, request = ‘<AssignServiceToRequest><MemberId>123456</MemberId><OrderBundle><BundleId>5080</BundleId></OrderBundle></AssignServiceToRequest>'[/code]
you can use grep and awk to find the number of times a unique bundleID appears by running
[code]grep -i bundleID LOG_FILE_NAME | awk ‘{ split ($11,a,">"); print a[6]}’ | sort | uniq -c | sort -rn [/code]
breaking down the commands
grep -i : tells grep to only show the lines from the file (LOG_FILE_NAME) containing the text bundleID and makes the search case insensitive
awk ‘{ split ($11,a,”>”); print a[6]}’ : tells awk to grab the input from grep and take the 11th item (by default awk separates content with a space) and split the string into an array (a) using > as a delimiter. And finally print out the value of the array a’s sixth member
sort : sorts the output from awk into ascending order
uniq -c : takes the output from sort and counts uniq items
sort -qn : takes the output from uniq and does a reverse order sort
The output looked like this
[code]
173 5080</BundleId
12 5090</BundleId
8 2833</BundleId
1 2412</BundleId
1 2038</BundleId
1 1978</BundleId
1 1924</BundleId
[/code]
HOW TO : Configure tcpdump to rotate capture files based on size
quick note for self. If you are capturing traffic using tcpdump, you can rotate the capture files based on size
[code]sudo tcpdump -i INTERFACE_TO_CAPTURE_TRAFFIC_ON -C 10 -s0 -W NO_OF_FILES_TO_ROTATE_THROUGH -w /PATH_TO_CAPTURE_FILE [/code]
explanation of the options used
-i : specify the interface you want to capture the traffic on. If not specified, tcpdump will listen on the lowest numbered interface. i.e. eth0
-C : specify the size of the file multiplied by 1000000 bytes. In this example, the file created would be 10000000 bytes. Or ~9.8MB
-s : specify the packet length to capture. 0 (zero) tells tcpdump to capture the entire packet
-W : specify the number of files to rotate through once the files size specified in -C is reached. The files keep rotating throughout the capture
-w : Specify the path to the capture file. tcpdump appends an integer to the end of the file based on the number of files it has to rotate through.
HOW TO : Restrict access to proxied content in Apache
If you are using the mod_proxy feature in Apache to forward requests for certain content to a backend server, but want to restrict access to that content to clients originating from certain IP addresses, you can use the location feature in Apache.
The Location directive limits the scope of the enclosed directives by URL. This is very similar to the Directory directive, but the difference is that you can put controls based on the URL rather than the location of the content.
In this example, I am forwarding content destined to https://kudithipudi.org/testLocation to an internal server at http://127.0.0.1:8080/testLocation. I am going to use the Location directive to restrict access to just requests originating from IP Address 10.10.10.10
[code]
<Location /testLocation>
Order Deny,Allow
Deny from all
Allow from 10.10.10.10
</Location>
ProxyPass /testLocation http://127.0.0.1:8080/testLocation
ProxyPassReverse /testLocation http://127.0.0.1:8080/testLocation [/code]
HOW TO : Run web servers using one liners
A collection of one liners using different tools and programming languages to run a full fledged web server on any machine. They can be used to
- Serve files located on the server
- Act as server listening on a particular port. This is especially helpful if you are trying to setup a load-balancer and/or firewall and need to test access to the end points.
The criteria for the on liners was that you don’t need any additional modules other than the standard modules included with the language distributions.
NC : Netcat
netcat (nc) is pretty powerful network utility. You can start a web server running on port 8080 by simply running
[code]nc -l 8080[/code]
If you want to serve a particular file, you can do so by running
[code]while :; do nc -l 8080 < SAMPLE_FILE ; done [/code]
Python
You can start a web server in python by running
Python 2.x
[code] python -m SimpleHTTPServer 8080 [/code]
Python 3.x
[code]python -m http.server 8080 [/code]
This command will serve up a page with listing of all the files in the directory that the command was executed in. Pretty nifty way to quickly share files
Perl
You can start a web server in perl by running
[code]perl -MIO::All -e ‘io(":8080")->fork->accept->(sub { $_[0] < io(-x $1 ? "./$1 |" : $1) if /^GET \/(.*) / })’ [/code]
Ruby
You can start a web server using Ruby by running
sudo ruby -rwebrick -e ‘server = WEBrick::HTTPServer.new : Port = >8080
server.start’
I haven’t been able to figure out how to pass an end of line in the command line. So you need to literally pass the commands in two lines.
Scratch that.. My friend, Ray, showed me the right way to pass a line delimiter in the same command.
[code]ruby -rwebrick -e ‘server = WEBrick::HTTPServer.new(:Port => 8080) ; server.start’ [/code]
He even provided an additional option to define the directory you want to serve files from
[code]: DocumentRoot => ‘/some/shit’ [/code]
PHP
Starting with PHP 5.4 you can initiate a web server by running
[code]php -S localhost:8080[/code]
All of these options should work on any operating system. But I have only tried them on Linux.
Do you know how to do the same thing in other languages? Please share them in the comments section.
Credits: I collected these bits of code from the following sites
Python : http://www.garyrobinson.net/2004/03/one_line_python.html
Perl : http://www.perlmonks.org/?node_id=470397
Ruby : http://phrogz.net/simplest-possible-ruby-web-server
PHP : http://php.net/manual/en/features.commandline.webserver.php
HOW TO : Combine landscape and portrait page layouts in Microsoft Word
Blogging this as a “memory” note for myself 🙂
I was putting together a report for work and needed one of the pages in the word document to be in landscape mode, instead of the regular portrait mode. I thought it was a simple thing of adding a page break and applying the “landscape” layout in the page setup. But ended up either having all pages in landscape mode or in portrait mode. A bit of googling finally helped out :). Looks like the trick is to use section breaks instead of page breaks.
Here are the steps to do it in Microsoft Word 2010
- Add the content you want into word. In this example, I created two paragraphs, test landscape and test portrait
- At the place you want to split the page format, insert a section break, by going to Page Layout –> Breaks –> Section Breaks –> Next Page
- Now change the page orientation by going to Page Layout -> Orientation –> Landscape. This will only change the orientation for the current section.
- And voila you document now has two different page orientations 🙂
HOW TO : Use curl to check the impact of DNS changes
Ran into an interesting scenario at work today. We had to check the impact of a DNS change on a certain hostname. Normally, you would edit your host file entry to reflect the DNS change and do your testing. Here is another way you can do it using cURL. In this particular example, I am checking the SSL certificate details of the hostname .
[code]curl –insecure –trace-ascii debug.txt https://HOSTNAME:PORT –resolve HOSTNAME:PORT:IP_ADDRESS [/code]
That’s a pretty convoluted command :). Let’s try to break it down
[code]–insecure [/code]
: tells cURL to ignore certificate warnings. This is helpful if you are using self signed certs
[code]–trace-ascii [/code]
: tells cURL to save the SSL connection details (in debug mode) to a file called debug.txt
[code]–resolve [/code]
: tells cURL to use the options mentioned after it to resolve the hostname, rather than using DNS. The format for resolve is <host:port:address>
NOTE: You need to have version 7.21.3 or higher of cURL to use this option
Here’s a real world example. Say, I want to see how the IP address 72.30.38.140 would reacts if www.google.com requests are routed to it
[code]
samurai@samurai:~$ curl –insecure –trace-ascii debug.txt https://www.google.com –resolve www.google.com:443:72.30.38.140
The document has moved <A HREF="http://www.google.com/?s=https">here</A>.<P>
<!– ir2.fp.sp2.yahoo.com uncompressed/chunked Mon Nov 12 22:44:41 UTC 2012 –>
samurai@samurai:~$ more debug.txt
== Info: Added www.google.com:443:72.30.38.140 to DNS cache
== Info: About to connect() to www.google.com port 443 (#0)
== Info: Trying 72.30.38.140… == Info: connected
== Info: Connected to www.google.com (72.30.38.140) port 443 (#0)
== Info: successfully set certificate verify locations:
== Info: CAfile: none
CApath: /etc/ssl/certs
== Info: SSLv3, TLS handshake, Client hello (1):
=> Send SSL data, 223 bytes (0xdf)
0000: ……P.|v..1..kA…….=J.xr.=ft.3.|…Z…..9.8………5…..
0040: …………….3.2…..E.D…../…A………………………
0080: …….W………www.google.com………..4.2……………….
00c0: ………………………….
== Info: SSLv3, TLS handshake, Server hello (2):
<= Recv SSL data, 42 bytes (0x2a)
0000: …&..P.{.I"L….3x..N…9…./<n….A..5.
== Info: SSLv3, TLS handshake, CERT (11):
<= Recv SSL data, 1272 bytes (0x4f8)
0000: ……….0…0..S……….0…*.H……..0N1.0…U….US1.0…
0040: U….Equifax1-0+..U…$Equifax Secure Certificate Authority0…1
0080: 00401230014Z..150703045000Z0..1)0′..U… 2g8aO5wI1bKJ2ZD588UsLvD
00c0: e3gTbg8DU1.0…U….US1.0…U….California1.0…U….Sunnyvale1
0100: .0…U….Yahoo Inc.1.0…U….www.yahoo.com0.."0…*.H……..
0140: …..0……….5.p./……..O…k.C…9E+.J..H.s….Bm.T.E.-..<
0180: ^…m…r.v<\…&Qq..l………. @'(q.m..ZJ.*kt…!.AWU…….M.
01c0: …n…O….0.._…H….4……>.m..K…….Z…:.Df%.lR.!…(!.
0200: .FV.dQ…f.V….P,.J9.c..dM.s>C=….Y..#…47#2…..cP.{….g.rU
0240: .d…P……………..0…0…U………..0…U………….t5.
0280:……U..0:..U…3010/.-.+.)http://crl.geotrust.com/crls/secure
02c0: ca.crl0..[..U…..R0..N..www.yahoo.com..yahoo.com..us.yahoo.com.
0300: .kr.yahoo.com..uk.yahoo.com..ie.yahoo.com..fr.yahoo.com..in.yaho
0340: o.com..ca.yahoo.com..br.yahoo.com..de.yahoo.com..es.yahoo.com..m
0380: x.yahoo.com..it.yahoo.com..sg.yahoo.com..id.yahoo.com..ph.yahoo.
03c0: com..qc.yahoo.com..tw.yahoo.com..hk.yahoo.com..cn.yahoo.com..au.
0400: yahoo.com..ar.yahoo.com..vn.yahoo.com0…U.#..0…H.h.+….G.# .
0440: O3….0…U.%..0…+………+…….0…*.H……………2..0.
0480: S.’.y….GD.Q…=…K+..q..kv…….<h…….ZLE.h$..M2^.C..IT..
04c0: ".5j….Vc7.4……1.Wu.[.a>+………9..{.a:………
== Info: SSLv3, TLS handshake, Server finished (14):
<= Recv SSL data, 4 bytes (0x4)
0000: ….
== Info: SSLv3, TLS handshake, Client key exchange (16):
=> Send SSL data, 262 bytes (0x106)
0000: …….R…..b.,.&.. s.Ob;.E_.EnSw../D…’…..(aB<<……F..]..
0040: o………~…*..r?.C..%..22…J.bu&.x(j|…….>A5..OF.G…C.$.
0080: .9u9n.z…K…..u…..~:W.{Sii.{2..6……..<…..i…8y$y…..6
00c0: …1.(M…fx….#k..r….47..t.q…..A.?.0. .D…..~…G+.,….~
0100: ..=.#y
== Info: SSLv3, TLS change cipher, Client hello (1):
=> Send SSL data, 1 bytes (0x1)
0000: .
== Info: SSLv3, TLS handshake, Finished (20):
=> Send SSL data, 16 bytes (0x10)
0000: ….!9)…6…+.
== Info: SSLv3, TLS change cipher, Client hello (1):
<= Recv SSL data, 1 bytes (0x1)
0000: .
== Info: SSLv3, TLS handshake, Finished (20):
<= Recv SSL data, 16 bytes (0x10)
0000: …..(qN..l.]…
== Info: SSL connection using AES256-SHA
== Info: Server certificate:
== Info: subject: serialNumber=2g8aO5wI1bKJ2ZD588UsLvDe3gTbg8DU; C=US; ST=California; L=Sunnyvale; O=Yahoo Inc.; CN=www.yahoo.com
== Info: start date: 2010-04-01 23:00:14 GMT
== Info: expire date: 2015-07-03 04:50:00 GMT
== Info: subjectAltName does not match www.google.com
=> Send header, 167 bytes (0xa7)
0000: GET / HTTP/1.1
0010: User-Agent: curl/7.21.6 (x86_64-pc-linux-gnu) libcurl/7.21.6 Ope
0050: nSSL/1.0.0e zlib/1.2.3.4 libidn/1.22 librtmp/2.3
0082: Host: www.google.com
0098: Accept: */*
00a5:
<= Recv header, 32 bytes (0x20)
0000: HTTP/1.1 301 Moved Permanently
<= Recv header, 37 bytes (0x25)
0000: Date: Mon, 12 Nov 2012 22:44:41 GMT
<= Recv header, 42 bytes (0x2a)
0000: Location: http://www.google.com/?s=https
<= Recv header, 23 bytes (0x17)
0000: Vary: Accept-Encoding
<= Recv header, 19 bytes (0x13)
0000: Connection: close
<= Recv header, 28 bytes (0x1c)
0000: Transfer-Encoding: chunked
<= Recv header, 40 bytes (0x28)
0000: Content-Type: text/html; charset=utf-8
<= Recv header, 24 bytes (0x18)
0000: Cache-Control: private
<= Recv header, 2 bytes (0x2)
0000:
<= Recv data, 173 bytes (0xad)
0000: 000009d
0009: The document has moved <A HREF="http://www.google.com/?s=https">
0049: here</A>.<P>.<!– ir2.fp.sp2.yahoo.com uncompressed/chunked Mon
0089: Nov 12 22:44:41 UTC 2012 –>.
00a8: 0
00ab:
== Info: Closing connection #0
== Info: SSLv3, TLS alert, Client hello (1):
=> Send SSL data, 2 bytes (0x2)
0000: ..
[/code]