quick note for self
hdparm -tT /dev/sdx
sdx : actual device you want to test
Linux
HOW TO : Use awk to print values larger than certain number
Quick how to on using awk to filter results if a certain value (column) is larger than a set value.
For example, if you have a file (servers.txt) with lines in this format
a_datacenter, servers 20 error, servers xyz b_datacenter, servers 21 c_datacenter, servers 50
and you want to show only the lines that have server value larger than 20, you can do this in awk by running
grep datacenter servers.txt | awk '$3 > 20 {print ;}' | more
breaking down the commands
grep – parsing down the output to just show the lines containing datacenter
awk – $3 > 20 : Get the third variable (awk seperates text using spaces by default) and check if it is greater than 20
print – print the entire line
HOW TO : Use grep and awk to find count of unique entries
I have use grep extensively before to analyze data in log files before. A good example is this post about using grep and sort to find the unique hits to a website. Here is another way to do it using grep and awk.
Say the log file you are analyzing is in the format below and you need to get the unique number of BundleIDs
[code]2013-02-25 12:00:06,684 ERROR [com.blahblah.sme.command.request.CustomCommand] Unable to execute AssignServiceCommand, request = ‘<AssignServiceToRequest><MemberId>123456</MemberId><OrderBundle><BundleId>5080</BundleId></OrderBundle></AssignServiceToRequest>'[/code]
you can use grep and awk to find the number of times a unique bundleID appears by running
[code]grep -i bundleID LOG_FILE_NAME | awk ‘{ split ($11,a,">"); print a[6]}’ | sort | uniq -c | sort -rn [/code]
breaking down the commands
grep -i : tells grep to only show the lines from the file (LOG_FILE_NAME) containing the text bundleID and makes the search case insensitive
awk ‘{ split ($11,a,”>”); print a[6]}’ : tells awk to grab the input from grep and take the 11th item (by default awk separates content with a space) and split the string into an array (a) using > as a delimiter. And finally print out the value of the array a’s sixth member
sort : sorts the output from awk into ascending order
uniq -c : takes the output from sort and counts uniq items
sort -qn : takes the output from uniq and does a reverse order sort
The output looked like this
[code]
173 5080</BundleId
12 5090</BundleId
8 2833</BundleId
1 2412</BundleId
1 2038</BundleId
1 1978</BundleId
1 1924</BundleId
[/code]
HOW TO : Configure tcpdump to rotate capture files based on size
quick note for self. If you are capturing traffic using tcpdump, you can rotate the capture files based on size
[code]sudo tcpdump -i INTERFACE_TO_CAPTURE_TRAFFIC_ON -C 10 -s0 -W NO_OF_FILES_TO_ROTATE_THROUGH -w /PATH_TO_CAPTURE_FILE [/code]
explanation of the options used
-i : specify the interface you want to capture the traffic on. If not specified, tcpdump will listen on the lowest numbered interface. i.e. eth0
-C : specify the size of the file multiplied by 1000000 bytes. In this example, the file created would be 10000000 bytes. Or ~9.8MB
-s : specify the packet length to capture. 0 (zero) tells tcpdump to capture the entire packet
-W : specify the number of files to rotate through once the files size specified in -C is reached. The files keep rotating throughout the capture
-w : Specify the path to the capture file. tcpdump appends an integer to the end of the file based on the number of files it has to rotate through.
HOW TO : Use curl to check the impact of DNS changes
Ran into an interesting scenario at work today. We had to check the impact of a DNS change on a certain hostname. Normally, you would edit your host file entry to reflect the DNS change and do your testing. Here is another way you can do it using cURL. In this particular example, I am checking the SSL certificate details of the hostname .
[code]curl –insecure –trace-ascii debug.txt https://HOSTNAME:PORT –resolve HOSTNAME:PORT:IP_ADDRESS [/code]
That’s a pretty convoluted command :). Let’s try to break it down
[code]–insecure [/code]
: tells cURL to ignore certificate warnings. This is helpful if you are using self signed certs
[code]–trace-ascii [/code]
: tells cURL to save the SSL connection details (in debug mode) to a file called debug.txt
[code]–resolve [/code]
: tells cURL to use the options mentioned after it to resolve the hostname, rather than using DNS. The format for resolve is <host:port:address>
NOTE: You need to have version 7.21.3 or higher of cURL to use this option
Here’s a real world example. Say, I want to see how the IP address 72.30.38.140 would reacts if www.google.com requests are routed to it
[code]
samurai@samurai:~$ curl –insecure –trace-ascii debug.txt https://www.google.com –resolve www.google.com:443:72.30.38.140
The document has moved <A HREF="http://www.google.com/?s=https">here</A>.<P>
<!– ir2.fp.sp2.yahoo.com uncompressed/chunked Mon Nov 12 22:44:41 UTC 2012 –>
samurai@samurai:~$ more debug.txt
== Info: Added www.google.com:443:72.30.38.140 to DNS cache
== Info: About to connect() to www.google.com port 443 (#0)
== Info: Trying 72.30.38.140… == Info: connected
== Info: Connected to www.google.com (72.30.38.140) port 443 (#0)
== Info: successfully set certificate verify locations:
== Info: CAfile: none
CApath: /etc/ssl/certs
== Info: SSLv3, TLS handshake, Client hello (1):
=> Send SSL data, 223 bytes (0xdf)
0000: ……P.|v..1..kA…….=J.xr.=ft.3.|…Z…..9.8………5…..
0040: …………….3.2…..E.D…../…A………………………
0080: …….W………www.google.com………..4.2……………….
00c0: ………………………….
== Info: SSLv3, TLS handshake, Server hello (2):
<= Recv SSL data, 42 bytes (0x2a)
0000: …&..P.{.I"L….3x..N…9…./<n….A..5.
== Info: SSLv3, TLS handshake, CERT (11):
<= Recv SSL data, 1272 bytes (0x4f8)
0000: ……….0…0..S……….0…*.H……..0N1.0…U….US1.0…
0040: U….Equifax1-0+..U…$Equifax Secure Certificate Authority0…1
0080: 00401230014Z..150703045000Z0..1)0′..U… 2g8aO5wI1bKJ2ZD588UsLvD
00c0: e3gTbg8DU1.0…U….US1.0…U….California1.0…U….Sunnyvale1
0100: .0…U….Yahoo Inc.1.0…U….www.yahoo.com0.."0…*.H……..
0140: …..0……….5.p./……..O…k.C…9E+.J..H.s….Bm.T.E.-..<
0180: ^…m…r.v<\…&Qq..l………. @'(q.m..ZJ.*kt…!.AWU…….M.
01c0: …n…O….0.._…H….4……>.m..K…….Z…:.Df%.lR.!…(!.
0200: .FV.dQ…f.V….P,.J9.c..dM.s>C=….Y..#…47#2…..cP.{….g.rU
0240: .d…P……………..0…0…U………..0…U………….t5.
0280:……U..0:..U…3010/.-.+.)http://crl.geotrust.com/crls/secure
02c0: ca.crl0..[..U…..R0..N..www.yahoo.com..yahoo.com..us.yahoo.com.
0300: .kr.yahoo.com..uk.yahoo.com..ie.yahoo.com..fr.yahoo.com..in.yaho
0340: o.com..ca.yahoo.com..br.yahoo.com..de.yahoo.com..es.yahoo.com..m
0380: x.yahoo.com..it.yahoo.com..sg.yahoo.com..id.yahoo.com..ph.yahoo.
03c0: com..qc.yahoo.com..tw.yahoo.com..hk.yahoo.com..cn.yahoo.com..au.
0400: yahoo.com..ar.yahoo.com..vn.yahoo.com0…U.#..0…H.h.+….G.# .
0440: O3….0…U.%..0…+………+…….0…*.H……………2..0.
0480: S.’.y….GD.Q…=…K+..q..kv…….<h…….ZLE.h$..M2^.C..IT..
04c0: ".5j….Vc7.4……1.Wu.[.a>+………9..{.a:………
== Info: SSLv3, TLS handshake, Server finished (14):
<= Recv SSL data, 4 bytes (0x4)
0000: ….
== Info: SSLv3, TLS handshake, Client key exchange (16):
=> Send SSL data, 262 bytes (0x106)
0000: …….R…..b.,.&.. s.Ob;.E_.EnSw../D…’…..(aB<<……F..]..
0040: o………~…*..r?.C..%..22…J.bu&.x(j|…….>A5..OF.G…C.$.
0080: .9u9n.z…K…..u…..~:W.{Sii.{2..6……..<…..i…8y$y…..6
00c0: …1.(M…fx….#k..r….47..t.q…..A.?.0. .D…..~…G+.,….~
0100: ..=.#y
== Info: SSLv3, TLS change cipher, Client hello (1):
=> Send SSL data, 1 bytes (0x1)
0000: .
== Info: SSLv3, TLS handshake, Finished (20):
=> Send SSL data, 16 bytes (0x10)
0000: ….!9)…6…+.
== Info: SSLv3, TLS change cipher, Client hello (1):
<= Recv SSL data, 1 bytes (0x1)
0000: .
== Info: SSLv3, TLS handshake, Finished (20):
<= Recv SSL data, 16 bytes (0x10)
0000: …..(qN..l.]…
== Info: SSL connection using AES256-SHA
== Info: Server certificate:
== Info: subject: serialNumber=2g8aO5wI1bKJ2ZD588UsLvDe3gTbg8DU; C=US; ST=California; L=Sunnyvale; O=Yahoo Inc.; CN=www.yahoo.com
== Info: start date: 2010-04-01 23:00:14 GMT
== Info: expire date: 2015-07-03 04:50:00 GMT
== Info: subjectAltName does not match www.google.com
=> Send header, 167 bytes (0xa7)
0000: GET / HTTP/1.1
0010: User-Agent: curl/7.21.6 (x86_64-pc-linux-gnu) libcurl/7.21.6 Ope
0050: nSSL/1.0.0e zlib/1.2.3.4 libidn/1.22 librtmp/2.3
0082: Host: www.google.com
0098: Accept: */*
00a5:
<= Recv header, 32 bytes (0x20)
0000: HTTP/1.1 301 Moved Permanently
<= Recv header, 37 bytes (0x25)
0000: Date: Mon, 12 Nov 2012 22:44:41 GMT
<= Recv header, 42 bytes (0x2a)
0000: Location: http://www.google.com/?s=https
<= Recv header, 23 bytes (0x17)
0000: Vary: Accept-Encoding
<= Recv header, 19 bytes (0x13)
0000: Connection: close
<= Recv header, 28 bytes (0x1c)
0000: Transfer-Encoding: chunked
<= Recv header, 40 bytes (0x28)
0000: Content-Type: text/html; charset=utf-8
<= Recv header, 24 bytes (0x18)
0000: Cache-Control: private
<= Recv header, 2 bytes (0x2)
0000:
<= Recv data, 173 bytes (0xad)
0000: 000009d
0009: The document has moved <A HREF="http://www.google.com/?s=https">
0049: here</A>.<P>.<!– ir2.fp.sp2.yahoo.com uncompressed/chunked Mon
0089: Nov 12 22:44:41 UTC 2012 –>.
00a8: 0
00ab:
== Info: Closing connection #0
== Info: SSLv3, TLS alert, Client hello (1):
=> Send SSL data, 2 bytes (0x2)
0000: ..
[/code]
HOW TO : Compare two directories in Linux
Quick post on using diff to compare two directories in Linux. This will show the list of files and subdirectories that are different in either directories
[code]diff /PATH_TO_FIRST_DIRECTORY /PATH_TO_SECOND_DIRECTORY -r –brief [/code]
Options used
- r : Searched recursively through the directory
- –brief : Only shows the names of the files that differ. If you want details of the content that differs, remove this option
HOW TO : grep for response codes in apache logs
If you want to grep for certain http response codes in a apache log file
- Look for all access requests with a 200 response code[code] grep -i "[: ]200[: ]" HTTP_ACCESS_LOG [/code]
- Look for all access requests that do NOT have a 200 response code[code] grep -i -v "[: ]200[: ]" HTTP_ACCESS_LOG [/code]
Details of the options
- [code]"[: ]"[/code]
tells grep to look for space or tab before the specified string, which in this case is 200.
HOW TO : Find files, search for content in them, replace the content
The title pretty much says it all :). Here is a quick one liner, using multiple tools, to look for files in a directory, search for certain content in them and replace them with other content
[code]find -type f | xargs grep -l ORIGINAL_CONTENT | xargs perl -p -i -e ‘s/ORIGINAL_CONTENT/NEW_CONTENT/g’ [/code]
You can theoretically take out the grep (second command) and directly pipe the find output to perl and get the same outcome.
Going over list of the options used
find
- “-type f” lists all objects of type file in the directory (and sub directories)
grep
- “-l” lists the names of the files (with relative path) which have the text ORIGINAL_CONTENT in them
perl
- “-p” forces perl to loop through requests. In this case files
- “-e” tells perl that the next argument is a perl statement
- “-i” tells perls to edit the file in place (i.e. no need for an output file)
HOW TO : Use iostat to check network share performance
Use the following option to check out how network (NFS) shares are performing on your sever
[code]iostat -n 2[/code]
The -n option tells iostat to just show the network shares
2 tells iostat to refresh the stats every 2 seconds
HOW TO : Use screen to multitask
Want your terminal to look like this? 
In addition to the coolness, factor it also helps you do (monitor) multiple things at a time. In this screenshot
- I am checking the resource utilization on my web server
- Tailing the web server logs to look for errors
- have a small console for me to run any commands
You can achieve this by using the nifty screen utility. Screen allows you to multiplex between multiple consoles. So you can open one terminal and have multiple consoles on it. The commands for using screen are a bit hard to get used to. Here are the shortcuts I used to achieve the screen above
- Install the screen package
- Create a new screen session by running[code] screen [/code]
- Add a new screen console by executing[code] ctrl + a [/code]
[code]c[/code]
- Split the screen by executing[code]ctrl + a [/code]
[code] Shift + s [/code]
- Name the different consoled by executing[code]ctrl + a[/code]
[code]Shift + a[/code]
As you might have figured out by now, “ctrl + a” puts you into screen command mode. You can get a list of all available options by executing
[code]ctrl +a[/code]
[code]?[/code]
Here’s a quick reference guide that has more details http://aperiodic.net/screen/quick_reference
Have fun multitasking 🙂