If you want to search for a pattern at the end of a line, you can use
tail -f logfile | grep -v "0$"
breaking down the commands
tail -f : standard tail command. Continuous output to console as the file grows (or until it ends)
grep -v : -v command forces grep to show content that doesn’t match pattern
0$ : This regex is specifically looking for a 0 at the end of the line, which is denoted by $.
varnishlog, one of the tools provided with varnish cache, uses VSL Query Expressions (https://www.varnish-cache.org/docs/trunk/reference/vsl-query.html) to provide some powerful insights into the requests and responses.
Here is a how you can use varnishlog to show all client requests that are ending up with a 404 response.
sudo varnishlog -g request -i ReqURL -q "BerespStatus != 200"
Technically, this particular query shows all client requests with a response other than 200.
Breaking down the commands
-g request : shows all entries related to the request
-i ReqURL : forces varnishlog to only display the Requesting URL
-q “BerespStatus != 200” : query filter to only match non 200 responses. Note that the query has to be enclosed in “”.
We were trying to modify some ACL (access control lists) in squid to allow traffic to certain websites. Instead of adding each individual hostnames in a domain, we wanted to add all traffic to a certain domain.
Document on the interwebs is old or not clear on how to achieve this.
After some trial and error, here is what works
say you want to allow all traffic to the google.com domain, you create a access list using dstdomain like below
acl name_of_acl dstdomain .google.comThe “.” before the domain name acts as a wildcard
Then you use the acl to allow http access to it like below
http_access allow name_of_aclLet’s say you are running a command as sudo and need to pass the output to a different command using pipe, you would run
sudo command1 | command 2this usually results in the following error
-bash: /command2: Permission deniedThe trick to fix is to run sudo with -c and enclose the commands in ” like below
sudo -c 'command1 | command 2'essentially you are opening a shell with sudo and running the commands
If you don’t have tcpdump installed on your solaris server, you can use the “snoop” system command to capture network traffic.
Here is the command line option to capture 1000 packets of network traffic from IP 192.168.10.10 on a solaris server using inteface e1000g1 and write the output to /tmp/capture.pcap
snoop -d e1000g1 -c 10000 -o /tmp/capture.pcap host 192.168.10.10
Details of the command options
More details at https://docs.oracle.com/cd/E23824_01/html/821-1453/gexkw.html
PS : You have to have root privileges to run this command.
quick note for self
hdparm -tT /dev/sdx
sdx : actual device you want to test
Quick how to on using awk to filter results if a certain value (column) is larger than a set value.
For example, if you have a file (servers.txt) with lines in this format
a_datacenter, servers 20 error, servers xyz b_datacenter, servers 21 c_datacenter, servers 50
and you want to show only the lines that have server value larger than 20, you can do this in awk by running
grep datacenter servers.txt | awk '$3 > 20 {print ;}' | more
breaking down the commands
grep – parsing down the output to just show the lines containing datacenter
awk – $3 > 20 : Get the third variable (awk seperates text using spaces by default) and check if it is greater than 20
print – print the entire line
We had a recent challenge at work which required us to execute different actions based on which office a particular workstation was located in. Since we have unique network ranges per office, I thought this would be a good variable to use. Just for future reference, here is how we accomplished this in a batch file. The workstations were running Windows 7
[code]
@ECHO OFF
FOR /f "tokens=3" %%I IN (
‘netsh interface ip show address "Local Area Connection" ^| findstr "IP Address"’
) DO SET ipAddress=%%I
REM "Office 1"
IF NOT x%ipAddress:10.130=%==x%ipAddress% (
ECHO "Office 1" + %ipAddress%
ECHO "do_something_else" )
REM "Office 2"
IF NOT x%ipAddress:10.140=%==x%ipAddress% (
ECHO "Office 2" + %ipAddress%
ECHO "do_something_else" )
[/code]
Details of function used
Following up from my post earlier this month regarding building a security application that scans publicly available data (Google) and report on potential information leakage from a hostname.
I created a repo on github if anyone is interested in contributing. First thing any good developer does is to check code in early and often :). The repo is at https://github.com/kudithipudi/security-domainscan
Here’s the sudo code I put together as a framework to build on
[code]
functions
read_file(file)
open file;
for each line
process_line(hostname)
process_line(hostname)
search_google(hostname)
write to log
search_google (hostname)
connect to google api
get results for hostname
return number of results
main
read_file(input)
[/code]
Quick entry for my own records.
MongoDB is one of the popular open source document database that is part of the nosql movement. One of the applications we deployed at work uses MongoDB as an internal storage engine. We ran into an issue where MongoDB was trying to replicate data to MySQL and the replication stopped because of a size mismatch for an object between MongoDB and MySQL. Essentially MongoDB was trying to insert a record into MySQL that was larger than the defined length.
Here is the query we used to find the culprit objects. We used the awesome Robomongo client to connect to the MongoDB instance.
[code]db.some_table_to_search.find({$where:"this.some_column_to_search.length > 40"})[/code]
Breaking down the command
db -> Specifies the database you are trying to search
some_table_to_search -> Specifie the table you are trying to search
some_column_to_search -> Specified the particular column you are trying to search.
In this specific example, we were looking for entries longer than 40 characters for this column.
If you come from the traditional RDBMS world, here is a link from MongoDB comparing terminology between RDBMS and MongoDB.