Posted on

How data streams work (AKA queue design)

Good blog post by Timothy Downs on how queues and data streams work with a layman example at https://hackernoon.com/introduction-to-redis-streams-133f1c375cd3

Quoting the example here

We have a very long book which we would like many people to read. Some can read during their lunch hour, some read on Monday nights, others take it home for the weekend. The book is so long that at any point in time, we have hundreds of people reading it.

Readers of our book need to keep track of where they are up to in our book, so they keep track of their location by putting a bookmark in the book. Some readers read very slow, leaving their bookmark close to the beginning. Other readers give up halfway, leaving theirs in the middle and never coming back to it.

To make matters even worse, we are adding pages to this book every day. Nobody can actually finish this book.

Eventually our book fills up with bookmarks, until finally one day it is too heavy to carry and nobody can read it any more.

A very clever person then decided that readers should not be allowed to place bookmarks inside the book, and must instead write down the page they are up to on their diary.

This is the design of Apache Kafka, and it is a very resilient design. Readers are often not responsible citizens and often will not clean up after themselves, and the book may be the log of all the important events that happen in our company.

Posted on

Overheard : Communication

Mark Horstman, on his podcast about when not to use email (https://www.manager-tools.com/2017/05/when-not-use-email-part-2)

No delivery of information is purely about information. Every delivery of information has some effect on the relationship that is formed during the communication or (relationship) that previously existed.

Posted on

Overheard : Quote about Servant Leadership

Quote (or rather statement) by Arthur Brooks on Servant Leadership. Credits for capturing the statement and documenting it, goes to Bret Simmons.

“We in this country are facing a lack of visionary servant leadership. Any leader you can think of will say they are fighting for people, and this is a necessary but insufficient condition for being a leader, to fight for people that need you. But what we really need for real vision is level two and level three servant leadership. What’s level two servant leadership? It’s fighting for people that need you that you don’t need. Level three servant leadership is fighting for people who don’t like you. This is the problem, where we split into tribes where leaders only lead their followers.” Arthur Brooks, February 18, 2016, TED 2016, Vancouver, British Columbia

Posted on

HOW TO : Configure nginx for WordPress permalinks

Over the last week, I moved this blog from a LAMP (Linux, Apache, MySQL, PHP) stack to LEMP (Linux, Nginx, MySQL, PHP) stack. Have a blog post in the works with all the gory details, but wanted to quick document a quirk in the WordPress + Nginx combination that broke permalinks on this site.

Permalinks are user friendly permanent static URLs for a blog post. So for example this particular blog post’ URL is

https://kudithipudi.org/2017/02/24/how-to-configure…press-permalinks/

instead of

https://kudithipudi.org/?p=1762

This works by default in Apache because WordPress puts in the required rewrite rules.

To get it work in Nginx, you have to add the following config in the Nginx site configuration

Under the / location context, add the following

try_files $uri $uri/ /index.php?$args;

This is essentially telling Nginx to try to display the URI as is, and if it fails that, pass the URI as an argument to index.php.

Posted on

HOW TO : Configure maven to work with a proxy server

We ran into an issue at work recently, when trying to deploy a spring boot based micro service. The application included some libraries that were downloaded by the application using maven during runtime.

The environment that this application was running in had controlled access to the Internet. We configured the jvm parameters using the following parameters

-Dhttp.useProxy=true -Dhttp.proxyHost=PROXY_SERVER_NAME -Dhttp.proxyPort=xxxx

when running the application, we were getting failed dependency errors that stated something like this

Caused by: org.eclipse.aether.resolution.ArtifactResolutionException: Could not transfer artifact com.xxxxx.boot:xxxx
-boot-autoconfigure:pom:0.0.1-SNAPSHOT from/to central (http://repo1.maven.org/maven2/): Connection refused at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolve(DefaultArtifactResolver.java:444) at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifacts(DefaultArtifactResolver.java:246) at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifact(DefaultArtifactResolver.java:223) at org.apache.maven.repository.internal.DefaultArtifactDescriptorReader.loadPom(DefaultArtifactDescriptorReader.java:334)

Lot of head scratching, because the server running the application had connection to the proxy server and the JVM parameters were accurate.

Finally, we figured out Maven doesn’t use the JVM startup parameters and had to have proxy access configured specifically. There are two ways to do this

  • Add an environment variable called “MAVEN_OPTS” and pass the proxy server settings.. something like this
set MAVEN_OPTS= -Dhttp.proxyHost=PROXY_SERVER_NAME -Dhttp.proxyPort=xxxx
  • Configure the proxy server settings in Maven’s settings.xml… something like this
 <proxies>
 <proxy>
 <id>genproxy</id>
 <active>true</active>
 <protocol>http</protocol>
 <!--username>proxyuser</username-->
 <!--password>proxypass</password-->
 <host>myproxy.mycompany.com</host>
 <port>8080</port>
 <nonProxyHosts>*.mycompany.com|127.0.0.1</nonProxyHosts>
 </proxy>
 </proxies>
Posted on

HOW TO : Query varnishlogs to show backend server

Command parameters for varnishlog to view the backend server that is processing the request. In this particular case, I wanted to see the request URL and backend server for any responses with HTTP code 401 (unauthorized access)

sudo varnishlog -i BackendOpen,BereqURL -q "BerespStatus == 401"