HOW TO : Delete duplicate lines in Linux/Unix

If you have a large text file and want to quickly delete any duplicate lines, you can use the following option of sort in *nix..

sort -u filename > newfilename

If you want to get even funkier :), like only checking duplicates on a particular column, feel free to check out this link