If you have a large text file and want to quickly delete any duplicate lines, you can use the following option of sort in *nix..
sort -u filename > newfilename
If you want to get even funkier 
, like only checking duplicates on a particular column, feel free to check out this link http://www.gnu.org/software/coreutils/manual/html_node/sort-invocation.html#sort-invocation.