bash remove duplicate lines from a file 1

bash remove duplicate lines from a file

# Basic syntax:
sort input_file | uniq --unique
# Sort the file first because uniq requires a sorted file to work. Uniq
# then eliminates all duplicated lines in the file, keeping one instance
# of each duplicated line

# Note, this doesn't return only non-duplicated lines. It returns
# unique instances of all lines, whether or not they are duplicated

# Note, if you want to return only one instance of all lines but see
# the number of repetitions for each line, run:
sort input_file | uniq -c

Here is what the above code is Doing:
1. sort input_file
– sort the input file
2. |
– pipe the output of the previous command into the next command
3. uniq –unique
– uniq is a command that takes a sorted file and returns only unique
instances of each line
– –unique is an option that tells uniq to return only unique lines
(i.e. lines that are not duplicated)

Similar Posts