实例详细说明linux下去除重复行命令uniq

 

一，uniq干什么用的

文本中的重复行，基本上不是我们所要的，所以就要去除掉。linux下有其他命令可以去除重复行，但是我觉得uniq还是比较方便的一个。使用uniq的时候要注意以下二点

1，对文本操作时，它一般会和sort命令进行组合使用，因为uniq 不会检查重复的行，除非它们是相邻的行。如果您想先对输入排序，使用sort -u。

2，对文本操作时，若域中为先空字符(通常包括空格以及制表符)，然后非空字符，域中字符前的空字符将被跳过

二，uniq参数说明

1.[zhangy@BlackGhost ~]$ uniq --help 
2.用法：uniq [选项]... [文件] 
3.从输入文件或者标准输入中筛选相邻的匹配行并写入到输出文件或标准输出。 
4.不附加任何选项时匹配行将在首次出现处被合并。 
5.长选项必须使用的参数对于短选项时也是必需使用的。 
6. -c, --count //在每行前加上表示相应行目出现次数的前缀编号 
7. -d, --repeated //只输出重复的行 
8. -D, --all-repeated //只输出重复的行，不过有几行输出几行 
9. -f, --skip-fields=N //-f 忽略的段数，-f 1 忽略第一段 
10. -i, --ignore-case //不区分大小写 
11. -s, --skip-chars=N //根-f有点像，不过-s是忽略，后面多少个字符 -s 5就忽略后面5个字符 
12. -u, --unique //去除重复的后，全部显示出来，根mysql的distinct功能上有点像 
13. -z, --zero-terminated end lines with 0 byte, not newline 
14. -w, --check-chars=N //对每行第N 个字符以后的内容不作对照 
15. --help //显示此帮助信息并退出 
16. --version //显示版本信息并退出 

 

 

[zhangy@BlackGhost ~]$ uniq --help
用法：uniq [选项]... [文件]
从输入文件或者标准输入中筛选相邻的匹配行并写入到输出文件或标准输出。

不附加任何选项时匹配行将在首次出现处被合并。

长选项必须使用的参数对于短选项时也是必需使用的。
 -c, --count              //在每行前加上表示相应行目出现次数的前缀编号
 -d, --repeated          //只输出重复的行
 -D, --all-repeated      //只输出重复的行，不过有几行输出几行
 -f, --skip-fields=N     //-f 忽略的段数，-f 1 忽略第一段
 -i, --ignore-case       //不区分大小写
 -s, --skip-chars=N      //根-f有点像，不过-s是忽略，后面多少个字符 -s 5就忽略后面5个字符
 -u, --unique            //去除重复的后，全部显示出来，根mysql的distinct功能上有点像
 -z, --zero-terminated   end lines with 0 byte, not newline
 -w, --check-chars=N      //对每行第N 个字符以后的内容不作对照
 --help              //显示此帮助信息并退出
 --version              //显示版本信息并退出

三，测试文本文件uniqtest

this is a test

this is a test

this is a test

i am tank

i love tank

i love tank

this is a test

whom have a try

WhoM have a try

you have a try

i want to abroad

those are good men

we are good men

四，实例详解

 

1.[zhangy@BlackGhost mytest]$ uniq -c uniqtest 
2. 3 this is a test 
3. 1 i am tank 
4. 2 i love tank 
5. 1 this is a test //和第一行是重复的 
6. 1 whom have a try 
7. 1 WhoM have a try 
8. 1 you have a try 
9. 1 i want to abroad 
10. 1 those are good men 
11. 1 we are good men 

 

[zhangy@BlackGhost mytest]$ uniq -c uniqtest
 3 this is a test
 1 i am tank
 2 i love tank
 1 this is a test           //和第一行是重复的
 1 whom have a try
 1 WhoM have a try
 1 you  have a try
 1 i want to abroad
 1 those are good men
 1 we are good men

从上例子中我们可以看出，uniq的一个特性，检查重复行的时候，只会检查相邻的行。重复数据，肯定有很多不是相邻在一起的。

1.[zhangy@BlackGhost mytest]$ sort uniqtest |uniq -c 
2. 1 WhoM have a try 
3. 1 i am tank 
4. 2 i love tank 
5. 1 i want to abroad 
6. 4 this is a test 
7. 1 those are good men 
8. 1 we are good men 
9. 1 whom have a try 
10. 1 you have a try 

 

[zhangy@BlackGhost mytest]$ sort uniqtest |uniq -c   
 1 WhoM have a try   
 1 i am tank   
 2 i love tank   
 1 i want to abroad   
 4 this is a test   
 1 those are good men   
 1 we are good men   
 1 whom have a try   
 1 you  have a try

这样就可以解决上个例子中提到的问题

[zhangy@BlackGhost mytest]$ uniq -d -c uniqtest
 3 this is a test
 2 i love tank

[zhangy@BlackGhost mytest]$ uniq -d -c uniqtest   
 3 this is a test   
 2 i love tank  

uniq -d 只显示重复的行

[zhangy@BlackGhost mytest]$ uniq -D uniqtest
this is a test
this is a test
this is a test
i love tank
i love tank

[zhangy@BlackGhost mytest]$ uniq -D uniqtest   
this is a test   
this is a test   
this is a test   
i love tank   
i love tank  

uniq -D 只显示重复的行，并且把重复几行都显示出来。他不能和-c一起使用

 

[zhangy@BlackGhost mytest]$ uniq -f 1 -c uniqtest
 3 this is a test
 1 i am tank
 2 i love tank
 1 this is a test
 2 whom have a try
 1 you have a try
 1 i want to abroad
 2 those are good men //只有一行，显示二行

[zhangy@BlackGhost mytest]$ uniq -f 1 -c uniqtest
 3 this is a test
 1 i am tank
 2 i love tank
 1 this is a test
 2 whom have a try
 1 you  have a try
 1 i want to abroad
 2 those are good men   //只有一行，显示二行

在这里those只有一行，显示的却是重复了，这是因为，-f 1 忽略了第一列，检查重复从第二字段开始的。

[zhangy@BlackGhost mytest]$ uniq -i -c uniqtest
 3 this is a test
 1 i am tank
 2 i love tank
 1 this is a test
 2 whom have a try //一个大写，一个小写
 1 you have a try
 1 i want to abroad
 1 those are good men
 1 we are good men

[zhangy@BlackGhost mytest]$ uniq -i -c uniqtest
 3 this is a test
 1 i am tank
 2 i love tank
 1 this is a test
 2 whom have a try  //一个大写，一个小写
 1 you  have a try
 1 i want to abroad
 1 those are good men
 1 we are good men

检查的时候，不区分大小写

[zhangy@BlackGhost mytest]$ uniq -s 4 -c uniqtest
3 this is a test
1 i am tank
2 i love tank
1 this is a test
3 whom have a try //根上一个例子有什么不同
1 i want to abroad
1 those are good men
1 we are good men

 [zhangy@BlackGhost mytest]$ uniq -s 4 -c uniqtest
 3 this is a test
 1 i am tank
 2 i love tank
 1 this is a test
 3 whom have a try   //根上一个例子有什么不同
 1 i want to abroad
 1 those are good men
 1 we are good men

检查的时候，不考虑前4个字符，这样whom have a try 就和 you have a try 就一样了。

[zhangy@BlackGhost mytest]$ uniq -u uniqtest
i am tank
this is a test
whom have a try
WhoM have a try
you have a try
i want to abroad
those are good men
we are good men

[zhangy@BlackGhost mytest]$ uniq -u uniqtest   
i am tank   
this is a test   
whom have a try   
WhoM have a try   
you  have a try   
i want to abroad   
those are good men   
we are good men

去重复的项，然后全部显示出来

[zhangy@BlackGhost mytest]$ uniq -w 2 -c uniqtest
 3 this is a test
 3 i am tank
 1 this is a test
 1 whom have a try
 1 WhoM have a try
 1 you have a try
 1 i want to abroad
 1 those are good men
 1 we are good men

[zhangy@BlackGhost mytest]$ uniq -w 2 -c uniqtest   
 3 this is a test   
 3 i am tank   
 1 this is a test   
 1 whom have a try   
 1 WhoM have a try   
 1 you  have a try   
 1 i want to abroad   
 1 those are good men   
 1 we are good men  

对每行第2个字符以后的内容不作检查，所以i am tank 根 i love tank就一样了。