SQL重复记录查询(转载) ,及对大数据量的重复数据优化

1、查找表中多余的重复记录，重复记录是根据单个字段（peopleId）来判断
select * from people
where peopleId in (select peopleId from people group by peopleId having count(peopleId) > 1)

例二：

select * from testtable

where numeber in (select number from people group by number having count(number) > 1 )

可以查出testtable表中number相同的记录

2、删除表中多余的重复记录，重复记录是根据单个字段（peopleId）来判断，只留有rowid最小的记录
delete from people
where peopleId in (select peopleId from people group by peopleId having count(peopleId) > 1)
and rowid not in (select min(rowid) from people group by peopleId having count(peopleId )>1)

3、查找表中多余的重复记录（多个字段）
select * from vitae a
where (a.peopleId,a.seq) in (select peopleId,seq from vitae group by peopleId,seq having count(*) > 1)

4、删除表中多余的重复记录（多个字段），只留有rowid最小的记录
delete from vitae a
where (a.peopleId,a.seq) in (select peopleId,seq from vitae group by peopleId,seq having count(*) > 1)
and rowid not in (select min(rowid) from vitae group by peopleId,seq having count(*)>1)


5、查找表中多余的重复记录（多个字段），不包含rowid最小的记录
select * from vitae a
where (a.peopleId,a.seq) in (select peopleId,seq from vitae group by peopleId,seq having count(*) > 1)
and rowid not in (select min(rowid) from vitae group by peopleId,seq having count(*)>1)

(二)
比方说
在A表中存在一个字段“name”，
而且不同记录之间的“name”值有可能会相同，
现在就是需要查询出在该表中的各记录之间，“name”值存在重复的项；
Select Name,Count(*) From A Group By Name Having Count(*) > 1

如果还查性别也相同大则如下:
Select Name,sex,Count(*) From A Group By Name,sex Having Count(*) > 1


(三)
方法一

declare @max integer,@id integer

declare cur_rows cursor local for select 主字段,count(*) from 表名 group by 主字段 having count(*) >； 1

open cur_rows

fetch cur_rows into @id,@max

while @@fetch_status=0

begin

select @max = @max -1

set rowcount @max

delete from 表名 where 主字段 = @id

fetch cur_rows into @id,@max

end

close cur_rows

set rowcount 0

方法二

　　有两个意义上的重复记录，一是完全重复的记录，也即所有字段均重复的记录，二是部分关键字段重复的记录，比如Name字段重复，而其他字段不一定重复或都重复可以忽略。

1、对于第一种重复，比较容易解决，使用

select distinct * from tableName

就可以得到无重复记录的结果集。

如果该表需要删除重复的记录（重复记录保留1条），可以按以下方法删除

select distinct * into #Tmp from tableName

drop table tableName

select * into tableName from #Tmp

drop table #Tmp

发生这种重复的原因是表设计不周产生的，增加唯一索引列即可解决。

2、这类重复问题通常要求保留重复记录中的第一条记录，操作方法如下

假设有重复的字段为Name,Address，要求得到这两个字段唯一的结果集

select identity(int,1,1) as autoID, * into #Tmp from tableName

select min(autoID) as autoID into #Tmp2 from #Tmp group by Name,autoID

select * from #Tmp where autoID in(select autoID from #tmp2)

最后一个select即得到了Name，Address不重复的结果集（但多了一个autoID字段，实际写时可以写在select子句中省去此列）

(四)

查询重复

select * from tablename where id in (

select id from tablename

group by id

having count(id) > 1

)

============================分割线,以下是优化SQl==============================================

需要从表A中取得所有字段b相同的数据
如下，该SQL用于查询出数据库中某表的所有重复记录!

比如表A

—————————————————————— 
ID USER     PASS 
—————————————————————— 
1 test1        a 
2 test2        b 
3 test3        c 
4 test3        cc 
5 test3        ccc 
6 test4      d 
7 test5      d 
8 test5      e 
—————————————————————— 

 
用什么方法找出其中USER重复的数据，即

—————————————————————— 
ID USER     PASS 
—————————————————————— 
3    test3     c 
4    test3     cc 
5    test3     ccc 
7    test5     d 
8    test5     e 

 

这只是一个例子，如果表中的数据有几百万条，如何提高查询的效率

SQL语句写了很多种了（如上面所转载的），但速度都很慢。
例如如下SQL，在表中数据量很少时执行速度还是很快的，但是当数据量打到50W时，我至今没有等到它的返回结果....

select id,user,pass 
from A where user in (
 select user  from A   group by user having count(user)>1); 

 
方法一：再ORACLE的环境下
可以考虑用EXISTS条件语句代替IN语句 （速度有所提升）

SELECT A1.ID, A1.USER, A1.PASS    
FROM A A1   
WHERE EXISTS   
(   
    SELECT 1    
   FROM A A2   
   WHERE A1.USER = A2.USER  
    AND A1.ID <> A2.ID   
)  
----------------------------------------------------------
SELECT A1.ID, A1.USER, A1.PASS 
FROM A A1 
WHERE EXISTS 
( 
    SELECT 1 
    FROM A A2 
    WHERE A1.USER = A2.USER 
    AND A1.ID <> A2.ID 
) 

 



方法二：如果是BATCH或者SP，可以考虑作一个如下的中间表B

CREATE TABLE B   
AS    
SELECT    
 USER, COUNT(*) AS USER_COUNT   
FROM A   
GROUP BY USER    
HAVING COUNT(*) > 1   
-------------------------------------
ALTER TABLE B   
 ADD PRIMARY KEY (USER)   
------------------------------------
CREATE TABLE B 
AS 
SELECT 
    USER, COUNT(*) AS USER_COUNT 
FROM A 
GROUP BY USER 
HAVING COUNT(*) > 1 
---------------------------------------
ALTER TABLE B 
  ADD PRIMARY KEY (USER) 

 

然后用以下SQL进行检索

SELECT A.*   FROM A   INNER JOIN B ON  
A.USER = B.USER  

这条执行的效率怎样？

select user    
         from A    
         group by user having count(user)>1  
select user 
          from A 
          group by user having count(user)>1 

 
还有重复的几率有多大，如果超过一半的话，优化死也优化不到哪里去

方法三：可以用left join试试（mysql测试通过），把重复记录做为主表left join 表 

select  A.id, A.user, A.pass    
from  
       (select user    
            from A    
           group by user having count(user)>1   
       ) AS B left join A on B.user=A.user  

 
这几种方法中，根据不同的环境进行优化了，如果重复记录比较大的情况下，推荐使用临时表，但有时增加临时表又需要进行维护，第三种方法left join也是很好的选择了