数据库表被锁死

  3月3日下午，系统发生了严重故障，后台程序大量抛出如下异常：

DBCP object created 2011-03-03 12:35:19 by the following code was never closed:
java.lang.Exception
        at org.apache.commons.dbcp.AbandonedTrace.setStackTrace(AbandonedTrace.java:157)
        at org.apache.commons.dbcp.AbandonedObjectPool.borrowObject(AbandonedObjectPool.java:76)
        at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:95)
        at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:540)
        at org.springframework.orm.hibernate3.LocalDataSourceConnectionProvider.getConnection(LocalDataSourceConnectionProvider.java
:82)
        at org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:423)
        at org.hibernate.jdbc.ConnectionManager.getConnection(ConnectionManager.java:144)
        at org.hibernate.jdbc.JDBCContext.connection(JDBCContext.java:119)
        at org.hibernate.transaction.JDBCTransaction.begin(JDBCTransaction.java:57)
        at org.hibernate.impl.SessionImpl.beginTransaction(SessionImpl.java:1326)
        at org.springframework.orm.hibernate3.HibernateTransactionManager.doBegin(HibernateTransactionManager.java:509)
        at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionMana
ger.java:377)
        at org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSuppor
t.java:263)
        at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:101)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
        at org.springframework.aop.framework.adapter.AfterReturningAdviceInterceptor.invoke(AfterReturningAdviceInterceptor.java:50)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
        at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:50)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:89)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
        at $Proxy44.saveContentSync(Unknown Source)
        at com.wondertek.mobiletv.sync.SyncCMSDataSQLOperate.saveObjectBean(SyncCMSDataSQLOperate.java:1

运维人员初步认定是连接池耗尽引起的，重启系统几次，每次过几分钟之后又出同样问题。通过google,网上提供解决方案如下:

引用

分析：
这是removeAbandoned=true导致的，使用这个配置的时候将会使用AbandonedObjectPool．
而AbandonedObjectPool只在开发阶段使用，因为AbandonedObjectPool能帮你发现占用连接过长的代码．

log信息说明了getCustomerOnline占用连接超过removeAbandonedTimeout设定的时间．所以设置removeAbandoned=false就行了。
AbandonedObjectPool只在开发使用，在后续版本会去掉，现在api都deprecated了

我们参照此方法修改配置文件，重启后还是没有解决问题。运营组那边报过来说模板一直不能保存，也不能给栏目分配模板。我们通过本地运行系统，直接连接生产数据库进行调试，同样不能保存模板。这样得出一个结论系统没有问题，数据库出问题。通过debug,代码里调用下列语句的时候一直过不去。

select body from template_file where tplid=9441 for update

用这个语句在plsql里模拟调用，一直处于等待之中。这样得出结论template_file表被锁了。
用下列语句查行锁定情况,并进行解锁。

select b.owner,b.object_name,l.session_id,l.locked_mode 
from v$locked_object l, dba_objects b 
where  b.object_id=l.object_id and l.LOCKED_MODE > 0 
and (b.OBJECT_NAME = 'NODE_TEMPLATE' or b.OBJECT_NAME = 'CONTENT_TEMPLATE'
 or b.OBJECT_NAME = 'TEMPLATE_FILE')

select t2.sid,t2.serial#
from v$locked_object t1,v$session t2 
where t1.session_id=t2.sid order by t2.logon_time;
--解锁
alter system kill session 'sid,serial' alter system kill session '146,21177';

将所有的锁打开后几分钟后，还是没有改善，又产生很多锁。最后经过一段时间，因为业务量少了，锁自动解开了，系统恢复正常。

事后通过生成awr report查看数据库的健康情况。


可以看出enq: TM - contention等待事件很高。网上说外键约束上没有建立索引引起的问题。

SELECT * FROM (
   SELECT c.table_name, cc.column_name, cc.position column_position
   FROM   user_constraints c, user_cons_columns cc
   WHERE  c.constraint_name = cc.constraint_name
   AND    c.constraint_type = 'R'
   MINUS
   SELECT i.table_name, ic.column_name, ic.column_position
   FROM   user_indexes i, user_ind_columns ic
   WHERE  i.index_name = ic.index_name
   )
ORDER BY table_name, column_position;

同时从awr report上发现了很多执行效率很差的语句，经过了优化，并对调用频率很高的表建立了索引，系统性能得到了很大的提高 。

引用

最近一个客户反映数据库在执行远程INSERT的时候应用反映很慢，在做AWR后发现enq: TM - contention等待事件很高，应该是外键约束上没有建立索引的问题。
下面我来做个实验还原一下当时的现象。

CREATE TABLE supplier
    (     supplier_id     number(10)     not null,
        supplier_name     varchar2(50)     not null,
        contact_name     varchar2(50),   
        CONSTRAINT supplier_pk PRIMARY KEY (supplier_id)
    );

INSERT INTO supplier VALUES (1, 'Supplier 1', 'Contact 1');
INSERT INTO supplier VALUES (2, 'Supplier 2', 'Contact 2');
COMMIT;

CREATE TABLE product
    (     product_id     number(10)     not null,
        product_name    varchar2(50)    not null,
        supplier_id     number(10)     not null,
        CONSTRAINT fk_supplier
          FOREIGN KEY (supplier_id)
         REFERENCES supplier(supplier_id)
         ON DELETE CASCADE
    );

INSERT INTO product VALUES (1, 'Product 1', 1);
INSERT INTO product VALUES (2, 'Product 2', 1);
INSERT INTO product VALUES (3, 'Product 3', 2);
COMMIT;

然后去执行几条相关的语句
User 1: DELETE supplier WHERE supplier_id = 1;
User 2: DELETE supplier WHERE supplier_id = 2;(现象HANG住)
User 3: INSERT INTO supplier VALUES (5, 'Supplier 5', 'Contact 5');(现象HANG住)

现在检查锁的情况

col event format a20
col type format a10
col object_name a15
col object_type a15
SELECT l.sid, s.blocking_session blocker, s.event, l.type, l.lmode, l.request, o.object_name, o.object_type
FROM v$lock l, dba_objects o, v$session s
WHERE UPPER(s.username) = UPPER('&User')
AND   l.id1        = o.object_id (+)
AND   l.sid        = s.sid
ORDER BY sid, type;
输入 user 的值:  test
原值    3: WHERE UPPER(s.username) = UPPER('&User')
新值    3: WHERE UPPER(s.username) = UPPER('test')

       SID    BLOCKER EVENT                TYPE            LMODE    REQUEST
---------- ---------- -------------------- ---------- ---------- ----------
OBJECT_NAME     OBJECT_TYPE
--------------- ---------------
       144        158 enq: TM - contention TM                  3          0
SUPPLIER        TABLE

       144        158 enq: TM - contention TM                  0          2
PRODUCT         TABLE

       153            SQL*Net message from TM                  3          0
                       client
SUPPLIER        TABLE

       SID    BLOCKER EVENT                TYPE            LMODE    REQUEST
---------- ---------- -------------------- ---------- ---------- ----------
OBJECT_NAME     OBJECT_TYPE
--------------- ---------------

       153            SQL*Net message from TM                  3          0
                       client
PRODUCT         TABLE

       153            SQL*Net message from TX                  6          0
                       client



       SID    BLOCKER EVENT                TYPE            LMODE    REQUEST
---------- ---------- -------------------- ---------- ---------- ----------
OBJECT_NAME     OBJECT_TYPE
--------------- ---------------
       158        153 enq: TM - contention TM                  0          5
PRODUCT         TABLE

       158        153 enq: TM - contention TM                  3          0
SUPPLIER        TABLE

已选择7行。

SQL>

能看到enq: TM - contention TM  等待事件

检查没有索引的外键

SELECT * FROM (
   SELECT c.table_name, cc.column_name, cc.position column_position
   FROM   user_constraints c, user_cons_columns cc
   WHERE  c.constraint_name = cc.constraint_name
   AND    c.constraint_type = 'R'
   MINUS
   SELECT i.table_name, ic.column_name, ic.column_position
   FROM   user_indexes i, user_ind_columns ic
   WHERE  i.index_name = ic.index_name
   )
ORDER BY table_name, column_position;
TABLE_NAME
------------------------------
COLUMN_NAME
--------------------------------------------------------------------------------

COLUMN_POSITION
---------------
PRODUCT
SUPPLIER_ID
              1

建立索引后：

CREATE INDEX fk_supplier ON product (supplier_id);

INSERT INTO supplier VALUES (6, 'Supplier 6', 'Contact 6');

INSERT INTO supplier VALUES (7, 'Supplier 7', 'Contact 7');

User 1: DELETE supplier WHERE supplier_id = 6;
User 2: DELETE supplier WHERE supplier_id = 7;
User 3: INSERT INTO supplier VALUES (8, 'Supplier 8', 'Contact 8');

相关的等待事件消失。

awr入门
http://blog.csdn.net/netbios333/archive/2009/06/12/4264114.aspx