Cause if this I/O issue, both the databases were down.The SysAdmin team, dismounted and then mounted back these two disk,and the I/O issue was no more.It was time to startup the database.One of the database started up successfully.The other one, started throwing ORA-00600: [4194], [38], [20]
Successfully onlined Undo Tablespace 29. Sun May 16 01:52:16 2010 SMON: enabling tx recovery Sun May 16 01:52:16 2010 Database Characterset is WE8ISO8859P1 Sun May 16 01:52:17 2010 Errors in file /disk1/oracle/admin/idcdb/udump/idc_ora_22161.trc: ORA-00600: internal error code, arguments: [4194], [38], [20], [], [], [], [], [] Doing block recovery for file 66 block 213923 Block recovery from logseq 15340, block 64 to scn 5970802072110 Sun May 16 01:52:20 2010 Recovery of Online Redo Log: Thread 1 Group 4 Seq 15340 Reading mem 0 Mem# 0: /disk1/oradata/redo/redolog6.log Mem# 1: /disk1/oradata/redo/redomlplx/redolog6a.log Block recovery stopped at EOT rba 15340.66.16 Block recovery completed at rba 15340.66.16, scn 1390.797530529 Doing block recovery for file 66 block 361 Block recovery from logseq 15340, block 64 to scn 5970802071968 Sun May 16 01:52:20 2010 Recovery of Online Redo Log: Thread 1 Group 4 Seq 15340 Reading mem 0 Mem# 0: /disk1/oradata/redo/redolog6.log Mem# 1: /disk1/oradata/redo/redomlplx/redolog6a.log Block recovery completed at rba 15340.66.16, scn 1390.797530529 replication_dependency_tracking turned off (no async multimaster replication found) Starting background process QMNC QMNC started with pid=20, OS id=23281 Sun May 16 01:52:25 2010 Completed: ALTER DATABASE OPENAfter few minutes, the instance was terminated by PMON process
Errors in file /disk1/oracle/admin/idcdb/bdump/idc_mmon_21649.trc: ORA-00600: internal error code, arguments: [4194], [38], [20], [], [], [], [], [] ORA-00600: internal error code, arguments: [4194], [38], [20], [], [], [], [], [] ORA-00600: internal error code, arguments: [4194], [38], [20], [], [], [], [], [] Sun May 16 01:52:31 2010 Errors in file /disk1/oracle/admin/idcdb/bdump/idc_pmon_21599.trc: ORA-00600: internal error code, arguments: [4194], [9], [5], [], [], [], [], [] Sun May 16 01:52:31 2010 Errors in file /disk1/oracle/admin/idcdb/udump/idc_ora_24006.trc: ORA-00600: internal error code, arguments: [4194], [54], [39], [], [], [], [], [] Sun May 16 01:52:32 2010 Errors in file /disk1/oracle/admin/idcdb/bdump/idc_pmon_21599.trc: ORA-00600: internal error code, arguments: [4194], [9], [5], [], [], [], [], [] PMON: terminating instance due to error 472 Sun May 16 01:52:32 2010 DEBUG: Replaying xcb 0x47fb9e970, pmd 0x47fde0728 for failed op 8 Doing block recovery for file 66 block 293914 Block recovery from logseq 15340, block 76 to scn 5970802072130 Sun May 16 01:52:32 2010 Recovery of Online Redo Log: Thread 1 Group 4 Seq 15340 Reading mem 0 Mem# 0: /disk1/oradata/redo/redolog6.log Mem# 1: /disk1/oradata/redo/redomlplx/redolog6a.log Block recovery completed at rba 15340.78.16, scn 1390.797530691 Sun May 16 01:52:38 2010 Instance terminated by PMON, pid = 21599As per MOS
DESCRIPTION: A mismatch has been detected between Redo records and rollback (Undo) records.Now, it was time to come in action
We are validating the Undo record number relating to the change being applied against the maximum undo record number recorded in the undo block.
This error is reported when the validation fails.
ARGUMENTS:
Arg [a] Maximum Undo record number in Undo block
Arg [b] Undo record number from Redo block
FUNCTIONALITY:
Kernel Transaction Undo called from Cache layer
IMPACT:
PROCESS FAILURE
POSSIBLE ROLLBACK SEGMENT CORRUPTION
SUGGESTIONS:This error may indicate a rollback segment corruption.
This may require a recovery from a database backup depending on the situation.
. Below is the workaround steps :-1. Startup database in mount mode
SQL> startup mount ORACLE instance started. Total System Global Area 4294967296 bytes Fixed Size 2078264 bytes Variable Size 4160752072 bytes Database Buffers 117440512 bytes Redo Buffers 14696448 bytes Database mounted.2. Check the values for Undo related parameter
SQL> show parameter undo NAME TYPE VALUE ------------------------------------ ----------- ------------------------------ undo_management string AUTO undo_retention integer 900 undo_tablespace string UNDOTBS13. Change the UNDO_MANAGEMENT to ‘MANUAL’
SQL> SQL> alter system set undo_management='MANUAL' scope=spfile;4. As, the above parameter is a static one, bounce the database
SQL> shu immediate; ORA-01109: database not open Database dismounted. ORACLE instance shut down. SQL> startup ORACLE instance started. Total System Global Area 4294967296 bytes Fixed Size 2078264 bytes Variable Size 4160752072 bytes Database Buffers 117440512 bytes Redo Buffers 14696448 bytes Database mounted. Database opened.5. In the meantime, keep checking the alert log.Check if there is some error after opening the database successfully with undo_management set to manual.
6.If everything seems fine, then go ahead and create a new undo tablespace.
SQL> create undo tablespace UNDOTBS datafile '/disk1/oradata/idcdb/undotbs1.dbf' size 1G autoextend on; Tablespace created. SQL> show parameter undo NAME TYPE VALUE ------------------------------------ ----------- ------------------------------ undo_management string MANUAL undo_retention integer 900 undo_tablespace string UNDOTBS1 SQL>7. Set the undo_tablespace to the new Undo Tablespace name and undo_management back to ‘AUTO’
SQL> alter system set undo_tablespace='UNDOTBS' scope=spfile; System altered. SQL> alter system set undo_management='AUTO' scope=spfile; System altered.8. Bounce the database
SQL> shu immediate; Database closed. Database dismounted. ORACLE instance shut down. SQL> startup ORACLE instance started. Total System Global Area 4294967296 bytes Fixed Size 2078264 bytes Variable Size 4160752072 bytes Database Buffers 117440512 bytes Redo Buffers 14696448 bytes Database mounted. Database opened.9. Check the Alert log for if any errors and check the undo parameters
SQL> show parameter undo NAME TYPE VALUE ------------------------------------ ----------- ------------------------------ undo_management string AUTO undo_retention integer 900 undo_tablespace string UNDOTBSAfter this the database was open successfully without any error.
0 comments:
Post a Comment