# /opt/oracle.SupportTools/CheckHWnFWProfile
[WARNING] The hardware and firmware are not supported. See details below

[PCISlot:HBA:LSIModel:LSIhw:MPThw:LSIfw:MPTBios:DOM:OSDevice:DOMMake:DOMModel:DOMfw:CountAuraCountDOM]
Requires:
   AllSlots_AllHBAs SAS1068E B3orC0 105 011b5c00 06.26.00.00 AllDOMs_NotApplicable MARVELL SD88SA02 D20Y 4_16
Found:
   AllSlots_AllHBAs SAS1068E B3orC0 105 011b5b00 06.26.00.00 AllDOMs_NotApplicable MARVELL SD88SA02 D20Y 4_16

[WARNING] The hardware and firmware are not supported. See details above

CheckHWnFWProfile script is expecting firmware version to be 011b5c00 where it is found to be 011b5b00, so it reports the check failure. This is due to BUG 13088963. All flash card firmware versions are not being upgraded due to bug in CheckHWnFwProfile  script, which is also responsible for firmware upgrade on the cells.

MOS Note 1372320.1  - Problem : [Warning] The Hardware And Firmware Are Not Supported on 11.2.2.4

There is a patch 13089037 available for this problem. This patch also fixes following bugs.

This update will:

• Update ibdiagtools to latest revision (Bug 13089037).
•  STORAGE EXPANSION RACK SUPPORT AMISS IN IBDIAGTOOLS IN 11.2.2.4.0
• Correct the flash firmware update regression (Bug 13088963).
•  CHECKHWNFWPROFILE HARDWARE/FIRMWARE NOT SUPPORTED REQUIRES LSIFW 011B5C00
• Correct the missing support for X4800 for special bios update package (Bug 13089050)
•  FLASH_BIOS PACKAGE DOES NOT WORK FOR X4800* MODELS

Critical Issues Discovered Post-Release

1) Bug 13083530 - 10Gb Ethernet network interfaces shut down unexpectedly.

For environments configured with 10GigE Ethernet for the database server hosts running Oracle Linux, do NOT apply the 11.2.2.4 minimal pack to the database server hosts. A loss of connectivity problem for 10GigE was reported and confirmed. An update on this will be available soon and will be published as an update to the patch README and the corresponding patch MOS Note. Customers who are not already running with 11.2.2.3.5 on their compute nodes are recommended to applying the 11.2.2.3.5 minimal pack (available in Patch 12849110) until further updates are available.

Apply the 11.2.2.4 cell patch as usual. The cells do not use 10GigE. It is supported to run with the latest 11.2.2.4 version on the cells together with an earlier minimal pack release.

For example for compute node, please note we navigate into dl380 for compute node

# tar -pxvf computeImageMaker_11.2.1.2.3_LINUX.X64_100305-1.x86_64.tar
# cd dl380
# makeImageMedia.sh computeNodeImage.iso

Or for cell node, please note we navigate into dl180 directory for cell.

# tar -pxvf cellImageMaker_11.2.1.2.3_LINUX.X64_100305-1.x86_64.tar
# cd dl180
# makeImageMedia.sh cellImage.iso

Would it not have been nice, if one could just mount this iso file via ILOM as virtual cdrom, change the boot order in BIOS by booting into bios, which can also be force either via ILOM gui or via /usr/bin/biosconfig -set_boot_override <xmlfile> command, and choose virtual cdrom as first boot device. But problem with this approach is cell/compute node imaging process resets the ILOM, which means that even our virtual cdrom iso image is also removed during this process and this results in imaging process not completing properly.

One could try leaving a external USB permanently attached on lab exadata, and then via ILOM try to reimage, so that during ILOM reset boot image device is not detached and once can simply reconnect to ILOM to continue answering the on-screen messages. But one problem with this approach is, as part of imaging one has to remove the external USB stick otherwise automated configuration scripts dont funtion properly. I tried it on the lab exadata and could get the cell to reimage properly with external USB still attached. Even after manually changing the boot order to internal USB and then harddrive, which is what cell checks for during each reboot validation, cell just sat on a blinking prompt without going forward. I left the cell overnight as well, thinking that properly some processing is going on, but with no luck. Once i removed the external USB stick, automated configuration scripts were able to properly complete the imaging process and came to automated run of ipconf script, which set the various setting on the cell at first boot after reimaging process.

Would it have been nice, it all this could be done remotely via ILOM once ilom has been connected to the network, as ilom network configuration is not reset during re-imaging process. But that would be wishful thinking !!!

I was trying this with 11.2.1.2.3 image, as that was my starting image on compute/cell nodes in prod/dr/test, so wanted to replicated the same history. I have not tried with later image versions yet to see if process has been improved in this regard, that trial would be for some time later.

Regards,

Vishal

Non-interactive shell issue for Database Host minimal pack 

Recently i set about patching Exadata Storage Server software to from 11.2.2.x.x to 11.2.2.4.0, which is the latest patch from Oracle Corporation. I was testing and documenting the process for one of my client and wanted to automate this as much as possible, as in past people actually executing the commands had missed running few commands on certain nodes. As with any Exadata storage server software patch, there is cell node component of patch which is patched using patchmgr either in rolling or non-rolling fashion. And there is database host component, called database minimal pack. Release note of 11.2.2.4.0 asks the install the patch (after running some prerequisites) using ./install.sh -force option. I was taking the approach of install patch on one cell node and if successful, then apply on rest of the cell nodes in parallel. Similarly apply the database minimal pack patch on one compute node, then if successful, apply it on rest of the compute nodes in parallel. And what could be more convenient to run command in parallel on many nodes than dcli command. So i programatically created an dbs_group_withoutfirstnode file with all the compute nodes apart from first compute node. Then installed the patch on first compute node, which was successful. After that using dcli i transferred the patch to other nodes, extracted its contents in parallel. Then using dcli command ran the (cd <patch_directory>; ./install.sh -force) command on rest of the compute nodes. But guess what, compute node patch does not like the running via dcli. DCLI simply runs the command on a remote host using “ssh command” method in simple terms. Though its slightly more complex. Effect of running command via dcli is that, all command are run in non-interactive session i.e. without tty terminal or standard output/error. It means that if your script is not redirecting all standard output and standard error messages to a file, then it will exit with a non-zero (i.e unsuccessful) exit code. install.sh script gives a call to dopatch.sh, which in turn calls a series of functions listed. As part of one of the function, it tries to set update the image version and adds it to image history. In this function, it tries to output the error messages explicitly to /dev/stderr device. As a result of this, if compute node patch is run via some automated script, it exits at this step and fails to run any further steps which include firmware update to ILOM and BIOS upgrade etc.

Now after this has happened, imageinfo command will show the new version, but there will be empty status and activation date. imagehistory will also not show the new image version. If you try to rollback the patch using ./install.sh -rollback-ib command, it will complain that version is not valid, as it is not set with success status. So if you try run /opt/oracle.cellos/imagestatus -set success , then it will complain. But you can force it by using /opt/oracle.cellos/imagestatus -set success -force db_patch. After this you will be able to use the rollback. And then you can install the patch again using an interactive shell.

grub.conf Symoblic link Issue

I also noticed that symbolic link /etc/grub.conf which points to /boot/grub/grub.conf is missing on OEL5.5 compute/cell nodes. OEL5.5 is installed starting with 11.2.1.3.1 cell image.

Suggestions for Oracle Exadata Development

Exadata development team could write their upgrade/patching so that they are compatible with dcli, it allows to automated the patch and save lot of hassle.

Summary

- Don’t use non-interactive shell or dcli to run compute node patching commands.
- Check your /etc/grub.conf symbolic link exists which needs to point to /boot/grub/grub.conf.

Hopefully this will save some hassle to someone out there patching production Exadata’s.

[Update, 05-Nov-2011]

One can redirect all the standard output and standard error to a file, then it will be possibile to run install.sh to install compute minimal patch via dcli.

cd /opt/oracle.Support/onecommand/
dcli -l root -g dbs_group "mkdir -p /opt/oracle.Support/onecommand/patches/patch_11.2.2.4.0.110929"

# Transfer the compute node minimal patch file
dcli -l root -g dbs_group -d /opt/oracle.Support/onecommand/patches/patch_11.2.2.4.0.110929/ -f /opt/oracle.Support/onecommand/patches/patch_11.2.2.4.0.110929/db_patch_11.2.2.4.0.110929.zip

# Unzip the compute node patch file
dcli -l root -g dbs_group  "(cd /opt/oracle.Support/onecommand/patches/patch_11.2.2.4.0.110929/; unzip -o db_patch_11.2.2.4.0.110929.zip)"

# Run the compute node patch
dcli -l root -g dbs_group "(cd /opt/oracle.Support/onecommand/patches/patch_11.2.2.4.0.110929/db_patch_11.2.2.4.0.110929 ; ./install.sh >> install.sh.log 2>&1)"

Cheers,
Vishal Gupta

I feel if more and more people know about Exadata, they would be able to advocate about Exadata and try to convince their company’s decision makers into buying Exadata machines. So making Exadata documentation publicly available should help oracle rather than their competitors. Also a knowledgable dba would be able to manage Exadata better than un-knowledgable dba.

Many Oracle DBA managing Exadata are not aware that Exadata documentation is available on every cell under /usr/share/doc/oracle/Exadata/doc folder. But if you dont have access to a Exadata cell, then how do you view the Exadata documentation? Well you have few options for that.

Ways to obtain Exadata Documentation

1.  Ask your friendly DBA/SystemAdmin who has access to Exadata cell to give your entire (including sub-directories) contents of /usr/share/doc/oracle/Exadata/doc folder. Then using your favourite internet browser open the index.html file to view documentation.

2.  Extract the exadata documentation from cell images. You don’t have to install the Exadata Storage server software to obtain its documentation.

1. Create an Oracle Web Account at Oracle Technology Network aka OTN . Account creation direct link.
2. Wait for account validation email to come, click on link to verify your Oracle single sign on account.
3. Go to Oracle eDelivery site, now know as Oracle Software Delivery Cloud at http://eDelivery.oracle.com and login using your Oracle Web account.
4. Under Media pack search, select product pack as “Oracle Database” and platform as “Linux x86-64″ and hit Go.
5. It will list the Oracle Exadata Storage Server (CELL) Software Media pack links, download the storage server software. 11.2.2 media pack contains all 11.2.2.x.x cell and compute images.
6.  Extract the files.
7. For older cell images (upto 11.2.2.3.3), you will find the cellImageMaker_<imageversion>.x86-64.tar.zip.
8. Extract the tar file from zipfile. In older cell images (upto 11.2.2.2.2), it will prompt you for a password which can be obtained from Oracle support. This password is common on all cellimage zipfiles. In newer cellimage from 11.2.2.3.2 onwards, zipfile is no longer password protected.
9. On any linux/unix machine extract the Exadata documentation from tar file using “tar -pxvf cellImageMaker_<imageversion>.x86-64.tar dl180/boot/cellbits/doclib.zip” command. This extracts the doclib.zip file containing Exadata documentation for that cell image version.

Disclaimer – Please ensure that you or your company has the Exadata software license before obtaining the Exadata documentation, as it is Oracle’s copyright protected material and only meant to exadata license holders. I will not be liable for any license or copyright breaches due to you or your company not having a valid license before obtaining the Exadata documentation.

Just a quick note to say that RedHat submitted certification test result of Oracle 11gR2 on RHEL6 to Oracle Corporation on 09-Aug-2011, so we should expect the formal certification around last week of Sep-2011.

More news at - http://www.redhat.com/about/news/blog/Red-Hat-Submits-Oracle-11gR2-on-Red-Hat-Enterprise-Linux-6-Certification-Test-Results-to-Oracle

In 11g, full table scan do not always go through the buffer cache. Decision to read via direct path or through cache is based on the size of the table, buffer cache and various other stats. Table is considered to be small or large based value of _small_table_threshold internal parameter.  Default value of this parameter is 2% of buffer cache size and is specified in blocks. This means any object (table) smaller than 2% of the buffer cache will be read via the buffer cache and not using direct path read. And tables larger than 2% of the buffer cache are read via direct path read and not via buffer cache. With AMM (Automatic Memory Management) or ASMM (Automatic Shared Memory Management), buffer cache could drop to a lower value if memory is falling short for shared pool. In such a case after restart of instance, _small_table_threshold parameter would become even lower due to decreased buffer cache.

By enabling event 10949 at either session or system/instance level one can disable the autotuning of direct path reads for full table scans. It means full table scan on every table whether small or large, will go via buffer cache. This could flush out the already cached cached objects. I would strongly advise against setting this for all the databases in our standard build.

[/source]SQL> alter system set events '10949 trace name context forever' scope=spfile;

$ oerr ora 10949 10949, 00000, "Disable autotune direct path read for full table scan"
 // *Cause:
 // *Action: Disable autotune direct path read for serial full table scan.

If you are doing full table scans on the same LARGE table too often, then I would suggest to tune the query or create an index on the table. You could also set the minimum value for db_cache_size, so that it does not fall below the minimum (normal workload) levels and small_table_threshold parameter value does not fall below desired threshold.

Related Oracle MOS Notes

Doc ID 793845.1 – High ‘direct path read’ waits in 11g

Doc ID 787373.1 – How does Oracle load data into the buffer cache for table scans ?

http://asanga-pradeep.blogspot.com/2011/06/restore-point-prerequisites-change-in.html

I have raised a Oracle Service Request to correct the documentation. Documentation Bug – 12763133.

[Update 19-Jul-2011]

Following bug has been raised by Oracle Support.

Bug 12768808: FLASHBACK DATABASE DO NOT REQUIRE TO BE IN MOUNT MODE FROM 112 ONWARDS

Excerpt from 11gR2 documentation, SQL Reference for “ALTER DATABASE” statement  (as of 18-Jul-2011) - Click Here

flashback_mode_clause

Use this clause to put the database in or take the database out of FLASHBACK mode. You can specify this clause only if the database is in ARCHIVELOG mode and you have already prepared a fast recovery area for the database.You can specify this clause when the database is mounted but not open. This clause cannot be specified on a physical standby database if redo apply is active.

See Also:

Oracle Database Backup and Recovery User’s Guide for information on preparing the fast recovery area for Flashback operations

FLASHBACK ON Use this clause to put the database in FLASHBACK mode. When the database is in FLASHBACK mode, Oracle Database automatically creates and manages Flashback Database logs in the fast recovery area. Users with SYSDBA system privilege can then issue a FLASHBACK DATABASE statement.

FLASHBACK OFF Use this clause to take the database out of FLASHBACK mode. Oracle Database stops logging Flashback data and deletes all existing Flashback Database logs. Any attempt to issue a FLASHBACK DATABASE will fail with an error.

11gR2

$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.1.0 Production on Mon Jul 18 04:56:02 2011

Copyright (c) 1982, 2009, Oracle.  All rights reserved.

Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

SQL> select flashback_on, log_mode, open_mode from v$database;

FLASHBACK_ON	   LOG_MODE	OPEN_MODE
------------------ ------------ --------------------
NO		   ARCHIVELOG	READ WRITE

SQL> alter database flashback on;

Database altered.

SQL> select flashback_on, log_mode, open_mode from v$database;

FLASHBACK_ON	   LOG_MODE	OPEN_MODE
------------------ ------------ --------------------
YES		   ARCHIVELOG	READ WRITE

SQL> alter database flashback off;

Database altered.

SQL> select flashback_on, log_mode, open_mode from v$database;

FLASHBACK_ON	   LOG_MODE	OPEN_MODE
------------------ ------------ --------------------
NO		   ARCHIVELOG	READ WRITE

11gR1

$ sqlplus / as sysdba

SQL*Plus: Release 11.1.0.6.0 - Production on Mon Jul 18 05:15:52 2011

Copyright (c) 1982, 2007, Oracle.  All rights reserved.

Connected to:
Oracle Database 11g Enterprise Edition Release 11.1.0.6.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

SQL> select flashback_on, log_mode, open_mode from v$database;

FLASHBACK_ON	   LOG_MODE	OPEN_MODE
------------------ ------------ ----------
NO		   ARCHIVELOG	READ WRITE

SQL> alter database flashback on;
alter database flashback on
*
ERROR at line 1:
ORA-38759: Database must be mounted by only one instance and not open.

SQL> shutdown immediate;
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL> startup mount;
ORACLE instance started.

Total System Global Area  217157632 bytes
Fixed Size		    2142976 bytes
Variable Size		  159386880 bytes
Database Buffers	   50331648 bytes
Redo Buffers		    5296128 bytes
Database mounted.
SQL> select flashback_on, log_mode, open_mode from v$database;

FLASHBACK_ON	   LOG_MODE	OPEN_MODE
------------------ ------------ ----------
NO		   ARCHIVELOG	MOUNTED

SQL> alter database flashback on;

Database altered.

SQL> select flashback_on, log_mode, open_mode from v$database;

FLASHBACK_ON	   LOG_MODE	OPEN_MODE
------------------ ------------ ----------
YES		   ARCHIVELOG	MOUNTED

SQL> alter database open;

Database altered.

SQL> select flashback_on, log_mode, open_mode from v$database;

FLASHBACK_ON	   LOG_MODE	OPEN_MODE
------------------ ------------ ----------
YES		   ARCHIVELOG	READ WRITE

SQL> alter database flashback off;

Database altered.

SQL> select flashback_on, log_mode, open_mode from v$database;

FLASHBACK_ON	   LOG_MODE	OPEN_MODE
------------------ ------------ ----------
NO		   ARCHIVELOG	READ WRITE

[oracle@linux2 ~]$ . oraenv
ORACLE_SID = [oracle] ? 11202
The Oracle base has been set to /opt/oracle/product/rhel4/database/11.2.0.2
[oracle@linux2 ~]$ rman
 

After entering above rman command, nothing happened. It just hung on the command prompt. I did not receive any error at all. Hmm… whats happening? I issued an CTRL-C to cancel it and tried again. Still same result. Okay lets trace it using Linux strace utility.

oracle@linux2 ~]$ strace rman
execve("/usr/X11R6/bin/rman", ["rman"], [/* 31 vars */]) = 0
uname({sys="Linux", node="linux2.vishalgupta.com", ...}) = 0
brk(0)                                  = 0x5a3000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2a95556000
access("/etc/ld.so.preload", R_OK)      = 0
open("/etc/ld.so.preload", O_RDONLY)    = 3
fstat(3, {st_mode=S_IFREG|0444, st_size=23, ...}) = 0
mmap(NULL, 23, PROT_READ|PROT_WRITE, MAP_PRIVATE, 3, 0) = 0x2a95557000
close(3)                                = 0
readlink("/proc/self/exe", "/usr/X11R6/bin/rman", 4096) = 19
open("/stub/lib64/libcwait.so", O_RDONLY) = 3
read(3, "\177ELF\2\1\1\3>\1\20\6"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=4176, ...}) = 0
mmap(NULL, 1051008, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x2a95558000
mprotect(0x2a95559000, 1046912, PROT_NONE) = 0
mmap(0x2a95658000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0) = 0x2a95658000
close(3)                                = 0
munmap(0x2a95557000, 23)                = 0
open("/opt/oracle/product/rhel4/database/11.2.0.2/home/lib/tls/x86_64/libc.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
stat("/opt/oracle/product/rhel4/database/11.2.0.2/home/lib/tls/x86_64", 0x7fbfffec30) = -1 ENOENT (No such file or directory)
open("/opt/oracle/product/rhel4/database/11.2.0.2/home/lib/tls/libc.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
stat("/opt/oracle/product/rhel4/database/11.2.0.2/home/lib/tls", 0x7fbfffec30) = -1 ENOENT (No such file or directory)
open("/opt/oracle/product/rhel4/database/11.2.0.2/home/lib/x86_64/libc.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
stat("/opt/oracle/product/rhel4/database/11.2.0.2/home/lib/x86_64", 0x7fbfffec30) = -1 ENOENT (No such file or directory)
open("/opt/oracle/product/rhel4/database/11.2.0.2/home/lib/libc.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
stat("/opt/oracle/product/rhel4/database/11.2.0.2/home/lib", {st_mode=S_IFDIR|0755, st_size=12288, ...}) = 0
open("/etc/ld.so.cache", O_RDONLY)      = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=146311, ...}) = 0
mmap(NULL, 146311, PROT_READ, MAP_PRIVATE, 3, 0) = 0x2a95659000
close(3)                                = 0
open("/lib64/tls/libc.so.6", O_RDONLY)  = 3
read(3, "\177ELF\2\1\1\3>\1p\305\1"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=1636072, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2a9567d000
mmap(NULL, 2330696, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x2a9567e000
mprotect(0x2a957ae000, 1085512, PROT_NONE) = 0
mmap(0x2a958ad000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x12f000) = 0x2a958ad000
mmap(0x2a958b3000, 16456, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x2a958b3000
close(3)                                = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2a958b8000
mprotect(0x2a958ad000, 16384, PROT_READ) = 0
mprotect(0x3e48a14000, 4096, PROT_READ) = 0
arch_prctl(ARCH_SET_FS, 0x2a958b86e0)   = 0
munmap(0x2a95659000, 146311)            = 0
brk(0)                                  = 0x5a3000
brk(0x5c4000)                           = 0x5c4000
fstat(0, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 1), ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2a95659000
read(0,  <unfinished ...>
[oracle@linux2 ~]$

Had to again issue a CTRL-c to come out of trace, as it was just hanging again. I tried connecting to database using sqlplus, that worked fine.

oracle@linux2 ~]$ sqlplus / as sysdba
SQL*Plus: Release 11.2.0.2.0 Production on Mon Jun 20 06:08:58 2011
Copyright (c) 1982, 2010, Oracle.  All rights reserved.

Connected to:

Oracle Database 11g Enterprise Edition Release 11.2.0.2.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
SQL>

Hmm… Lets recompile the Oracle binaries, it case something funny was going on with libraries. And then we will re-run our RMAN cli.

[oracle@linux2 tns_admin]$ relink
writing relink log to: /opt/oracle/product/rhel4/database/11.2.0.2/home/install/relink.log
[oracle@linux2 tns_admin]$ rman

Hmm… still the same.

It turns out the there is Linux RMAN command as well, which is installed by xorg-x11-devel rpm package.

[oracle@linux2 ~]$ which rman
/usr/X11R6/bin/rman
[oracle@linux2 bin]$ man rman
PolyglotMan(1)                                                                                                                                                    PolyglotMan(1)

NAME
       PolyglotMan, rman - reverse compile man pages from formatted form to a number of source formats

So lets run the actual Oracle RMAN command.

[oracle@linux2 bin]$ $ORACLE_HOME/bin/rman

Recovery Manager: Release 11.2.0.2.0 - Production on Mon Jun 20 07:14:58 2011

Copyright (c) 1982, 2009, Oracle and/or its affiliates.  All rights reserved.

RMAN>

So, moral of the story is set your PATH correctly. I hope someone will find it useful, in case their rman command is hung.

It gives following details and more.

• At Exadata level,
• number/model of compute nodes and cell nodes.
• number of Infiniband switches and their port count.
• Key capabilities ( as per data sheets)
• For each database compute node and cell node.
• Server Model
• CPU Type, CPU Speed, CPU Cores, CPU Threads
• RAM
• Network ports – type and count
• Flash RAM (Smart Cache)
• Disk types

I hope someone will find it useful.

Exadata Model Comparison Spreadsheet – link

Here is the list of schemas:-

ANONYMOUS
APEX_PUBLIC_USER
CTXSYS
DBSNMP
DIP
EXFSYS
FLOWS_%
FLOWS_FILES
LBACSYS
MDDATA
MDSYS
MGMT_VIEW
OLAPSYS
ORACLE_OCM
ORDDATA
ORDPLUGINS
ORDSYS
OUTLN
OWBSYS
SI_INFORMTN_SCHEMA
SPATIAL_CSW_ADMIN_USR
SPATIAL_WFS_ADMIN_USR
SYS
SYSMAN
SYSTEM
WKPROXY
WKSYS
WK_TEST
WMSYS
XDB
XS$NULL

On a RHEL OS using LVM volume manager, you can extend the underlying disk and its associated filesytem (non-root) can be extended while its mounted.

• Extend the underlying virtual/SAN disks.

• Check the size of disk

#   fdisk -l

• Force the re-scan of disk, so that OS kernel can see the new size

# echo 1 > /sys/block/sdb/device/rescan

where disk extended was /dev/sdb

• Extend LVM physical volume size.

# pvs

# pvresize /dev/sdb

# pvs

• Extend LVM logical volume

# lvdisplay

# lvextend /dev/ora_vg/ora /dev/sdb

# lvdisplay

• Extend filesystem (online)

# df –h

# resize2fs /dev/ora_vg/ora

# df –h

Filesystem is extended.

For this sometimes init.ora has following parameter

# cat $ORACLE_HOME/dbs/init<ORACLE_SID>.ora
SPFILE='+DATA/ORACLESID/spfileORACLESID.ora'
 

Now if you want to move spfile between ASM and filesystem, here how you can do it.

From ASM -> Filesystem

</span>
<pre>SQL> CREATE pfile='/tmp/initORACLESID.ora' from spfile='+DATA/ORACLESID/spfileORACLESID.ora';
SQL> CREATE SPFILE='/tmp/spfileORACLESID.ora' from pfile='/tmp/initORACLESID.ora';</pre>
<span style="font-family: Times New Roman; font-size: small;"> 

From Filesystem -> ASM

</span>
<pre>SQL> CREATE pfile='/tmp/initORACLESID.ora' from spfile='$ORACLE_HOME/dbs/spfileORACLESID.ora';
SQL> CREATE SPFILE='+DATA/ORACLESID/spfileORACLESID.ora' from pfile='/tmp/initORACLESID.ora';</pre>

# cat /etc/init.d/oracle
case $1 in
start)
 /opt/oracle/product/11.1.0/bin/dbstart;
 ;;
stop)
 /opt/oracle/product/11.1.0/bin/dbshut;
 ;;
esac

# ln -s /etc/init.d/Oracle /etc/rc.d/rc0.d/K01oracle
# ln -s /etc/init.d/Oracle /etc/rc.d/rc3.d/S99oracle
# ln -s /etc/init.d/Oracle /etc/rc.d/rc4.d/S99oracle
# ln -s /etc/init.d/Oracle /etc/rc.d/rc5.d/S99oracle
# ln -s /etc/init.d/Oracle /etc/rc.d/rc6.d/K01oracle

This works fine on non-Linux operating systems. But on Linux operating system, there is an requirement to for start/stop script to create/remove a /var/lock/subsys/ file, where is name of the file in /etc/init.d. This ensures that, when init is changing runlevel as part of server start and stop routine, a service which is already started does not start again. And a service which is already stopped does not stop again. If /var/lock/subsys/ is not created then inspite of K(ill) symbolic links being present in rc*.d folder for runlevel zero (halt) and 6 (reboot), stop script is not called. This means process are killed instead of being shutdown cleanly by your stop script.

On Linux Operating system, start/stop functions for a service init.d script should look like below. Important part is highlighted in red below

# chkconfig: 2345 99 01

# description: This script start/stop Oracle

case $1 in
start)
   /opt/oracle/product/11.1.0/bin/dbstart;
    touch /var/lock/subsys/Oracle;
    # where  is same as name of file in /etc/init.d
    ;;
stop)
    /opt/oracle/product/11.1.0/bin/dbshut;
    rm –f /var/lock/subsys/Oracle; 
    ;;
esac

# ln -s /etc/init.d/Oracle /etc/rc.d/rc0.d/K01oracle
# ln -s /etc/init.d/Oracle /etc/rc.d/rc3.d/S99oracle
# ln -s /etc/init.d/Oracle /etc/rc.d/rc4.d/S99oracle
# ln -s /etc/init.d/Oracle /etc/rc.d/rc5.d/S99oracle
# ln -s /etc/init.d/Oracle /etc/rc.d/rc6.d/K01oracle

This applies to not only oracle but all start/stop created for init.d on Linux.

This requirement to create subsystem lock file in /var/lock/subsys can be verified by looking at /etc/rc script, which is called everytime there is runlevel change. See below the portion of /etc/rc script which is checking for presense of /var/lock/subsys/ file before calling Kill/stop scripts.

# First, run the KILL scripts.
for i in /etc/rc$runlevel.d/K* ; do
   check_runlevel "$i" || continue

   # Check if the subsystem is already up.
   subsys=${i#/etc/rc$runlevel.d/K??}
   [ -f /var/lock/subsys/$subsys -o -f /var/lock/subsys/$subsys.init ] \
    || continue

   # Bring the subsystem down.
   if LC_ALL=C egrep -q "^..*init.d/functions" $i ; then
      $i stop
   else
      action $"Stopping $subsys: " $i stop
   fi
done

[Update:2009-05-03 : Jonathan Lewis mentioned in a email to me that version_outline gives version in which a particular hint can be used in an outline. ]

This even has a column to give you inverse of an hint.

This view is a undocumented view.

SQL> desc v$sql_hint
Name                                      Null?    Type
----------------------------------------- -------- ----------------------------
NAME                                               VARCHAR2(64)
SQL_FEATURE                                        VARCHAR2(64)
CLASS                                              VARCHAR2(64)
INVERSE                                            VARCHAR2(64)
TARGET_LEVEL                                       NUMBER
PROPERTY                                           NUMBER
VERSION                                            VARCHAR2(25)
VERSION_OUTLINE                                    VARCHAR2(25)

Jonanthan Lewis – Log File Write

Riyaj Shamsudeen – Log file sync tuning

Christian Bilien – Log file sync wait

For quite some time we had been experiencing slow statspack snapshots, taking about 300sec. In a worst case scenario it took 7 hours. My colleague was investigating it, it turned out that on this particular database “_optimizer_ignore_hints” was set to true. So it was ignoring all the optimization put in by Oracle in statspack snapshot code.

Environment

OS – Linux

Database – 10.2.0.3

Disable Optimizer Hints

SQL> set timing on

SQL> alter session set “_optimizer_ignore_hints” = true;

Session altered.

Elapsed: 00:00:00.03

SQL> exec statspack.snap

PL/SQL procedure successfully completed.

Elapsed: 00:00:20.73

Enable Optimizer Hints

SQL> alter session set “_optimizer_ignore_hints” = false;

Session altered.

Elapsed: 00:00:00.01

SQL> exec statspack.snap

PL/SQL procedure successfully completed.

Elapsed: 00:00:01.75

SQL> exec statspack.snap

PL/SQL procedure successfully completed.

Elapsed: 00:00:01.32

Nobody knows why this parameter was set at instance level. DBA who was handling these databases has left. We will follow up with the vendor, if there is no specific reason to disable optimizer hints then _optimizer_ignore_hints hidden parameter will removed from init.ora. For the time being statspack snapshot dbms job has been altered to include following statements.

alter session set “_optimizer_ignore_hints” = false;

statspack.snap;

Yesterday I received a request from a developer to load file into a BLOB. To be honest I have never loaded a file into a BLOB in past. I had some idea that I would have to use DBMS_LOB pl/sql package to achieve this.

Here are the steps to do this.

1. Create an oracle directory object.

2. Load the file into BLOB

I was asked by application team to refresh 6 schemas from 2 other schemas over the weekend. How hard could it be I thought? Seemed like a simple request. Just take a datapump export of source schema, drop the destination schema and import using datapump with remapped schema and tablespace names.

So i set out exporting source schemas, it was taking more than usual. But i did not pay too much attention as i was very busy and it eventually finished in couple of hours. So onto next job of dropping existing users. Now this is where real problem started. Simple “drop user cascade;” had been running for more than 1 hour. I looked at database session was doing “db sequential read”. Session wait sequence was changing every now and then, which suggests that it was doing sequential read again and again for different data. So I started an 10046 trace of session.

It turns out that peoplesoft had about 100,000 objects for each copy. And this database had 10 copies of it. So there were over 1 million objects (tables, indexes, views etc) in the database !!! Thats a huge number of objects for a database. On top that there were some old master datapump import tables in SYSTEM tables, which were over 1GB each. So SYSTEM tablespace was also 23GB in size. That got me even more suspicious. For each import job of each copy, datapump was creating over 1GB size master table. And it was not cleaning up the master tables somehow. I tried to attach to dp jobs, but could not attach to them as they were not running. So only way to get rid of them was to drop the datapump master table for all old jobs. That reduced the size of SYSTEM tablespace. But it still did not speed up the drop user command. It was still running for over 24 hours. Trace of drop user session showed that session was doing fetch on obj$ table after dropping each object to get next get object details. And this was slowing things down.

Ultimately i generated the a script to drop all (100,000) objects individually and ran it. After that i dropped user with no objects. It reduced the 24 hours to 45 minutes to get rid of the user.

This definitely looks like the inefficient code of “DROP USER CASCADE;” command. I would be raising an TAR with oracle to see what they have to say about it.

Watch this space for some interesting oracle reading.