Hi All,
We've started having a few issues of last with SDU (4.2P2) running very slowly on one of our HPUX boxes. We're using SMO, but I'm fairly sure the issue is with SDU rather than SMO.
Running simple snapdrive commands such as snapdrive storage show, or snap create can take a long time to complete. Last night, for example, the SMO log show the snap create command taking >2hours!! As you can see below it take nearly half an hour to do a snapdrive storage show command.
The problem is much worse overnight where there are a lot of SMO jobs running (there are ~50 oracle DB's on this box). Has anyone out there seen anything similar before?
smo log:
2012-04-02 20:56:59,024 [default0 8ac6508459d0e901a6b2cc2467a6ff6c] [DEBUG]: EXE-00000: Executing shell command:
0:/bin/sh -c "/usr/sbin/snapdrive" "storage" "show" "-fs" "/oranetbak/dba1" "-v"
1:/usr/sbin/snapdrive storage show -fs /oranetbak/dba1 -v
2012-04-02 21:23:57,743 [Execution Monitor Thread [/usr/sbin/snapdrive storage show -fs /oranetbak/dba1 -v]] [DEBUG]: EXE-00001: Shell result [0:26:58.716] (Exit Value: 0):
NFS device: filer02-nfs:/vol/LDN_ORA_ARCH_08/dba1_hpuxora4 mount point: /oranetbak/dba1 (persistent)
Protocol: NFS Actual Path: filer02-nfs:/vol/LDN_ORA_ARCH_08/dba1_hpuxora4
2012-04-02 21:23:57,809 [default0 8ac6508459d0e901a6b2cc2467a6ff6c] [DEBUG]: EXE-00000: Executing shell command:
0:/bin/sh -c "/usr/sbin/snapdrive" "version"
1:/usr/sbin/snapdrive version
2012-04-02 21:23:59,908 [Execution Monitor Thread [/usr/sbin/snapdrive version]] [DEBUG]: EXE-00001: Shell result [0:00:02.097] (Exit Value: 0):
snapdrive Version 4.2P2
Snapdrive Daemon Version 4.2P2
2012-04-02 21:23:59,909 [default0 8ac6508459d0e901a6b2cc2467a6ff6c] [DEBUG]: EXE-00000: Executing shell command:
0:/bin/sh -c "/usr/sbin/snapdrive" "version"
1:/usr/sbin/snapdrive version
2012-04-02 21:24:01,761 [Execution Monitor Thread [/usr/sbin/snapdrive version]] [DEBUG]: EXE-00001: Shell result [0:00:01.852] (Exit Value: 0):
snapdrive Version 4.2P2
Snapdrive Daemon Version 4.2P2
2012-04-02 21:24:01,763 [default0 8ac6508459d0e901a6b2cc2467a6ff6c] [INFO ]: SD-00017: Finished storage discovery for /oranetbak/dba1.
2012-04-02 21:24:01,960 [default0 8ac6508459d0e901a6b2cc2467a6ff6c] [INFO ]: SD-00001: Beginning snapshot with name smo_dba1_dba1_hpuxora4_20120402204005_f_h_2_8ae676193674791a0136747921bd0001_0 for [/
oranetbak/dba1, /oranetctl/dba1].
2012-04-02 21:24:01,988 [default0 8ac6508459d0e901a6b2cc2467a6ff6c] [DEBUG]: EXE-00000: Executing shell command:
0:/bin/sh -c "/usr/sbin/snapdrive" "snap" "create" "-fs" "/oranetbak/dba1" "-fs" "/oranetctl/dba1" "-snapname" "smo_dba1_dba1_hpuxora4_20120402204005_f_h_2_8ae676193674791a0136747921bd0001_0" "-u
nrelated"
1:/usr/sbin/snapdrive snap create -fs /oranetbak/dba1 -fs /oranetctl/dba1 -snapname smo_dba1_dba1_hpuxora4_20120402204005_f_h_2_8ae676193674791a0136747921bd0001_0 -unrelated
2012-04-02 23:37:54,758 [Execution Monitor Thread [/usr/sbin/snapdrive snap create -fs /oranetbak/dba1 -fs /oranetctl/dba1 -snapname smo_dba1_dba1_hpuxora4_20120402204005_f_h_2_8ae676193674791a01
36747921bd0001_0 -unrelated]] [DEBUG]: EXE-00001: Shell result [2:13:52.769] (Exit Value: 0):
Starting snap create /oranetbak/dba1, /oranetctl/dba1
WARNING: DO NOT CONTROL-C!
If snap create is interrupted, incomplete snapdrive
generated data may remain on the filer volume(s)
which may interfere with other snap operations.
Successfully created snapshot smo_dba1_dba1_hpuxora4_20120402204005_f_h_2_8ae676193674791a0136747921bd0001_0 on 2 filer volumes:
filer02-nfs:/vol/LDN_ORA_ARCH_08
filer02-nfs:/vol/LDN_ORA_CTRL_08
snapshot smo_dba1_dba1_hpuxora4_20120402204005_f_h_2_8ae676193674791a0136747921bd0001_0 contains:
file system: /oranetbak/dba1
file system: /oranetctl/dba1
filer directory: filer02-nfs:/vol/LDN_ORA_ARCH_08/dba1_hpuxora4
filer directory: filer02-nfs:/vol/LDN_ORA_CTRL_08/dba1_hpuxora4
0001-904 Warning: Attempting to take a crash consistent snapshot.
Switching to the best-effort mode
2012-04-02 23:37:54,975 [default0 8ac6508459d0e901a6b2cc2467a6ff6c] [WARN ]: SD-10015: Snapdrive command "/usr/sbin/snapdrive snap create -fs /oranetbak/dba1 -fs /oranetctl/dba1 -snapname smo_dba1_c
ds3dev_hpuxora4_20120402204005_f_h_2_8ae676193674791a0136747921bd0001_0 -unrelated" generated warning: 0001-904 Warning: Attempting to take a crash consistent snapshot.
Switching to the best-effort mode
2012-04-02 23:37:54,999 [default0 8ac6508459d0e901a6b2cc2467a6ff6c] [INFO ]: SD-00002: Created snapshot with name smo_dba1_dba1_hpuxora4_20120402204005_f_h_2_8ae676193674791a0136747921bd0001_0 for [/or
anetbak/dba1, /oranetctl/dba1].
Many thanks,
Craig
