Data Backup and Recovery

SDU very slow, HP-UX 11, NFS only

GARDINEC_EBRD
2,584 Views

Hi All,

We've started having a few issues of last with SDU (4.2P2) running very slowly on one of our HPUX boxes.  We're using SMO, but I'm fairly sure the issue is with SDU rather than SMO.

Running simple snapdrive commands such as snapdrive storage show, or snap create can take a long time to complete.  Last night, for example, the SMO log show the snap create command taking >2hours!!  As you can see below it take nearly half an hour to do a snapdrive storage show command. 

The problem is much worse overnight where there are a lot of SMO jobs running (there are ~50 oracle DB's on this box).   Has anyone out there seen anything similar before?

smo log:

2012-04-02 20:56:59,024 [default0 8ac6508459d0e901a6b2cc2467a6ff6c] [DEBUG]: EXE-00000: Executing shell command:

0:/bin/sh -c "/usr/sbin/snapdrive" "storage" "show" "-fs" "/oranetbak/dba1" "-v"

1:/usr/sbin/snapdrive storage show -fs /oranetbak/dba1 -v

2012-04-02 21:23:57,743 [Execution Monitor Thread [/usr/sbin/snapdrive storage show -fs /oranetbak/dba1 -v]] [DEBUG]: EXE-00001: Shell result [0:26:58.716] (Exit Value: 0):

NFS device: filer02-nfs:/vol/LDN_ORA_ARCH_08/dba1_hpuxora4 mount point: /oranetbak/dba1 (persistent)

Protocol: NFS   Actual Path: filer02-nfs:/vol/LDN_ORA_ARCH_08/dba1_hpuxora4

2012-04-02 21:23:57,809 [default0 8ac6508459d0e901a6b2cc2467a6ff6c] [DEBUG]: EXE-00000: Executing shell command:

0:/bin/sh -c "/usr/sbin/snapdrive" "version"

1:/usr/sbin/snapdrive version

2012-04-02 21:23:59,908 [Execution Monitor Thread [/usr/sbin/snapdrive version]] [DEBUG]: EXE-00001: Shell result [0:00:02.097] (Exit Value: 0):

snapdrive Version 4.2P2

Snapdrive Daemon Version 4.2P2

2012-04-02 21:23:59,909 [default0 8ac6508459d0e901a6b2cc2467a6ff6c] [DEBUG]: EXE-00000: Executing shell command:

0:/bin/sh -c "/usr/sbin/snapdrive" "version"

1:/usr/sbin/snapdrive version

2012-04-02 21:24:01,761 [Execution Monitor Thread [/usr/sbin/snapdrive version]] [DEBUG]: EXE-00001: Shell result [0:00:01.852] (Exit Value: 0):

snapdrive Version 4.2P2

Snapdrive Daemon Version 4.2P2

2012-04-02 21:24:01,763 [default0 8ac6508459d0e901a6b2cc2467a6ff6c] [INFO ]: SD-00017: Finished storage discovery for /oranetbak/dba1.

2012-04-02 21:24:01,960 [default0 8ac6508459d0e901a6b2cc2467a6ff6c] [INFO ]: SD-00001: Beginning snapshot with name smo_dba1_dba1_hpuxora4_20120402204005_f_h_2_8ae676193674791a0136747921bd0001_0 for [/

oranetbak/dba1, /oranetctl/dba1].

2012-04-02 21:24:01,988 [default0 8ac6508459d0e901a6b2cc2467a6ff6c] [DEBUG]: EXE-00000: Executing shell command:

0:/bin/sh -c "/usr/sbin/snapdrive" "snap" "create" "-fs" "/oranetbak/dba1" "-fs" "/oranetctl/dba1" "-snapname" "smo_dba1_dba1_hpuxora4_20120402204005_f_h_2_8ae676193674791a0136747921bd0001_0" "-u

nrelated"

1:/usr/sbin/snapdrive snap create -fs /oranetbak/dba1 -fs /oranetctl/dba1 -snapname smo_dba1_dba1_hpuxora4_20120402204005_f_h_2_8ae676193674791a0136747921bd0001_0 -unrelated

2012-04-02 23:37:54,758 [Execution Monitor Thread [/usr/sbin/snapdrive snap create -fs /oranetbak/dba1 -fs /oranetctl/dba1 -snapname smo_dba1_dba1_hpuxora4_20120402204005_f_h_2_8ae676193674791a01

36747921bd0001_0 -unrelated]] [DEBUG]: EXE-00001: Shell result [2:13:52.769] (Exit Value: 0):

Starting snap create /oranetbak/dba1, /oranetctl/dba1

  WARNING:  DO NOT CONTROL-C!

            If snap create is interrupted, incomplete snapdrive

         generated data may remain on the filer volume(s)

         which may interfere with other snap operations.

Successfully created snapshot smo_dba1_dba1_hpuxora4_20120402204005_f_h_2_8ae676193674791a0136747921bd0001_0 on 2 filer volumes:

        filer02-nfs:/vol/LDN_ORA_ARCH_08

        filer02-nfs:/vol/LDN_ORA_CTRL_08

        snapshot smo_dba1_dba1_hpuxora4_20120402204005_f_h_2_8ae676193674791a0136747921bd0001_0 contains:

        file system: /oranetbak/dba1

        file system: /oranetctl/dba1

        filer directory: filer02-nfs:/vol/LDN_ORA_ARCH_08/dba1_hpuxora4

        filer directory: filer02-nfs:/vol/LDN_ORA_CTRL_08/dba1_hpuxora4

0001-904 Warning: Attempting to take a crash consistent snapshot.

Switching to the best-effort mode

2012-04-02 23:37:54,975 [default0 8ac6508459d0e901a6b2cc2467a6ff6c] [WARN ]: SD-10015: Snapdrive command "/usr/sbin/snapdrive snap create -fs /oranetbak/dba1 -fs /oranetctl/dba1 -snapname smo_dba1_c

ds3dev_hpuxora4_20120402204005_f_h_2_8ae676193674791a0136747921bd0001_0 -unrelated" generated warning: 0001-904 Warning: Attempting to take a crash consistent snapshot.

Switching to the best-effort mode

2012-04-02 23:37:54,999 [default0 8ac6508459d0e901a6b2cc2467a6ff6c] [INFO ]: SD-00002: Created snapshot with name smo_dba1_dba1_hpuxora4_20120402204005_f_h_2_8ae676193674791a0136747921bd0001_0 for [/or

anetbak/dba1, /oranetctl/dba1].

Many thanks,

Craig

2 REPLIES 2

crocker
2,584 Views

  is this something your team handles or should this be asked in the NetApp Support Community?

Mike

GARDINEC_EBRD
2,584 Views

Hi,

We've removed and re-installed SDU, and for the past two days things have been much better.  Not sure why it had issues, but we'll run with it for a few days and I'll post an update.

Thanks,

Craig

Public