Re: Snap Mirror Compression and OnCommand 5

rdenyer001 · ‎2011-12-19

Does anyone know if its possible to enable Snap Mirror Compression with OnCommand 5 ?

I have found a document TR-3790 "SnapMirror Network Compression" which explains how to use compression with DFM 3.8 but it relies on the /etc/snapmirror.conf file and the compression=enable option.

Since I have upgraded to OnComand 5.0 my /etc/snapmirror.conf files seem to have been purged of any snapmirror relationships that are setup and managed via OnCommand.

smoot · ‎2011-12-19

I'm not aware of any changes in OC5 that would cause it to purge snapmirror.conf, but perhaps something changed that I'm not aware of.

If you enable SnapMirror compression by editing snapmirror.conf, those changes can be erased if you use the hostPreferredAddr[12] options. This is a known problem and should have been mentioned in the TR.

adaikkap · ‎2011-12-20

Hi Richard,

Can you get the output of the following command for your source and destination filers of snapmirror ?

dfm host get -q <filerid/name>

Regards

adai

rdenyer001 · ‎2011-12-20

Here is the output from dfm host get -q <filerid/name>

Destination

C:\Windows\system32>dfm host get -q HOST1
host=HOST1.dc.local
hostLogin=XXXX
hostPassword=********
hostLoginProtocol=ssh
hostPreferredAddr1=
hostPreferredAddr2=
hostPrimaryAddress=XX.X.X.XX
hostRLMAddress=
hostConsoleAddress=
cpuTooBusyThreshold=95
cpuBusyThresholdInterval=00:15:00
hostPingMethod=all
hostAdminTransport=http
hostNdmpLogin=XXXX
hostNdmpPassword=********
hostAdminPort=80
perfAdvisorTransport=httpOnly
prefsnmpVersion=1
useHostsEquiv=No
maxActiveDataTransfers=160
perfDataExportEnabled=No
defaultvFilerInterface=
processHostIP=
autoClientStatEnabled=No
hostClockSkewedThreshold=1 minute
hostClockNearlySkewedThreshold=30 seconds

Source
C:\Windows\system32>dfm host get -q HOST2
host=HOST2.dc.local
hostLogin=XXXX
hostPassword=********
hostLoginProtocol=ssh
hostPreferredAddr1=
hostPreferredAddr2=
hostPrimaryAddress=XX.X.X.XX
hostRLMAddress=XX.X.X.XX
hostConsoleAddress=
cpuTooBusyThreshold=95
cpuBusyThresholdInterval=00:15:00
hostPingMethod=all
hostAdminTransport=http
hostNdmpLogin=XXXX
hostNdmpPassword=********
hostAdminPort=80
perfAdvisorTransport=httpOnly
prefsnmpVersion=1
useHostsEquiv=No
maxActiveDataTransfers=160
perfDataExportEnabled=No
defaultvFilerInterface=
processHostIP=
autoClientStatEnabled=No
hostClockSkewedThreshold=1 minute
hostClockNearlySkewedThreshold=30 seconds

adaikkap · ‎2011-12-21

Hi Richard,

I created a compression enabled relationship on the filer and imported the same into a dataset in DFM 4.0.2D1.

C:\>dfm version

dfbm.exe 4.0.2.7809 (4.0.2D1)

dfdrm.exe 4.0.2.7809 (4.0.2D1)

dfpm.exe 4.0.2.7809 (4.0.2D1)

dfm.exe 4.0.2.7809 (4.0.2D1)

dfmcheck.exe 4.0.2.7809 (4.0.2D1)

dfmconfig.exe 4.0.2.7809 (4.0.2D1)

dfmconsole.exe 4.0.2.7809 (4.0.2D1)

dfmmonitor.exe 4.0.2.7809 (4.0.2D1)

dfmperf.exe 4.0.2.7809 (4.0.2D1)

dfmscheduler.exe 4.0.2.7809 (4.0.2D1)

dfmserver.exe 4.0.2.7809 (4.0.2D1)

dfmwatchdog.exe 4.0.2.7809 (4.0.2D1)

eventd.exe 4.0.2.7809 (4.0.2D1)

grapher.exe 4.0.2.7809 (4.0.2D1)

C:\>dfm controller list

ID Type Host Name Host Address ProductId Deleted

---- ------------------------ ----------------------------------- ----------------------------------------- ------------ --------

90 Active/Active Controller fas-sim-1 192.168.98.10 0099931871 No

91 Active/Active Controller fas-sim-2 192.168.98.11 0099931872 No

C:\>dfm host get -q 90

host=fas-sim-1

hostLogin=

hostPassword=********

hostLoginProtocol=rsh

hostPreferredAddr1=

hostPreferredAddr2=

hostPrimaryAddress=192.168.98.10

hostRLMAddress=

hostConsoleAddress=

cpuTooBusyThreshold=95

cpuBusyThresholdInterval=00:15:00

hostPingMethod=echo_snmp

hostAdminTransport=http

hostNdmpLogin=

hostNdmpPassword=********

hostAdminPort=80

perfAdvisorTransport=httpOnly

prefsnmpVersion=1

useHostsEquiv=No

maxActiveDataTransfers=

perfDataExportEnabled=No

defaultvFilerInterface=

processHostIP=

autoClientStatEnabled=No

C:\>dfm host get -q 91

host=fas-sim-2

hostLogin=

hostPassword=********

hostLoginProtocol=rsh

hostPreferredAddr1=

hostPreferredAddr2=

hostPrimaryAddress=192.168.98.11

hostRLMAddress=

hostConsoleAddress=

cpuTooBusyThreshold=95

cpuBusyThresholdInterval=00:15:00

hostPingMethod=echo_snmp

hostAdminTransport=http

hostNdmpLogin=

hostNdmpPassword=********

hostAdminPort=80

perfAdvisorTransport=httpOnly

prefsnmpVersion=1

useHostsEquiv=No

maxActiveDataTransfers=

perfDataExportEnabled=No

defaultvFilerInterface=

processHostIP=

autoClientStatEnabled=No

C:\>

Ran Multiple updates and found the conf entry to be intact.

Now upgraded my dfm server to 5.0 and ran multiple update and still find my conf entry to be intact in snapmirror.conf.

C:\>dfpm dataset list -x compression

Id: 392

Name: compression

Protection Policy: Mirror

Application Policy:

Description:

Owner:

Contact:

Volume Qtree Name Prefix:

Snapshot Name Format:

Primary Volume Name Format:

Secondary Volume Name Format:

Secondary Qtree Name Format:

DR Capable: No

Requires Non Disruptive Restore: No

Node details:

Node Name: Primary data

Resource Pools:

Provisioning Policy:

Time Zone:

DR Capable: No

vFiler:

Node Name: Mirror

Resource Pools:

Provisioning Policy:

Time Zone:

DR Capable: No

vFiler:

C:\>dfpm dataset list -m compression

Id Node Name Dataset Id Dataset Name Member Type Name

---------- -------------------- ---------- -------------------- -------------------------------------------------- -------------------------------------------------------

383 Primary data 392 compression volume fas-sim-1:/comp_vsm_src

386 Mirror 392 compression volume fas-sim-2:/comp_vsm_dst

C:\>dfpm dataset list -R compression

Id Name Protection Policy Provisioning Policy Relationship Id State Status Hours Source Destination

---------- --------------------------- --------------------------- ------------------- --------------- ------------ ------- ----- ---------------------------- ----------------------------

392 compression Mirror 389 snapmirrored idle 0.0 fas-sim-1:/comp_vsm_src fas-sim-2:/comp_vsm_dst

C:\>dfpm backup list compression

Backup Id Backup Version Retention Type Retention Duration (in seconds) Node Name Description Properties(Name=Value)

--------- --------------------- -------------- --------------------------------- -------------------- ----------------------------------- -------------------------

12 22 Dec 2011 00:42:08 unlimited Mirror

10 22 Dec 2011 00:42:08 unlimited Primary data

11 22 Dec 2011 00:41:54 monthly Mirror

9 22 Dec 2011 00:41:54 monthly Primary data

8 22 Dec 2011 00:19:35 unlimited Mirror

7 22 Dec 2011 00:19:21 weekly 3024000 Mirror

5 22 Dec 2011 00:19:21 weekly 3024000 Primary data

3 22 Dec 2011 00:16:51 daily 604800 Mirror test

1 22 Dec 2011 00:16:51 daily 604800 Primary data test

C:\>dfm run cmd fas-sim-2 " rdfile /etc/snapmirror.conf"

Job: 1

Host: fas-sim-2

Command: rdfile /etc/snapmirror.conf

Status: done

Submitted: Thu Dec 22 00:45:28 2011

Attempts: 1

Login Protocol: rsh

Use RLM Port: No

Started: Thu Dec 22 00:45:28 2011

Completed: Thu Dec 22 00:45:29 2011

Stdout: #Regenerated by registry Tue Sep 27 08:58:29 GMT 2011

Stdout: fas-sim-1:/vol/qsm_src/qsm_qt1 fas-sim-2:/vol/qsm_dst/qsm_qt1 restart=never 0-59/3 * * *

Stdout: fas-sim-1:/vol/qsm_src/qsm_qt2 fas-sim-2:/vol/qsm_dst/qsm_qt2 restart=never 0-59/4 * * *

Stdout: fas-sim-1:/vol/qsm_src/- fas-sim-2:/vol/qsm_dst/qsm_dst_non_qtree_data - 0-59/3 * * *

Stdout: fas-sim-1:vsm_src fas-sim-2:vsm_dst restart=never * 0-23 * *

Stdout: comp_con=multi(192.168.98.12,192.168.98.13)

Stdout: comp_con:comp_vsm_src fas-sim-2:comp_vsm_dst compression=enable - - - - -

C:\>

C:\>dfm version

dfbm.exe 5.0.0.7636 (5.0)

dfdrm.exe 5.0.0.7636 (5.0)

dfpm.exe 5.0.0.7636 (5.0)

dfm.exe 5.0.0.7636 (5.0)

dfmcheck.exe 5.0.0.7636 (5.0)

dfmconfig.exe 5.0.0.7636 (5.0)

dfmconsole.exe 5.0.0.7636 (5.0)

dfmmonitor.exe 5.0.0.7636 (5.0)

dfmperf.exe 5.0.0.7636 (5.0)

dfmscheduler.exe 5.0.0.7636 (5.0)

dfmserver.exe 5.0.0.7636 (5.0)

dfmwatchdog.exe 5.0.0.7636 (5.0)

eventd.exe 5.0.0.7636 (5.0)

grapher.exe 5.0.0.7636 (5.0)

C:\>

Can you provide the steps to reproduce ?

Regards

adai

fas-sim-2> rdfile /etc/snapmirror.conf

#Regenerated by registry Tue Sep 27 08:58:29 GMT 2011

fas-sim-1:/vol/qsm_src/qsm_qt1 fas-sim-2:/vol/qsm_dst/qsm_qt1 restart=never 0-59/3 * * *

fas-sim-1:/vol/qsm_src/qsm_qt2 fas-sim-2:/vol/qsm_dst/qsm_qt2 restart=never 0-59/4 * * *

fas-sim-1:/vol/qsm_src/- fas-sim-2:/vol/qsm_dst/qsm_dst_non_qtree_data - 0-59/3 * * *

fas-sim-1:vsm_src fas-sim-2:vsm_dst restart=never * 0-23 * *

comp_con=multi(192.168.98.12,192.168.98.13)

comp_con:comp_vsm_src fas-sim-2:comp_vsm_dst compression=enable - - - - -

fas-sim-2>

rdenyer001 · ‎2011-12-21

Thanks Adai,

Most of my Snapmirror relationships have been created directly from the "NetApp Management Console" and dont have entries in the snapmirror.conf file.

So I guess there are two options 1. wait until the NetApp management tools catch-up with ALL the features availible on a NetApp system or 2. Edit my snapmirror.conf files to add the compression option

Regards,

Richard

adaikkap · ‎2011-12-21

Hi Richard,

I think there is some misunderstanding. Let me try to clarify some.

PM does not support creation of compression enabled snapmirror relationships.But it interoperates
PM does not make a snapmirror.conf entry for all relationship its creates, except for those that are created using preferred interface and uses connections.

Case 1:Where PM creates the relationships:

================================

So if you create a normal QSM or VSM relationship using PM it neither makes a snapmirror.conf entry nor schedules in the same as below.

fas-sim-1:/vol/qsm_src/- fas-sim-2:/vol/qsm_dst/qsm_dst_non_qtree_data - 0-59/3 * * *

Instead the schedule are driven by DFM server scheduler service which does the scheduling as per the policy you use in PM.

If you have the option hostPreferredAddr1/2 set in DFM for the source and destination filer of Snapmirror.And create a VSM or QSM relationship using PM then PM creates a connection and make these entries in snapmirror.conf

fas-sim-1_fas-sim-2_1208568606=multi(192.168.98.12,192.168.98.13)

fas-sim-1_fas-sim-2_1208568606:comp_vsm_src fas-sim-2:comp_vsm_dst compression=enable - - - - -

(Note if 1 is set on primary then 1 in secondary is used. Its the respective interface and not 1 in source and 2 in destination is used)

Even in this case the schedule are driven by DFM server scheduler service and not ontap.This applies to both QSM and VSM.

So PM is not capable of creating a compression enabled QSM or VSM relationship.

The same applies to SV also where we don't create the snapvault snap schedules. In order to have the SV traffic in a preferred interface the following options in filer should be set.

ndpmd_preferred.interface.

Case2:Where relationships are imported.

First and foremost rule:

PM does not delete any snapmirror.conf or snapvault schedule when the corresponding relationships are imported into a Dataset.This enables PM to interoperate with Compression enabled

snapmirror.

So to have a compression enabled relationship in PM have the relationship created in the filer and the corresponding conf entries.Import them into a data and have the schedule in PM

fas-sim-1_fas-sim-2_1208568606=multi(192.168.98.12,192.168.98.13)

fas-sim-1_fas-sim-2_1208568606:comp_vsm_src fas-sim-2:comp_vsm_dst compression=enable - - - - -

It is advised to remove the schedule in the snapmirror.conf to remove the redundant scheduling. Make sure that hostPreferredAddr1/2 are not set on the source and destination filer of the compression enabled snapmirror filers in DFM.

When a relationship is imported say like below which has snapmirror.conf entry as follows

fas-sim-1:/vol/qsm_src/- fas-sim-2:/vol/qsm_dst/qsm_dst_non_qtree_data - 0-59/3 * * *

and hostPreferredAddr1/2 are correspondingly set on the source and destination filer then PM makes an entry in the snapmirror.conf as follows

fas-sim-1_fas-sim-2_1208568606=multi(192.168.187.12,192.168.187.13)

fas-sim-1_fas-sim-2_1208568606:/vol/qsm_src/- fas-sim-2:/vol/qsm_dst/qsm_dst_non_qtree_data - - - - -

Still the old entry remains.

The reason why hostPreferredAddr1/2 should not be set for compression enabled snapmirror is because before every transfer we rewrite the snapmirror.conf with the above two entry in this process compression options is deleted as there is no support for it in NDMP or api.

Hope this clarifies.

Regards

adai

localizedinsanity · ‎2012-10-10

I had originally asked....how would you go about retrofitting existing PM created relationships to enable compression? I subsequently worked that out....here are the steps...

* add dfm host set <hostname> hostPreferredAddr1=x.x.x. to each source node's configuration within DFM

* manually edit the /etc/snapmirror.conf file to populate it using the output of "snapmirror status" for source/destination pairings

* remember to add the "compression=enable" in the options field and to keep the schedule - - - -

* remember to configure your connection string settings

* I use a text editor to fully create the file structure, then paste it into the CLI using "wrfile /etc/snapmirror.conf" & ctrl-c

Once you do those steps, Protection Manager will use compression the next time it updates. This process assumes you have pre-existing datasets with policies defined and mirrors baselined/updating. If you need to configure new relationships, Protection Manager will populate the snapmirror.conf file (thanks to the hostPreferredAddr1 setting) and all you need to do is edit the configuration file to change the source node name to the connection string and add the compression option.

NOTE - an initialize process will NOT be compressed because Protection Manager cannot enable compression as pointed out in previous comments. Unless you create the relationship using System Manager without initializing, make the changes to snapmirror.conf, manually initialize, then import as outlined in Adai's post, you cannot initialize using compression. I hope this helps the next person to ask the question!

Thanks to Adai for getting me on the right track with his posting about preferred addresses.

adaikkap · ‎2012-10-10

Hi,

All you have to do is the following.

For all your Volume SnapMirror relationships pls make an entry in your snapmirror.conf file as follows.

comp_con=multi(192.168.98.12,192.168.98.13)

comp_con:comp_vsm_src fas-sim-2:comp_vsm_dst compression=enable - - - - -

Import those relationships to PM Mirror dataset.

Make sure you dont have the option hostPerferredAddr1 set on your filer otherwise on first update from PM the compression options will be erased from snapmirror.conf.

Read this public report for the same which affects compression option

http://support.netapp.com/NOW/cgi-bin/bol?Type=Detail&Display=381872

Regards

adai