Re: repeating non-conformant dataset after datamotion + thin prov

m_schuren · ‎2010-10-18

Hi,

maybe someone else has this issue?

Two of my testlab datasets show "nonconformant" every couple of minutes and generate alarms. These datasets use a "thin/nofat san" provisioning policy (no guarantee, no space reserve).

The NMC "Conform Now" says it needs to disable the lun space reserv, and a manual conformance run solves it, dataset goes "conformant", but at next monitoring interval, the same thing happens again - nonconformant and wanting to turn the lun reservation off...

I get these errors in DFM 40D15 conformance log:

Oct 18 22:24:44 [dfmserver:DEBUG]: [532:0x90c]: Conformance checker started scanning at Mon Oct 18 22:24:43 2010
Oct 18 22:24:44 [dfmserver:DEBUG]: [532:0x90c]: Finished scanning dataset UCN Testlab dmotionVMDS with protection policy: UNPROTECTED, unresolvable tasks: 0, resolvable tasks: 0, resolvable-need confirm tasks: 1
Oct 18 22:41:09 [dfmserver:ERROR]: [532:0xb04]: VolLUNSpaceReserveValidate::handle_error(): Volume dmotionSAN:/dmotionSAN (115905) has failed lun-space-reservation validation. Error Code: 141 Reason: This volume's lun-space-reservation option value does not match the policy option values Suggestion: Set lun-space-reservation option as per policy option values.
Oct 18 22:41:09 [dfmserver:DEBUG]: [532:0xb04]: Conformance checker started scanning at Mon Oct 18 22:41:09 2010
Oct 18 22:41:09 [dfmserver:DEBUG]: [532:0xb04]: Finished scanning dataset UCN Testlab dmotionSAN with protection policy: UNPROTECTED, unresolvable tasks: 0, resolvable tasks: 0, resolvable-need confirm tasks: 1
Oct 18 22:41:16 [dfmserver:ERROR]: [532:0xd80]: VolLUNSpaceReserveValidate::handle_error(): Volume dmotionVMDS:/dmotionVMDS (117344) has failed lun-space-reservation validation. Error Code: 141 Reason: This volume's lun-space-reservation option value does not match the policy option values Suggestion: Set lun-space-reservation option as per policy option values.
Oct 18 22:41:16 [dfmserver:DEBUG]: [532:0xd80]: Conformance checker started scanning at Mon Oct 18 22:41:15 2010
Oct 18 22:41:16 [dfmserver:DEBUG]: [532:0xd80]: Finished scanning dataset UCN Testlab dmotionVMDS with protection policy: UNPROTECTED, unresolvable tasks: 0, resolvable tasks: 0, resolvable-need confirm tasks: 1
Oct 18 22:57:37 [dfmserver:ERROR]: [532:0xf0c]: VolLUNSpaceReserveValidate::handle_error(): Volume dmotionSAN:/dmotionSAN (115905) has failed lun-space-reservation validation. Error Code: 141 Reason: This volume's lun-space-reservation option value does not match the policy option values Suggestion: Set lun-space-reservation option as per policy option values.
Oct 18 22:57:37 [dfmserver:DEBUG]: [532:0xf0c]: Conformance checker started scanning at Mon Oct 18 22:57:37 2010
Oct 18 22:57:37 [dfmserver:DEBUG]: [532:0xf0c]: Finished scanning dataset UCN Testlab dmotionSAN with protection policy: UNPROTECTED, unresolvable tasks: 0, resolvable tasks: 0, resolvable-need confirm tasks: 1
Oct 18 22:57:44 [dfmserver:ERROR]: [532:0xb04]: VolLUNSpaceReserveValidate::handle_error(): Volume dmotionVMDS:/dmotionVMDS (117344) has failed lun-space-reservation validation. Error Code: 141 Reason: This volume's lun-space-reservation option value does not match the policy option values Suggestion: Set lun-space-reservation option as per policy option values.
Oct 18 22:57:45 [dfmserver:DEBUG]: [532:0xb04]: Conformance checker started scanning at Mon Oct 18 22:57:44 2010
Oct 18 22:57:45 [dfmserver:DEBUG]: [532:0xb04]: Finished scanning dataset UCN Testlab dmotionVMDS with protection policy: UNPROTECTED, unresolvable tasks: 0, resolvable tasks: 0, resolvable-need confirm tasks: 1

The filer LUN within this vFilers's volume is definitely NOT space reserved, and has never been:

dmotionSAN@filer9> vol status -v dmotionSAN
         Volume State           Status            Options
     dmotionSAN online          raid_dp, flex     nosnap=on, nosnapdir=off,
                                redirect          minra=off, no_atime_update=on,
                                                  nvfail=off,
                                                  ignore_inconsistent=off,
                                                  snapmirrored=off,
                                                  create_ucode=on,
                                                  convert_ucode=on,
                                                  maxdirsize=18350,
                                                  schedsnapname=ordinal,
                                                  fs_size_fixed=off,
                                                  compression=off, guarantee=none,
                                                  svo_enable=off, svo_checksum=off,
                                                  svo_allow_rman=off,
                                                  svo_reject_errors=off, no_i2p=on,
                                                  fractional_reserve=100,
                                                  extent=off,
                                                  try_first=volume_grow,
                                                  read_realloc=off,
                                                  snapshot_clone_dependency=on
                Containing aggregate: 'aggr0'

Plex /aggr0/plex0: online, normal, active
RAID group /aggr0/plex0/rg0: normal

        Snapshot autodelete settings for dmotionSAN:
                                        state=on
                                                commitment=destroy
                                        trigger=volume
                                        target_free_space=5%
                                        delete_order=oldest_first
                                        defer_delete=prefix
                                        prefix=dfpm
                                        destroy_list=lun_clone,vol_clone,cifs_share
        Volume autosize settings:
                                        state=off

dmotionSAN@filer9> lun show -v
        /vol/dmotionSAN/o10g_lun0/o10g_lun0   40.0g (42953867264)   (r/w, online, mapped)
                Serial#: P3I4dZZTXv1j
                Share: none
                Space Reservation: disabled
                Multiprotocol Type: linux
                Maps: dfpm_dmotionSAN=0

It is just like it was created by ProvMgr.

The CLI on DFM server shows:

C:\>dfpm dataset conform -D 115778
Dataset dry run results
----------------------------------
Do: Disable space reservation for LUN dmotionSAN:/dmotionSAN/o10g_lun0/o10g_lun0
Effect: Disable space reservation of the LUN.

I also found:

C:\> dfpm dataset get 115778

Allow custom volume settings on provisioned volumes: No
Enable periodic write guarantee checks on SAN datasets: No

But however I try to set this option, it does not do anything:

C:\>dfpm dataset set 115778 isEnableWGChecks=Never
Left settings unchanged for 115778.

C:\>dfpm dataset set 115778 isEnableWGChecks=No
Left settings unchanged for 115778.

C:\>dfpm dataset set 115778 isEnableWGChecks=Yes
Left settings unchanged for 115778.

C:\>dfpm dataset set 115778 isEnableWGChecks=On
Left settings unchanged for 115778.

These 2 datasets toggle to nonconformant all the time since I performed datamotion on them in the past (successfully and cleand up).

They have been fine before the first datamotion (nondisruptive migration) operation AFAICT.

Ideas, anyone?

adaikkap · ‎2010-10-18

Can you get the output of the following command for the provisioning policy you are using?

dfpm policy get <provisioning policy name or id>

Also what was the ONTAP version when you didnt have problem ?

I am asuming you are currently running 7.3.3

Regards

adai

m_schuren · ‎2010-10-19

Hi adai,

C:\>dfpm policy get 115705
Name:                                     LAB SAN/NoFat non-SnapDrive
Description:                              Testlab SAN LUNs - no SnapDrive available, no dedupe
Type:                                     san
Disk Failure Reliability:                 any
Sub System Failure Reliability:           Disabled
Controller Failure Reliability:           Disabled
Resource Label:
Dataset Member Deduplication Option:      Disabled
Dataset Member Deduplication Schedule:    auto
Thin Provision Storage Space:             Enabled
Guarantee Space For LUN Writes:           Disabled
Thin Provisioning Configuration:          none
Storage Container To Provision:           lun
Dataset Member Nearly Full Threshold (%): 85
Dataset Member Full Threshold (%):        91
Custom Provisioning Script Path:

I saw it in the past with DOT 7.3.3Px already, the actual problem now is with 7.3.4P2, this does not seem to make a difference. These two datasets/vfilers have been created under 7.3.4P1, DFM 4.0D15.

When i change the policy to "more-thick" provisiong (space reservation on, guarantee on, frac_reserve 0, autosize on), the error goes away permanently after one conformance run.

When I switch back to "do not guarantee" (non-space-reserved, non-guaranteed) the error hits me again and again...

This does not happen when dataset/vfiler dataset has never been datamotion'ed before. At least this is what I observe. I can't find any obvious cause for this... 😕

sinhaa · ‎2010-10-20

@m.schuren

I'm trying your scenario but I'm not seeing it. My dfm server is 4.1 but I don't think there has been any change in conformance handling from 4.0 to 4.1. My filers are 7.3.5X1.

I have a san policy:

C:\Documents and Settings\Administrator>dfpm policy get san1
Name:                                     San1
Description:
Type:                                     san
Disk Failure Reliability:                 any
Sub System Failure Reliability:           Disabled
Controller Failure Reliability:           Disabled
Resource Label:
Dataset Member Deduplication Option:      Disabled
Dataset Member Deduplication Schedule:    auto
Thin Provision Storage Space:             Enabled
Guarantee Space For LUN Writes:           Disabled
Thin Provisioning Configuration:          none
Storage Container To Provision:           lun
Dataset Member Nearly Full Threshold (%): 80
Dataset Member Full Threshold (%):        90
Custom Provisioning Script Path:

C:\Documents and Settings\Administrator>

can you give the out put of the following command one after the other.

1. dfpm dataset conform <your_dataset_name>

2. dfm dataset conform -D <your_dataset_name>

I'll try with DFM 4.0D145 tomorrow.

warm regards,

Abhishek

If this post resolved your issue, help others by selecting ACCEPT AS SOLUTION or adding a KUDO.

adaikkap · ‎2010-10-20

I think I know this issue, there has been a change in the default space reservation of ONTAP from 7.3.3 or 7.3.4 or latter.

Will investigate more and let you know.

Regards

adia

m_schuren · ‎2010-10-23

Hi adai,

you are probably right. I never tested this provisioning policy with DOT < 7.3.3 , I think.

What can I do to solve it (besides switching to thick-provisioned luns)?

m_schuren · ‎2010-10-23

Hi sinhaa,

C:\>dfpm dataset conform 117293
Started conformance check for dataset 117293.
Check the status of the dataset with Management Console
or by viewing the conformance.log file.

C:\>dfpm dataset conform -D 117293
Dataset dry run results
----------------------------------
Do: Disable space reservation for LUN dmotionVMDS:/dmotionVMDS/dmotionVMDS01/dmotionVMDS01
Effect: Disable space reservation of the LUN.

C:\>dfpm dataset conform -a 117293
Started conformance check for dataset 117293.
Check the status of the dataset with Management Console
or by viewing the conformance.log file.

C:\>dfpm dataset conform -D 117293
Dataset dry run results
----------------------------------
No dry run results available.

...wait 10 minutes or so...

C:\>dfpm dataset conform -D 117293
Dataset dry run results
----------------------------------
Do: Disable space reservation for LUN dmotionVMDS:/dmotionVMDS/dmotionVMDS01/dmotionVMDS01
Effect: Disable space reservation of the LUN.

sinhaa · ‎2010-11-03

Hello m.schuren,

We tried to investigate your issue and its seems that this is a bug. Dataset doesn't readilty become non-conformant, even after a forceful conformance run, but after some time it does.

Nov 03 14:47:25 [dfmserver:ERROR]: [17884:0xf19dbba0]: VolLUNSpaceReserveValidate::handle_error(): Volume November_2010_1:/November_2010_2 (7490) has failed lun-space-reservation validation. Error Code: 141 Reason: This volume's lun-space-reservation option value does not match the policy option values Suggestion: Set lun-space-reservation option as per policy option values.

I'll raise a burt for the same and fwd you the burt id.

warm regards,

Abhishek

If this post resolved your issue, help others by selecting ACCEPT AS SOLUTION or adding a KUDO.

sinhaa · ‎2010-11-03

Th burt is the following:

460374

thanks.

If this post resolved your issue, help others by selecting ACCEPT AS SOLUTION or adding a KUDO.

m_schuren · ‎2010-11-04

Hi sinhaa,

thanks for taking the efforts.

I will track this burt ID.

Were you able to reproduce the issue?

sinhaa · ‎2010-11-04

Hello m.schuren,

I was able to reproduce the probelm. I'm trying to find a workaround. I tried few things but they didn't work.

I'll update you as soon as I get it.

warm regards,

Abhishek

If this post resolved your issue, help others by selecting ACCEPT AS SOLUTION or adding a KUDO.

mark_schuren · ‎2010-12-16

Just curious, is there any progress on this issue?

The problem is still there with DFM 4.0.1. The only workaround seems to be a more or less "thick provisioned" SAN policy. Not really what customers want.

The bug is still non-public, so partners (like us) cannot see anything in there...

Cheers,

Mark

sinhaa · ‎2010-12-16

Yes the problem is still there with 4.0.1. The burt fix in in target for the next DFM release. In the mean we are trying to find a feasible workaround to get rid of this.

I'll update you on this.

warm regards,

Abhsihk

If this post resolved your issue, help others by selecting ACCEPT AS SOLUTION or adding a KUDO.

adaikkap · ‎2010-12-16

Hi

Pls raise a case against it so that a public report is made available.

Regards

adai

mark_schuren · ‎2010-12-17

Hi Adai,

since this is a test/demo environement in our own lab, I do not have a customer (or valid customer serial number) to open a case for this.

I also will not recommend or implement provisioning manager at a customer, unless these obvious issues are fixed.

This burt is very bad for demos, and so we are not going to install provisioning manager in the field in such environments (because THEN we would have to create support cases, and would cause customer inconvenience, and a bad customer perception…)

So if possible, please fix this issue BEFORE weg o out into the field with it, without an official case from our side.

We can do remote sessions / webex if you like. I can get any required output or do testing if you need our lab to troubleshoot this problem.

Best regards,

Mark

Von: adaikkap

Gesendet: Freitag, 17. Dezember 2010 06:18

An: Mark Schuren

Betreff: "repeating non-conformant dataset after datamotion + thin prov"

<http://communities.netapp.com/index.jspa>

repeating non-conformant dataset after datamotion + thin prov

reply from Adaikkappan Arumugam<http://communities.netapp.com/people/adaikkap> in OnCommand Mgmt Software - View the full discussion<http://communities.netapp.com/message/45404#45404

sprakash · ‎2010-12-17

Marc,

Thank you for the feedback, we are looking into the issue.

Thanks,

Prakash.