Snapdrive issue when MSCS do failover. "Failed to enumerate virtual disks, error code 0x800703f0"

[chao]

Hi Gurus

IHAC met some strange error.    when they do MSCS failover testing, sometimes, for some resource group, the failover is failed.  they found in SnapDrive log, there are following error messages:

01/27-00:35:45.985 PID:7120 TID:6208 Helpers.cpp@2307  is_supported_scsi_inquiry_id_predicate_t::operator() - vendor_id 'netapp' is 'supported'
01/27-00:35:45.985 PID:7120 TID:6208 Helpers.cpp@2293  parsed_windows_device_path_t::parsed_windows_device_path_t() - windows device path is parsed '\\?\scsi#disk&ven_hp&prod_logical_volume&rev_4.12#5&c8b13fa&0&000400#{53f56307-b6bf-11d0-94f2-00a0c91efb8b}',parse_info::hit: 'false', parse_info::full: 'false',scsi_vendor_id: 'hp', scsi_product_id: 'logical', scsi_revision: ''

01/27-00:35:45.985 PID:7120 TID:6208 Helpers.cpp@2307  is_supported_scsi_inquiry_id_predicate_t::operator() - vendor_id 'hp' is 'not supported'
01/27-00:35:45.985 PID:7120 TID:6208 Helpers.cpp@2293  parsed_windows_device_path_t::parsed_windows_device_path_t() - windows device path is parsed '\\?\scsi#disk&ven_hp&prod_logical_volume&rev_4.12#5&c8b13fa&0&000500#{53f56307-b6bf-11d0-94f2-00a0c91efb8b}',parse_info::hit: 'false', parse_info::full: 'false',scsi_vendor_id: 'hp', scsi_product_id: 'logical', scsi_revision: ''

01/27-00:35:45.985 PID:7120 TID:6208 Helpers.cpp@2307  is_supported_scsi_inquiry_id_predicate_t::operator() - vendor_id 'hp' is 'not supported'
01/27-00:35:45.985 PID:7120 TID:6208 System.cpp@117  Finish CSystem::enumerate_disk_devices
01/27-00:35:45.985 PID:7120 TID:6208 host_volume_info.cpp@537  Start get_volumes_info()
01/27-00:35:46.000 PID:7120 TID:6208 mscs_cluster.cpp@151  mscs_cluster_t::initialize() - adding mscs disk cluster resource #1, name:'Disk J:\', owner_node_name:'SCCPRMDB02101', drive_letter:'J', windows_volume_name:'\\?\Volume{b5ea2501-43a8-11de-9883-001e0bcfaa3a}\', disk_signature:'0x39B467E5'

01/27-00:35:46.000 PID:7120 TID:6208 mscs_cluster.cpp@151  mscs_cluster_t::initialize() - adding mscs disk cluster resource #5, name:'Disk S:\', owner_node_name:'SCCPRMDB02101', drive_letter:'S', windows_volume_name:'\\?\Volume{b5ea251d-43a8-11de-9883-001e0bcfaa3a}\', disk_signature:'0x39B467E7'

01/27-00:35:46.016 PID:7120 TID:6208 mscs_cluster.cpp@151  mscs_cluster_t::initialize() - adding mscs disk cluster resource #6, name:'Disk T:\', owner_node_name:'SCCPRMDB02101', drive_letter:'T', windows_volume_name:'\\?\Volume{4e13c01d-4478-11de-b08b-001e0bcfaa3a}\', disk_signature:'0xB9454FB3'

01/27-00:35:46.016 PID:7120 TID:6208 mscs_cluster.cpp@151  mscs_cluster_t::initialize() - adding mscs disk cluster resource #7, name:'Disk I:\', owner_node_name:'SCCPRMDB02101', drive_letter:'I', windows_volume_name:'\\?\Volume{4e13c021-4478-11de-b08b-001e0bcfaa3a}\', disk_signature:'0xB9454FA3'

01/27-00:35:46.016 PID:7120 TID:6208 mscs_cluster.cpp@151  mscs_cluster_t::initialize() - adding mscs disk cluster resource #8, name:'Disk K:\', owner_node_name:'SCCPRMDB02101', drive_letter:'K', windows_volume_name:'\\?\Volume{4e13c025-4478-11de-b08b-001e0bcfaa3a}\', disk_signature:'0xB9454FA5'

01/27-00:35:46.031 PID:7120 TID:6208 mscs_cluster.cpp@151  mscs_cluster_t::initialize() - adding mscs disk cluster resource #9, name:'Disk L:\', owner_node_name:'SCCPRMDB02101', drive_letter:'L', windows_volume_name:'\\?\Volume{4e13c029-4478-11de-b08b-001e0bcfaa3a}\', disk_signature:'0xB9454FA7'

01/27-00:35:46.031 PID:7120 TID:6208 mscs_cluster.cpp@151  mscs_cluster_t::initialize() - adding mscs disk cluster resource #10, name:'Disk M:\', owner_node_name:'SCCPRMDB02101', drive_letter:'M', windows_volume_name:'\\?\Volume{4e13c02d-4478-11de-b08b-001e0bcfaa3a}\', disk_signature:'0xB9454FA9'

01/27-00:35:46.047 PID:7120 TID:6208 mscs_cluster.cpp@151  mscs_cluster_t::initialize() - adding mscs disk cluster resource #11, name:'Disk N:\', owner_node_name:'SCCPRMDB02101', drive_letter:'N', windows_volume_name:'\\?\Volume{4e13c031-4478-11de-b08b-001e0bcfaa3a}\', disk_signature:'0xB9454FAB'

01/27-00:35:46.047 PID:7120 TID:6208 mscs_cluster.cpp@151  mscs_cluster_t::initialize() - adding mscs disk cluster resource #12, name:'Disk O:\', owner_node_name:'SCCPRMDB02101', drive_letter:'O', windows_volume_name:'\\?\Volume{4e13c035-4478-11de-b08b-001e0bcfaa3a}\', disk_signature:'0xB9454FAD'

01/27-00:35:46.047 PID:7120 TID:6208 mscs_cluster.cpp@151  mscs_cluster_t::initialize() - adding mscs disk cluster resource #13, name:'Disk R:\', owner_node_name:'SCCPRMDB02101', drive_letter:'R', windows_volume_name:'\\?\Volume{4e13c039-4478-11de-b08b-001e0bcfaa3a}\', disk_signature:'0xB9454FAF'

01/27-00:35:46.063 PID:7120 TID:6208 mscs_cluster.cpp@151  mscs_cluster_t::initialize() - adding mscs disk cluster resource #14, name:'Disk U:\', owner_node_name:'SCCPRMDB02101', drive_letter:'U', windows_volume_name:'\\?\Volume{4e13c03d-4478-11de-b08b-001e0bcfaa3a}\', disk_signature:'0xB9454FB1'

01/27-00:35:46.063 PID:7120 TID:6208 mscs_cluster.cpp@151  mscs_cluster_t::initialize() - adding mscs disk cluster resource #15, name:'Disk W:\', owner_node_name:'SCCPRMDB02101', drive_letter:'W', windows_volume_name:'\\?\Volume{4e13c041-4478-11de-b08b-001e0bcfaa3a}\', disk_signature:'0xB9454FB5'

01/27-00:35:46.078 PID:7120 TID:6208 mscs_cluster.cpp@151  mscs_cluster_t::initialize() - adding mscs disk cluster resource #18, name:'Disk P:\', owner_node_name:'SCCPRMDB02101', drive_letter:'P', windows_volume_name:'\\?\Volume{b5ea2520-43a8-11de-9883-001e0bcfaa3a}\', disk_signature:'0x39B467E9'

01/27-00:35:46.078 PID:7120 TID:6208 mscs_cluster.cpp@151  mscs_cluster_t::initialize() - adding mscs disk cluster resource #21, name:'Disk Q:\', owner_node_name:'SCCPRMDB02101', drive_letter:'Q', windows_volume_name:'\\?\Volume{b5ea24f7-43a8-11de-9883-001e0bcfaa3a}\', disk_signature:'0x7DD1EB03'

01/27-00:35:46.094 PID:7120 TID:6208 mscs_cluster.cpp@151  mscs_cluster_t::initialize() - adding mscs disk cluster resource #23, name:'Disk F:\', owner_node_name:'SCCPRMDB02101', drive_letter:'F', windows_volume_name:'\\?\Volume{b5ea24fb-43a8-11de-9883-001e0bcfaa3a}\', disk_signature:'0x39B467E1'

01/27-00:35:46.094 PID:7120 TID:6208 mscs_cluster.cpp@151  mscs_cluster_t::initialize() - adding mscs disk cluster resource #27, name:'Disk H:\', owner_node_name:'SCCPRMDB02102', drive_letter:'H', windows_volume_name:'\\?\Volume{b5ea24ef-43a8-11de-9883-001e0bcfaa3a}\', disk_signature:'0x7DD1EB05'

01/27-00:35:46.094 PID:7120 TID:6208 mscs_cluster.cpp@75  An attempt was made to reference a token that does not exist.
WIN32 error code 0x3f0.
01/27-00:35:46.094 PID:7120 TID:6208 System.cpp@2014  Failed to enumerate virtual disks, error code 0x800703f0, error description 'An attempt was made to reference a token that does not exist.
'
01/27-00:35:46.094 PID:7120 TID:6208 EventLog.cpp@330  Start NeedMuteEvent(): dwEventID 304 dwNumStrings 2 enumEvtMuteGrp -1

01/27-00:35:46.094 PID:7120 TID:6208 EventLog.cpp@417  End NeedMuteEvent(): dwEventID 304 dwNumStrings 2 enumEvtMuteGrp -1 bRet=00

01/27-00:35:46.094 PID:7120 TID:6208 VirtualDisk.cpp@916  Finish EnumVirtualDisksInternal
01/27-00:35:46.094 PID:7120 TID:6208 VirtualDisk.cpp@3557  Start CVirtualDisk::NTFSHolePunchProgress
01/27-00:35:46.094 PID:7120 TID:6208 FCPVdisk.cpp@8776  Start CFCPVdisk::NTFSHolePunchProgress
01/27-00:35:46.094 PID:7120 TID:6208 VirtualDisk.cpp@3573  Finish CVirtualDisk::NTFSHolePunchProgress

In Snapdrive log, it report "Failed to enumerate virtual disks, error code 0x800703f0".

Customer environment is windows 2003, SDW 5.0.1. there are about 16 Luns in this cluster.  sometimes, the failover failed, but sometimes failover worked well.  My questions are

1) why SDW report  "Failed to enumerate virtual disks, error code 0x800703f0"?

2) Is it necessary that SDW service must be up when do MSCS failover?

3) is it the root cause that 16 LUNs is too much for SDW?

4) Any workaround?

Of coz, I already ask them to raise a case.  if I get the case#, I will update here.

Any input is appreciated!

Thanks and Best Regards!

TC

[watan]

Hi Chao,

1) why SDW report  "Failed to enumerate virtual disks, error code 0x800703f0"?

May want to look at the host side logs to see if there are any other errors.  iScsi network errors could cause some problems.

2) Is it necessary that SDW service must be up when do MSCS failover?

SDW service needs to be up and running for snapdrive to work and not for MSCS failover.

3) is it the root cause that 16 LUNs is too much for SDW?

16 luns should be fine.

4) Any workaround?

Found a KB article that may help

https://now.netapp.com/Knowledgebase/solutionarea.asp?id=kb34470

[chao]

Hi Watan

Thanks for your kindly reply.  I am just back from a long biz-trip.

Customer has already followed the KB34470 but it seems sometimes it worked very well and sometime could not work. custoemr environment is FCP.

some log msg are like:

Event Type: Warning
Event Source: ClusSvc
Event Category: (5)
Event ID: 1123
Date:  1/27/2010
Time:  12:36:18 AM
User:  N/A
Computer: SCCPRMDB02101
Description:
The description for Event ID ( 1123 ) in Source ( ClusSvc ) cannot be found. The local computer may not have the necessary registry information or message DLL files to display messages from a remote computer. You may be able to use the /AUXSOURCE= flag to retrieve this description; see Help and Support for details. The following information is part of the event: SCCPRMDB02102, heartbeat.

Event Type: Warning
Event Source: ClusSvc
Event Category: (5)
Event ID: 1135
Date:  1/27/2010
Time:  12:36:22 AM
User:  N/A
Computer: SCCPRMDB02101
Description:
The description for Event ID ( 1135 ) in Source ( ClusSvc ) cannot be found. The local computer may not have the necessary registry information or message DLL files to display messages from a remote computer. You may be able to use the /AUXSOURCE= flag to retrieve this description; see Help and Support for details. The following information is part of the event: SCCPRMDB02102.

Event Type: Error
Event Source: HP Ethernet
Event Category: (9)
Event ID: 2
Date:  1/27/2010
Time:  12:37:04 AM
User:  N/A
Computer: SCCPRMDB02101
Description:
The description for Event ID ( 2 ) in Source ( HP Ethernet ) cannot be found. The local computer may not have the necessary registry information or message DLL files to display messages from a remote computer. You may be able to use the /AUXSOURCE= flag to retrieve this description; see Help and Support for details. The following information is part of the event: A network interface has failed. An Ethernet adapter's status changed from OK to Error, or an Ethernet team's status changed to Error from another state., Check the network cables, switches, and hubs for the cause of the failure., AlertingElementFormat: 2 0x2 (CIMObjectPath)
AlertingManagedElement: "\\SCCPRMDB02101\ROOT\HPQ:HP_WinEthernetPort.DeviceID="{ED26CE84-2AC3-42BF-BE99-3DC7B7684FCC}",CreationClassName="HP_WinEthernetPort",SystemCreationCla...""
AlertType: 2 0x2 (Communications Alert)
Description: "A network interface has failed. An Ethernet adapter's status changed from OK to Error, or an Ethernet team's status changed to Error from another state."
EventCategory: 9 0x9 (Network Device)
EventID: "2"
EventSubCategory: 11 0xb (Ethernet Port)
ImpactedDomain: 4 0x4 (System)
IndicationIdentifier: "{0EA18AA4-898C-43E2-81F7-0DAD8D0BDA25}"
IndicationTime: "20100127003659.263000+480"
NetworkAddresses[0]: "192.168.0.1"
NetworkAddresses[1]: "10.48.74.12"
NetworkAddresses[2]: "10.48.74.16"
NetworkAddresses[3]: "10.48.74.11"
NetworkAddresses[4]: "10.48.74.3"
NetworkAddresses[5]: "10.48.74.17"
NetworkAddresses[6]: "10.48.74.15"
OSType: 69 0x45 (Microsoft Windows Server 2003)
OSVersion: "5.2.3790"
PerceivedSeverity: 6 0x6 (Critical)
ProbableCause: 7 0x7 (Communications Subsystem Failure)
ProbableCauseDescription: "Network Interface Lost Connectivity"
ProviderName: "HP Ethernet"
ProviderVersion: "2.1.0.0"
RecommendedActions[0]: "Check the network cables, switches, and hubs for the cause of the failure."
Summary: "Network interface lost connectivity"
SystemCreationClassName: "HP_WinComputerSystem"
SystemFirmwareVersion[0]: "2008.09.23"
SystemFirmwareVersion[1]: "2007.12.07"
SystemGUID: "32323534-3139-4e43-4738-32315333374b"
SystemModel: "ProLiant DL580 G5"
SystemName: "sccprmdb02101.cmbccd.cmbchina.com"
SystemProductID: "452291-B21"
SystemSerialNumber: "CNG821S37K"
TIME_CREATED: 129089974244553023 0x1ca9ea5cae9313f
UserComment: "2"
, For more information, please contact HP Support..

Event Type: Error
Event Source: SnapDrive
Event Category: (2)
Event ID: 304
Date:  1/27/2010
Time:  12:35:46 AM
User:  N/A
Computer: SCCPRMDB02101
Description:
The description for Event ID ( 304 ) in Source ( SnapDrive ) cannot be found. The local computer may not have the necessary registry information or message DLL files to display messages from a remote computer. You may be able to use the /AUXSOURCE= flag to retrieve this description; see Help and Support for details. The following information is part of the event: 800703f0, An attempt was made to reference a token that does not exist.
.

[watan]

Try setting the swsvc service to depend on the clussvc service so SnapDrive doesn't start before the cluster service.