Fas2040 Vsphere 4.1 NFC error - nfs storage

slinuxuzer · ‎2011-02-28

Hello everyone, we recently purchased a Netapp FAS2040 Filer to replace an ageing FAS940 filer, I am currently running Vsphere 4.1 with ESXI 4.1 on two new DL 380 G6's.

I have both filers mounted and am in the process of migrating my vm's over to the new filer.

I am able to upload large files to the FAS940 through the Vsphere Datastore browser, but I am unable to upload the exact same files to the FAS2040, I have tested Svmotion'ing over vm's and they move over fine, I have even CIFS shared a datastore on the 2040 and uploaded these same files (7GB) to the 2040 this way, I feel confident that my NFS storage connections were setup properly, as a Netapp engineer whom I believe is competent set them up, he has told me that this feature doesn't work alot of the time, so he CIFS shares a tools datastore to upload tools to, I would normally leave the issue at that, but I recently tried to deploy a .ovf template to the 2040 and it would not upload, it would however upload to the FAS940.

I had opened a case with VMware, but so far they were not able to solve the issue, we are in agreement that there is a filer issue at some level, (I believe this simply because both issues work on the FAS940)

So far I have tried,

1. hard coding GIG-E full and Auto-negotiate for my nics (managment and storage) at all levels

2. deleting and re-creating port groups an virtual switches

3. I have tested it with various flow control settings.

Here is a snippet of from tail -f /var/log/vmware/vpx/vpxa.og (apparently the upload gets transferred from the client to the managment network interface of ESXI, I confirmed this with wireshark)

I would be deeply endebted to anyone who could assist in solving this, as I have been trying to solve this for almost two weeks now.

Thanks in advance.

A couple of lines that concern me are

[2011-02-26 22:17:08.209 15BFDB90 info 'Libs'] UUID: Unable to open /dev/mem: No such file or directory

[2011-02-26 22:16:58.026 160B5B90 warning 'Libs'] [NFC ERROR] NfcBufWrite: failed to write file

[2011-02-26 22:16:58.029 15CC0B90 warning 'Libs' opID=task-internal-4905-b765816c] [NFC ERROR] NfcBuf_Recv: session error 4

[2011-02-26 22:16:58.029 15CC0B90 warning 'Libs' opID=task-internal-4905-b765816c] [NFC ERROR] NfcServerLoop: failed to receive file data

[2011-02-26 22:16:58.153 15CC0B90 error 'App' opID=task-internal-4905-b765816c] [VPXNFCSERVER] Nfc server failed: File error -- NfcBufWrite: failed to write file

[2011-02-26 22:16:58.153 15CC0B90 verbose 'App' opID=task-internal-4905-b765816c] [VPXNFCSERVER] Closing NFC session

-------------------------------------------------------------------------------------------------------------------------------------------

[2011-02-26 22:16:31.594 15CC0B90 verbose 'App'] [VpxaVMAP] CheckThreshold percent changes (0,0,0,0) - threshold 5
[2011-02-26 22:16:32.679 15D83B90 verbose 'SoapAdapter.HTTPService'] User agent is 'VMware-client/4.1.0'
[2011-02-26 22:16:32.679 15D83B90 verbose 'SoapAdapter.HTTPService'] HTTP Response: Client: NeedsContentLength: false UnderstandsChunking: true CanKeepAlive: true (PresetContentLength -1)
[2011-02-26 22:16:32.679 15B3AB90 info 'App' opID=task-internal-4897-a5a86df9] [VpxLRO] -- BEGIN task-internal-4897 -- -- vpxapi.VpxaService.fetchQuickStats -- 52146ae9-e337-9f23-12ea-0f2067427e66
[2011-02-26 22:16:32.680 15B3AB90 verbose 'SoapAdapter.HTTPService' opID=task-internal-4897-a5a86df9] HTTP Response: Complete (processed 972 bytes)
[2011-02-26 22:16:32.680 15B3AB90 info 'App' opID=task-internal-4897-a5a86df9] [VpxLRO] -- FINISH task-internal-4897 -- -- vpxapi.VpxaService.fetchQuickStats -- 52146ae9-e337-9f23-12ea-0f2067427e66
[2011-02-26 22:16:35.239 15C3EB90 warning 'App'] [VpxaHalStats] Unexpected return result. Expect 1 sample, receive 2
[2011-02-26 22:16:38.200 15AF9B90 verbose 'App'] [VpxaMoVm::CheckMoVm] did not find a VM with ID 7 in the vmList
[2011-02-26 22:16:38.200 15AF9B90 verbose 'App'] [VpxaAlarm] VM with vmid = 7 not found
[2011-02-26 22:16:48.203 15AF9B90 verbose 'App'] [VpxaMoVm::CheckMoVm] did not find a VM with ID 7 in the vmList
[2011-02-26 22:16:48.203 15AF9B90 verbose 'App'] [VpxaAlarm] VM with vmid = 7 not found
[2011-02-26 22:16:51.595 15DC4B90 verbose 'App'] [VpxaVMAP] Checking Node Resources
[2011-02-26 22:16:51.602 15DC4B90 verbose 'App'] [VpxaVMAP] CheckThreshold values stored:(20384,43045,20384,43045) - retrieved:(20384,42974,20384,42974)
[2011-02-26 22:16:51.602 15DC4B90 verbose 'App'] [VpxaVMAP] CheckThreshold percent changes (0,0,0,0) - threshold 5
[2011-02-26 22:16:53.090 15D42B90 verbose 'App'] [VpxaVMAP] Monitoring AAM health: vpxdDasStateOnLastInvocation(running) currentVpxdDasState(running) forceRunOfListNodes(0) isDasEnabled(1) skipOperation(0)
[2011-02-26 22:16:53.090 15D42B90 verbose 'App'] [VpxaVMAP::Invoke]Command to invoke is /opt/vmware/aam/bin/aamPerl /opt/vmware/aam/ha/aam_config_util.pl -z -shortname=txoesx1 -uname=VMkernel -cmd=monitornodes -domain=vmware
[2011-02-26 22:16:53.091 15D42B90 info 'SysCommandPosix'] ForkExec(/opt/vmware/aam/bin/aamPerl) 613699
[2011-02-26 22:16:54.896 15CC0B90 verbose 'SoapAdapter.HTTPService'] User agent is 'VMware-client/4.1.0'
[2011-02-26 22:16:54.896 15CC0B90 verbose 'SoapAdapter.HTTPService'] HTTP Response: Client: NeedsContentLength: false UnderstandsChunking: true CanKeepAlive: true (PresetContentLength 129)
[2011-02-26 22:16:54.896 15CC0B90 verbose 'SoapAdapter.HTTPService'] HTTP Response: Auto-completing at 129/129 bytes
[2011-02-26 22:16:54.896 15C3EB90 verbose 'SoapAdapter.HTTPService'] User agent is 'VMware-client/4.1.0'
[2011-02-26 22:16:54.897 15C3EB90 verbose 'SoapAdapter.HTTPService'] HTTP Response: Client: NeedsContentLength: false UnderstandsChunking: true CanKeepAlive: true (PresetContentLength -1)
[2011-02-26 22:16:54.897 159EBB90 info 'App' opID=C45D4F31-0000008A-16] [VpxLRO] -- BEGIN task-internal-4898 -- -- vpxapi.VpxaService.MakeDirectory -- 52146ae9-e337-9f23-12ea-0f2067427e66
[2011-02-26 22:16:54.899 159EBB90 verbose 'App' opID=C45D4F31-0000008A-16] [VpxaDatastoreContext] Resolved URL netfs://170.4.216.181//vol/test/ISO/SUSE to localPath /vmfs/volumes/c8cd2f66-895d1e33/ISO/SUSE
[2011-02-26 22:16:54.902 159EBB90 verbose 'SoapAdapter.HTTPService' opID=C45D4F31-0000008A-16] HTTP Response: Complete (processed 396 bytes)
[2011-02-26 22:16:54.902 159EBB90 info 'App' opID=C45D4F31-0000008A-16] [VpxLRO] -- FINISH task-internal-4898 -- -- vpxapi.VpxaService.MakeDirectory -- 52146ae9-e337-9f23-12ea-0f2067427e66
[2011-02-26 22:16:54.909 15B3AB90 verbose 'SoapAdapter.HTTPService'] User agent is 'VMware-client/4.1.0'
[2011-02-26 22:16:54.909 15B3AB90 verbose 'SoapAdapter.HTTPService'] HTTP Response: Client: NeedsContentLength: false UnderstandsChunking: true CanKeepAlive: true (PresetContentLength -1)
[2011-02-26 22:16:54.910 15BFDB90 info 'App' opID=C45D4F31-0000008C-1b] [VpxLRO] -- BEGIN task-internal-4899 -- -- vpxapi.VpxaService.nfcMakeTicketByDatastore -- 52146ae9-e337-9f23-12ea-0f2067427e66
[2011-02-26 22:16:54.911 15BFDB90 verbose 'App' opID=C45D4F31-0000008C-1b] [VpxaDatastoreContext] Resolved URL netfs://170.4.216.181//vol/test/ to localPath /vmfs/volumes/c8cd2f66-895d1e33/
[2011-02-26 22:16:54.912 15BFDB90 info 'Libs' opID=C45D4F31-0000008C-1b] UUID: Unable to open /dev/mem: No such file or directory
[2011-02-26 22:16:54.913 15BFDB90 verbose 'SoapAdapter.HTTPService' opID=C45D4F31-0000008C-1b] HTTP Response: Complete (processed 601 bytes)
[2011-02-26 22:16:54.913 15BFDB90 info 'App' opID=C45D4F31-0000008C-1b] [VpxLRO] -- FINISH task-internal-4899 -- -- vpxapi.VpxaService.nfcMakeTicketByDatastore -- 52146ae9-e337-9f23-12ea-0f2067427e66
[2011-02-26 22:16:54.918 15BBCB90 verbose 'SoapAdapter.HTTPService'] User agent is 'VMware-client/4.1.0'
[2011-02-26 22:16:54.918 15BBCB90 verbose 'SoapAdapter.HTTPService'] HTTP Response: Client: NeedsContentLength: false UnderstandsChunking: true CanKeepAlive: true (PresetContentLength -1)
[2011-02-26 22:16:54.919 15B3AB90 info 'App' opID=C45D4F31-0000008D-ab] [VpxLRO] -- BEGIN task-internal-4900 -- -- vpxapi.VpxaService.searchDatastore -- 52146ae9-e337-9f23-12ea-0f2067427e66
[2011-02-26 22:16:54.920 15B3AB90 verbose 'App' opID=C45D4F31-0000008D-ab] [VpxaDatastoreContext] Resolved URL netfs://170.4.216.181//vol/test/ISO to localPath /vmfs/volumes/c8cd2f66-895d1e33/ISO
[2011-02-26 22:16:54.942 15B3AB90 verbose 'App' opID=C45D4F31-0000008D-ab] [VpxURL] Resolved localPath /vmfs/volumes/c8cd2f66-895d1e33/ISO/ to URL netfs://170.4.216.181//vol/test/ISO/
[2011-02-26 22:16:54.944 15B3AB90 verbose 'App' opID=C45D4F31-0000008D-ab] [VpxURL] Resolved localPath /vmfs/volumes/c8cd2f66-895d1e33/ISO/ to URL netfs://170.4.216.181//vol/test/ISO/
[2011-02-26 22:16:54.944 15B3AB90 verbose 'SoapAdapter.HTTPService' opID=C45D4F31-0000008D-ab] HTTP Response: Complete (processed 637 bytes)
[2011-02-26 22:16:54.944 15B3AB90 info 'App' opID=C45D4F31-0000008D-ab] [VpxLRO] -- FINISH task-internal-4900 -- -- vpxapi.VpxaService.searchDatastore -- 52146ae9-e337-9f23-12ea-0f2067427e66
[2011-02-26 22:16:55.239 15B7BB90 warning 'App'] [VpxaHalStats] Unexpected return result. Expect 1 sample, receive 2
[2011-02-26 22:16:55.240 159EBB90 verbose 'App'] Accepted NFC connection (useSSL=0), spawning server
[2011-02-26 22:16:55.241 159EBB90 info 'App'] PROXY connection to NFC: found session ticket:
[N9VimShared20NfcTicketByDatastoreE:0xffb134ac]
[2011-02-26 22:16:55.241 15C3EB90 info 'App' opID=task-internal-4901-e4437697] [VpxLRO] -- BEGIN task-internal-4901 -- -- VpxNfcServerLro --
[2011-02-26 22:16:55.241 15C3EB90 verbose 'App' opID=task-internal-4901-e4437697] VpxNfcServer::StartWork: useSSL=0

[2011-02-26 22:16:55.241 15C3EB90 verbose 'App' opID=task-internal-4901-e4437697] [VPXNFCSERVER] Starting NFC server loop
[2011-02-26 22:16:55.355 15C3EB90 verbose 'App' opID=task-internal-4901-e4437697] [VpxaVmprovUtil] DsPathToLocalPath conversion: [TEST] ISO/SUSE/SUSE MBRAlign.mf -> /vmfs/volumes/c8cd2f66-895d1e33/ISO/SUSE/SUSE MBRAlign.mf
[2011-02-26 22:16:55.355 15C3EB90 verbose 'App' opID=task-internal-4901-e4437697] [VpxaDatastoreContext] Resolved DsPath [TEST] ISO/SUSE/SUSE MBRAlign.mf to localPath /vmfs/volumes/c8cd2f66-895d1e33/ISO/SUSE/SUSE MBRAlign.mf
[2011-02-26 22:16:55.360 15C3EB90 verbose 'App' opID=task-internal-4901-e4437697] [VpxaVmprovUtil] DsPathToLocalPath conversion: [TEST] ISO/SUSE/SUSE MBRAlign.mf -> /vmfs/volumes/c8cd2f66-895d1e33/ISO/SUSE/SUSE MBRAlign.mf
[2011-02-26 22:16:55.360 15C3EB90 verbose 'App' opID=task-internal-4901-e4437697] [VpxaDatastoreContext] Resolved DsPath [TEST] ISO/SUSE/SUSE MBRAlign.mf to localPath /vmfs/volumes/c8cd2f66-895d1e33/ISO/SUSE/SUSE MBRAlign.mf
[2011-02-26 22:16:55.366 15C3EB90 verbose 'App' opID=task-internal-4901-e4437697] [VpxaDatastoreContext] No conversion for localpath /vmfs/volumes/c8cd2f66-895d1e33/ISO/SUSE/SUSE MBRAlign.mf
[2011-02-26 22:16:55.373 15C3EB90 verbose 'App' opID=task-internal-4901-e4437697] [VpxaVmprovUtil] DsPathToLocalPath conversion: [TEST] ISO/SUSE/SUSE MBRAlign.mf -> /vmfs/volumes/c8cd2f66-895d1e33/ISO/SUSE/SUSE MBRAlign.mf
[2011-02-26 22:16:55.373 15C3EB90 verbose 'App' opID=task-internal-4901-e4437697] [VpxaDatastoreContext] Resolved DsPath [TEST] ISO/SUSE/SUSE MBRAlign.mf to localPath /vmfs/volumes/c8cd2f66-895d1e33/ISO/SUSE/SUSE MBRAlign.mf
[2011-02-26 22:16:55.376 15C3EB90 verbose 'App' opID=task-internal-4901-e4437697] [VPXNFCSERVER] Closing NFC session
[2011-02-26 22:16:55.376 15C3EB90 info 'App' opID=task-internal-4901-e4437697] [VpxLRO] -- FINISH task-internal-4901 -- -- VpxNfcServerLro --
[2011-02-26 22:16:55.379 15BFDB90 verbose 'SoapAdapter.HTTPService'] User agent is 'VMware-client/4.1.0'
[2011-02-26 22:16:55.379 15BFDB90 verbose 'SoapAdapter.HTTPService'] HTTP Response: Client: NeedsContentLength: false UnderstandsChunking: true CanKeepAlive: true (PresetContentLength -1)
[2011-02-26 22:16:55.380 15C3EB90 info 'App' opID=C45D4F31-00000091-68] [VpxLRO] -- BEGIN task-internal-4902 -- -- vpxapi.VpxaService.nfcMakeTicketByDatastore -- 52146ae9-e337-9f23-12ea-0f2067427e66
[2011-02-26 22:16:55.381 15C3EB90 verbose 'App' opID=C45D4F31-00000091-68] [VpxaDatastoreContext] Resolved URL netfs://170.4.216.181//vol/test/ to localPath /vmfs/volumes/c8cd2f66-895d1e33/
[2011-02-26 22:16:55.382 15C3EB90 info 'Libs' opID=C45D4F31-00000091-68] UUID: Unable to open /dev/mem: No such file or directory
[2011-02-26 22:16:55.382 15C3EB90 verbose 'SoapAdapter.HTTPService' opID=C45D4F31-00000091-68] HTTP Response: Complete (processed 601 bytes)
[2011-02-26 22:16:55.382 15C3EB90 info 'App' opID=C45D4F31-00000091-68] [VpxLRO] -- FINISH task-internal-4902 -- -- vpxapi.VpxaService.nfcMakeTicketByDatastore -- 52146ae9-e337-9f23-12ea-0f2067427e66
[2011-02-26 22:16:55.711 15D83B90 verbose 'App'] Accepted NFC connection (useSSL=0), spawning server
[2011-02-26 22:16:55.712 15D83B90 info 'App'] PROXY connection to NFC: found session ticket:
[N9VimShared20NfcTicketByDatastoreE:0xffa4cba4]
[2011-02-26 22:16:55.712 15B3AB90 info 'App' opID=task-internal-4903-dc6e02fc] [VpxLRO] -- BEGIN task-internal-4903 -- -- VpxNfcServerLro --
[2011-02-26 22:16:55.712 15B3AB90 verbose 'App' opID=task-internal-4903-dc6e02fc] VpxNfcServer::StartWork: useSSL=0

[2011-02-26 22:16:55.712 15B3AB90 verbose 'App' opID=task-internal-4903-dc6e02fc] [VPXNFCSERVER] Starting NFC server loop
[2011-02-26 22:16:55.716 15B3AB90 verbose 'App' opID=task-internal-4903-dc6e02fc] [VpxaVmprovUtil] DsPathToLocalPath conversion: [TEST] ISO/SUSE/SUSE MBRAlign.ovf -> /vmfs/volumes/c8cd2f66-895d1e33/ISO/SUSE/SUSE MBRAlign.ovf
[2011-02-26 22:16:55.716 15B3AB90 verbose 'App' opID=task-internal-4903-dc6e02fc] [VpxaDatastoreContext] Resolved DsPath [TEST] ISO/SUSE/SUSE MBRAlign.ovf to localPath /vmfs/volumes/c8cd2f66-895d1e33/ISO/SUSE/SUSE MBRAlign.ovf
[2011-02-26 22:16:55.721 15B3AB90 verbose 'App' opID=task-internal-4903-dc6e02fc] [VpxaVmprovUtil] DsPathToLocalPath conversion: [TEST] ISO/SUSE/SUSE MBRAlign.ovf -> /vmfs/volumes/c8cd2f66-895d1e33/ISO/SUSE/SUSE MBRAlign.ovf
[2011-02-26 22:16:55.721 15B3AB90 verbose 'App' opID=task-internal-4903-dc6e02fc] [VpxaDatastoreContext] Resolved DsPath [TEST] ISO/SUSE/SUSE MBRAlign.ovf to localPath /vmfs/volumes/c8cd2f66-895d1e33/ISO/SUSE/SUSE MBRAlign.ovf
[2011-02-26 22:16:55.727 15B3AB90 verbose 'App' opID=task-internal-4903-dc6e02fc] [VpxaDatastoreContext] No conversion for localpath /vmfs/volumes/c8cd2f66-895d1e33/ISO/SUSE/SUSE MBRAlign.ovf
[2011-02-26 22:16:55.734 15B3AB90 verbose 'App' opID=task-internal-4903-dc6e02fc] [VpxaVmprovUtil] DsPathToLocalPath conversion: [TEST] ISO/SUSE/SUSE MBRAlign.ovf -> /vmfs/volumes/c8cd2f66-895d1e33/ISO/SUSE/SUSE MBRAlign.ovf
[2011-02-26 22:16:55.734 15B3AB90 verbose 'App' opID=task-internal-4903-dc6e02fc] [VpxaDatastoreContext] Resolved DsPath [TEST] ISO/SUSE/SUSE MBRAlign.ovf to localPath /vmfs/volumes/c8cd2f66-895d1e33/ISO/SUSE/SUSE MBRAlign.ovf
[2011-02-26 22:16:55.737 15B3AB90 verbose 'App' opID=task-internal-4903-dc6e02fc] [VPXNFCSERVER] Closing NFC session
[2011-02-26 22:16:55.737 15B3AB90 info 'App' opID=task-internal-4903-dc6e02fc] [VpxLRO] -- FINISH task-internal-4903 -- -- VpxNfcServerLro --
[2011-02-26 22:16:55.740 15BFDB90 verbose 'SoapAdapter.HTTPService'] User agent is 'VMware-client/4.1.0'
[2011-02-26 22:16:55.740 15BFDB90 verbose 'SoapAdapter.HTTPService'] HTTP Response: Client: NeedsContentLength: false UnderstandsChunking: true CanKeepAlive: true (PresetContentLength -1)
[2011-02-26 22:16:55.741 FF937B10 info 'App' opID=C45D4F31-00000092-14] [VpxLRO] -- BEGIN task-internal-4904 -- -- vpxapi.VpxaService.nfcMakeTicketByDatastore -- 52146ae9-e337-9f23-12ea-0f2067427e66
[2011-02-26 22:16:55.742 FF937B10 verbose 'App' opID=C45D4F31-00000092-14] [VpxaDatastoreContext] Resolved URL netfs://170.4.216.181//vol/test/ to localPath /vmfs/volumes/c8cd2f66-895d1e33/
[2011-02-26 22:16:55.743 FF937B10 info 'Libs' opID=C45D4F31-00000092-14] UUID: Unable to open /dev/mem: No such file or directory
[2011-02-26 22:16:55.744 FF937B10 verbose 'SoapAdapter.HTTPService' opID=C45D4F31-00000092-14] HTTP Response: Complete (processed 601 bytes)
[2011-02-26 22:16:55.744 FF937B10 info 'App' opID=C45D4F31-00000092-14] [VpxLRO] -- FINISH task-internal-4904 -- -- vpxapi.VpxaService.nfcMakeTicketByDatastore -- 52146ae9-e337-9f23-12ea-0f2067427e66
[2011-02-26 22:16:56.070 15B7BB90 verbose 'App'] Accepted NFC connection (useSSL=0), spawning server
[2011-02-26 22:16:56.071 15B7BB90 info 'App'] PROXY connection to NFC: found session ticket:
[N9VimShared20NfcTicketByDatastoreE:0xffb0d53c]
[2011-02-26 22:16:56.071 15CC0B90 info 'App' opID=task-internal-4905-b765816c] [VpxLRO] -- BEGIN task-internal-4905 -- -- VpxNfcServerLro --
[2011-02-26 22:16:56.071 15CC0B90 verbose 'App' opID=task-internal-4905-b765816c] VpxNfcServer::StartWork: useSSL=0

[2011-02-26 22:16:56.071 15CC0B90 verbose 'App' opID=task-internal-4905-b765816c] [VPXNFCSERVER] Starting NFC server loop
[2011-02-26 22:16:56.076 15CC0B90 verbose 'App' opID=task-internal-4905-b765816c] [VpxaVmprovUtil] DsPathToLocalPath conversion: [TEST] ISO/SUSE/SUSE_MBRAlign-disk1.vmdk -> /vmfs/volumes/c8cd2f66-895d1e33/ISO/SUSE/SUSE_MBRAlign-disk1.vmdk
[2011-02-26 22:16:56.076 15CC0B90 verbose 'App' opID=task-internal-4905-b765816c] [VpxaDatastoreContext] Resolved DsPath [TEST] ISO/SUSE/SUSE_MBRAlign-disk1.vmdk to localPath /vmfs/volumes/c8cd2f66-895d1e33/ISO/SUSE/SUSE_MBRAlign-disk1.vmdk
[2011-02-26 22:16:56.081 15CC0B90 verbose 'App' opID=task-internal-4905-b765816c] [VpxaVmprovUtil] DsPathToLocalPath conversion: [TEST] ISO/SUSE/SUSE_MBRAlign-disk1.vmdk -> /vmfs/volumes/c8cd2f66-895d1e33/ISO/SUSE/SUSE_MBRAlign-disk1.vmdk
[2011-02-26 22:16:56.081 15CC0B90 verbose 'App' opID=task-internal-4905-b765816c] [VpxaDatastoreContext] Resolved DsPath [TEST] ISO/SUSE/SUSE_MBRAlign-disk1.vmdk to localPath /vmfs/volumes/c8cd2f66-895d1e33/ISO/SUSE/SUSE_MBRAlign-disk1.vmdk
[2011-02-26 22:16:56.085 15CC0B90 verbose 'App' opID=task-internal-4905-b765816c] [VpxaVmprovUtil] DsPathToLocalPath conversion: [TEST] ISO/SUSE/SUSE_MBRAlign-disk1.vmdk -> /vmfs/volumes/c8cd2f66-895d1e33/ISO/SUSE/SUSE_MBRAlign-disk1.vmdk
[2011-02-26 22:16:56.085 15CC0B90 verbose 'App' opID=task-internal-4905-b765816c] [VpxaDatastoreContext] Resolved DsPath [TEST] ISO/SUSE/SUSE_MBRAlign-disk1.vmdk to localPath /vmfs/volumes/c8cd2f66-895d1e33/ISO/SUSE/SUSE_MBRAlign-disk1.vmdk
[2011-02-26 22:16:57.542 15B3AB90 verbose 'VpxaHalCnxHostagent'] Received callback in WaitForUpdatesDone
[2011-02-26 22:16:57.542 15B3AB90 verbose 'VpxaHalCnxHostagent'] [VpxaHalCnxHostagent::ProcessUpdate] Applying updates from 5539 to 5540 (at 5539)
[2011-02-26 22:16:57.542 15B3AB90 verbose 'VpxaHalResourcePool'] [VpxaHostdSpecSync] Received change notification from hostd ...
[2011-02-26 22:16:57.542 15B3AB90 verbose 'VpxaHalResourcePool'] [VpxaHostdSpecSync] [IsVmListChangeUpdate] returning false
[2011-02-26 22:16:57.542 15B3AB90 verbose 'VpxaHalResourcePool'] [VpxaHostdSpecSync] Setting _syncPending = true ...
[2011-02-26 22:16:57.542 15B3AB90 verbose 'VpxaHalResourcePool'] [VpxaHostdSpecSync] Launching ProcessResourceNotification in new thread...
[2011-02-26 22:16:57.542 15B7BB90 verbose 'VpxaHalResourcePool'] [VpxaHostdSpecSync] Operation is transform ...
[2011-02-26 22:16:57.549 15B7BB90 verbose 'VpxaHalResourcePool'] [VpxaHostdSpecSync] Copied root values ...
[2011-02-26 22:16:57.549 15B7BB90 verbose 'VpxaHalResourcePool'] [VpxaHostdSpecSync] Generated list of transforming operations ...
[2011-02-26 22:16:57.549 15B7BB90 verbose 'VpxaHalResourcePool'] [OverrideTreeOnHostd] oldOperations != new operations, trying with new operation set...
[2011-02-26 22:16:57.549 15B7BB90 info 'VpxaHalResourcePool'] [OverrideTreeOnHostd] Trees are identical, nothing to do
[2011-02-26 22:16:57.549 15B7BB90 verbose 'VpxaHalResourcePool'] [OverrideTreeOnHostd] currOverrideState = 0, application succeeded, exiting...
[2011-02-26 22:16:57.549 15B7BB90 verbose 'VpxaHalResourcePool'] [ProcessResourceNotification] No syncs pending, exiting loop...
[2011-02-26 22:16:58.026 160B5B90 warning 'Libs'] [NFC ERROR] NfcBufWrite: failed to write file

[2011-02-26 22:16:58.029 15CC0B90 warning 'Libs' opID=task-internal-4905-b765816c] [NFC ERROR] NfcBuf_Recv: session error 4

[2011-02-26 22:16:58.029 15CC0B90 warning 'Libs' opID=task-internal-4905-b765816c] [NFC ERROR] NfcServerLoop: failed to receive file data

[2011-02-26 22:16:58.153 15CC0B90 error 'App' opID=task-internal-4905-b765816c] [VPXNFCSERVER] Nfc server failed: File error -- NfcBufWrite: failed to write file

[2011-02-26 22:16:58.153 15CC0B90 verbose 'App' opID=task-internal-4905-b765816c] [VPXNFCSERVER] Closing NFC session
[2011-02-26 22:16:58.153 15CC0B90 info 'App' opID=task-internal-4905-b765816c] [VpxLRO] -- FINISH task-internal-4905 -- -- VpxNfcServerLro --
[2011-02-26 22:16:58.153 15CC0B90 info 'App' opID=task-internal-4905-b765816c] [VpxLRO] -- ERROR task-internal-4905 -- -- VpxNfcServerLro: vmodl.fault.SystemError:
Result:
(vmodl.fault.SystemError) {
   dynamicType = <unset>,
   faultCause = (vmodl.MethodFault) null,
   reason = "Nfc server failed: File error -- NfcBufWrite: failed to write file
",
   msg = "",
}
Args:

[2011-02-26 22:16:58.203 15AF9B90 verbose 'App'] [VpxaMoVm::CheckMoVm] did not find a VM with ID 7 in the vmList
[2011-02-26 22:16:58.204 15AF9B90 verbose 'App'] [VpxaAlarm] VM with vmid = 7 not found
[2011-02-26 22:16:58.404 15D42B90 verbose 'App'] [VpxaVMAP::Invoke] task percent done is 100
[2011-02-26 22:16:59.130 15D42B90 verbose 'App'] [VpxaVMAP::Invoke] Command output:
02/26/11 22:16:53 [print_args ] Invoked command:

02/26/11 22:16:53 [print_args ] /opt/vmware/aam/bin/ftPerl /opt/vmware/aam/ha/aam_config_util.pl -z -shortname=txoesx1 -uname=VMkernel -cmd=monitornodes -domain=vmware

-------------------------------------------------------------------------------------------------------------------------------------------

vstewart · ‎2011-02-28

A great way to audit your configuration is to use the VMware vSphere plug-in, the NetApp VSC. It will audit an dcorrect your current settings and ensure they are set to best practices (as defined in TR-3749).

We make the tools to make your life easier.

Vaughn Stewart

Director, NetApp

slinuxuzer · ‎2011-03-01

Hey Vaughn thanks for your quick response, I forgot to mention I have already installed the latest version of the VSC and configured the recommneded values for the FAS2040.

I also believe that I have the correct configs in place in accordance with Tr-3749

vstewart · ‎2011-03-01

Have you used the VSC to make a net-new datastore and tested your copy operation with that new datastore?

slinuxuzer · ‎2011-03-01

I am not sure what you mean by "net-new" datastore, but I have created a new datastore throught the VSC, I have also created volumes and manually mounted them through V-center and I get the same results from each tests.

slinuxuzer · ‎2011-03-07

Vaughn, I have actually narrowed the issue down to the network card Hp's NC375T, when I switch over to a NC382i card the transfer runs fine. I have found another post where someone apparently was having the same issue.

http://communities.netapp.com/thread/9127

I also noticed from running the nfs stat command on my filer that I am seeing the badcalls and xdrcall fields get incremented by one each time I try the upload and it fails.

Also running the netdiag command on the filer revealed a message.

The TCP receive window advertised by NFS client {Esx-Hostname-NFS2} is 4163.
This is less than the recommended value of 8760 bytes.
You should increase the TCP receive buffer size for NFS on the client.

I called Vmware to determine how to change this setting in my ESXI host and they advised me this is set under the host configuration tab, advanced settings (Software) and that the field is labeled NFS.ReceiveBufferSize

The field allows for 64 - 264 KB, mine was set to 128. Changing this to 264 and remounting the storage did get rid of the warning message on the filer and allowed the upload to run longer before failure, but did ultimatley still fail.

Again, the transfer runs successfully without changing any ESXI settings provided I change over to a different card, the NC375T is a Vmware HCL supported card.

I have an open case with Netapp regarding this issue. Case#2002030957

eric_barlier · ‎2011-03-07

have you thought about trying other methods to copy data across?

ndmpcopy

snapmirror

vol copy

from the 940 to the new controller.

Hope this helps.

Eric

slinuxuzer · ‎2011-03-08

I actually solved this issue, in January I downloaded the latest version of ESXI, apparently this had an older driver for NC375T bundled with it, I downloaded the newest driver for ESXI from Hp and installed it and the card started working like a champ.

This driver was only one revision newer.