Subscribe

Could not allocate 4096 bytes ... FAS2050C as NFS datastore for vSphere 4.x

We have a vSphere cluster (two ESX 4.0 hosts and one ESXi 4.1 host) with a FAS2050C (actually IBM N3600) sitting behind them as an NFS datastore.  The filer is running ONTAP 7.3.1.

From time to time, we get VM's that don't reboot or don't restart properly.  We're seeing the following errors in the VM's logfile:

> Oct 11 13:34:39.281: vmx| Vix: [125168 mainDispatch.c:3248]: VMAutomation_ReportPowerOpFinished: statevar=1, newAppState=1873, success=1
Oct 11 13:34:39.284: vmx| MONITOR MODE: allowed modes          : BT HV
Oct 11 13:34:39.288: vmx| MONITOR MODE: user requested modes   : BT HV HWMMU
Oct 11 13:34:39.289: vmx| MONITOR MODE: guestOS preferred modes: HWMMU HV BT
Oct 11 13:34:39.291: vmx| MONITOR MODE: filtered list          : HV BT
Oct 11 13:34:39.293: vmx| HV Settings: virtual exec = 'hardware'; virtual mmu = 'software'
Oct 11 13:34:41.391: vmx| VMMon_ConfigMemSched: vmmon.numVCPUs=1
Oct 11 13:34:41.393: vmx| CreateVM: Swap: generating normal swap file name.
Oct 11 13:34:41.415: vmx| Swap file path: '/vmfs/volumes/0a8368f6-3849b6aa/ContentStudioVM/ContentStudioVM-6c349bc3.vswp'
Oct 11 13:34:41.428: vmx| Using swap file '/vmfs/volumes/0a8368f6-3849b6aa/ContentStudioVM/ContentStudioVM-6c349bc3.vswp'
Oct 11 13:34:41.430: vmx| VMXVmdb_LoadRawConfig: Loading raw config
Oct 11 13:34:41.623: vmx| Failed to extend memory file from 0x0 bytes -> 0x1000 bytes.
Oct 11 13:34:41.625: vmx| BusMem: Failed to allocate frames for region BusError.
Oct 11 13:34:41.627: vmx| Msg_Post: Error
Oct 11 13:34:41.628: vmx| [msg.memVmnix.ftruncateFailed] Could not allocate 4096 bytes of anon memory: No space left on device.
Oct 11 13:34:41.630: vmx| [msg.moduletable.powerOnFailed] Module PhysMem power on failed.
Oct 11 13:34:41.631: vmx| ----------------------------------------
Oct 11 13:34:42.140: vmx| VMX_PowerOn: ModuleTable_PowerOn = 0
Oct 11 13:34:42.166: vmx| Vix: [125168 mainDispatch.c:3248]: VMAutomation_ReportPowerOpFinished: statevar=1, newAppState=1873, success=1
Oct 11 13:34:42.167: vmx| Vix: [125168 mainDispatch.c:3254]: VMAutomation: Ignoring ReportPowerOpFinished because the VMX is shutting down.
Oct 11 13:34:42.168: vmx| Vix: [125168 mainDispatch.c:3248]: VMAutomation_ReportPowerOpFinished: statevar=0, newAppState=1870, success=1
Oct 11 13:34:42.169: vmx| Vix: [125168 mainDispatch.c:3254]: VMAutomation: Ignoring ReportPowerOpFinished because the VMX is shutting down.
Oct 11 13:34:42.541: vmx| Transitioned vmx/execState/val to poweredOff
Oct 11 13:34:42.711: vmx| Vix: [125168 mainDispatch.c:3248]: VMAutomation_ReportPowerOpFinished: statevar=0, newAppState=1870, success=0
Oct 11 13:34:42.713: vmx| Vix: [125168 mainDispatch.c:3254]: VMAutomation: Ignoring ReportPowerOpFinished because the VMX is shutting down.
Oct 11 13:34:42.715: vmx| VMX idle exit
Oct 11 13:34:43.104: vmx| Vix: [125168 mainDispatch.c:599]: VMAutomation_LateShutdown()
Oct 11 13:34:43.107: vmx| Vix: [125168 mainDispatch.c:549]: VMAutomationCloseListenerSocket. Closing listener socket.
Oct 11 13:34:43.841: vmx| Flushing VMX VMDB connections
Oct 11 13:34:44.387: vmx| IPC_exit: disconnecting all threads
Oct 11 13:34:44.388: vmx| VMX exit (0).
Oct 11 13:34:44.390: vmx| AIOMGR-S : stat o=1522 r=3152 w=44 i=866 br=50065408 bw=2793472
Oct 11 13:34:44.392: vmx| VMX has left the building: 0.

Since all of these ESX hosts are configured in a cluster, the VM swapfiles live on the NetApp.  There is plenty of space available on this particular NetApp volume, so the "No space" error is incorrect.

If we fire up this VM again after the error message, it comes up fine -- so this problem is intermittent.

Anyone have any thoughts?  At first glance it seems liek the NetApp isn't able to honor the allocate request in time.  Maybe there is some internal ESX timeout setting we can increase?  Or maybe someone knows of a VMware KB that mentions this issue or that an NFS setting could help make a difference here?

Thanks in advance...

Ray

Re: Could not allocate 4096 bytes ... FAS2050C as NFS datastore for vSphere 4.x

Hi Ray,

I'm not 100% sure whether that may be the culprit, but what are your NFS settings on the ESX side?

In particular NetApp recommends setting these values:

NFS.HeartbeatFrequency = 12
NFS.HeartbeatMaxFailures = 10

Regards,

Radek

Re: Could not allocate 4096 bytes ... FAS2050C as NFS datastore for vSphere 4.x

Thanks for the reply.  Those are the settings we're currently using per TR3428.