RCU Guest Customization problem

[admstaceya]

Hi, I'm having a problem with the Guest customization with RCU.  I can clone the VMs fine (flexclone elements, etc), but the customization fails to take.  I've tried Sysprep and standard customisation profiles.  These profiles do work fine when I clone a VM through the usual Vsphere cloning wizard.

I have tried both RCU 3.0 and 3.1 (VSC 2.0)

Platform setup:

Vcenter 4.0 U2 (I have also reproduced with 4.0 U1)
ESXi 4.0 U2 (I have not attempted with previous versions)
Vcenter is running on Windows 2008 R2 (64bit)
VMFS volumes are in use on FC.

Cloned machine: Windows XP or Windows 2003 32-bit (tried both to ensure its not the image)

There are no errors in the kamino.log file, other than a delay and retry on the customisation, which may be normal.  The other logs I get are related to vmount2.exe which seem to amount to the fact that it is being used.  The VPXD logs (attached) show that it has attempted to fail back to vmount due to not being able to detect vmware tools for the VM.

I've attached snippets of the vpxd log files that relate to customisation - one for a singe machine cloned with RCU and another by using clone in the traditional vpshere way.

Pretty stumped on this; all the hard bits work!

[costea]

What version of VMware Tools do you have installed on the guest OS template?  I just tested with Version 4.0.0, build-208167 and it worked for me.

[admstaceya]

Tools is Version 4.0.0 build-261974.

I've taken another look and I can see a few differences between the VMs when created by each method - this is looking at the machine folders without starting any machines up.

Vsphere method: (machine is called w2k3-2vsph)

w2k3-2vsph.vmdk

vmware.log

w2k3-2vsph.nvram

w2k3-2vsph.vmx

w2k3-2vsph.vmxf

w2k3-2vsph.vmsd

imc433e.tmp

RCU Method:

w2k3-2clo.vmdk

w2k3-2clo.vmx

w2k3-2clo.vmxf

w2k3-2clo.vmsd

I've attached the contents of these folders barring the .tmp and the vmdk.

You can see in the vsph method that the customisaton file is in the vmx file ready for deployment and there are a few other differences, but both do state the same details for the vmware tools version.

[chrisgeb]

There is a VMware KB  that provides info on how to verify that the cust spec is running...  It may be useful to get the  %SYSTEMROOT%\Temp\guestcust.log file and see what that says if anything. 

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1006842

[admstaceya]

In the guest c:\windows\temp contains two files:

bootrunmsg.dll

bootrun-script.log

C:\vmware-vmount-bootrun-temp-7d50>move /y \vmware-vmount-bootrun-temp-7d50\autochk.bak C:\WINDOWS\system32\autochk.exe

          1 file(s) moved.

C:\vmware-vmount-bootrun-temp-7d50>deployPkg.exe -deploy imcC452.tmp

There is also a vmprep.log file which I've attached.  So something has happened but not clear what exactly.

On the Vcenter server, the only log file in temp thats relevant is vmount2.log, extract below.  Can someone take a look at the Application event logs on their Vcenter server and see if there are any Information events referring to vmount2.exe around the customisation time of a deployment to see if the problem is with vmount, or the fact that vmount is being used in this case.

vmount2.log: (this information is also what appears in the event log)

[07/29/10 10:18:02] Status:  (VmountDisk:226) disk vpxa-nfc://sanfs://vmfs_uuid:4c2cb500-e98b1d60-25c6-00215e221476/w2k3-2clo/w2k3-2clo.vmdk@vmd-02.domain.local:902 was opened
[07/29/10 10:18:02] Status:  (VmountDevice:1492) device of type volume with offset 32256 was created
[07/29/10 10:18:03] Status:  (VmountDevice:142) device of type volume with offset 32256 was closed
[07/29/10 10:18:03] Status:  (VmountDevice:1492) device of type volume with offset 32256 was created
[07/29/10 10:18:05] Status:  (VmountDisk:406) disk vpxa-nfc://sanfs://vmfs_uuid:4c2cb500-e98b1d60-25c6-00215e221476/w2k3-2clo/w2k3-2clo.vmdk@vmd-02.domain.local:902 was closed
[07/29/10 10:18:05] Status:  (VmountDevice:142) device of type volume with offset 32256 was closed

[admstaceya]

Can anyone confirm if vmount2 is invoked in a working environment?  It would really help to know if this is the correct mechanism used when using RCU.  You will see Information events in the vcenter event log.

thanks

Ant

[chrisgeb]

Do you have a case with the GSC on this issue?  They may be able to try to replicate it and make sure we are collecting the right information...  I tried to replicate this and couldn't create the error nor could I find what vmount2.exe was doing during NetApp cloning vs VMware Cloning.

Can you look and see if the VMware Mount Service for Virtual Center is running on the vCenter Server?  If so is it set to manual or automatic?  Can you stop the service and try the clone again.  The vmountvpx service is the vmount2.exe application.  It doesnt seem that this should need to be running.  There may be other apps like VCB or converter that may require it but not VSC.  I would try that first and then work with the GSC and VMware support to run some more tests and figure out what is happening in your environment.

Thanks,

Chris

[admstaceya]

Hi, back from holiday, sorry for delayed response.

Checked the box and the "VMware mount service for VirtualCenter" is set to manual and is not started at this time. I don't have VCB or Converter installed on this box as I've only installed the server and the client.

I'm going to eliminate 2008 R2 from the pot as a last ditch.

Could you confirm you do see the application log events for vmount, though?

Unfortunately logging a call presumably won't be possible as I'm an IBM (value-minus) customer and even if they did support it they never get anywhere.

[admstaceya]

OK I've tested the following environments:

VC 4.0 U2 in all cases

W2k3 x32 - Guest customisation works fine

W2k8 x64 R1 (SP2) - Guest customisation does not work

W2k8 x64 R2 - Guest customisation does not work

Looks like tomorrow will be testing W2k8 x32 and W2k3 x64.  Might also go for a clean build of one of the x64 servers to eliminate something in the template.

On the server that works I still get in the vpxd log file that Vmware tools version was not detected and the .tmp file is not included in the VM's folder so must have been put inside the vmdk.  Vmount2 is invoked on the working server and vcenter logs look no different from the systems where the customisation does not occur.

More investigation tomorrow.

[admstaceya]

I've built 6 or 7 Vcenter servers in the last day or so to test every permutation.

Guest Customisation does not seem to work with VC 4.0 when run on a Windows 2008 platform (any version or architecture).  It works fine with Windows 2003 based servers.

The good news is that VC 4.1 does work on Windows 2008 R2 (I'm not bothering to test on earlier versions)  though I need to prove its robust as the very first attempt failed to join the domain, but subsequent ones did.  I now need to get my VDI environment properly up and running.

[admstaceya]

For info I have now tried with a guest that has Build 208167 (4.0U1 without patches) though the host and vcenter were still U2.  Still getting no joy.  As I don't know if the vmount element is the issue or not I'm getting stuck as to what to try beyond changing OS and downgrading ESXi.

[costea]

Do you have any updates on this issue?  Have you found out what was going wrong with the vmount?