Terminology first. You are concerned about the LUN "ostype", not protocol. Protocol for block is either iSCSI or Fibre Channel. I mention this for clarity for everyone reading.
Both LUNs and igroups have a "ostype". The "ostype" value is tied to the controlling Operating System and adjusts certain behaviors and settings that are unique to the Operating System. That's why the "ostype" exists.
For igroups, the "ostype" controls behaviors associated with how the controlling OS talks over the wire - either iSCSI or Fibre Channel - using the SCSI-2/3 command set. For LUNs, the ostype controls behaviors associated with how the controlling OS reads and writes data to the logical "disk". For instance, a LUN with an "ostype" of "windows" will trigger the NetApp controller to fake the starting offset to get better partition alignment for better performance, whereas an "ostype" of "windows_2008" won't do that because W2k8 changed the default alignment. So the LUN "ostype" is all about the contents, while the igroup "ostype" is about the wire protocol. In general of course.
Typically, the "ostype" for both LUN and igroup are essentially the same or within the same family - for instance an igroup of ostype "windows" and a LUN of ostype "windows_2008" are within the same family. Natural of course because typically the controlling server is the same for both the igroup and the LUN.
In virtualization, you introduce the possibility that the controlling server is different between the igroup and the LUN. With RDMs, ESX acts as a pass through. That is, the ESX host takes the read/write request from the Guest OS and puts it out on the wire essentially as is (there are a couple of exceptions, of course). So - what you have is ESX (VMWare) controlling the "wire" (igroup) while the Guest OS controls the contents of the disk (LUN). So for an RDM that is mounted to a Windows guest, the igroup should be of ostype "vmware" and the LUN should be of ostype "windows" (or one of the other ostypes in the windows family as appropriate like "windows_2008").
Bonus question - if you are not using SnapDrive, why use an RDM at all in this example? Do you have some other need to snapshot this LUN independently of ESX control at the NetApp level? Need to take NetApp FlexClones for some reason? If not, why RDM at all? It does introduce a level of complexity but shouldn't be done just because. If it was done for "performance" I'll say it is a common misconception that RDMs are, in general, faster. In most cases you can't get a difference in performance, since you still go through both the guest OS kernel and the ESX kernel to process any disk request. Of course there are rare exceptions, but RDMs are primarily for when you need advanced "native" storage capabilities.
Bob Greenwald
Lead Storage Engineer, Huron Legal
Huron Consulting Group
NCDA | NCIE-SAN Clustered Data OnTap