I'm not familiar with your configuration but if you work with thick provisioning, temporarily switching to thin provisioning (turning off space reservation of the lun and changing the space guarantee of the volume to none could save you alot of space and enable you to create a new lun for VMFS5. Migrate everything needed to the new datastore and delete/undo the changes for the old datastore. Helped me a lot of times, hope it will be useful for you as well.
We had the same problem with 2008 server with RDM luns. We was not able to mount a snapshot of the lun with the same error message.
We had to upgrade to VMFS5 to make it work. For us it worked without problem but before doing it live we "tested" creating a new test lun with a VMFS5 datastore using it to store the RDM configuration while mounting the iscsi snapshot lun.