The only environment variable that makes sense is "smbios.memory.enabled".
Prior to changing the vmware guest memory size I noticed that "smbios.memory.enabled" was the same as the guest vm memory.
The process I used to increase the memory on my 9.8 sim - note: this must be done while the simulator is shutdown - you won't be able to set the memory size of the vmware guest os while it is in a running state (whether it is hung/non-responsive or not)
1) Increase the vmware guest memory size to 8192 (this equates to 8192000 which is what you will use at the VLOADER prompt
2) Start the vmware guest (simulate onta 9.8) - when prompted to do so interrupt the simulator from booting - watch the screen closely as there will be a message which states "Hit any key to abort"
3) When the simulator is at the "VLOADER>" prompt use the following command to match the simulate ontap memory size to the vmware guest os memory size:
setenv smbios.memory.enabled 8192000
4) Next confirm that the change was committed using this command:
If they match then continue
5) Check the value of "bootarg.rdb_corrupt" - if this environment variable does not exist the type "boot"
At this point the simulator should boot and should become stable (not sporadic hangs).
HTH (Hope This Helps)