I have both 8.1 and 8.1.1 Cluster-mode Sims running on ESX 5 without any issues. It looks almost like the download file is corrupted. Below is the image of my file list for 8.1 from browsing the ESX datastore. I would try to re-download the file.
In our case we originally had a problem too with the 7-mode. The files in the downloaded .zip seemed to be made for VMware Workstation. Different files are used in the case of VMware vCenter. To load to a ESX host via vCenter, we had to use a VMware converter.
Were you ever able to resolve your installation issue?
I struggled with the same issue for hours. I was confused because the .tar file was 8 GB but only 2 x 1 GB files were extracted. I was using izarc which I have used for years. I decided to give 7-zip a shot and manually extract everything. It turns out all of the files are visible when using 7-zip. This would probably be the same for other compression utilities.
One note: After powering on, I did not have a root volume. I just did a new install using the boot utilities menu.