this is the response from VMwar:
2019-03-24T23:04:41.380Z cpu22:33383)NMP: nmp_ThrottleLogForDevice:3298: Cmd 0x2a (0x439e4b6ccd00, 36451) to dev "naa.600a098038303841635d4a2d2d575a63" on path "vmhba2:C0:T1:L6" Failed: H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL
2019-03-24T23:04:41.380Z cpu22:33383)ScsiDeviceIO: 2613: Cmd(0x439e4b6ccd00) 0x2a, CmdSN 0x8000004b from world 36451 to dev "naa.600a098038303841635d4a2d2d575a63" failed H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2019-03-24T23:05:01.627Z cpu14:33383)ScsiDeviceIO: 2613: Cmd(0x439e4b7a09c0) 0x2a, CmdSN 0x80000073 from world 36451 to dev "naa.600a098038303841635d4a2d2d575a63" failed H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
2019-03-24T23:16:25.753Z cpu10:33383)ScsiDeviceIO: 2613: Cmd(0x439e40b95680) 0x2a, CmdSN 0x800e0017 from world 35697 to dev "naa.600a098038303772735d4a37734f484c" failed H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.
-- We tried to bring the production back up by registeing orpahned VM's on the host .170 and powered on successfully
-- In the mean time the host .171 went into non-responding state which houses the vCenter and which had the VM's also
-- We rebooted the VM and it came back online and once it was online the host .170 went into not-responding state
-- The VM LMS has 3 disk with 2 levels of snapshots
[root@localhost:/vmfs/volumes/583f0917-bc1e434e-c61b-000af74ef094/LMS] ls -ltrh | grep -i delta
-rw------- 1 root root 948.7G Dec 18 16:46 LMS_1-000001-delta.vmdk
-rw------- 1 root root 908.4G Mar 24 16:47 LMS_1-000002-delta.vmdk
-rw------- 1 root root 215.2G Mar 24 16:47 LMS-000001-delta.vmdk
[root@localhost:/vmfs/volumes/583f0917-bc1e434e-c61b-000af74ef094/LMS] ls -ltrh /vmfs/volumes/59a78fc2-1606aed3-675e-000af74ef094/LMS | grep -i delta
-rw------- 1 root root 46.7G Dec 15 23:56 LMS-000002-delta.vmdk
-rw------- 1 root root 46.5G Mar 24 16:47 LMS-000003-delta.vmdk
-- Even if we try to power on it will stuck at 33% and told the customer we will have to wait till it completes or we need to do a consolidation
-- Since the reads and writes are failing on all the datastores told the customer to engage storage Vendor on this to check further
-- Checked the driver and firmware on HBA and they are compatible
-- With storage commands failing with above errors, we see host .170 and .171 is going into not-responding state one after the other and it is difficult for us to troubleshoot until we fix the storage issue.
the second stack has been added 6 month after the first one and the first stack is connected to IBM chassis blade, it was working fine all those month but the problem start when we add the second stack wich is connected to a hp blade ( LUN on the second stack are access by HP blade).
we have a replication site where the configuration is same and there everything is fine
