> Above mentioned issued does not occur when some other vendor SANs are used.
I don't think that's related to ONTAP SAN vs "other SANs" but simply is your MPIO properly configured on the OS level. If it's not, then when failures happen, they won't be detected by GPFS either.
https://www.ibm.com/docs/en/spectrum-scale/5.0.4?topic=issues-gpfs-is-not-using-underlying-multipath-device
Key MPIO settings are OS-level, some still have to be done in Spectrum Scale (NSD-related configuration steps).
GPFS should use DevMapper to detect that some disks that are down, and use the replicas, unless strict replication is configured but isn't possible.
https://www.ibm.com/docs/en/spectrum-scale/5.0.4?topic=failure-strict-replication
https://www.ibm.com/docs/en/spectrum-scale/5.0.4?topic=failures-disk-connectivity-failure-recovery
What you could do is create one test VM, connect it to ONTAP (I assume this is FC or iSCSI?) and create a single node Spectrum Scale cluster in the VM, then first confirm correct behavior (failure, recovery) on the OS level, and then confirm that Spectrum Scale properly uses those multipath devices. If Spectrum Scale isn't correctly handling disk (and complete path disconnection) failures, NSDs are not correctly configured (or DevMapper isn't, but I assume you wouldn't test GPFS if all DevMapper tests didn't pass).
https://www.ibm.com/docs/en/spectrum-scale/5.0.4?topic=dmf-error-numbers-specific-gpfs-application-calls-when-disk-failure-occurs
Generally speaking E-Series would be a better choice of storage for Spectrum Scale back-end, but just like ONTAP, MPIO would have to work, and Spectrum Scale be correctly configured, for a similar setup to survive failures of a replica system.
