It is tricky to change MTU to a smaller value because that means all nodes can't send non-fragmented packets to the node that changes to a smaller MTU, and especially with SolidFire (or clusters in general) by doing that you essentially take a node's interface down and with it put the node out of service (iSCSI interface)
On top of that there's also VMware which is configured to use MTU 9000 so once you change SF to 1500 your ESXi and VMs and vCenter will drop.
It's best to read them, create a plan and check with Support if the steps are OK, then do it during scheduled downtime.
> The default Gateway "The gateway is not valid." For the MGMT network on compute
Maybe your gateway isn't pingable or is on a different (unreachable) VLAN.
Depending on VMware vSwitch configuration and one of several approaches to networking (VDS/VSS, 2 cable, 4 cable, all Host Tagged VLANs, or some Host Tagged VLANs), there are different problems that people run into.
You'd have to describe which approach you took and how you configured your network for it, etc.