The cluster interconnect from FAS nodes are 4x 10Gbit and from AFF 2x 10Gbit.
The work load on the AFF will be only via NFS and all the load from the FAS will be moved over to the new AFF system and then the older FAS will be used for not so performance intensive tasks like backups or so.
On the AFF, we will host our VMs and this will be a lot of them.
That's why I'm asking, if it would not be better, to stay local on a HA pair when lifs are failing over. I don't know, if the FAS will be able in the future to take the extra load from the AFF or at least if it will be able to forward the data through its network stack and the interconnect.
At the moment, the FAS is about 70% CPU for both nodes, sometimes more.
Are you able to make the AFF 4x10GbE for the cluster interconnect? (I didn't see a model mentioned some just don't have the ports for it)
I personally would leave them at broadcast domain wide, one downside to it is that it the SFO partner could already be running higher output through it's ports and lifs failing over to the SFO parters ports could push over 100%, where if it' broadcast domain wide they'd fail over to what the system feels is the better port.
Also, do you mean batch upgrades, not rolling? That I could potentially see an issue, but rolling is only going to do one of the HA pairs and then the next before it goes on to the next pair. where as a batch (8 nodes or higher) you'll have more than one node failing over.