with the option ip.path_mtu_discovery.enable on it shouldnt matter much.
As long as the max framesize is supported by all the switch devices along the path they should be able to negotiate the MTU at session start.
Personally what we do normally is to split the 9000 and 1500mtu traffic onto there own network generally the 9000 goes onto a non-routeable network we do this mainly historically now. i.e. if the MTU's were mismatched you could end up with fragmentation at the router layer which would kill the router
It depends what happens when client gets packet over 1500 long. if it is able to process it - probably, nothing happens. If it will discard it - communication will simply fail. If it never gets frames over 1500 - everything will work, but why would you enable JF in this case in the first place?
The same applies to filer BTW. I am not sure what happens when filer gets oversized frame. Interesting is, our filers (and Cisco switches) do log quite a number of oversized frames although MTU is set to standard everywhere. I still have not got around to really investigate it.
Jumbo frames are really too much pain to administer for the little gain you get.
Its depends on what you're doing, we saw a moderate improvement in our testing with 10GbE vs 10GbE with TOE vs 10GbE with Jumbo vs 10GbE with Jumbo and TOE. It wasnt enough for us to warrant the change, but depending on traffic proof it might help.
Jumbo frames if you decide to use them IMO at least need to either be run everywhere (sometimes impractical with the likes of network attached printers and the like) or kept to an isolated segment. We have a dedicated 10GbE segment that connects all our filers VM hosts and backup servers. Its isolated and works fine for what we use it for.
Although "option ip.path_mtu_discovery.enable on" SHOULD sort out the issues, i've never been game enough to trust it