If you are not using ODM , we would suggest to use mincache=direct (ie. direct mounting - no cache).
Could also mount with noatime to speed it up.
Oracle does cache and does this in the SGA.
The problem is that Oracle cache and any filesystem cache is not the same at all.
Filesystems have different blocks sizes and also have to update metadata (inode information and allocation information specific to the filesystem).
If you mount the Oracle database files with cache, you get a "clash" of the 2 caches , and will actually get a slow down.
The mincahce=direct mount option will mount the VxFS filesystem without filesystem cache. The problem with this is that all other files (on the same mount point and filesystem) will need some cache as well to perform better (Oracle does cache for database tables).
The other option to mount with, is noatime . This will prevent VxFS from modifying the access time of the DBF (I assume dbf) files. This will also save a little time as the access time does not need to change every time a file is being accessed. (would not do the same with the modification time -- mtime ; for backup purposes).
Now, modsort .....
Any filesystem caches metadata (regardless). So, when the filesystem reads an inode from disk, it will keep this in memory (to access later - disk being 100 times slower than memory).
Big problem is, how many are you going to keep in memory ?
And, when you release some of it, which inodes (or the memory associated with the inode) will you release?
The solution is obviously to find the inodes (in memory) that has been least accessed (or last accessed).
This means that you will have to go through the inodes that you have in memory, sort them by last date/time accessed, and the release the "oldest" ones.
With any sorting (does not matter if you use bubble sort or mod sort or ......) this does take time.
Solaris has actually put all of this sorting in the kernel (so you do not have to do it). Well, at least certain versions...
All you have to do, is tell the filesystem to use this.
This is why you can enable vmodsort for vxfs. Just make sure that you have the correct versions for this (Solaris and VxFS)
Last thing ...
The qio mount option does not actually enable quick IO. All it does is check if you have a license to use it.
A long time ago, Veritas could look at IO to raw volumes and to filesystems to see what the IO looks like. (size , order, ....)
Veritas then took what they learned, and did quickio to allow VxFS to do IO in a way that is very close to raw performance.
ODM is just a formal specification (from Oracle) for this. So, now any filesystem vendor (zfs or ufs or ...) can write odm routines (that Oracle will call to do IO) and put this into a library and link into Oracle.
Veritas ODM is just this. A library of functions that does IO in a way Oracle expects it to do (not going into too much of the specifications now, but there are a lot and public).
If you do not link the Veritas ODM, Oracle uses the "old" way of doing IO (it calls ODM function calls, but these map back to standard IO routines and system calls)