I have some volumes that are unix primarily but still shared on cifs. The problem is the default nt user right now is root which I want to change but I have to fix a lot of volumes that were carried over first. For right now though when I create a cifs share of a unix qtree it isn't enforcing the share permissions at all. Any domain user can access the share and since they are root the unix permissions give them full access. Even enabling accessbasedenum doesn't help. So now viruses that write to open shares are hitting these and it's causing a large problem. If anyone knows what I can do with this please let me know.
I have also seen now if I create a cifs share and put a unix qtree under it that the share permissions are enforced at the top level but not on the qtree
for example if I create a unix qtree under a share that is ntfs and that qtree is unix then a user can't open \\share but can open \\share\unixqtree even though there is no share for that spot specifically.
You confuse share level and filesystem level permissions.
To be able to access server\share\folder<file:/// server\share\folder> user must first have permissions to connect to server\share<file:/// server\share>. If (s)he is not allowed to do it, there will be error right away and no way to access any file and/or folder inside this share at all. Your initial question was about share level access. And this is controlled by share ACL.
Once user connects to share (assuming necessary permissions are granted) access to individual file(s) and/or folder(s) in this share is controlled by file ACL. For Unix qtree these ACLs are reduced to standard Unix file owner, group and mode bits. For access check Windows user is mapped to Unix user and access is verified using standard Unix rules. If all your users are mapped to root, then every user has access to every file (on Unix qtree).
Please read TR-3490 about multiprotocol access to NetApp.