I know in the past Windows Groups where not supported for quotas only UNIX Groups are, this is true upto ONTAP 184.108.40.206. Does anyone know if this has changed in newer ONTAP versions or even if there is a plan to modify it?
However for one use case of this there is a work-around. The case is where you have a flexvol or qtree that will be a "workgroup share" for a particular group and you want to limit how much they write.
In the case of a volume, just set the size and put a CIFS share-level ACL that only allows that group to access it. In the case of a qtree, put a tree quota on the qtree and so the same with the share-level ACL.