We have made improvements to better support rquota with ONTAP 9.6 and more so in ONTAP 9.7. Which ONTAP version are you using? There are other options that must be configured as well for quotas to work. Have you configured a quota policy for the volume specifically using the command "volume quota policy rule create"?
You can reference this for more details on the command: