Again, even with the adjusted size of 100GB for each of 50 users, that will "consume" all 5TB. I hope you are considering usable vs raw storage. Also, if you define user quotas, you may not need to apply group quotas, unless you want to restrict the group to something less than 5TB.
Assuming the 5TB is in one volume, if more than one volume, then you would have to define quotas for each volume involved with this; here is a possible solution:
/vol/uservol = 5TB user space (after all the right sizing and Snap Reserve, etc)
You could simply make each user accessing that volume have a 100GB limit.
/etc/quotas file (Example 1)
#Quota Target Type Disk Files Threshhold SDisk SFiles
* user@/vol/uservol 100G - - - -
/etc/quotas file (Example 2)
#Quota Target Type Disk Files Threshhold SDisk SFiles
* user@/vol/uservol 100G - - - -
UserGrp group@/vol/uservol 4T - - - -
I didn't put any values for the other fields (Files, Thres, etc) because that would be up to you to decide what to put there. As far as the Group_name is concerned, you can use the Windows group name or the SID of the group or groups.
Example 1 allows each of the users accessing /vol/uservol 100GB of space, and with 50 users, that would be a total of 5TB. Example 2 allows each user 100GB, but collectively, as a group, they can only fill the volume to 4 TB. There are many ways that you could successfully configure quota restrictions, the method used would depend on what you need to end up with. You can apply quotas against users, groups, or qtrees. Of course, these could be configured with the NetApp System Manager tool.