It appears that you're asking to do several things here, and I believe you are trying to set quotas (not quotes?). First of all, unless I've done my math incorrectly, if you take the 5TB as totally usable space, you can only define 20 quotas of 250GB each - you would not be able to have 100. If you set a Group Quota of 250GB, That means that all the users of a specified group collectively can have a total of 250GB. This means that if User1 in Group1(20 users, for example) uses 200GB, there would only be 50GB available for the other 19 members of Group1. You can combine quotas by applying a group quota, plus you can define a general user quota, so that no one user consumes all the allowed space of the group. The general rules for quotas is that a specific quota will override a general quota (using the wildcard character (*)), and if multiple specific quotas exist, the most restrictive will take effect.
So if you want each user to have 250GB of space and then specify group quotas, there is not enough available space to do that.
Again, even with the adjusted size of 100GB for each of 50 users, that will "consume" all 5TB. I hope you are considering usable vs raw storage. Also, if you define user quotas, you may not need to apply group quotas, unless you want to restrict the group to something less than 5TB.
Assuming the 5TB is in one volume, if more than one volume, then you would have to define quotas for each volume involved with this; here is a possible solution:
/vol/uservol = 5TB user space (after all the right sizing and Snap Reserve, etc)
You could simply make each user accessing that volume have a 100GB limit.
/etc/quotas file (Example 1)
#Quota Target Type Disk Files Threshhold SDisk SFiles
I didn't put any values for the other fields (Files, Thres, etc) because that would be up to you to decide what to put there. As far as the Group_name is concerned, you can use the Windows group name or the SID of the group or groups.
Example 1 allows each of the users accessing /vol/uservol 100GB of space, and with 50 users, that would be a total of 5TB. Example 2 allows each user 100GB, but collectively, as a group, they can only fill the volume to 4 TB. There are many ways that you could successfully configure quota restrictions, the method used would depend on what you need to end up with. You can apply quotas against users, groups, or qtrees. Of course, these could be configured with the NetApp System Manager tool.