For each pairing you need separate SRM and SRA servers at both the PROTECTED and RECOVERY sites.
So if I've interpreted your situation correctly, you will need to implement SRM and SRA appliances as follows:
SITE A
SRM/SRA servers for SVM1 protection - paired with "a)" in SITE B
SRM/SRA servers for SVM2 protection - paired with "b)" in SITE B
SITE B
a) SRM/SRA servers for SVM1 protction
b) SRM/SRA servers for SVM2 protction
This is very similar to my existing pairings - I am protecting two (2) groups of resources to a single DR location.