You are correct to send Remove-NaSnapmirror to the destination controller. However, this code you posted has a bug that isn't immediately obvious:
$filer2=connect-nacontroller -name filer2
remove-nasnapmirror -destination filer1:vol1 -source filer2:vol2 -sourcecontroller $filer2
In this snippet, I think you are trying to send Remove-NaSnapmirror to the destination controller. However, the preceding line contains Connect-NaController, which will set the value of $global:CurrentNaController to filer2. This will cause Remove-NaSnapmirror to be sent to filer2.
There are various ways to fix this, the simplest being to use the -Transient switch:
$filer2 = Connect-NaController -Name filer2 -Transient
Remove-NaSnapmirror -Destination filer1:vol1 -Source filer2:vol2 -SourceController $filer2