Here's the process I use for removing a configured vfiler IP address:
First we add and configure the new address using Set-NaVfilerAddress:
PS C:\PowerShell> Set-NaVfilerAddress lemon -AddAddresses 192.168.0.200 -Interface ns0 -Netmask 255.255.255.0 
Name                      Status         Ipspace              VfnetCount VfstoreCount AdminHost
----                      ------         -------              ---------- ------------ ---------
lemon                     running        default-ipspace               2            1 
PS C:\PowerShell> Get-NaVfiler lemon | select -ExpandProperty vfnets 
Interface                               Ipaddress                               Netmask
---------                               ---------                               -------
ns0                                     192.168.0.117                           255.255.255.0
ns0                                     192.168.0.200                           255.255.255.0
The address 192.168.0.117 is the one we want to remove.  First, we need to unconfigure it.  This can be accomplished by removing the alias from the interface the address is bound to:
PS C:\PowerShell> Get-NaNetInterface ns0 | select Interface, Aliases
Interface                                                   Aliases
---------                                                   -------
ns0                                                         {192.168.0.201, 192.168.0.117, 192.168.0.200}
PS C:\PowerShell> Set-NaNetInterface ns0 -RemoveAlias 192.168.0.117
Interface            Ipspace               Enabled   PrimaryAddresses
---------            -------               -------   ----------------
ns0                  default-ipspace         True    192.168.0.102
Now that we've done this, we can use Set-NaVfilerAddress -RemoveAddresses:
PS C:\PowerShell> Get-NaVfiler lemon | select -ExpandProperty vfnets 
Interface                               Ipaddress                               Netmask
---------                               ---------                               -------
                                        192.168.0.117
ns0                                     192.168.0.200                           255.255.255.0
PS C:\PowerShell> Set-NaVfilerAddress lemon -RemoveAddresses 192.168.0.117 
Name                      Status         Ipspace              VfnetCount VfstoreCount AdminHost
----                      ------         -------              ---------- ------------ ---------
lemon                     running        default-ipspace               1            1
PS C:\PowerShell> get-navfiler lemon | select -ExpandProperty vfnets 
Interface                               Ipaddress                               Netmask
---------                               ---------                               -------
ns0                                     192.168.0.200                           255.255.255.0
Now we are left with the address we wanted.
I hope that helps!
-Steven