1. Make sure SQL Server 2005 backward compatibility components was installed.
2. Make sure the following VB script can be run correctly. If no, resolve the script issue first with MSFT (you should get same error as SMSQL):
'script starts here…
Set oServer = CreateObject("SQLDMO.SQLServer2")
Set InstanceList = oServer.ListInstalledInstances
wscript.echo "Total installed instances : " & InstanceList.Count
wscript.echo "========================================="
For i=1 To InstanceList.Count
wscript.echo InstanceList.Item(i)
Next
Set app = oServer.Application
set serverlist = app.ListAvailableSQLServers
wscript.echo "Total instances on network : " & serverlist.Count
wscript.echo "========================================="
For i=1 To serverlist.Count
wscript.echo serverlist.Item(i)
Next
'script ends here