VIBE can not connect to filer

[mheimberg]

Hi all

After using VIBE successfull in several installations I am facing a problem:

vibe.exe --verbose gives return code 11, meaning something wrong with userid/password.

System: Windows Server 2003 SP2, FAS3020 Ontap 7.2.4P9, vibe.exe 1.0.3

When using default protocol ZAPI the log states: ERROR: ZAPI connection error, when using --protocol rsh access is denied. All other operations terminate successfully.

BUT on command prompt rsh access works fine with the credentials given in vibe. Also other access methods (eg. ssh by putty) works fine.

Any idea what could be wrong or should be checked?

THX

Mark

[systemsengineering]

Hello Mark - Have you found a solution to this issue. I am experiencing a similar issue and wonder if it is patch related. We ran the following patches on our vcenter server last weekend. The vcenter server is where we run vibe.

KB951072

KB953839

KB951066

KB951066

KB951748

KB952954

KB950974

KB953938

KB950760

KB951698

KB950762

KB950759

Thank you

Rob

[kusek]

Markus, Rob,

I've seen cases where people were experiencing issues with vibe running on *nix boxes connecting and trying to execute but they did not experience any issues using the Windows version of the vibe to connection.

For troubleshooting - Do you experience the same problem both connecting from a Windows as well as a *nix host?

It'd be nice to find out what is causing this condition (patch or otherwise) and have a clear recourse

Thanks!

Christopher

[rmatt]

Hi, Markus. If you're having problems with 'rsh' protocol access but it works through command line, I would test a couple of things:

1) Login to your NetApp controller where you're trying to 'rsh' or open up FilerView.

2) Run the VIBE operation again.

3) Look at the messages (either on the console or through FilerView). If you don't see anything, you can't reach the NetApp controller. If you see authentication errors, it's because you're coming in as a different user than what you think.

In nearly 100% of the cases users aren't running with the same environment or user as what they are using on the command prompt. For example, if you run both commands through a command prompt, does it work? If you use the 'set' command from the Command Prompt, and compare that to 'runas /user:<username> cmd' and from that Command Prompt window run 'set', do the 'set' outputs match?

Feel free to copy/paste the VIBE log here for review and we can take a look.

Thanks,

--Matt

[BrendonHiggins]

I am liking what Matt Robinson is saying and it is what I would do.

Get a console session on the filer then use a CMD to run the command

rsh filername version

The command will return the version and the console will be happy.

Now run VIBE and it will say user aname tried to log on from IP. Add this user and machine to the RSH list.

Another problem you may be having with rsh is caused by HP teaming, if the filer is on the same IP subnet as the machine you are using. Turn off one nic and try again. If this does fix it. Let me know and I will post the KB and details for this issue.

Good luck.

[mheimberg]

Thanks to all for your kind help.

rsh <filername> version

is working with all the accounts we defined in the hosts.equiv file. But regardless of whatever account we give in the vibe parameter list, we get this error in the last lines of the vibe log:

Running remote shell command: version -b ...

ERROR: Running 'version -b' failed: 512

On the console of the filer we have no message, so it seems, that our request doesn't even reach the filer. We'll check for some IP/DNS issues like mentioned in this case http://communities.netapp.com/thread/1841? by Matt Robinson.

stay tuned

Markus

[BrendonHiggins]

Are you on a HP server with teaming enabled?

If so have a quick look at the properties of the NIC. 1 of the two Nics will have 0 for throughput against its card. This will be were the return rsh traffic will be being sent. Try turning off this card.

[mheimberg]

Virtual Center runs on a FSC L200 with only one NIC enabled and only IP address configured.

Regards
Markus

[mheimberg]

Slowly I am getting frustrated and a bit desperate...

This is the infrastructure:

[Virtual Center Server]-172.22.8.209 ----- 172.22.8.212-[ESX]-192.168.8.212 ------ 192.168.8.130-[FAS3020]-172.22.8.130-

So: we use a dedicated "Server LAN" (192.x.x.x), but the FAS3020 has also a connection to 172.22.8.0/24, the ServiceConsole Port of the ESX is 172.22.8.212

Now: if I issue on the VCenter server

cmd>rsh filername -l root:passwd version -b

I get the answer from fiilername.

Tracing the connection with pktt on the filer I can clearly see the rsh communication.

When starting vibe.exe with default ZAPI-protocol, I get the error:

ERROR: ZAPI connection to 172.22.8.130 failed!

thats why we try protocol rsh.

When running vibe with protocol rsh, then I get:

[12:08:33] Ping of storage appliance 172.22.8.130 successful.
[12:08:33] Testing shell login to storage appliance 172.22.8.130 ...
[12:08:33] Running remote shell command: version -b ...
[12:08:33] ERROR: Running 'version -b' failed: 512
Exiting with return code: 11

11 means. A NetApp API initialization call failed. Confirm the --sauser and --sapasswd options are correct

but the given sauser (root) und sapasswd are correct!

Tracing the packets on the filer shows, that there is no rsh connectin coming in.

Using wireshark on the VCenter server shows, that there is *no outgoing rsh* session!!! Yes: no single packet with proto rsh leaves the server... instead I see a lot of sslv3 traffic between VCenter server and the filer - no clue, what the systems are talking...

maybe the attached logs, used config and this description takes us a step further.

Thanks to everyone.

Markus

[robertquast]

Hi Markus, we had the same issue in another thread.  Are you trying to run Vibe with the --config option?  Robf Keyworth and I were getting the same unexplainable 512 error and by chance tried everything on command line and it worked fine.  I'd recommend forgoing the config file if you're using it.

   -Rob Quast

[mheimberg]

Hi Rob,

sorry, no difference: wether using --config.cfg or directly from the cmd-prompt, the result is the same: no connection, no rsh communication.

Meanwhile we reconfigured the VCenter server with a second connection to the server VLAN, so we can connect directly via IP 192.168.8.x - but still getting no connection.

THX anyway.

Markus

[rmatt]

Hi, all. I spoke with Markus and it looks like 1.0.7 does the trick for his RSH testing. 1.0.8 should be on the NOW ToolChest shortly (they are doing some maintenance work which has caused some delays).

[mheimberg]

Yes, it really solved my problems.

Markus

[rmatt]

1.0.8 is available on the NOW ToolChest for download. Thanks,

--Matt

[vjohn]

Hi - We are experiencing the same issue. We are using 1.0.8 and backups via rsh is working fine. However, we can not do restores using ZAPI. ZAPI wasn't working either for backups.

We added a second NIC to VC like Mark did and still having issues with Error Code 11; however, we know the sauser and sapassword is correct because we are using it for backups.

Any chance we can use rsh for restores?

[vjohn]

I upgraded to version 1.0.10 and I am still having issues with restore via ZAPI. I am getting the following error:

[14:35:00] Virtual Center login successful.
[14:35:00] Attempting to ping storage appliance 192.168.152.10 ...
[14:35:00] Ping of storage appliance 192.168.152.10 successful.
[14:35:00] Testing login to storage appliance 192.168.152.10 ...
[14:35:01] ERROR: ZAPI connection to 192.168.152.10 failed: in Zapi::invoke, cannot connect to socket
Exiting with return code: 11

[vjohn]

Fixed ZAPI issue by enabling options httpd.admin.enable on netapp

[rmatt]

I think we've answered the 'httpd.admin.enable' to 'on' option, but in terms of RSH for restores, we felt that restores would be an uncommon, unscheduled event, and as a result we focused on the ZAPI functionality instead. 1.0.10 now offers the ability to see why the failure occurred which should be sufficient for most to figure out to turn that on.

Thanks,

--Matt

[vjohn]

Thanks Matt... The error reporting for 1.0.10is definitely better and now that we know what the issue with ZAPI was, we are revertng to ZAPI for backups as well. The cryptpasswd feature you added is reason enough for us to switch to ZAPI.