NetApp VSC 9.7 and VMware 6.7u3

MicheleD · ‎2020-05-01

Hi All,

I'm struggling to complete the install of the latest versions of VSC (tried both 9.7 and 9.7p1) and connect to the VASA Provider. I have checked multiple times the networking / settings from all components to find out plugin files are not even downloaded. Actually when I even try to browse to https://<vsc_server>:8143/webclient_deployment_bundle I get an HTTP Error 500. No firewall in between and I'm baffled why this is happening. Can anyone please shed a light? Happy to upload logs in case.

If I try to download the zip file using a web browser I get the following:

HTTP ERROR 500

Problem accessing /webclient_deployment_bundle. Reason:

    Server Error

Caused by:

java.io.FileNotFoundException: /opt/netapp/vscserver/tmp/jetty-0.0.0.0-8143-nvpf.war-_-any-9825894675574059877.dir/webapp/vsc.zip (No such file or directory)
	at java.base/java.io.FileInputStream.open0(Native Method)
	at java.base/java.io.FileInputStream.open(FileInputStream.java:213)
	at java.base/java.io.FileInputStream.<init>(FileInputStream.java:155)
	at com.netapp.nvpf.server.vsphere.WebClientDownloadRequestHandler.handleRequest(WebClientDownloadRequestHandler.java:60)
	at org.springframework.web.context.support.HttpRequestHandlerServlet.service(HttpRequestHandlerServlet.java:67)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:873)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1623)
	at com.netapp.nvpf.container.vsphere.request.VSphereClientRequestContextFilter.doFilterInternal(VSphereClientRequestContextFilter.java:49)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346)
	at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:262)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
	at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1700)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1667)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
	at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220)
	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:152)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
	at org.eclipse.jetty.server.Server.handle(Server.java:505)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:370)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:267)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
	at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:427)
	at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:321)
	at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:159)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
	at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:698)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:804)
	at java.base/java.lang.Thread.run(Thread.java:835)

Any help would be greatly appreciated.

Thanks

Ontapforrum · ‎2020-05-01

Hi,

My knowledge around this issue is limited but lets get started with few queries and hopefully we can get some help from experts.

Few queries/suggestions:
1) what is the Ontap version? - This question is wrt to compatibility matrix for VSC,ESXi & ONTAP. I am guessing you must have already checked that out but no harm in getting it re-checked. http://support.netapp.com/matrix

2) http 500 error is usually not specific and can occur for a number of different reasons, so it's difficult to nail it until you diagone the logs. However, there is an error you have mentioned aroud 'java' extensions.

This one : java.io.FileNotFoundException: /opt/netapp/vscserver/tmp/jetty-0.0.0.0-8143-nvpf.war-_-any-9825894675574059877.dir/webapp/vsc.zip (No such file or directory)

Looks like it's complaining about file not found (jetty-0.0.0.0-8143-nvpf.war)?

vCenter will reach out to the VSC server and pull across a "webclient_deployment_bundle". UI extensions from this bundle are copied to directories on the vCenter server. Probably it is unable to download the bundle usign vsc address, and therefore filenotfound error.

This KB talks about 'UI extensions unable to download:
https://kb.netapp.com/app/answers/answer_view/a_id/1005302

3) Which was the previous version that was working ? so, is it 9.7 which broke it or is the upgrade to 6.7U3 ?

4) (Not likely the case but no harm in suggesting) A certificate is required for communication between vCenter, ESXi hosts and the VASA Provider of a storage array, this certificate is setup when you register a VASA Provider in vCenter Server. However, as I am reading on the google, there is a statement : By default, 6.7 U3 does not allow self-signed certificates by Storage VASA provider. So, is it a self-signed ones?
https://kb.vmware.com/s/article/74756

Thanks!

MicheleD · ‎2020-05-02

Hello there and thanks for stepping in!

quick answers to your questions:

1. ONTAP 9.7P3 / VSC 9.7 and 9.7p1 / VCSA 6.7.0.43000 build 15976714

2. Tried to download bundle file from Chrome Web Browser (v 81) from a laptop on the same network where the Management and Cluster Management are sitting. I can browse the Register.HTML and CLI pages with no issues at all. When I try directly from VCSA using wget it gets stuck at this:

root@vcsa [ ~ ]# wget https://netappvsc:8143/webclient_deployment_bundle
--2020-05-02 10:39:11-- https://netappvsc:8143/webclient_deployment_bundle
Resolving netappvsc... 192.168.30.40
Connecting to netappvsc|192.168.30.40|:8143...

I have checked FQDN forward/reverse name resolution as well. FW Ports too

3. This is a clean install of all mentioned components #1. ONTAP is working as a champ. Also both VCS and VASA services are up and running. Just the VASA Provider not registering!

4. This is an interesting point. I will take a look at this as well. Thing is HTTP Server error is shown even when trying to browse that URL from a standard browser on the same network. I have also regenerated the Certificates (both VSC and VASA) rebooted the VCS VM and still same behavior.

To add on that I have also checked/cleaned the plugin installation folders on VCSA (which are not created as the process breaks before), UnRegistered the only 2 extensions that are created in the VCSA using the "MOB" page and restarted the vShpere UI service.

Any suggestion would be great! I'm running out of ideas. Happy to provide logs as required.

Thanks,

Michele

Ontapforrum · ‎2020-05-02

As far as compatibility is concerned all components looks good.

On the VSC Server, are you able to find VSC Process listening.
C:\>netstat -ano | find "8143"

I am wondering if this port is already in use, may be.

This KB is for removing UI extensions and re-registering VSC with vCenter: I just glanced and looks like the steps are well documented.
https://kb.netapp.com/app/answers/answer_view/a_id/1029905

I am not sure if it applies to your version but you can try the steps mentioned.

While you try that out, please raise a ticket with NetApp and keep trying, sometimes it's just a matter of little more patience.

Thanks!

MicheleD · ‎2020-05-02

Hello,

Just to confirm port 8143 is open and in listening mode. It couldn't be otherwise or the Register.HTML wouldn't work either.

I'm doing a bit of more digging and enabled the Diag user for troubleshooting on the VSC appliance itself.

The reason for the Error 500 is because the .zip file is trying to upload to VCSA is not there AND has a different name!

I took a couple of screenshot showing the existing ones right now.

In this screenshot the desired directory does not exist! Error log shows a different directory name. Log shows directory starting with 9825 and the file system shows a similar directory starting with 3200

In this second screenshot even the name of the ".zip" file is different. At least judging from what the web browser was expecting to find: vsc.zip vs. vsch5.zip

Now this begs the question: Is there a safe way to delete existing generated files and safely create new ones?

Would the Hard Reset Database (option 9) clear things up? Since this is a "clean install" there's no much data to loose.

Or is there a different option? I'm baffled about the file name it is expected to be found. Which one should be the right one vsc.zip or vsch5.zip? Or maybe is just me talking non-sense?

Thanks

MicheleD · ‎2020-05-02

Ok further updates:

I have tried to Hard Reset Database and regenerate both VSC and VASA certificates. New temporary files have been created. When I try to test the previous commands everything is the same.

Definitely the error also consists in the fact that webclient_deployment_bundle looks at the vsc.zip and not to vsch5.zip which presumably is the one for the HTML5 vShpere Client?

After creating a copy of vsch5.zip to vsc.zip:

The good news: VASA Provider service can now upload zip file to VCSA

The bad news: Still doesn't work in any of the vSphere Clients (HTML5 or Flex)

Any hint would be greatly appreciated.

Thanks

Ontapforrum · ‎2020-05-02

Not bad, you might be getting closer. keep trying!

I was reading this thread:(Don't know if it is of any help)

Due to the transition from the Flash vSphere Web Client to the HTML5 vSphere Client, it is important to clear the browser cache, and then logout and close browser sessions during the upgrade."
https://community.netapp.com/t5/Ask-The-Experts/VSC-9-6-plugin-in-flash-based-vSphere-6-5u2-u3-Web-Client/td-p/151235

I understand you are not 'upgrading' here but may be clearing cache/closing browser may be ..?

Ontapforrum · ‎2020-05-02

Could you take another look at the 'steps' mentioned in the 9.7 VSC: [Section:Deploying the virtual appliance for VSC, VASA Provider, and SRA]
https://docs.netapp.com/vapp-97/index.jsp

https://docs.netapp.com/vapp-97/index.jsp?topic=%2Fcom.netapp.doc.vsc-dsg%2Fhome.html

"You must have logged out of and closed all of the browser sessions of vSphere Client, and deleted the browser cache to avoid any browser cache issue during the deployment of the virtual appliance for VSC, VASA Provider, and SRA."

MicheleD · ‎2020-05-03

Hi all,

First thanks for stepping in and help. I managed to solve the issue now but I had to reinstall the VSC from scratch (it was a clean install anyway!) It all works fine now. I have added the ONTAP clusters and all works great.

Before reinstalling the appliance (and yes cleaning the browser cache as well) I was still experiencing the same issue.

It turns out a network configuration was not working as expected. Fixing that I was then able to download locally the zip file for testing (using curl which yes complains about the self-signed cert) and on the VCSA later on to progress with the VASA Registration and setup. The VSC could talk/see the VCSA but not the other way round as they are on separate networks and VLANS. So I would say the network misconfiguration originated the issue.

Thanks

TMACMD · ‎2020-05-02

Customer has their own certificate authority

Had an issue a few months ago. Turned out to be a certificate issue. VASA requires correct and valid certificates.

it pulls in the entire chain from the CA

my customer had two certificates that expired and were being propagated through vcenter and passed on to VASA. The expired certificates will cause the VASA enablement to fail.

worth checking if you have or are using your own CA