SnapCreator Upgrade from 3.3 to 3.4

jntballard · ‎2011-06-30

To upgrade to v3.4 the guide indicates to use the following variables in the exisint 3.3 configs:

CONFIG_TYPE -PLUGIN|STANDARD
USE_PROXY - (Y|N).
NTAP_CONSISTENCY_GROUP_WAFL_SYNC - (Y|N)
NTAP_SNAPSHOT_RESTORE_AUTO_DETECT - (Y|N
MOUNT_CMD<##>
UMOUNT_CMD<##> APP_DEFINED_CLONE -(Y|N)

Are these the only variables needed? Seems like there are many more variables in 3.4...

For the upgrade, this is the procedure I was going to follow:

1. Stop 3.3 GUI on scServer

2. Install new 3.4 Snap Creator

3. Copy the unique DB config files from 3.3 to the 3.4 config directory

4. Add variables as indicated above

5. Update scripts to point to new 3.4 directory

6. Start the java GUI from the 3.4 directory

7. Update Snap Creator Agent on remote servers

8. Remove old 3.3 directories

Is that all I need? Just want to ensure I have the VARs needed and the process correct.

Thanks for the help -

ktenzer · ‎2011-06-30

Yep that is about right

For SC 3.3.0 to SC 3.4.0 both Agent and Server must be upgraded.

The only thing maybe missing is the GUI scheduler. I am going to check with some of the other devs if there is a way to copy the DB tables it uses over otherwise you need to re-create all the schedules in the GUI. If you aren't using them ignore this. We are going to improve this in the future.

Regards,

Keith

jntballard · ‎2011-06-30

Thanks Keith,

I am currenlty not using the GUI schedular in 3.3 but was thinking about it for 3.4. For now, everything is scheduled via cron.

For the GUI schedular, is the GUI required to be running via '/usr/bin/java -jar snapcreator.jar -gui_port 6060'; in order for the jobs to run? Just curious as cron is running as a deamon, I would imagine the java process must be running to manage the GUI schedular jobs.

charly · ‎2011-06-30

Hi Keith,

Yes we can port the DB tables created in 3.3 to 3.4.

1. Install the SnapCreator server.

2. If you have selected "Start server as service", then go to Windows services and stop the service "SnapCreatorServerService" and delete <scServer Home>/gui/snapcreator folder.

3. copy <scServer 3.3 home>/gui/snapcreator folder to <scServer 3.4 home>/gui/

4. copy the content of configs folder to <scServer 3.3 home>/configs except "default" folder. If you copy "default" folder, you might loose some of the new features while creating new config.

5. restart the server using java -jar snapcreator.jar -gui_port. If snapcreator is installed as service, then go to Windows services and start the service "SnapCreatorServerService".

Little manual work is required in this case.

regards

avbohemen · ‎2011-07-01

We did the upgrade in the same way, added the variables to our existing configs, copied the schedule-database, changed the startup scripts, upgraded the agents (and also changed them to run multithreaded). Everything seems to be running fine, but we have a big problem: scServer cannot reach any scAgent...

It throws a java exception like this:

"Caused by: java.net.SocketException: SocketException invoking http://rs24.asp.local:9089/SnapCreator/Daemon/Agent: Unexpected end of file from server"

I also tried changing the agent back to single threaded, but that makes no difference. So... help?

ktenzer · ‎2011-07-01

Can you telnet to agent:port "telnet rs24.asp.local 9089" from scServer?

Also a java error is very strange? The snapcreator binary meaning the agent and the backend server is not written in Java. Please try things from CLI to rule out Java.

The only place I could imagine you seeing a java error is if you do an agent ping from GUI since that java client in that case talks to agent. But snapcreator server binary to snapcreator agent binary = NO JAVA CODE.

Another thing check the /scAgent3.4.0/config/agent.conf? Make sure if you had any security things there from 3.3.0 that you update that, the agent.conf changed so please read about it. Here is what you should have:

############################################

### SnapCreator Agent Configuration File ###

############################################

######################################################################

### Command to allow or wildcard "*" can be used to allow anything ###

######################################################################

host: scServer@*

If you are running any commands or scripts through agent then you need to do something like this

############################################

### SnapCreator Agent Configuration File ###

############################################

######################################################################

### Command to allow or wildcard "*" can be used to allow anything ###

######################################################################

host: scServer@*

command: echo

command: snapdrive

command: script.sh

Regards,

Keith

~

avbohemen · ‎2011-07-01

Yes, telnet from the server works fine. I checked the agent.conf: it was the original from the tar-file, but there were ^M charaters in the AIX version, so I removed them and restarted the agent, but it is still not reachable.

I need to clarify the java error: that shows up if I do a "test agent connection" from the web gui.

Running a profile from the command line gives this error:

[root@linux scServer3.4.0]# ./snapcreator --profile <profilename> --config <configname> --action snap --policy daily --verbose

########## Parsing Environment Parameters ##########

########## PRE APPLICATION QUIESCE COMMANDS ##########

[Fri Jul 1 11:02:32 2011] INFO: No commands defined

########## PRE APPLICATION QUIESCE COMMANDS FINISHED SUCCESSFULLY ##########

########## Application quiesce ##########

[Fri Jul 1 11:02:32 2011] ERROR: No valid response

[Fri Jul 1 11:02:32 2011] ERROR: [scf-00053] Application quiesce for plugin oracle failed with exit code 1, Exiting!

########## Application unquiesce ##########

[Fri Jul 1 11:02:32 2011] ERROR: No valid response

[Fri Jul 1 11:02:32 2011] ERROR: [scf-00054] Application unquiesce for plugin oracle failed with exit code 1, Exiting!

########## PRE EXIT COMMANDS ##########

[Fri Jul 1 11:02:32 2011] INFO: No commands defined

########## PRE EXIT COMMANDS FINISHED SUCCESSFULLY ##########

ktenzer · ‎2011-07-01

Ok so this is consistent, it means the agent is not reachable which means there is some Network communication issues. Neither the GUI nor the CLI can talk to the agent. This can happen due to following:

1. Firewall (thought if you can telnet the port should be open)

2. Using the wrong agent port

3. Other Network issues

I am thinking maybe option 2? Are you sure you started agent on correct port? You are not using default port wich is 9090. Also are you sure the SC_AGENT setting is correct?

You can send us command you are using to start agent, the SC_AGENT setting in config file, and the output from your telnet command?

Please double check

Keith

ktenzer · ‎2011-07-01

Oh another thing if you have ^M in agent.conf that means you probably edited it in windows and copied to unix, is that possible? If you didn't that means they were there wich would be a build issue on our edb. Let us know?

Double check and make sure there are no ^M in your config file. This would explain this issue since SC_AGENT would have ^M and that would mess things up.

If in doubt do this, vi config file, edit select all. Create new file in vi, and paste config there. Then use the new config file.

Regards,

Keith

avbohemen · ‎2011-07-01

Yes, we explicitly configured port 9089 for all agents, because 9090 is in use on AIX by the Web System Management daemon. I did find another thing: using telnet, I get a normal (open) connection if I use the single-threaded agent, but the connection is opended and closed immediately if I start the multithreaded agent.

As for the ^M: they are in the .tar.gz package on the NOW site, I did not change anything to the agent.conf file. Oh, and also, the files on NOW are not .tar.gz files, just plain .tar files. gunzip told me they were not in gz format, and tar would not unpack them if I used the -z option. And yes, I removed the ^M from the files.

avbohemen · ‎2011-07-01

Found another file with ^Ms in it: /scAgent3.4.0/wsdl/agent.wsdl

Removed them, now the java error from the GUI is: java.io.IOException: HTTP response '401: Unauthorized' invoking http://rs24.asp.local:9089/SnapCreator/Daemon/Agent with authorization username 'scServer'

So, it now looks like an authentication error. This is my Agent.conf:

############################################

### SnapCreator Agent Configuration File ###

############################################

######################################################################

### Command to allow or wildcard "*" can be used to allow anything ###

######################################################################

host: scServer@*

command:

By the way, besides AIX agents we also have some Linux (x64) agents, and they show the same behaviour. I did not find any ^Ms in the linux files (so that package seems to be more ok than the AIX one), but the errors are the same. To be more specific, both on AIX and Linux, I get:

- scServer Unauthorized when I run the single-threaded agent

- Unexpected end of file when I run the multithreaded agent.

On the agent machine, the multi-threaded agent also shows this warning when I do a telnet or "test connection" from the GUI:

Use of uninitialized value in numeric lt (<) at

/</opt/netapp/scServer3.4.0/snapcreator>HTTP/Daemon.pm line 380 (#1)

(W uninitialized) An undefined value was used as if it were already

defined. It was interpreted as a "" or a 0, but maybe it was a mistake.

To suppress this warning assign a defined value to your variables.

To help you figure out what was undefined, perl will try to tell you the

name of the variable (if any) that was undefined. In some cases it cannot

do this, so it also tells you what operation you used the undefined value

in. Note, however, that perl optimizes your program and the operation

displayed in the warning may not necessarily appear literally in your

program. For example, "that $foo" is usually optimized into "that "

. $foo, and the warning will refer to the concatenation (.) operator,

even though there is no . in your program.

Any thoughts what more can be wrong?

ktenzer · ‎2011-07-01

This is a strange issue

We have double checked and the AIX build does have ^Ms in the agent.conf (it appears to be only build with ^Ms anywhere), the wsdl.conf isnt used that is for creating your own APIs so that can be ignored. However we verified this was QA'ed and ^Ms didn't have any effect. Also I just tested downloaded from NOW AIX and Linux package. Installed Linux scServer and AIX scAgent. I am not testing DB just APP_QUIESCE_CMDs and APP_UNQUIESCE_CMDs to agent. Everything works with and without control ^Ms.

With Linux x64 I have a DB2 database and I tested that and everything works as expected, again using packages on NOW

Here is my agent.conf. I did notice if you delete ^M and then add your own ^M then it will break but otherwise they seem to be safely ignored. I would recommend removing them though since you have issue:

############################################^M

### SnapCreator Agent Configuration File ###^M

############################################^M

######################################################################^M

### Command to allow or wildcard "*" can be used to allow anything ###^M

######################################################################^M

host: scServer@*^M

command: echo^M

If I do cat agent.conf then I dont see ^Ms.

# uname -a

AIX ibm520b 1 6 00009D2CD700

# ./snapcreator --start-multithreaded-agent 9089 --verbose --debug

[Fri Jul 1 07:19:53 2011] INFO: Starting NetApp Snap Creator Framework Agent [multi-threaded] in Debug Mode

[Fri Jul 1 07:19:53 2011] DEBUG: Listening on provided port: 9089

[Fri Jul 1 07:19:53 2011] INFO: NetApp Snap Creator Framework Agent [multi-threaded], running with pid 23134360, is listening on port 9089 of all configured network interfaces

[Fri Jul 1 07:20:01 2011] DEBUG: Reloading configuration from /keith/scAgent3.4.0/config/agent.conf

[Fri Jul 1 07:20:01 2011] DEBUG: Reloading configuration finished with

exit code: [0]

stdout: []

stderr: []

[Fri Jul 1 07:20:01 2011] DEBUG: 10.65.55.80 is allowed to send requests

[Fri Jul 1 07:20:01 2011] INFO: Authorized request from [scServer@*]

[Fri Jul 1 07:20:01 2011] DEBUG: Reloading configuration from /keith/scAgent3.4.0/config/agent.conf

[Fri Jul 1 07:20:01 2011] DEBUG: Reloading configuration finished with

exit code: [0]

stdout: []

stderr: []

[Fri Jul 1 07:20:01 2011] DEBUG: Adding [echo] to the allowed regex pattern

[Fri Jul 1 07:20:01 2011] DEBUG: Using regex [echo] for allowing commands

[Fri Jul 1 07:20:01 2011] DEBUG: Validating command/sequence [echo quiesce]

[Fri Jul 1 07:20:01 2011] DEBUG: Command [echo quiesce] is allowed

[Fri Jul 1 07:20:01 2011] DEBUG: Command/sequence [echo quiesce] validated successfully

[Fri Jul 1 07:20:01 2011] DEBUG: Executing command [echo quiesce]

[Fri Jul 1 07:20:02 2011] DEBUG: Command [echo quiesce] finished with

exit code: [0]

stdout: [quiesce]

stderr: []

[Fri Jul 1 07:20:02 2011] DEBUG: Command [echo quiesce] finished successfully with message

[quiesce]

[Fri Jul 1 07:20:03 2011] DEBUG: Reloading configuration from /keith/scAgent3.4.0/config/agent.conf

[Fri Jul 1 07:20:03 2011] DEBUG: Reloading configuration finished with

exit code: [0]

stdout: []

stderr: []

[Fri Jul 1 07:20:03 2011] DEBUG: 10.65.55.80 is allowed to send requests

[Fri Jul 1 07:20:03 2011] INFO: Authorized request from [scServer@*]

[Fri Jul 1 07:20:04 2011] INFO: Authorized request from [scServer@*]

[Fri Jul 1 07:20:04 2011] DEBUG: Reloading configuration from /keith/scAgent3.4.0/config/agent.conf

[Fri Jul 1 07:20:04 2011] DEBUG: Reloading configuration finished with

exit code: [0]

stdout: []

stderr: []

[Fri Jul 1 07:20:04 2011] DEBUG: Adding [echo] to the allowed regex pattern

[Fri Jul 1 07:20:04 2011] DEBUG: Using regex [echo] for allowing commands

[Fri Jul 1 07:20:04 2011] DEBUG: Validating command/sequence [echo unquiesce]

[Fri Jul 1 07:20:04 2011] DEBUG: Command [echo unquiesce] is allowed

[Fri Jul 1 07:20:04 2011] DEBUG: Command/sequence [echo unquiesce] validated successfully

[Fri Jul 1 07:20:04 2011] DEBUG: Executing command [echo unquiesce]

[Fri Jul 1 07:20:04 2011] DEBUG: Command [echo unquiesce] finished with

exit code: [0]

stdout: [unquiesce]

stderr: []

[Fri Jul 1 07:20:04 2011] DEBUG: Command [echo unquiesce] finished successfully with message

[unquiesce]

ktenzer · ‎2011-07-01

The issues you are seeing would happen if you used an scServer 3.4.0 with scAgent 3.3.0, that isn't supported but as you mentioned you upgraded both.

Is it possible you didn't kill the process for the old SC 3.3.0 agent? Something very wierd is happening here

Keith

avbohemen · ‎2011-07-01

Found the problem.................

For SnapCreator3.3.0, I used the scServer3.3.0 directory to setup the start/stop script for the agent, copied that to /etc/rc.d/init.d, and linked it to /etc/rcX.d/S99scagent.

For SnapCreator3.4.0, I changed the /etc/rc.d/init.d/scAgent script to reflect the new version, so now the SNAPCREATOR_PATH was set (again) to /opt/netapp/scServer3.4.0

I also changed the options to start the multi-threaded agent of course.

Because there was only comments in agent.conf in 3.3, everything apparently worked fine.

Now 3.4 uses some form of authentication (scServer@*), agent.conf could not be found in /opt/netapp/scServer3.4.0/config.... giving me all the trouble above.

Changed SNAPCREATOR_PATH in the startup script to the scAgent3.4.0 directory, and everything is fine.

I will go and stand in the corner in shame.

Thanks for your help!

ktenzer · ‎2011-07-01

PHEW

You had us all running around like chickens without a head ;D

Still you did uncover the ^M issue, that wasn't expected...we are looking into it but it sounds like no big deal since the ^Ms went through our full QA process so we would have seen an issue if there was one. Still it is something we will want to fix, having ^Ms is not necessary and we will probably BURT it since it is unexpected (well at least from a user perspective).

Thank you so much!

Good luck with SC 3.4.0 I hope you enjoy the improvements we made one of them being much better security controls on agent side

Keith

avbohemen · ‎2011-07-01

Perhaps it would be nice to warn the user when starting the agent in normal mode, that agent.conf is missing. Now I saw the message when I started the agent in debug-mode, thanks to your example. Otherwise I would probably still be searching...

One of the biggest improvements for us in 3.4.0 is support for HP-UX and the multithreaded agent. We have machine with lots of Oracle instances, in which SMO is a pita to use. I'm glad 3.4 was released on the scheduled date, we will be deploying several hp-ux machines with snapcreator.

ktenzer · ‎2011-07-01

Good Feedback, I opened a feature request for this. When in doubt --debug on server and agent

Let us know how multi-threaded agent works out and that is exact reason we added multi-threaded agent

Regards,

Keith

pascalc · ‎2011-07-13

I did the upgrade under win2008R2 64 bits following these advices.

All went well except 2 things :

You also have to copy the content of <scServer 3.3 home>/log/ folder to <scServer 3.4 home>/log/
otherwise you'll end up with no logs at all (snapcreator does not recreate the sub directory when logging again)
the snapcreator service installed by the installer does not work.
The start error code is "0"
The system event log error message is
"Le service SnapCreatorServerService s’est arrêté avec l’erreur service particulière L’opération a réussi.." (french)
"The SnapCreatorServerService service stopped with service error : operation successfull" (english)

FOr those in the same case, I have set a scheduled task with "start at boot" trigger to start snapcreator automatically using the following bat file content :

cd /d <scServer3.4 home>\gui

"c:\Program Files (x86)\Java\jre6\bin\java.exe" -jar snapcreator.jar

ktenzer · ‎2011-07-13

Your first point is correct, thanks for bringing this up

The second point, the service does work and this is first report that it doesnt work. I am guessing this is because of the language (french). Unfortunately it is really hard to test all languages but we will look into this issue and see if there is anything we can do.

Also can you please install on more systems see if it is issue with just the server? Can you install on an english system and see if it works? This will help us narrow down the problem.

We appreciate all your support and help to make SC better!

Keith

charly · ‎2011-07-14

Hi,

Please run 'java -version' and see whether its 64bit

If you are getting

java version "1.6.0_24"

Java(TM) SE Runtime Environment (build 1.6.0_24-b07)

Java HotSpot(TM) Client VM (build 19.1-b02, mixed mode, sharing)

then need to update the JDK to 64bit. SnapCreator 64 bit installer will work only with 64bit Java.

64bit java version will show as

java version "1.6.0_24"

Java(TM) SE Runtime Environment (build 1.6.0_24-b07)

Java HotSpot(TM) 64-Bit Server VM (build 19.1-b02, mixed mode)

Between please send us the log under <scServer home>/gui/*.txt

regards

pascalc · ‎2011-07-15

java64 was the fix I needed - thanks charly

now the snapcreator service is starting nicely.

SnapCreator Upgrade from 3.3 to 3.4

Portfolio Updates | NetApp ONAIR