Snapcreator 3.3.0 for Mysql 5.1.51

lingaraj_j · ‎2011-02-09

Hi,

I am using SnapCreator 3.3.0 to perform Mysql db(lun snapshot)backup and integration with protection manager.SnapCreator Server installed on DFM4.0 and agents installed on Mysql DB servers(2 servers in cluster installed on RHEL 5.2).As per SC admin/install doc done the install and config but while trying to run SC snapshot we are getting below error:

26 [Thu Feb 3 16:21:24 2011] [192.168.2.219:9090(3.3.0.1)] INFO: Quiescing databases

27 [Thu Feb 3 16:21:24 2011] [192.168.2.219:9090(3.3.0.1)] INFO: Quiescing database mysql

28 [Thu Feb 3 16:21:24 2011] [192.168.2.219:9090(3.3.0.1)] DEBUG: Connection to mysql successfully established

29 [Thu Feb 3 16:21:24 2011] [192.168.2.219:9090(3.3.0.1)] DEBUG: Executing sql command 'flush tables with read lock' for database mysql

30 [Thu Feb 3 16:21:24 2011] [192.168.2.219:9090(3.3.0.1)] ERROR: #08S01Got packets out of order

31 [Thu Feb 3 17:21:12 2011] [192.168.2.219:9090(3.3.0.1)] ERROR: Application quiesce for plugin mysql failed with exit code 1, Exiting!

Environment:

- Both source and destination arrays are running ONTAP 8.0.1

- SC 3.3.0 and DFM 4.0 (installed on same server)

Thanks in advance for any thoughts.

-Raj

nkarthik · ‎2011-02-09

Config file please….

Thanks

Karthikeyan.N

lingaraj_j · ‎2011-02-09

Hi Karthik,

Please find the config details.Also i will attach the config files.

Configuration Name: DBbackup
Filer Name: 192.168.2.157
Filer User: ndmp
Filer Password: *****
Password Protection: yes

Volumes: snap_Test

Snapshot Name: Scdb
Snapshot Policy Name: daily
Snapshot Policy Retention: 1
Snapshot Policy Retention Age: 30
Snapshot Policy Name Convention: Timestamp
Consistency Group: Yes
Consistency Group Timeout: medium

SnapVault Update: Yes
SnapVault Retention: 60
SnapVault Retention Age: 60
SnapVault Wait Time: 60
SnapVault Volumes: snap_Test
Database: MySql
MySql Db:User/Password: snapdb:dfm/dfm123
Mysql Host: 192.168.2.219
Mysql Port:3306
Mysql Master Slave: No

Agent IP/DNS: 192.168.2.219
Agent Port: 9090
Server Timeout (secs): 1800
OM Host: 192.168.2.218
OM User: root
OM Password: *****
OM Port: 8088

OM Alerting: Yes

ktenzer · ‎2011-02-09

Can you please upload the config file so we can see it, feel free to remove passwords with "xxxxxx" if you arent using encrypted passwords.

Keith

lingaraj_j · ‎2011-02-09

Hi,

Please find the config files.Created 2 files but for both we are getting same error.

Thanks,

Raj

ktenzer · ‎2011-02-09

What do the following commands result with

mysql -u root -p (also try to log in remotely from scServer)

telnet 192.168.2.219 3306

Also what happens if you try things without agent so comment out SC_AGENT?

Regards,

Keith

ktenzer · ‎2011-02-09

Hi Raj,

The config file is missing in the scdump zip file for some reason. Can you send it? I would recommend opening NGS support case, that is quickest way to get you help but we would be happy to take a look as well.

It seems like either a network issue or rights of the mysql user (I have seen this error before but dont remember what it was). One thing to try would be not to use the agent, the MySQL plugin can run agentless. So comment out SC_AGENT line and see what happens.

Regards,

Keith

lingaraj_j · ‎2011-02-09

Keith,

I have attached config details.We installed SC agent on db servers and SC server on DFM server.Where we need to comment SC_AGENT line.We are planning to install SC server on db servers and try to take sc snapshot.Is this will make any difference?

Thanks,

Raj

ktenzer · ‎2011-02-09

You comment out SC_AGENT line in config file

#SC_AGENT=host:port

You can install scServer on DB host but why? You could just install scServer centrally. The scServer can talk to MySQL DB remotely or though agent, the agent is optional, depends on if port 3306 is open to scServer or not. If it isnt then you need scServer local or agent and then scServer central. If 3306 is open then you just need scServer central.

Again I would recommend opening a NGS support case.

Regards,

kEITH

nkarthik · ‎2011-02-09

Can you able to login to the mysql using

“Mysql –h 192.168.2.219 –Dsnapdb –udfm –Pdfm123 –port 3306” from where you are trying to take the backup.

Thanks

Karthikeyan.N

lingaraj_j · ‎2011-02-09

[root@cms-bkp ~]# mysql -h 192.168.2.219 mysqldb -udfm -pdfm123 -port 3306

mysql Ver 14.12 Distrib 5.0.45, for redhat-linux-gnu (x86_64) using readline 5.0

This software comes with ABSOLUTELY NO WARRANTY. This is free software,

and you are welcome to modify and redistribute it under the GPL license

Usage: mysql [OPTIONS] [database]

-?, --help Display this help and exit.

-I, --help Synonym for -?

--auto-rehash Enable automatic rehashing. One doesn't need to use

'rehash' to get table and field completion, but startup

and reconnecting may take a longer time. Disable with

--disable-auto-rehash.

nkarthik · ‎2011-02-09

Can you please check whether you are able to MySQL login with the credential from CLI.

Thanks

Karthikeyan.N

lingaraj_j · ‎2011-02-09

Hi Karhik,

Yes its happening.We tried with root and dfm login.both are working.NGS told that it is issue with mysql.But customer dont have sql admin to resolve this issue.Not This is new setup(mysqldb/snapcreator).

Thanks,

Raj

ktenzer · ‎2011-02-11

Hi Raj,

It looks like you are running MySQL 5.045 which doesn't appear to be supported

SC 3.3 supports following MySQL versions according to IMT http://now.netapp.com/matrix :

MySQL Enterprise Server 5.1.30
MySQL Enterprise Server 5.1.31
MySQL Enterprise Server 5.1.32
MySQL Enterprise Server 5.1.33
MySQL Enterprise Server 5.1.34
MySQL Enterprise Server 5.1.35
MySQL Enterprise Server 5.1.36
MySQL Enterprise Server 5.1.37
MySQL Enterprise Server 5.1.38
MySQL Enterprise Server 5.1.39
MySQL Enterprise Server 5.1.40
MySQL Enterprise Server 5.1.41
MySQL Enterprise Server 5.1.42
MySQL Enterprise Server 5.1.43
MySQL Enterprise Server 5.1.44
MySQL Enterprise Server 5.1.45
MySQL Enterprise Server 5.1.46
MySQL Enterprise Server 5.1.47
MySQL Enterprise Server 5.1.48
MySQL Enterprise Server 5.1.49
MySQL Enterprise Server 5.1.50
MySQL Enterprise Server 5.1.51
MySQL Enterprise Server 5.1.52

Other's have also had problems with 5.0.x with SC. It appears that there are some incompadibility issues w/Net::MySQL and the MySQL client libs.

A workaround solution to this problem is simply not use MYSQL plugin.

You can try using mysql command

APP_QUIESCE_CMD01=mysql <arguments> <sql query=flush tables with read lock>

APP_QUIESCE_CMD02=mysql <arguments> <sql query=flush logs>

APP_UNQUIESCE_CMD01=mysql <arguments> <sql query=unlock tables>

PRE_EXIT_CMD01=mysql <arguments> <sql query=unlock tables>

Quiesce SQL:

flush tables with read lock

flush logs

Unquiesce SQL:

unlock tables

Regards,

Keith

lingaraj_j · ‎2011-02-11

Hi Keith,

We are using Mysql 5.1.52 ver.Any way I will try using mysql command on monday.Not use MYSQL plugin - meance no need to start snapcreator agent on Mysqldb server right?

Regards,

Lingaraj

ktenzer · ‎2011-02-12

Hi Lingaraj,

If you are using MySQL version 5.1.52 that is supported depending on OS, take a look in Support Matrix (IMT) http://now.netapp.com/matrix

If your configuration is supported please open a NGS case, that is the best way to make sure problems like this get addressed.

I would certainly try the APP_QUIESCE_CMDS, APP_UNQUIESCE_CMDS, and PRE_EXIT_CMDS on monday as workaround to get something working

Let us know how we can help further

Keith

lingaraj_j · ‎2011-03-15

Hi All,

After recreating Mysql DB as per netapp best practice guide it started working.

As we were getting error with old DB setup cms-db1/cms-db2-cluster(agent installed on both cms-db1/cms-db2 and snap creator server installed on dfm server) we installed snap creator server in new DB server cms-as1(not in cluster) and set the variables as per netapp best practice and created new snap creator profile & config file for this. And we did the changes in my.cnf default value but while connecting it shows some permission issue with the Mysoldb.May be it requires agent for this I am not sure.

Later we changed the default password from 1 to 0 as recommendation in mentioned link. http://code.google.com/p/mythbox/issues/detail?id=72 – comment 8 for cms-db1 and cms-db2(both are in cluster) but still we were getting “ERROR: #08S01Got packets out of order”.

We requested customer to set the variables as per netapp best practice guide and they confirmed us that they did the mysql parameter changes in the cms-db1 and cms-db2 servers permanently using the options file. As a result they had to recreate the database as "innodb_data_file_path=ibdata1:100M:autoextend" creates errors in mysql server on startup.

Created a new test database snapdb with the account dfm.Granted all privileges on this account.

Since then the snap creator has connected successfully with the database.

Thanks for your suggestions.

ktenzer · ‎2011-03-16

This is great news!

Thank you for sharing this. Posting this feedback will help many others in addition to Snap Creator.

Regards,

Keith

nkarthik · ‎2011-03-16

Good Lingaraj, Nice response. Now I remembered How you have done.

Regards,

Karthikeyan.N

ASHWINPAWARTESL · ‎2012-11-23

Apparently, I was facing a same error :

[Fri Nov 23 12:46:57 2012] [192.168.1.96:9090(3.6.0.1)] INFO: Quiescing databases

[Fri Nov 23 12:46:57 2012] [192.168.1.96:9090(3.6.0.1)] INFO: Quiescing database mysql

[Fri Nov 23 12:46:57 2012] [192.168.1.96:9090(3.6.0.1)] ERROR: #08S01Got packets out of order

[Fri Nov 23 12:46:57 2012] [192.168.1.96:9090(3.6.0.1)] ERROR: [scf-00053] Application quiesce for plugin mysql failed with exit code 1, Exiting!

I did as much as I could to fix as per this thread, but it kept failing on this. I am using MySQL 5.1.30 on Centos 5.5 (2.6.18-308.16.1.el5) and I logged into MySQL as root user. All I did just now is - entered this command "GRANT ALL PRIVILEGES ON *.* TO 'username'@'localhost' IDENTIFIED BY 'mypass' WITH GRANT OPTION;

I stopped MySQL and restarted it. I logged into MySQL again.

I ran the snapcreator script and it Worked!!!

[Fri Nov 23 12:57:52 2012] [192.168.1.96:9090(3.6.0.1)] INFO: Quiescing databases

[Fri Nov 23 12:57:52 2012] [192.168.1.96:9090(3.6.0.1)] INFO: Quiescing database mysql

[Fri Nov 23 12:57:52 2012] [192.168.1.96:9090(3.6.0.1)] INFO: Quiescing database mysql finished successfully

[Fri Nov 23 12:57:52 2012] [192.168.1.96:9090(3.6.0.1)] INFO: Quiescing databases finished successfully

My Question : What is grant privileges got to do with packets out of order ? I am not an SQL expert. All I wanted to do is is simulate customer enviornment on my lab machine. I succecded in doing that, but I am wondering what's packet out of order means here and is there a way to get more verbose explanatio of the error.

Thanks,

-Ashwin

ktenzer · ‎2012-11-26

SC uses the MySQL API so if permissions arent correct then you get an error coming back from API. This error simply means communications problems and not having granted permissions would fall under that category.

Unfortunately we are using API so we just pass error back, maybe we can figure out how to get user friendly or at least inform user of things to check to troubleshoot.

Keith