Data Backup and Recovery
Data Backup and Recovery
I am using SnapCreator 3.3.0 to perform Mysql db(lun snapshot)backup and integration with protection manager.SnapCreator Server installed on DFM4.0 and agents installed on Mysql DB servers(2 servers in cluster installed on RHEL 5.2).As per SC admin/install doc done the install and config but while trying to run SC snapshot we are getting below error:
26 [Thu Feb 3 16:21:24 2011] [192.168.2.219:9090(18.104.22.168)] INFO: Quiescing databases
27 [Thu Feb 3 16:21:24 2011] [192.168.2.219:9090(22.214.171.124)] INFO: Quiescing database mysql
28 [Thu Feb 3 16:21:24 2011] [192.168.2.219:9090(126.96.36.199)] DEBUG: Connection to mysql successfully established
29 [Thu Feb 3 16:21:24 2011] [192.168.2.219:9090(188.8.131.52)] DEBUG: Executing sql command 'flush tables with read lock' for database mysql
30 [Thu Feb 3 16:21:24 2011] [192.168.2.219:9090(184.108.40.206)] ERROR: #08S01Got packets out of order
31 [Thu Feb 3 17:21:12 2011] [192.168.2.219:9090(220.127.116.11)] ERROR: Application quiesce for plugin mysql failed with exit code 1, Exiting!
- Both source and destination arrays are running ONTAP 8.0.1
- SC 3.3.0 and DFM 4.0 (installed on same server)
Thanks in advance for any thoughts.
Config file please….
Please find the config details.Also i will attach the config files.
Configuration Name: DBbackup
Filer Name: 192.168.2.157
Filer User: ndmp
Filer Password: *****
Password Protection: yes
Snapshot Name: Scdb
Snapshot Policy Name: daily
Snapshot Policy Retention: 1
Snapshot Policy Retention Age: 30
Snapshot Policy Name Convention: Timestamp
Consistency Group: Yes
Consistency Group Timeout: medium
SnapVault Update: Yes
SnapVault Retention: 60
SnapVault Retention Age: 60
SnapVault Wait Time: 60
SnapVault Volumes: snap_Test
MySql Db:User/Password: snapdb:dfm/dfm123
Mysql Host: 192.168.2.219
Mysql Master Slave: No
Agent IP/DNS: 192.168.2.219
Agent Port: 9090
Server Timeout (secs): 1800
OM Host: 192.168.2.218
OM User: root
OM Password: *****
OM Port: 8088
OM Alerting: Yes
Can you please upload the config file so we can see it, feel free to remove passwords with "xxxxxx" if you arent using encrypted passwords.
What do the following commands result with
mysql -u root -p (also try to log in remotely from scServer)
telnet 192.168.2.219 3306
Also what happens if you try things without agent so comment out SC_AGENT?
The config file is missing in the scdump zip file for some reason. Can you send it? I would recommend opening NGS support case, that is quickest way to get you help but we would be happy to take a look as well.
It seems like either a network issue or rights of the mysql user (I have seen this error before but dont remember what it was). One thing to try would be not to use the agent, the MySQL plugin can run agentless. So comment out SC_AGENT line and see what happens.
I have attached config details.We installed SC agent on db servers and SC server on DFM server.Where we need to comment SC_AGENT line.We are planning to install SC server on db servers and try to take sc snapshot.Is this will make any difference?
You comment out SC_AGENT line in config file
You can install scServer on DB host but why? You could just install scServer centrally. The scServer can talk to MySQL DB remotely or though agent, the agent is optional, depends on if port 3306 is open to scServer or not. If it isnt then you need scServer local or agent and then scServer central. If 3306 is open then you just need scServer central.
Again I would recommend opening a NGS support case.
Can you able to login to the mysql using
“Mysql –h 192.168.2.219 –Dsnapdb –udfm –Pdfm123 –port 3306” from where you are trying to take the backup.
[root@cms-bkp ~]# mysql -h 192.168.2.219 mysqldb -udfm -pdfm123 -port 3306
mysql Ver 14.12 Distrib 5.0.45, for redhat-linux-gnu (x86_64) using readline 5.0
Copyright (C) 2002 MySQL AB
This software comes with ABSOLUTELY NO WARRANTY. This is free software,
and you are welcome to modify and redistribute it under the GPL license
Usage: mysql [OPTIONS] [database]
-?, --help Display this help and exit.
-I, --help Synonym for -?
--auto-rehash Enable automatic rehashing. One doesn't need to use
'rehash' to get table and field completion, but startup
and reconnecting may take a longer time. Disable with
Can you please check whether you are able to MySQL login with the credential from CLI.
Yes its happening.We tried with root and dfm login.both are working.NGS told that it is issue with mysql.But customer dont have sql admin to resolve this issue.Not This is new setup(mysqldb/snapcreator).
It looks like you are running MySQL 5.045 which doesn't appear to be supported
SC 3.3 supports following MySQL versions according to IMT http://now.netapp.com/matrix :
MySQL Enterprise Server 5.1.30
MySQL Enterprise Server 5.1.31
MySQL Enterprise Server 5.1.32
MySQL Enterprise Server 5.1.33
MySQL Enterprise Server 5.1.34
MySQL Enterprise Server 5.1.35
MySQL Enterprise Server 5.1.36
MySQL Enterprise Server 5.1.37
MySQL Enterprise Server 5.1.38
MySQL Enterprise Server 5.1.39
MySQL Enterprise Server 5.1.40
MySQL Enterprise Server 5.1.41
MySQL Enterprise Server 5.1.42
MySQL Enterprise Server 5.1.43
MySQL Enterprise Server 5.1.44
MySQL Enterprise Server 5.1.45
MySQL Enterprise Server 5.1.46
MySQL Enterprise Server 5.1.47
MySQL Enterprise Server 5.1.48
MySQL Enterprise Server 5.1.49
MySQL Enterprise Server 5.1.50
MySQL Enterprise Server 5.1.51
MySQL Enterprise Server 5.1.52
Other's have also had problems with 5.0.x with SC. It appears that there are some incompadibility issues w/Net::MySQL and the MySQL client libs.
A workaround solution to this problem is simply not use MYSQL plugin.
You can try using mysql command
APP_QUIESCE_CMD01=mysql <arguments> <sql query=flush tables with read lock>
APP_QUIESCE_CMD02=mysql <arguments> <sql query=flush logs>
APP_UNQUIESCE_CMD01=mysql <arguments> <sql query=unlock tables>
PRE_EXIT_CMD01=mysql <arguments> <sql query=unlock tables>
flush tables with read lock
We are using Mysql 5.1.52 ver.Any way I will try using mysql command on monday.Not use MYSQL plugin - meance no need to start snapcreator agent on Mysqldb server right?
If you are using MySQL version 5.1.52 that is supported depending on OS, take a look in Support Matrix (IMT) http://now.netapp.com/matrix
If your configuration is supported please open a NGS case, that is the best way to make sure problems like this get addressed.
I would certainly try the APP_QUIESCE_CMDS, APP_UNQUIESCE_CMDS, and PRE_EXIT_CMDS on monday as workaround to get something working
Let us know how we can help further
After recreating Mysql DB as per netapp best practice guide it started working.
As we were getting error with old DB setup cms-db1/cms-db2-cluster(agent installed on both cms-db1/cms-db2 and snap creator server installed on dfm server) we installed snap creator server in new DB server cms-as1(not in cluster) and set the variables as per netapp best practice and created new snap creator profile & config file for this. And we did the changes in my.cnf default value but while connecting it shows some permission issue with the Mysoldb.May be it requires agent for this I am not sure.
Later we changed the default password from 1 to 0 as recommendation in mentioned link. http://code.google.com/p/mythbox/issues/detail?id=72 – comment 8 for cms-db1 and cms-db2(both are in cluster) but still we were getting “ERROR: #08S01Got packets out of order”.
We requested customer to set the variables as per netapp best practice guide and they confirmed us that they did the mysql parameter changes in the cms-db1 and cms-db2 servers permanently using the options file. As a result they had to recreate the database as "innodb_data_file_path=ibdata1:100M:autoextend" creates errors in mysql server on startup.
Created a new test database snapdb with the account dfm.Granted all privileges on this account.
Since then the snap creator has connected successfully with the database.
Thanks for your suggestions.
This is great news!
Thank you for sharing this. Posting this feedback will help many others in addition to Snap Creator.
Good Lingaraj, Nice response. Now I remembered How you have done.
Apparently, I was facing a same error :
[Fri Nov 23 12:46:57 2012] [192.168.1.96:9090(18.104.22.168)] INFO: Quiescing databases
[Fri Nov 23 12:46:57 2012] [192.168.1.96:9090(22.214.171.124)] INFO: Quiescing database mysql
[Fri Nov 23 12:46:57 2012] [192.168.1.96:9090(126.96.36.199)] ERROR: #08S01Got packets out of order
[Fri Nov 23 12:46:57 2012] [192.168.1.96:9090(188.8.131.52)] ERROR: [scf-00053] Application quiesce for plugin mysql failed with exit code 1, Exiting!
I did as much as I could to fix as per this thread, but it kept failing on this. I am using MySQL 5.1.30 on Centos 5.5 (2.6.18-308.16.1.el5) and I logged into MySQL as root user. All I did just now is - entered this command "GRANT ALL PRIVILEGES ON *.* TO 'username'@'localhost' IDENTIFIED BY 'mypass' WITH GRANT OPTION;
I stopped MySQL and restarted it. I logged into MySQL again.
I ran the snapcreator script and it Worked!!!
[Fri Nov 23 12:57:52 2012] [192.168.1.96:9090(184.108.40.206)] INFO: Quiescing databases
[Fri Nov 23 12:57:52 2012] [192.168.1.96:9090(220.127.116.11)] INFO: Quiescing database mysql
[Fri Nov 23 12:57:52 2012] [192.168.1.96:9090(18.104.22.168)] INFO: Quiescing database mysql finished successfully
[Fri Nov 23 12:57:52 2012] [192.168.1.96:9090(22.214.171.124)] INFO: Quiescing databases finished successfully
My Question : What is grant privileges got to do with packets out of order ? I am not an SQL expert. All I wanted to do is is simulate customer enviornment on my lab machine. I succecded in doing that, but I am wondering what's packet out of order means here and is there a way to get more verbose explanatio of the error.
SC uses the MySQL API so if permissions arent correct then you get an error coming back from API. This error simply means communications problems and not having granted permissions would fall under that category.
Unfortunately we are using API so we just pass error back, maybe we can figure out how to get user friendly or at least inform user of things to check to troubleshoot.