Subscribe

KVM plugin failure

[ Edited ]

Dear All,

I can't make the SC KVM plugin work, it always fails in the discovery phase with the following error about 10 minutes after the discover or backup operation is started:

Before the failure is returned, the KVM host is almost frozen due to the high amount on IOs observed on local disks. A couple of java processes (the SC agent) are apparently responsible for the freeze.

The exact error message is provided here:

[2014-03-03 15:05:34,345] DEBUG: Workflow : discover started with workflow id : 7

[2014-03-03 15:05:34,345] DEBUG: Version: Snap Creator Framework 4.1.0

[2014-03-03 15:05:34,345] DEBUG: Profile: init

[2014-03-03 15:05:34,345] DEBUG: Config: kvm

[2014-03-03 15:05:34,345] DEBUG: Action: discover

[2014-03-03 15:05:34,346] DEBUG: Plugin: kvm

[2014-03-03 15:05:34,346] DEBUG: Policy: null

[2014-03-03 15:05:34,346] DEBUG: Volume Name: detect

[2014-03-03 15:05:34,346] DEBUG: Snapshot Name: kvm_20140303150534

########## Agent validation ##########

[2014-03-03 15:05:34,559] INFO: Agent validation completed successfully for agent localhost:9090

########## Plugin validation ##########

[2014-03-03 15:05:34,584] INFO: Plugin validation completed successfully for plugin kvm

########## Application Discovery ##########

[2014-03-03 15:05:34,757] INFO: Performing discovery on : kvm

[2014-03-03 15:09:09,013] ERROR: SCF-00028: Auto Discovery for plug-in [kvm] failed with error [java.net.ConnectException: ConnectException invoking https://localhost:9090/SnapCreator/operations/status/059225cf-e871-4ee4-af7b-14b5e37776e4/7: Connection refused] and exit code [-1], Exiting!

[2014-03-03 15:09:09,802] DEBUG: Workflow : discover_OnFailure started with workflow id : 8

[2014-03-03 15:09:09,968] DEBUG: Version: Snap Creator Framework 4.1.0

[2014-03-03 15:09:09,969] DEBUG: Profile: init

[2014-03-03 15:09:09,969] DEBUG: Config: kvm

[2014-03-03 15:09:09,969] DEBUG: Action: discover_OnFailure

[2014-03-03 15:09:09,998] DEBUG: Plugin: kvm

[2014-03-03 15:09:09,998] DEBUG: Policy: null

[2014-03-03 15:09:10,034] DEBUG: Volume Name: detect

[2014-03-03 15:09:10,034] DEBUG: Snapshot Name: kvm_20140303150534

########## Agent Workflow Finalization ##########

[2014-03-03 15:10:42,366] INFO: Agent Workflow Finalization started

[2014-03-03 15:11:41,584] WARN: SCF-00080: Agent finalization for workflow failed with warning [java.net.SocketException: SocketException invoking http://localhost:9090/SnapCreator/Daemon/Agent: Unexpected end of file from server]. Continuing.

########## Snap Creator Framework 4.1.0 failed ##########

[2014-03-03 15:11:49,679] INFO: Pre Exit commands are not defined. Skipping !

I have no idea where to look at for debugging this as no log file (except an unknown.log) is created under the logs/plugins directory of the scAgent installation path.

Any help would be highly appreciated.

Kind regards

Pierre Kestremond

Re: KVM plugin failure

The only strange message I found in the agent.log for the above job is shown underlined below:

[2014-03-03T15:05:41,684+0100] DEBUG [pool-2-thread-3] com.netapp.snapcreator.agent.nextgen.plugin.legacyservice.ExecutorImpl - stdoutLogger running=true

[2014-03-03T15:05:41,685+0100] DEBUG [pool-2-thread-3] com.netapp.snapcreator.agent.nextgen.plugin.legacyservice.ExecutorImpl - stderrLogger running=true

[2014-03-03T15:05:43,645+0100] DEBUG [Thread-15] com.netapp.snapcreator.agent.nextgen.plugin.legacyservice.StreamLogger - run() - type=stdout, wid=7, p=kvm - 2014-03-03 15:05:43 (pid=17342) - Deleting request plugins/wrapper/in/7+kvm+1393855539683+discover

[2014-03-03T15:05:43,646+0100] DEBUG [Thread-15] com.netapp.snapcreator.agent.nextgen.plugin.legacyservice.StreamLogger - run() - type=stdout, wid=7, p=kvm - 2014-03-03 15:05:43 (pid=17342) - Will execute discover now (plugin=kvm, workflowId=7)

[2014-03-03T15:05:43,646+0100] DEBUG [Thread-15] com.netapp.snapcreator.agent.nextgen.plugin.legacyservice.StreamLogger - run() - type=stdout, wid=7, p=kvm - 2014-03-03 15:05:43 (pid=17342) - Config Parameters: ACTION, ALLOW_IP_ADDR, APP_AUTO_DISCOVERY, APP_CONF_PERSISTENCE, APP_DEFINED_BACKUP, APP_DEFINED_CLONE, APP_DEFINED_MOUNT, APP_DEFINED_RESTORE, APP_DEFINED_UMOUNT, APP_IGNORE_ERROR, APP_NAME, APP_VM, ARCHIVE_LOG_RECURSIVE_SEARCH, CMODE_SET, CMODE_SNAPSHOT_FORCE_DELETE, CONFDIR, CONFIG_NAME, CONFIG_TYPE, CONFNAME, KVM_CMD_RETRY_COUNT, KVM_VM_CONFIG_DIR, KVM_VM_MAPPING, LOG_NUM, LOG_TRACE_ENABLE, NTAP_ASUP_ERROR_ENABLE, NTAP_CIFS_EXPORT_ENABLE, NTAP_CLONE_FOR_BACKUP, NTAP_CLONE_SECONDARY, NTAP_CONSISTENCY_GROUP_SNAPSHOT, NTAP_CONSISTENCY_GROUP_TIMEOUT, NTAP_CONSISTENCY_GROUP_WAFL_SYNC, NTAP_EXTERNAL_SNAPSHOT_REGEX, NTAP_LUN_CLONE_RESERVATION, NTAP_NFS_EXPORT_ACCESS, NTAP_NFS_EXPORT_PERSISTENT, NTAP_NUM_VOL_CLONES, NTAP_OSSV_ENABLE, NTAP_PM_RUN_BACKUP, NTAP_PM_UPDATE, NTAP_PWD_PROTECTION, NTAP_SNAPMIRROR_CASCADING_UPDATE, NTAP_SNAPMIRROR_UPDATE, NTAP_SNAPMIRROR_USE_SNAPSHOT, NTAP_SNAPMIRROR_WAIT, NTAP_SNAPSHOT_CLEANUP, NTAP_SNAPSHOT_DELETE_BY_AGE_ONLY, NTAP_SNAPSHOT_DEPENDENCY_IGNORE, NTAP_SNAPSHOT_DISABLE, NTAP_SNAPSHOT_NODELETE, NTAP_SNAPSHOT_RESTORE_AUTO_DETECT, NTAP_SNAPSHOT_RETENTIONS, NTAP_SNAPSHOT_RETENTION_AGE, NTAP_SNAPVAULT_NODELETE, NTAP_SNAPVAULT_RESTORE_WAIT, NTAP_SNAPVAULT_SNAPSHOT, NTAP_SNAPVAULT_UPDATE, NTAP_SNAPVAULT_WAIT, NTAP_TIMEOUT, NTAP_USERS, NTAP_USE_EXTERNAL_SNAPSHOT, NTAP_VOL_CLONE_RESERVE, OM_EVENT_GENERATE, OPERATION_ID, PORT, PROFILE_NAME, RECORD_ID, SC_AGENT, SC_AGENT_LOG_ENABLE, SC_AGENT_TIMEOUT, SC_AGENT_UNQUIESCE_TIMEOUT, SC_AGENT_WATCHDOG_ENABLE, SC_ITERATION_ID, SC_SERVER_PORT, SC_USER_NAME, SNAME, SNAME_COMBINED, SNAPDRIVE, SNAPDRIVE_DISCOVERY, SNAP_TIME, SNAP_TIMESTAMP_ONLY, SUCCESS_MSG, SVNAME_COMBINED, TRANSPORT, USE_GLOBAL_CONFIG, USE_PROXY, VOLUMES, obj,

[2014-03-03T15:05:43,647+0100] DEBUG [Thread-15] com.netapp.snapcreator.agent.nextgen.plugin.legacyservice.StreamLogger - run() - type=stdout, wid=7, p=kvm - Bin dir: /opt/NetApp/scAgent4.1.0/plugins/wrapper

[2014-03-03T15:05:43,647+0100] DEBUG [Thread-15] com.netapp.snapcreator.agent.nextgen.plugin.legacyservice.StreamLogger - run() - type=stdout, wid=7, p=kvm - Plugin dir: /opt/NetApp/scAgent4.1.0/plugins/

[2014-03-03T15:05:43,647+0100] DEBUG [Thread-15] com.netapp.snapcreator.agent.nextgen.plugin.legacyservice.StreamLogger - run() - type=stdout, wid=7, p=kvm - Perl Plugin dir: /opt/NetApp/scAgent4.1.0/plugins/perl

[2014-03-03T15:05:43,647+0100] DEBUG [Thread-15] com.netapp.snapcreator.agent.nextgen.plugin.legacyservice.StreamLogger - run() - type=stdout, wid=7, p=kvm - Getting instance for kvm

[2014-03-03T15:05:43,685+0100] DEBUG [pool-2-thread-3] com.netapp.snapcreator.agent.nextgen.plugin.legacyservice.WrapperHelperImpl - execute() - Result file doesn't exist

[2014-03-03T15:05:43,685+0100] DEBUG [pool-2-thread-3] com.netapp.snapcreator.agent.nextgen.plugin.legacyservice.ExecutorImpl - stdoutLogger running=true

[2014-03-03T15:05:43,685+0100] DEBUG [pool-2-thread-3] com.netapp.snapcreator.agent.nextgen.plugin.legacyservice.ExecutorImpl - stderrLogger running=true

[2014-03-03T15:05:43,915+0100] DEBUG [Thread-15] com.netapp.snapcreator.agent.nextgen.plugin.legacyservice.StreamLogger - run() - type=stdout, wid=7, p=kvm - Executing discover

[2014-03-03T15:05:43,935+0100] DEBUG [Thread-15] com.netapp.snapcreator.agent.nextgen.plugin.legacyservice.StreamLogger - run() - type=stdout, wid=7, p=kvm - OS - execute() - trying to run ls /etc/libvirt/qemu

[2014-03-03T15:05:43,944+0100] DEBUG [Thread-16] com.netapp.snapcreator.agent.nextgen.plugin.legacyservice.StreamLogger - run() - type=stderr, wid=7, p=kvm - could not find ParserDetails.ini in /tmp/pdk-root-17342/XML/SAX

[2014-03-03T15:05:44,443+0100] DEBUG [Thread-15] com.netapp.snapcreator.agent.nextgen.plugin.legacyservice.StreamLogger - run() - type=stdout, wid=7, p=kvm - OS - execute() - trying to run mount | grep nfs

[2014-03-03T15:05:44,474+0100] DEBUG [Thread-16] com.netapp.snapcreator.agent.nextgen.plugin.legacyservice.StreamLogger - run() - type=stderr, wid=7, p=kvm - Deep recursion on subroutine "Net::SSLeay::AUTOLOAD" at

[2014-03-03T15:05:44,474+0100] DEBUG [Thread-16] com.netapp.snapcreator.agent.nextgen.plugin.legacyservice.StreamLogger - run() - type=stderr, wid=7, p=kvm -   /</opt/NetApp/scAgent4.1.0/plugins/wrapper/wrapper>Net/SSLeay.pm line 349 (#1)

[2014-03-03T15:05:44,474+0100] DEBUG [Thread-16] com.netapp.snapcreator.agent.nextgen.plugin.legacyservice.StreamLogger - run() - type=stderr, wid=7, p=kvm -     (W recursion) This subroutine has called itself (directly or indirectly)

[2014-03-03T15:05:44,474+0100] DEBUG [Thread-16] com.netapp.snapcreator.agent.nextgen.plugin.legacyservice.StreamLogger - run() - type=stderr, wid=7, p=kvm -     100 times more than it has returned.  This probably indicates an

[2014-03-03T15:05:44,475+0100] DEBUG [Thread-16] com.netapp.snapcreator.agent.nextgen.plugin.legacyservice.StreamLogger - run() - type=stderr, wid=7, p=kvm -     infinite recursion, unless you're writing strange benchmark programs, in

[2014-03-03T15:05:44,475+0100] DEBUG [Thread-16] com.netapp.snapcreator.agent.nextgen.plugin.legacyservice.StreamLogger - run() - type=stderr, wid=7, p=kvm -     which case it indicates something else.

[2014-03-03T15:05:44,475+0100] DEBUG [Thread-16] com.netapp.snapcreator.agent.nextgen.plugin.legacyservice.StreamLogger - run() - type=stderr, wid=7, p=kvm -

[2014-03-03T15:05:44,475+0100] DEBUG [Thread-16] com.netapp.snapcreator.agent.nextgen.plugin.legacyservice.StreamLogger - run() - type=stderr, wid=7, p=kvm -     This threshold can be changed from 100, by recompiling the perl binary,

[2014-03-03T15:05:44,475+0100] DEBUG [Thread-16] com.netapp.snapcreator.agent.nextgen.plugin.legacyservice.StreamLogger - run() - type=stderr, wid=7, p=kvm -     setting the C pre-processor macro PERL_SUB_DEPTH_WARN to the desired value.

[2014-03-03T15:05:44,475+0100] DEBUG [Thread-16] com.netapp.snapcreator.agent.nextgen.plugin.legacyservice.StreamLogger - run() - type=stderr, wid=7, p=kvm -

[2014-03-03T15:05:44,956+0100] DEBUG [qtp1279199758-20] com.netapp.snapcreator.agent.nextgen.security.AuthorizationValidatorImpl - isAuthorized() - Incoming request from user scServer2agent (127.0.0.1, 127.0.0.1)

Re: KVM plugin failure

Hello Pierre,

1) Please provide us with the Operating System detail.

2) Also, please provide us with the KVM configuration options from your snapcreator config file.

3) Please refer this thread https://communities.netapp.com/message/122252

     Does this file exist? ls /etc/libvirt/qemu

4) Are you able to run virsh commands?

    

Please let me know the details.

I shall forward the information to our KVM expert and get a feedback.

Thanks,
Siva Ramanathan

Re: KVM plugin failure

Hi Siva,

Here is the info you asked for:

1) Linux version

[root@treefrog ~]# cat /etc/redhat-release                                                                                                                                                               

Fedora release 19 (Schrödinger’s Cat)

[root@treefrog ~]# uname -a

Linux treefrog.netapp.com 3.12.11-201.fc19.x86_64 #1 SMP Fri Feb 14 19:08:33 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

[root@treefrog ~]#

2) the SC config file is included here:

[root@treefrog init]# grep -v "^#" kvm.conf

CONFIG_TYPE=STANDARD

VOLUMES=auto:detect

META_DATA_VOLUME=

LOG_NUM=30

SNAPDRIVE=N

SNAPDRIVE_DISCOVERY=N

LOG_TRACE_ENABLE=N

NTAP_PWD_PROTECTION=Y

USE_PROXY=N

NTAP_TIMEOUT=60

USE_GLOBAL_CONFIG=N

FEDERATED_APPLICATIONS=

ALLOW_IP_ADDR=N

SC_TMP_DIR=

NTAP_USERS=vs_kvm_pierrek_admin:vsadmin/793563786f416f4a6661735434714e3479484e3872413d3d0a

MANAGEMENT_INTERFACES=

PORT=443

TRANSPORT=HTTPS

SECONDARY_INTERFACES=

CMODE_CLUSTER_NAME=

CMODE_CLUSTER_USERS=

CMODE_SNAPSHOT_FORCE_DELETE=Y

CMODE_SET=Y

SNAME=kvm

SNAP_TIMESTAMP_ONLY=Y

NTAP_SNAPSHOT_POLICIES=

NTAP_USE_EXTERNAL_SNAPSHOT=N

NTAP_SNAPSHOT_NODELETE=N

NTAP_CONSISTENCY_GROUP_TIMEOUT=MEDIUM

NTAP_SNAPSHOT_RETENTIONS=DAILY:7

NTAP_SNAPSHOT_CREATE_CMD01=

NTAP_CONSISTENCY_GROUP_WAFL_SYNC=N

NTAP_CONSISTENCY_GROUP_SNAPSHOT=N

NTAP_SNAPSHOT_RETENTION_AGE=7

NTAP_METADATA_SNAPSHOT_CREATE_CMD=

NTAP_SNAPSHOT_DELETE_BY_AGE_ONLY=N

NTAP_SNAPSHOT_DEPENDENCY_IGNORE=N

NTAP_SNAPSHOT_RESTORE_AUTO_DETECT=N

NTAP_SNAPSHOT_DISABLE=N

NTAP_SNAPSHOT_LABEL=

NTAP_EXTERNAL_SNAPSHOT_REGEX=.*

NTAP_SNAPSHOT_CLEANUP=N

NTAP_SNAPVAULT_UPDATE=N

SNAPVAULT_QTREE_INCLUDE=

NTAP_SNAPVAULT_NODELETE=N

NTAP_SNAPVAULT_RESTORE_WAIT=N

NTAP_SNAPVAULT_RETENTION_AGE=

NTAP_SNAPVAULT_SNAPSHOT=N

NTAP_SNAPVAULT_MAX_TRANSFER=

NTAP_SNAPVAULT_WAIT=0

NTAP_SNAPVAULT_RETENTIONS=

SNAPVAULT_VOLUMES=

NTAP_SNAPMIRROR_WAIT=60

NTAP_SNAPMIRROR_USE_SNAPSHOT=N

NTAP_SNAPMIRROR_MAX_TRANSFER=

SNAPMIRROR_VOLUMES=

SNAPMIRROR_CASCADING_VOLUMES=

NTAP_SNAPMIRROR_CASCADING_UPDATE=N

NTAP_SNAPMIRROR_UPDATE=N

NTAP_CLONE_IGROUP_MAP=

NTAP_NFS_EXPORT_ACCESS=read-only

NTAP_LUN_CLONE_RESERVATION=N

NTAP_CLONE_SECONDARY_VOLUMES=

NTAP_NUM_VOL_CLONES=1

NTAP_NFS_EXPORT_PERSISTENT=N

NTAP_CLONE_FOR_BACKUP=N

NTAP_CIFS_EXPORT_ENABLE=N

NTAP_CLONE_SECONDARY=N

NTAP_NFS_EXPORT_HOST=

NTAP_VOL_CLONE_RESERVE=NONE

NTAP_DFM_DATA_SET=

NTAP_PM_RUN_BACKUP=N

NTAP_PM_UPDATE=N

NTAP_DFM_SNAPSHOT_FORMAT=

OM_TRANSPORT=

OM_HOST=

OM_EVENT_GENERATE=N

OM_USER=

OM_PWD=

OM_PORT=

NTAP_OSSV_FS_SNAPSHOT=

NTAP_OSSV_FS_SNAPSHOT_CREATE_CMD01=

NTAP_OSSV_HOMEDIR=

NTAP_OSSV_ENABLE=N

UMOUNT_CMD01=

ARCHIVE_CMD01=

APP_UNQUIESCE_CMD01=

MOUNT_CMD01=

APP_QUIESCE_CMD01=

PRE_CLONE_CREATE_CMD01=

PRE_RESTORE_CMD01=

PRE_NTAP_CMD01=

PRE_APP_UNQUIESCE_CMD01=

PRE_EXIT_CMD01=

PRE_APP_QUIESCE_CMD01=

PRE_NTAP_CLONE_DELETE_CMD01=

POST_NTAP_DATA_TRANSFER_CMD01=

POST_RESTORE_CMD01=

POST_APP_QUIESCE_CMD01=

POST_NTAP_CMD01=

POST_CLONE_CREATE_CMD01=

POST_APP_UNQUIESCE_CMD01=

SUCCESS_MSG=INFO: NetApp Snap Creator Framework finished successfully "(Action: %ACTION)"

NTAP_ASUP_ERROR_ENABLE=N

SENDTRAP=

SUCCESS_TRAP=

SC_AGENT_UNQUIESCE_TIMEOUT=305

SC_AGENT_WATCHDOG_ENABLE=N

SC_AGENT=localhost:9090

SC_AGENT_LOG_ENABLE=N

SC_AGENT_TIMEOUT=600

SC_CLONE_TARGET=

APP_NAME=kvm

APP_CONF_PERSISTENCE=Y

APP_IGNORE_ERROR=N

APP_DEFINED_RESTORE=Y

PYTHON_OPTS=

APP_DEFINED_CLONE=N

JAVA_HOME=

PYTHON_HOME=

FS_NAME=

JVM_ARGS=

PERL_OPTS=

PERL_HOME=

PLUGINS_FORWARD=

VALIDATE_VOLUMES=

APP_AUTO_DISCOVERY=Y

JAVA_CLASSPATH=

PLUGINS_RESTORE=

APP_DEFINED_BACKUP=N

APP_DEFINED_UMOUNT=N

PLUGINS_REVERSE=

APP_DEFINED_MOUNT=N

ARCHIVE_LOG_RECURSIVE_SEARCH=N

ARCHIVE_LOG_DIR=

ARCHIVE_LOG_EXT=

ARCHIVE_LOG_RETENTION=

ARCHIVE_LOG_ENABLE=

KVM_CMD_RETRY_COUNT=3

KVM_VM_MAPPING=auto:detect:detect

KVM_VM_CONFIG_DIR=/etc/libvirt/qemu

[root@treefrog init]#

3) here is the content of the /etc/libvirt/qemu directory:

[root@treefrog init]# ls -l /etc/libvirt/qemu

total 52

-rw-------. 1 root root 2868 Dec 24 11:23 arch-linux-01.xml

-rw-------. 1 root root 2802 Dec 20 16:32 arch-linux-02.xml

-rw-------. 1 root root 2852 Jan 26 08:12 fedora18-01.xml

-rw-------. 1 root root 2956 Jan 31 13:40 netapp7.xml

drwx------. 3 root root 4096 Jan 17 03:12 networks

-rw-------. 1 root root 5719 Jan 19 13:23 oel6-01.xml

-rw-------. 1 root root 5120 Dec 31 14:41 oel6-01.xml.save

-rw-------. 1 root root 2313 Jan  8 12:57 pierrek-lw7.xml

-rw-------. 1 root root 2915 Mar  3 11:28 tiny-centos64-template.xml  --> This one is located on NFS storage exported from my SVM

-rw-------. 1 root root 2877 Mar  3 09:15 tiny-centos64.xml

-rw-------. 1 root root 2546 Dec 21 12:57 win-2k8r2-02.xml

[root@treefrog init]#

I can run virsh commands as you can see here:

[root@treefrog init]# virsh list --all

Id    Name                           State

----------------------------------------------------

3     tiny-centos64-template         running

-     arch-linux-01                  shut off

-     arch-linux-02                  shut off

-     fedora18-01                    shut off

-     netapp7                        shut off

-     oel6-01                        shut off

-     pierrek-lw7                    shut off

-     tiny-centos64                  shut off

-     win-2k8r2-02                   shut off

[root@treefrog init]#

Thanks already for your help.

Pierre

Re: KVM plugin failure

We have not qualified or tested Fedora with Snap Creator. (

Meanwhile, could you please check by disbaling selinux?

http://docs.fedoraproject.org/en-US/Fedora/13/html/Security-Enhanced_Linux/sect-Security-Enhanced_Linux-Working_with_SELinux-Enabling_and_Disabling_SE...

I see that the files have ACL information

[root@treefrog init]# ls -l /etc/libvirt/qemu

total 52

-rw-------. 1 root root 2868 Dec 24 11:23 arch-linux-01.xml

Where is your snapcreator server running?

Also, please try with SC_AGENT=hostname:9090 (instead of localhost).

Please ensure /etc/hosts on the agent includes the storage controller IPs. (or in your DNS)

Thanks,
Siva Ramanathan

Re: KVM plugin failure

Hi Siva,

I changed the SC_AGENT parameter in the config file to the FQDN of the KVM host (in this test setup I am running the agent and server on the same machine, the KVM host).

Now I am getting the below error:

It is different from the one I had before. I will now disable SELINUX (I will do it only for troubleshooting but selinux should be in enforcing mode on my host). After a reboot I will let you know the result.

Kind regards

Pierre

Re: KVM plugin failure

Sorry the screenshot was not added. Here is the debug output of the job:

[2014-03-03 17:06:18,597] DEBUG: Workflow : discover started with workflow id : 9

[2014-03-03 17:06:18,600] DEBUG: Version: Snap Creator Framework 4.1.0

[2014-03-03 17:06:18,600] DEBUG: Profile: init

[2014-03-03 17:06:18,600] DEBUG: Config: kvm

[2014-03-03 17:06:18,600] DEBUG: Action: discover

[2014-03-03 17:06:18,600] DEBUG: Plugin: kvm

[2014-03-03 17:06:18,600] DEBUG: Policy: null

[2014-03-03 17:06:18,600] DEBUG: Volume Name: detect

[2014-03-03 17:06:18,600] DEBUG: Snapshot Name: kvm_20140303170618

########## Agent validation ##########

[2014-03-03 17:06:19,154] INFO: Agent validation completed successfully for agent treefrog.netapp.com:9090

########## Plugin validation ##########

[2014-03-03 17:06:19,268] INFO: Plugin validation completed successfully for plugin kvm

########## Application Discovery ##########

[2014-03-03 17:06:19,560] INFO: Performing discovery on : kvm

[2014-03-03 17:18:09,207] ERROR: [treefrog.netapp.com:9090 (4.1.0.1)] Operation failed. Reason: Executor timed out. Make sure plugins/wrapper/wrapper binary has execute permission set. Increasing WRAPPER_TIMEOUT_IN_MSEC in agent.properties may help, too.

[2014-03-03 17:18:09,549] ERROR: [treefrog.netapp.com:9090(4.1.0.1)] SCF-00028: Auto Discovery for plug-in [kvm] failed with error [validation failed] and exit code [102], Exiting!

[2014-03-03 17:18:10,285] DEBUG: Workflow : discover_OnFailure started with workflow id : 10

[2014-03-03 17:18:10,310] DEBUG: Version: Snap Creator Framework 4.1.0

[2014-03-03 17:18:10,310] DEBUG: Profile: init

[2014-03-03 17:18:10,310] DEBUG: Config: kvm

[2014-03-03 17:18:10,310] DEBUG: Action: discover_OnFailure

[2014-03-03 17:18:10,356] DEBUG: Plugin: kvm

[2014-03-03 17:18:10,356] DEBUG: Policy: null

[2014-03-03 17:18:10,397] DEBUG: Volume Name: detect

[2014-03-03 17:18:10,397] DEBUG: Snapshot Name: kvm_20140303170618

########## Agent Workflow Finalization ##########

[2014-03-03 17:18:10,715] INFO: Agent Workflow Finalization started

[2014-03-03 17:18:28,203] INFO: [treefrog.netapp.com:9090 (4.1.0.1)] Finalized workflow with id 9

[2014-03-03 17:18:28,203] INFO: Agent Workflow Finalization finished successfully

########## Snap Creator Framework 4.1.0 failed ##########

[2014-03-03 17:18:29,516] INFO: Pre Exit commands are not defined. Skipping !

Re: KVM plugin failure

From the treefrog host can you perform,

fping vs_kvm_pierrek_admin

Re: KVM plugin failure

Siva,

I have disabled selinux in /etc/selinux/config then the host has been rebooted.

[root@treefrog ~]# getenforce

Disabled

[root@treefrog ~]#

I have restarted the discovery job and it has failed the same way as with selinux in enforcing mode:

[2014-03-03 19:59:24,016] DEBUG: Workflow : discover started with workflow id : 11

[2014-03-03 19:59:24,018] DEBUG: Version: Snap Creator Framework 4.1.0

[2014-03-03 19:59:24,018] DEBUG: Profile: init

[2014-03-03 19:59:24,018] DEBUG: Config: kvm

[2014-03-03 19:59:24,018] DEBUG: Action: discover

[2014-03-03 19:59:24,018] DEBUG: Plugin: kvm

[2014-03-03 19:59:24,018] DEBUG: Policy: null

[2014-03-03 19:59:24,019] DEBUG: Volume Name: detect

[2014-03-03 19:59:24,019] DEBUG: Snapshot Name: kvm_20140303195923

########## Agent validation ##########

[2014-03-03 19:59:24,533] INFO: Agent validation completed successfully for agent treefrog.netapp.com:9090

########## Plugin validation ##########

[2014-03-03 19:59:24,629] INFO: Plugin validation completed successfully for plugin kvm

########## Application Discovery ##########

[2014-03-03 19:59:24,859] INFO: Performing discovery on : kvm

[2014-03-03 20:09:20,155] ERROR: SCF-00028: Auto Discovery for plug-in [kvm] failed with error [java.net.ConnectException: ConnectException invoking https://treefrog.netapp.com:9090/SnapCreator/operations/status/c19a2d6e-71d7-4195-ad28-30f78374053a/11: Connection refused] and exit code [-1], Exiting!

[2014-03-03 20:09:25,956] DEBUG: Workflow : discover_OnFailure started with workflow id : 12

[2014-03-03 20:09:25,956] DEBUG: Version: Snap Creator Framework 4.1.0

[2014-03-03 20:09:25,956] DEBUG: Profile: init

[2014-03-03 20:09:25,956] DEBUG: Config: kvm

[2014-03-03 20:09:25,956] DEBUG: Action: discover_OnFailure

[2014-03-03 20:09:25,956] DEBUG: Plugin: kvm

[2014-03-03 20:09:25,956] DEBUG: Policy: null

[2014-03-03 20:09:25,956] DEBUG: Volume Name: detect

[2014-03-03 20:09:25,956] DEBUG: Snapshot Name: kvm_20140303195923

########## Agent Workflow Finalization ##########

[2014-03-03 20:09:26,105] INFO: Agent Workflow Finalization started

[2014-03-03 20:09:46,525] WARN: SCF-00080: Agent finalization for workflow failed with warning [com.ctc.wstx.exc.WstxIOException: Connection refused]. Continuing.

########## Snap Creator Framework 4.1.0 failed ##########

[2014-03-03 20:09:47,977] INFO: Pre Exit commands are not defined. Skipping !

fping result:

[root@treefrog adm]# yum install fping

Loaded plugins: langpacks, refresh-packagekit, verify

Resolving Dependencies

--> Running transaction check

---> Package fping.x86_64 0:3.5-3.fc19 will be installed

--> Finished Dependency Resolution

Dependencies Resolved

=============================================================================================================================================================

Package                            Arch                                Version                                   Repository                            Size

=============================================================================================================================================================

Installing:

fping                              x86_64                              3.5-3.fc19                                updates                               45 k

Transaction Summary

=============================================================================================================================================================

Install  1 Package

Total download size: 45 k

Installed size: 102 k

Is this ok [y/d/N]: y

Downloading packages:

fping-3.5-3.fc19.x86_64.rpm                                                                                                           |  45 kB  00:00:00    

Running transaction check

Running transaction test

Transaction test succeeded

Running transaction

  Installing : fping-3.5-3.fc19.x86_64                                                                                                                   1/1

  Verifying  : fping-3.5-3.fc19.x86_64                                                                                                                   1/1

Installed:

  fping.x86_64 0:3.5-3.fc19                                                                                                                                 

Complete!

[root@treefrog adm]# df -h

Filesystem                        Size  Used Avail Use% Mounted on

/dev/mapper/fedora_treefrog-root   99G   19G   75G  20% /

devtmpfs                          2.9G     0  2.9G   0% /dev

tmpfs                             2.9G   84K  2.9G   1% /dev/shm

tmpfs                             2.9G  992K  2.9G   1% /run

tmpfs                             2.9G     0  2.9G   0% /sys/fs/cgroup

tmpfs                             2.9G  6.3M  2.9G   1% /tmp

/dev/sda1                         976M  149M  760M  17% /boot

/dev/mapper/fedora_treefrog-home  583G  321G  233G  59% /home

vs_kvm_pierrek:/kvm2               95G  1.3G   94G   2% /var/lib/libvirt/images/vs_kvm_pierrek

[root@treefrog adm]# fping vs_kvm_pierrek_admin

vs_kvm_pierrek_admin is alive

[root@treefrog adm]#

Re: KVM plugin failure

Thank you.

This sure looks like a network issue. The process waited for 10 minutes (600 seconds) and came back with a networking related error.

[2014-03-03 19:59:24,859] INFO: Performing discovery on : kvm

[2014-03-03 20:09:20,155] ERROR: SCF-00028: Auto Discovery for plug-in [kvm] failed with error [java.net.ConnectException: ConnectException invoking https://treefrog.netapp.com:9090/SnapCreator/operations/status/c19a2d6e-71d7-4195-ad28-30f78374053a/11: Connection refused] and exit code [-1], Exiting!

What is your java version? -- please try with a latest java 7 JRE (lesser than u51 update)

Please provide us with the below output.

netstat -tulpn

Can you try disable firewall and try a backup? (I am not sure if firewall is running)

/etc/init.d/iptables stop

Thanks,
Siva Ramanathan