00000d92.00732f2b 2437ac4e Wed Sep 25 2019 21:03:57 -06:00 [kern_mgwd:info:2162] 0x81d190100: 86ffff0000000034: WARNING: job_manager::job_store: run:src/job_store_support.cc:568 Job Mgr: rdb_transaction history: Timeout/0 Timeout/300 Timeout/600 00000d92.00732f34 2437b367 Wed Sep 25 2019 21:07:00 -06:00 [kern_mgwd:info:2162] 0x81d18ca00: 8603e90000305009: WARNING: job_manager::scheduler_other: get_next:src/scheduler.cc:440 onetime_scheduler::get_next: reschedule not allowed. line: 440 00000d92.00732f35 2437b367 Wed Sep 25 2019 21:07:00 -06:00 [kern_mgwd:info:2162] 0x81d18ca00: 8603e90000305009: WARNING: job_manager::job_record: get_next_run:src/job_record.cc:647 get_next_run(reschedule), returning 0, schedulenext is 0 for job: 21809 - 5689e6d4-e002-11e9-8fa8-00a098573067 00000d92.00732f36 2437b384 Wed Sep 25 2019 21:07:02 -06:00 [kern_mgwd:info:2162] 0x81b616100: 0: ERR: security_mgwd::tables::certificate::certificate: write_hash_files:src/tables/certificate/certificate.cc:1985 SSL: Failed to create symlink for file '/mroot/etc/vserver_4294967295/certificates/ssl/serverCa/99+4A538C28+EntrustRootCertificationAuthority-G2+455f1b52.0.pem' with '/mroot/etc/vserver_4294967295/certificates/ssl/serverCa/455f1b52.0'. Reason: 17 00000d92.00732f72 2437c402 Wed Sep 25 2019 21:14:05 -06:00 [kern_mgwd:info:2162] 0x81d18cf00: 8603e900003050b5: WARNING: job_manager::job_sched: modify_schedule:src/job_sched.cc:800 job_manager::job_sched::modify_schedule: Calling set_scheduler on the record: Current schedule = 0x81ca24200; New schedule = 0x81e2fd4e0 00000d92.00732f85 2437cfc3 Wed Sep 25 2019 21:19:05 -06:00 [kern_mgwd:info:2162] 0x81d190100: 86ffff0000000034: WARNING: job_manager::job_store: run:src/job_store_support.cc:568 Job Mgr: rdb_transaction history: Timeout/0 OK/300 Timeout/308 Timeout/608 00000d92.00732ff0 2437f2e2 Wed Sep 25 2019 21:34:05 -06:00 [kern_mgwd:info:2162] 0x81d190100: 86ffff0000000034: WARNING: job_manager::job_store: run:src/job_store_support.cc:568 Job Mgr: rdb_transaction history: Timeout/0 Timeout/300 Timeout/600 00000d92.00733070 24381894 Wed Sep 25 2019 21:50:10 -06:00 [kern_mgwd:info:2162] 0x81d190100: 86ffff0000000034: WARNING: job_manager::job_store: run:src/job_store_support.cc:568 Job Mgr: rdb_transaction history: Timeout/0 OK/300 Timeout/365 Timeout/665 00000d92.00733071 24381975 Wed Sep 25 2019 21:50:32 -06:00 [kern_mgwd:info:2162] A [src/rdb/HAM.cc 1233 (0x81b7aea00)]: checkHASFOConfigVerbose: haConfigured: True, result HA_AGENT_CONFIGURED. 00000d92.00733072 24381975 Wed Sep 25 2019 21:50:32 -06:00 [kern_mgwd:info:2162] update_phase_data: populateData - No phase information found entry doesn't exist { ANDU_LOG in populateData() at src/tables/cluster_image.cc:3824 } 00000d92.00733073 24381975 Wed Sep 25 2019 21:50:32 -06:00 [kern_mgwd:info:2162] update_phase_data: populateData - No phase information found entry doesn't exist { ANDU_LOG in populateData() at src/tables/cluster_image.cc:3824 } 00000d92.00733074 24381975 Wed Sep 25 2019 21:50:32 -06:00 [kern_mgwd:info:2162] update_phase_data: populateData - No phase information found entry doesn't exist { ANDU_LOG in populateData() at src/tables/cluster_image.cc:3824 } 00000d92.00733075 24381975 Wed Sep 25 2019 21:50:32 -06:00 [kern_mgwd:info:2162] A [src/rdb/HAM.cc 1233 (0x81b7b2100)]: checkHASFOConfigVerbose: haConfigured: True, result HA_AGENT_CONFIGURED. 00000d92.00733076 24381975 Wed Sep 25 2019 21:50:32 -06:00 [kern_mgwd:info:2162] Starting a backup of the var filesystem. 00000d92.00733077 243819f1 Wed Sep 25 2019 21:50:44 -06:00 [kern_mgwd:info:2162] A [src/rdb/HAM.cc 1233 (0x81b7b0800)]: checkHASFOConfigVerbose: haConfigured: True, result HA_AGENT_CONFIGURED. 00000d92.00733078 243819f1 Wed Sep 25 2019 21:50:44 -06:00 [kern_mgwd:info:2162] 0x81b148700: 8503e800000027c2: NOTICE: TABLES::disk: Added 168 disk recs to the shared reference cache (size 84) 00000d92.007330eb 24383bb2 Wed Sep 25 2019 22:05:09 -06:00 [kern_mgwd:info:2162] 0x81d190100: 86ffff0000000034: WARNING: job_manager::job_store: run:src/job_store_support.cc:568 Job Mgr: rdb_transaction history: Timeout/0 OK/300 Timeout/599 00000d92.007330f1 24384007 Wed Sep 25 2019 22:07:00 -06:00 [kern_mgwd:info:2162] 0x81d18b600: 8603e900003050a3: WARNING: job_manager::scheduler_other: get_next:src/scheduler.cc:440 onetime_scheduler::get_next: reschedule not allowed. line: 440 00000d92.007330f2 24384007 Wed Sep 25 2019 22:07:00 -06:00 [kern_mgwd:info:2162] 0x81d18b600: 8603e900003050a3: WARNING: job_manager::job_record: get_next_run:src/job_record.cc:647 get_next_run(reschedule), returning 0, schedulenext is 0 for job: 21810 - b9404fe1-e00a-11e9-8fa8-00a098573067 00000d92.007330f3 2438401d Wed Sep 25 2019 22:07:02 -06:00 [kern_mgwd:info:2162] 0x81b616100: 0: ERR: security_mgwd::tables::certificate::certificate: write_hash_files:src/tables/certificate/certificate.cc:1985 SSL: Failed to create symlink for file '/mroot/etc/vserver_4294967295/certificates/ssl/serverCa/99+4A538C28+EntrustRootCertificationAuthority-G2+455f1b52.0.pem' with '/mroot/etc/vserver_4294967295/certificates/ssl/serverCa/455f1b52.0'. Reason: 17 00000d92.00733145 24385ef9 Wed Sep 25 2019 22:20:12 -06:00 [kern_mgwd:info:2162] 0x81d190100: 86ffff0000000034: WARNING: job_manager::job_store: run:src/job_store_support.cc:568 Job Mgr: rdb_transaction history: Timeout/0 Timeout/300 OK/600 Timeout/603 00000d92.007331ae 24388221 Wed Sep 25 2019 22:35:12 -06:00 [kern_mgwd:info:2162] 0x81d190100: 86ffff0000000034: WARNING: job_manager::job_store: run:src/job_store_support.cc:568 Job Mgr: rdb_transaction history: Timeout/0 Timeout/300 Timeout/600 00000d92.00733229 2438a549 Wed Sep 25 2019 22:50:12 -06:00 [kern_mgwd:info:2162] 0x81d190100: 86ffff0000000034: WARNING: job_manager::job_store: run:src/job_store_support.cc:568 Job Mgr: rdb_transaction history: Timeout/0 Timeout/300 Timeout/600 00000d92.00733241 2438b1ca Wed Sep 25 2019 22:55:32 -06:00 [kern_mgwd:info:2162] Starting a backup of the var filesystem. 00000d92.0073328b 2438c831 Wed Sep 25 2019 23:05:05 -06:00 [kern_mgwd:info:2162] 0x81d190100: 86ffff0000000034: WARNING: job_manager::job_store: run:src/job_store_support.cc:568 Job Mgr: rdb_transaction history: Timeout/0 OK/300 Timeout/593 00000d92.007332a3 2438cca7 Wed Sep 25 2019 23:07:00 -06:00 [kern_mgwd:info:2162] 0x81d18c500: 8603e90000305137: WARNING: job_manager::scheduler_other: get_next:src/scheduler.cc:440 onetime_scheduler::get_next: reschedule not allowed. line: 440 00000d92.007332a4 2438cca7 Wed Sep 25 2019 23:07:00 -06:00 [kern_mgwd:info:2162] 0x81d18c500: 8603e90000305137: WARNING: job_manager::job_record: get_next_run:src/job_record.cc:647 get_next_run(reschedule), returning 0, schedulenext is 0 for job: 21811 - 1a0c3288-e013-11e9-8fa8-00a098573067 00000d92.007332a5 2438ccba Wed Sep 25 2019 23:07:01 -06:00 [kern_mgwd:info:2162] 0x81b616100: 0: ERR: security_mgwd::tables::certificate::certificate: write_hash_files:src/tables/certificate/certificate.cc:1985 SSL: Failed to create symlink for file '/mroot/etc/vserver_4294967295/certificates/ssl/serverCa/99+4A538C28+EntrustRootCertificationAuthority-G2+455f1b52.0.pem' with '/mroot/etc/vserver_4294967295/certificates/ssl/serverCa/455f1b52.0'. Reason: 17 00000d92.007332e8 2438eb8d Wed Sep 25 2019 23:20:11 -06:00 [kern_mgwd:info:2162] 0x81d190100: 86ffff0000000034: WARNING: job_manager::job_store: run:src/job_store_support.cc:568 Job Mgr: rdb_transaction history: Timeout/0 Timeout/300 OK/600 Timeout/606 00000d92.00733361 24390eb6 Wed Sep 25 2019 23:35:11 -06:00 [kern_mgwd:info:2162] 0x81d190100: 86ffff0000000034: WARNING: job_manager::job_store: run:src/job_store_support.cc:568 Job Mgr: rdb_transaction history: Timeout/0 Timeout/300 Timeout/600 00000d92.007333db 243931de Wed Sep 25 2019 23:50:11 -06:00 [kern_mgwd:info:2162] 0x81d190100: 86ffff0000000034: WARNING: job_manager::job_store: run:src/job_store_support.cc:568 Job Mgr: rdb_transaction history: Timeout/0 Timeout/300 Timeout/600 00000d92.0073341f 243948fa Thu Sep 26 2019 00:00:01 -06:00 [kern_mgwd:info:2162] 0x81b244200: 0: ERR: RPCServer: mgwd_xml_start_1_svc: Called for asup_pathname(/mroot/etc/log/autosupport/201909260000.0.files) 00000d92.00733420 243948fa Thu Sep 26 2019 00:00:01 -06:00 [kern_mgwd:info:2162] 0x81b244200: 0: ERR: RPCServer: mgwd_xml_start_1_svc: current_asup_xml exists. State: (2) 00000d92.00733421 243948fa Thu Sep 26 2019 00:00:01 -06:00 [kern_mgwd:info:2162] 0x81b244200: 0: ERR: RPCServer: mgwd_xml_start_1_svc: ASUP Started SUCCESS 00000d92.00733455 24394b2c Thu Sep 26 2019 00:00:58 -06:00 [kern_mgwd:info:2162] Starting a backup of the var filesystem. 00000d92.0073346a 24395506 Thu Sep 26 2019 00:05:11 -06:00 [kern_mgwd:info:2162] 0x81d190100: 86ffff0000000034: WARNING: job_manager::job_store: run:src/job_store_support.cc:568 Job Mgr: rdb_transaction history: Timeout/0 Timeout/300 Timeout/600 00000d92.00733486 24395947 Thu Sep 26 2019 00:07:00 -06:00 [kern_mgwd:info:2162] 0x81d18c500: 8603e900003051cc: WARNING: job_manager::scheduler_other: get_next:src/scheduler.cc:440 onetime_scheduler::get_next: reschedule not allowed. line: 440 00000d92.00733487 24395947 Thu Sep 26 2019 00:07:00 -06:00 [kern_mgwd:info:2162] 0x81d18c500: 8603e900003051cc: WARNING: job_manager::job_record: get_next_run:src/job_record.cc:647 get_next_run(reschedule), returning 0, schedulenext is 0 for job: 21812 - 7b947d7c-e01b-11e9-8fa8-00a098573067 00000d92.00733489 2439595b Thu Sep 26 2019 00:07:02 -06:00 [kern_mgwd:info:2162] 0x81b616100: 0: ERR: security_mgwd::tables::certificate::certificate: write_hash_files:src/tables/certificate/certificate.cc:1985 SSL: Failed to create symlink for file '/mroot/etc/vserver_4294967295/certificates/ssl/serverCa/99+4A538C28+EntrustRootCertificationAuthority-G2+455f1b52.0.pem' with '/mroot/etc/vserver_4294967295/certificates/ssl/serverCa/455f1b52.0'. Reason: 17 00000d92.007334a0 24396739 Thu Sep 26 2019 00:12:56 -06:00 [kern_mgwd:info:2162] 0x81d189d00: 8603e900003052aa: WARNING: job_manager::job_sched: process_modify:src/job_sched.cc:487 UIID:8847 UUID:4883e2f4-e024-11e9-8fa8-00a098573067 name:Local backup job rc:1 Doing nothing in modify callback for job, since it's pending cleanup 00000d92.007334a3 2439678a Thu Sep 26 2019 00:13:05 -06:00 [kern_mgwd:info:2162] 0x81d18b100: 8603e900003052b0: WARNING: FileReplication: Starting subscription (download) request for package nmg-fas8040-02.daily.2019-09-26.00_10_05.7z:latest with old API extract target '/mroot/etc/backups/.tmp/.tmp.bk.nmg-fas8040.daily.8.1569478205/nmg-fas8040-02'. 00000d92.007334a5 24396797 Thu Sep 26 2019 00:13:06 -06:00 [kern_mgwd:info:2162] 0x81d18a700: 8603e900003052b1: WARNING: FileReplication: Starting subscription (download) request for package nmg-fas8040-01.daily.2019-09-26.00_10_05.7z:latest with old API extract target '/mroot/etc/backups/.tmp/.tmp.bk.nmg-fas8040.daily.8.1569478205/nmg-fas8040-01'. 00000d92.007334a6 243967b2 Thu Sep 26 2019 00:13:08 -06:00 [kern_mgwd:info:2162] 0x81d18ca00: 8603e90000305275: WARNING: FileReplication: Initiating unpublish request of nmg-fas8040-02.daily.2019-09-26.00_10_05.7z on the cluster. 00000d92.007334a7 243967b2 Thu Sep 26 2019 00:13:08 -06:00 [kern_mgwd:info:2162] 0x81d18ca00: 8603e90000305275: WARNING: FileReplication: Initiating unpublish request of nmg-fas8040-01.daily.2019-09-26.00_10_05.7z on the cluster. 00000d92.007334ae 243968a2 Thu Sep 26 2019 00:13:32 -06:00 [kern_mgwd:info:2162] Info: smf-net: failed due to no more data or timeout. timeout=0 sec, fd=635, numFdReady=0, bytesRcvd=0, FD_ISSET=false, buflength=512 00000d92.007334b0 243968b7 Thu Sep 26 2019 00:13:35 -06:00 [kern_mgwd:info:2162] 0x81dc31900: 0: WARNING: FileReplication: The file replication service successfully distributed package nmg-fas8040.daily.2019-09-26.00_10_05.7z:0 to 169.254.23.82. 00000d92.007334b8 243969c3 Thu Sep 26 2019 00:14:02 -06:00 [kern_mgwd:info:2162] 0x81d18d400: 8603e90000305275: WARNING: FileReplication: Initiating unpublish request of nmg-fas8040.daily.2019-09-26.00_10_05.7z on the cluster. 00000d92.007334ba 24396a0b Thu Sep 26 2019 00:14:09 -06:00 [kern_mgwd:info:2162] 0x81d18c000: 8603e900003052c8: WARNING: job_manager::job_sched: modify_schedule:src/job_sched.cc:800 job_manager::job_sched::modify_schedule: Calling set_scheduler on the record: Current schedule = 0x81df1d2c0; New schedule = 0x81deb9dc0 00000d92.007334e5 24397824 Thu Sep 26 2019 00:20:10 -06:00 [kern_mgwd:info:2162] 0x81d190100: 86ffff0000000034: WARNING: job_manager::job_store: run:src/job_store_support.cc:568 Job Mgr: rdb_transaction history: OK/0 OK/299 OK/314 OK/361 OK/368 OK/434 OK/600 OK/605 00000d92.007334ea 24397832 Thu Sep 26 2019 00:20:11 -06:00 [kern_mgwd:info:2162] 0x81b7b4900: 0: ERR: RPCServer: mgwd_xml_start_1_svc: Called for asup_pathname(/mroot/etc/log/autosupport/201909260020.0.files) 00000d92.007334eb 24397832 Thu Sep 26 2019 00:20:11 -06:00 [kern_mgwd:info:2162] 0x81b7b4900: 0: ERR: RPCServer: mgwd_xml_start_1_svc: current_asup_xml exists. State: (2) 00000d92.007334ec 24397832 Thu Sep 26 2019 00:20:11 -06:00 [kern_mgwd:info:2162] 0x81b7b4900: 0: ERR: RPCServer: mgwd_xml_start_1_svc: ASUP Started SUCCESS 00000d92.00733566 24399b41 Thu Sep 26 2019 00:35:09 -06:00 [kern_mgwd:info:2162] 0x81d190100: 86ffff0000000034: WARNING: job_manager::job_store: run:src/job_store_support.cc:568 Job Mgr: rdb_transaction history: OK/0 Timeout/299 Timeout/599 00000d92.007335ec 2439be69 Thu Sep 26 2019 00:50:09 -06:00 [kern_mgwd:info:2162] 0x81d190100: 86ffff0000000034: WARNING: job_manager::job_store: run:src/job_store_support.cc:568 Job Mgr: rdb_transaction history: Timeout/0 Timeout/300 Timeout/600 00000d92.00733645 2439e17d Thu Sep 26 2019 01:05:07 -06:00 [kern_mgwd:info:2162] 0x81d190100: 86ffff0000000034: WARNING: job_manager::job_store: run:src/job_store_support.cc:568 Job Mgr: rdb_transaction history: Timeout/0 OK/300 Timeout/598 00000d92.00733662 2439e5ee Thu Sep 26 2019 01:07:00 -06:00 [kern_mgwd:info:2162] 0x81d18b600: 8603e9000030526e: WARNING: job_manager::scheduler_other: get_next:src/scheduler.cc:440 onetime_scheduler::get_next: reschedule not allowed. line: 440 00000d92.00733663 2439e5ee Thu Sep 26 2019 01:07:00 -06:00 [kern_mgwd:info:2162] 0x81d18b600: 8603e9000030526e: WARNING: job_manager::job_record: get_next_run:src/job_record.cc:647 get_next_run(reschedule), returning 0, schedulenext is 0 for job: 21813 - dd444cdb-e023-11e9-8fa8-00a098573067 00000d92.00733665 2439e5fa Thu Sep 26 2019 01:07:01 -06:00 [kern_mgwd:info:2162] 0x81b616100: 0: ERR: security_mgwd::tables::certificate::certificate: write_hash_files:src/tables/certificate/certificate.cc:1985 SSL: Failed to create symlink for file '/mroot/etc/vserver_4294967295/certificates/ssl/serverCa/99+4A538C28+EntrustRootCertificationAuthority-G2+455f1b52.0.pem' with '/mroot/etc/vserver_4294967295/certificates/ssl/serverCa/455f1b52.0'. Reason: 17 00000d92.0073368a 2439f6b3 Thu Sep 26 2019 01:14:10 -06:00 [kern_mgwd:info:2162] 0x81d18b100: 8603e9000030535f: WARNING: job_manager::job_sched: modify_schedule:src/job_sched.cc:800 job_manager::job_sched::modify_schedule: Calling set_scheduler on the record: Current schedule = 0x81ca81980; New schedule = 0x81b517400 00000d92.007336c1 243a026b Thu Sep 26 2019 01:19:10 -06:00 [kern_mgwd:info:2162] 0x81d190100: 86ffff0000000034: WARNING: job_manager::job_store: run:src/job_store_support.cc:568 Job Mgr: rdb_transaction history: Timeout/0 OK/300 OK/541 Timeout/543 00000d92.00733703 243a1c29 Thu Sep 26 2019 01:30:09 -06:00 [kern_mgwd:info:2162] 0x81d190100: 86ffff0000000034: WARNING: job_manager::job_store: run:src/job_store_support.cc:568 Job Mgr: rdb_transaction history: Timeout/0 Timeout/300 OK/600 00000d92.00733743 243a33a3 Thu Sep 26 2019 01:40:10 -06:00 [kern_mgwd:info:2162] 0x81d190100: 86ffff0000000034: WARNING: job_manager::job_store: run:src/job_store_support.cc:568 Job Mgr: rdb_transaction history: Timeout/0 Timeout/300 OK/600 00000d92.00733795 243a4af4 Thu Sep 26 2019 01:50:07 -06:00 [kern_mgwd:info:2162] 0x81d18d900: 86ffff000000002c: WARNING: job_manager::job_store: claim_job_tx:src/job_rec_int.cc:581 claim_job: Attempted to claim a job that was already owned: 489d5bff-b80c-11e7-964b-00a098573067 owner: f28b11a3-b769-11e7-9b07-00a09857ee8f 00000d92.00733796 243a4af4 Thu Sep 26 2019 01:50:07 -06:00 [kern_mgwd:info:2162] 0x81d18d900: 86ffff000000002c: WARNING: job_manager::job_sched: _check_claimable:src/job_sched.cc:2225 UIID:19 UUID:489d5bff-b80c-11e7-964b-00a098573067 name:Network Recovery Job - 5min rc:3 _check_claimable: failed to claim jobs: 12 00000d92.007337bd 243a56cb Thu Sep 26 2019 01:55:10 -06:00 [kern_mgwd:info:2162] 0x81d190100: 86ffff0000000034: WARNING: job_manager::job_store: run:src/job_store_support.cc:568 Job Mgr: rdb_transaction history: Timeout/0 Timeout/300 Timeout/600 00000d92.00733807 243a7286 Thu Sep 26 2019 02:07:00 -06:00 [kern_mgwd:info:2162] 0x81d18c000: 8603e9000030534c: WARNING: job_manager::scheduler_other: get_next:src/scheduler.cc:440 onetime_scheduler::get_next: reschedule not allowed. line: 440 00000d92.00733808 243a7286 Thu Sep 26 2019 02:07:00 -06:00 [kern_mgwd:info:2162] 0x81d18c000: 8603e9000030534c: WARNING: job_manager::job_record: get_next_run:src/job_record.cc:647 get_next_run(reschedule), returning 0, schedulenext is 0 for job: 21816 - 3f068dcf-e02c-11e9-8fa8-00a098573067 00000d92.0073380e 243a736b Thu Sep 26 2019 02:07:22 -06:00 [kern_mgwd:info:2162] 0x81b616100: 0: ERR: security_mgwd::tables::certificate::certificate: write_hash_files:src/tables/certificate/certificate.cc:1985 SSL: Failed to create symlink for file '/mroot/etc/vserver_4294967295/certificates/ssl/serverCa/99+4A538C28+EntrustRootCertificationAuthority-G2+455f1b52.0.pem' with '/mroot/etc/vserver_4294967295/certificates/ssl/serverCa/455f1b52.0'. Reason: 17 00000d92.00733839 243a79bb Thu Sep 26 2019 02:10:04 -06:00 [kern_mgwd:info:2162] A [src/rdb/TM.cc 2057 (0x81b7b4400)]: Time for txn TID <5,981331,981331> ('job_manager job_sched check claimable') local commit: 1339 msec; result: SUCCESS. 00000d92.0073383a 243a79d4 Thu Sep 26 2019 02:10:07 -06:00 [kern_mgwd:info:2162] 0x81d190100: 86ffff0000000034: WARNING: job_manager::job_store: run:src/job_store_support.cc:568 Job Mgr: rdb_transaction history: Timeout/0 OK/301 Timeout/597 00000d92.0073383b 243a7a25 Thu Sep 26 2019 02:10:15 -06:00 [kern_mgwd:info:2162] A [src/rdb/TM.cc 2057 (0x81b244c00)]: Time for txn TID <5,981333,981333> ('job_manager job tx loop') local commit: 2933 msec; result: SUCCESS. 00000d92.0073386a 243a8c2b Thu Sep 26 2019 02:17:55 -06:00 [kern_mgwd:info:2162] 0x81d189d00: 8603e90000305433: WARNING: job_manager::job_sched: process_modify:src/job_sched.cc:487 UIID:8850 UUID:c00618fe-e035-11e9-8fa8-00a098573067 name:Local backup job rc:1 Doing nothing in modify callback for job, since it's pending cleanup 00000d92.0073386e 243a8c95 Thu Sep 26 2019 02:18:07 -06:00 [kern_mgwd:info:2162] 0x81d18bb00: 8603e90000305435: WARNING: FileReplication: Starting subscription (download) request for package nmg-fas8040-02.8hour.2019-09-26.02_15_07.7z:latest with old API extract target '/mroot/etc/backups/.tmp/.tmp.bk.nmg-fas8040.8hour.7.1569485707/nmg-fas8040-02'. 00000d92.0073386f 243a8ca3 Thu Sep 26 2019 02:18:08 -06:00 [kern_mgwd:info:2162] 0x81d18ac00: 8603e90000305436: WARNING: FileReplication: Starting subscription (download) request for package nmg-fas8040-01.8hour.2019-09-26.02_15_07.7z:latest with old API extract target '/mroot/etc/backups/.tmp/.tmp.bk.nmg-fas8040.8hour.7.1569485707/nmg-fas8040-01'. 00000d92.00733871 243a8cbb Thu Sep 26 2019 02:18:10 -06:00 [kern_mgwd:info:2162] 0x81d18cf00: 8603e90000305404: WARNING: FileReplication: Initiating unpublish request of nmg-fas8040-02.8hour.2019-09-26.02_15_07.7z on the cluster. 00000d92.00733872 243a8cbb Thu Sep 26 2019 02:18:10 -06:00 [kern_mgwd:info:2162] 0x81d18cf00: 8603e90000305404: WARNING: FileReplication: Initiating unpublish request of nmg-fas8040-01.8hour.2019-09-26.02_15_07.7z on the cluster. 00000d92.0073387a 243a8dae Thu Sep 26 2019 02:18:35 -06:00 [kern_mgwd:info:2162] Info: smf-net: failed due to no more data or timeout. timeout=0 sec, fd=651, numFdReady=0, bytesRcvd=0, FD_ISSET=false, buflength=512 00000d92.0073387c 243a8dc6 Thu Sep 26 2019 02:18:37 -06:00 [kern_mgwd:info:2162] 0x81dc31900: 0: WARNING: FileReplication: The file replication service successfully distributed package nmg-fas8040.8hour.2019-09-26.02_15_07.7z:0 to 169.254.23.82. 00000d92.0073387e 243a8ed9 Thu Sep 26 2019 02:19:05 -06:00 [kern_mgwd:info:2162] 0x81d18d400: 8603e90000305404: WARNING: FileReplication: Initiating unpublish request of nmg-fas8040.8hour.2019-09-26.02_15_07.7z on the cluster. 00000d92.0073387f 243a8ed9 Thu Sep 26 2019 02:19:05 -06:00 [kern_mgwd:info:2162] 0x81d190100: 86ffff0000000034: WARNING: job_manager::job_store: run:src/job_store_support.cc:568 Job Mgr: rdb_transaction history: OK/0 OK/70 OK/233 OK/530 00000d92.007338c1 243aa8d3 Thu Sep 26 2019 02:30:10 -06:00 [kern_mgwd:info:2162] 0x81d190100: 86ffff0000000034: WARNING: job_manager::job_store: run:src/job_store_support.cc:568 Job Mgr: rdb_transaction history: Timeout/0 Timeout/300 OK/600 OK/615 00000d92.007338ff 243ac04c Thu Sep 26 2019 02:40:11 -06:00 [kern_mgwd:info:2162] 0x81d190100: 86ffff0000000034: WARNING: job_manager::job_store: run:src/job_store_support.cc:568 Job Mgr: rdb_transaction history: OK/0 OK/299 OK/599 00000d92.0073397a 243ae37f Thu Sep 26 2019 02:55:12 -06:00 [kern_mgwd:info:2162] 0x81d190100: 86ffff0000000034: WARNING: job_manager::job_store: run:src/job_store_support.cc:568 Job Mgr: rdb_transaction history: Timeout/0 OK/300 OK/600 Timeout/601 00000d92.007339c2 243b017e Thu Sep 26 2019 03:08:00 -06:00 [kern_mgwd:info:2162] 0x81d18c000: 8603e900003053ea: WARNING: job_manager::scheduler_other: get_next:src/scheduler.cc:440 onetime_scheduler::get_next: reschedule not allowed. line: 440 00000d92.007339c3 243b017e Thu Sep 26 2019 03:08:00 -06:00 [kern_mgwd:info:2162] 0x81d18c000: 8603e900003053ea: WARNING: job_manager::job_record: get_next_run:src/job_record.cc:647 get_next_run(reschedule), returning 0, schedulenext is 0 for job: 21817 - c74472c3-e034-11e9-8fa8-00a098573067 00000d92.007339c4 243b0190 Thu Sep 26 2019 03:08:01 -06:00 [kern_mgwd:info:2162] 0x81b616100: 0: ERR: security_mgwd::tables::certificate::certificate: write_hash_files:src/tables/certificate/certificate.cc:1985 SSL: Failed to create symlink for file '/mroot/etc/vserver_4294967295/certificates/ssl/serverCa/99+4A538C28+EntrustRootCertificationAuthority-G2+455f1b52.0.pem' with '/mroot/etc/vserver_4294967295/certificates/ssl/serverCa/455f1b52.0'. Reason: 17 00000d92.007339e7 243b067c Thu Sep 26 2019 03:10:07 -06:00 [kern_mgwd:info:2162] 0x81d190100: 86ffff0000000034: WARNING: job_manager::job_store: run:src/job_store_support.cc:568 Job Mgr: rdb_transaction history: OK/0 Timeout/295 OK/595 00000d92.00733a3d 243b29a6 Thu Sep 26 2019 03:25:08 -06:00 [kern_mgwd:info:2162] 0x81d190100: 86ffff0000000034: WARNING: job_manager::job_store: run:src/job_store_support.cc:568 Job Mgr: rdb_transaction history: OK/0 OK/296 OK/596 Timeout/601 00000d92.00733ab1 243b4cc4 Thu Sep 26 2019 03:40:07 -06:00 [kern_mgwd:info:2162] 0x81d190100: 86ffff0000000034: WARNING: job_manager::job_store: run:src/job_store_support.cc:568 Job Mgr: rdb_transaction history: OK/0 OK/295 OK/595 Timeout/599 00000d92.00733b2b 243b6ff3 Thu Sep 26 2019 03:55:07 -06:00 [kern_mgwd:info:2162] 0x81d190100: 86ffff0000000034: WARNING: job_manager::job_store: run:src/job_store_support.cc:568 Job Mgr: rdb_transaction history: OK/0 OK/295 OK/595 Timeout/600 00000d92.00733b2d 243b79be Thu Sep 26 2019 03:59:18 -06:00 [kern_mgwd:info:2162] A [src/rdb/rdb_latency_defs.cc 30 (0x81b244c00)]: _logLongLatency: Time to fsync: 7 sec. 00000d92.00733b2e 243b79be Thu Sep 26 2019 03:59:18 -06:00 [kern_mgwd:info:2162] A [src/rdb/fileops.cc 615 (0x81b244c00)]: Time to fsync /mroot/etc/cluster_config/rdb/Management/_unit.config: 7 sec. 00000d92.00733b2f 243b79be Thu Sep 26 2019 03:59:18 -06:00 [kern_mgwd:info:2162] A [src/rdb/cluster_events.cc 78 (0x81b244c00)]: Cluster event: node-event, epoch 0, site 1001 [long fsync: 7 sec]. 00000d92.00733b30 243b7b56 Thu Sep 26 2019 03:59:18 -06:00 [kern_mgwd:info:2162] A [src/rdb/TM.cc 2057 (0x81b244c00)]: Time for txn TID <5,981466,981466> ('job_manager job tx loop') local commit: 6993 msec; result: SUCCESS. 00000d93.00000235 243bb953 Thu Sep 26 2019 04:27:04 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 92 status calls. 00000d93.00000236 243bb95d Thu Sep 26 2019 04:27:05 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 93 status calls. 00000d93.00000237 243bb967 Thu Sep 26 2019 04:27:06 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 94 status calls. 00000d93.00000238 243bb971 Thu Sep 26 2019 04:27:07 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 95 status calls. 00000d93.00000239 243bb985 Thu Sep 26 2019 04:27:08 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 96 status calls. 00000d93.0000023b 243bb985 Thu Sep 26 2019 04:27:09 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 97 status calls. 00000d93.0000023c 243bb98f Thu Sep 26 2019 04:27:10 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 98 status calls. 00000d93.0000023d 243bb999 Thu Sep 26 2019 04:27:11 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 99 status calls. 00000d93.0000023e 243bb9a3 Thu Sep 26 2019 04:27:12 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 100 status calls. 00000d93.0000023f 243bb9b7 Thu Sep 26 2019 04:27:13 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 101 status calls. 00000d93.00000241 243bb9b7 Thu Sep 26 2019 04:27:14 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 102 status calls. 00000d93.00000242 243bb9c1 Thu Sep 26 2019 04:27:15 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 103 status calls. 00000d93.00000243 243bb9cb Thu Sep 26 2019 04:27:16 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 104 status calls. 00000d93.00000244 243bb9d5 Thu Sep 26 2019 04:27:17 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 105 status calls. 00000d93.00000245 243bb9e9 Thu Sep 26 2019 04:27:18 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 106 status calls. 00000d93.00000247 243bb9e9 Thu Sep 26 2019 04:27:19 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 107 status calls. 00000d93.00000248 243bb9f3 Thu Sep 26 2019 04:27:20 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 108 status calls. 00000d93.00000249 243bb9fd Thu Sep 26 2019 04:27:21 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 109 status calls. 00000d93.0000024a 243bba07 Thu Sep 26 2019 04:27:22 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 110 status calls. 00000d93.0000024b 243bba1b Thu Sep 26 2019 04:27:23 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 111 status calls. 00000d93.0000024d 243bba1b Thu Sep 26 2019 04:27:24 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 112 status calls. 00000d93.0000024e 243bba25 Thu Sep 26 2019 04:27:25 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 113 status calls. 00000d93.0000024f 243bba2f Thu Sep 26 2019 04:27:26 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 114 status calls. 00000d93.00000250 243bba39 Thu Sep 26 2019 04:27:27 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 115 status calls. 00000d93.00000251 243bba4d Thu Sep 26 2019 04:27:28 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 116 status calls. 00000d93.00000253 243bba4d Thu Sep 26 2019 04:27:29 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 117 status calls. 00000d93.00000254 243bba57 Thu Sep 26 2019 04:27:30 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 118 status calls. 00000d93.00000255 243bba61 Thu Sep 26 2019 04:27:31 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 119 status calls. 00000d93.00000256 243bba6b Thu Sep 26 2019 04:27:32 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 120 status calls. 00000d93.00000257 243bba7f Thu Sep 26 2019 04:27:33 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 121 status calls. 00000d93.00000259 243bba7f Thu Sep 26 2019 04:27:34 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 122 status calls. 00000d93.0000025a 243bba8a Thu Sep 26 2019 04:27:35 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 123 status calls. 00000d93.0000025b 243bba94 Thu Sep 26 2019 04:27:36 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 124 status calls. 00000d93.0000025c 243bba9e Thu Sep 26 2019 04:27:37 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 125 status calls. 00000d93.0000025d 243bbab2 Thu Sep 26 2019 04:27:38 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 126 status calls. 00000d93.0000025f 243bbab2 Thu Sep 26 2019 04:27:39 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 127 status calls. 00000d93.00000260 243bbabc Thu Sep 26 2019 04:27:40 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 128 status calls. 00000d93.00000261 243bbac6 Thu Sep 26 2019 04:27:41 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 129 status calls. 00000d93.00000262 243bbad0 Thu Sep 26 2019 04:27:42 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 130 status calls. 00000d93.00000263 243bbae4 Thu Sep 26 2019 04:27:43 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 131 status calls. 00000d93.00000265 243bbae4 Thu Sep 26 2019 04:27:44 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 132 status calls. 00000d93.00000266 243bbaee Thu Sep 26 2019 04:27:45 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 133 status calls. 00000d93.00000267 243bbaf8 Thu Sep 26 2019 04:27:46 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 134 status calls. 00000d93.00000268 243bbb02 Thu Sep 26 2019 04:27:47 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 135 status calls. 00000d93.00000269 243bbb16 Thu Sep 26 2019 04:27:48 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 136 status calls. 00000d93.0000026b 243bbb16 Thu Sep 26 2019 04:27:49 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 137 status calls. 00000d93.0000026c 243bbb20 Thu Sep 26 2019 04:27:50 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 138 status calls. 00000d93.0000026d 243bbb2a Thu Sep 26 2019 04:27:51 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 139 status calls. 00000d93.0000026e 243bbb34 Thu Sep 26 2019 04:27:52 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 140 status calls. 00000d93.0000026f 243bbb48 Thu Sep 26 2019 04:27:53 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 141 status calls. 00000d93.00000271 243bbb48 Thu Sep 26 2019 04:27:54 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 142 status calls. 00000d93.00000272 243bbb52 Thu Sep 26 2019 04:27:55 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 143 status calls. 00000d93.00000273 243bbb5c Thu Sep 26 2019 04:27:56 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 144 status calls. 00000d93.00000274 243bbb66 Thu Sep 26 2019 04:27:57 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 145 status calls. 00000d93.00000275 243bbb7a Thu Sep 26 2019 04:27:58 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 146 status calls. 00000d93.00000277 243bbb7a Thu Sep 26 2019 04:27:59 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 147 status calls. 00000d93.00000278 243bbb84 Thu Sep 26 2019 04:28:00 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 148 status calls. 00000d93.00000279 243bbb8e Thu Sep 26 2019 04:28:01 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 149 status calls. 00000d93.0000027a 243bbb98 Thu Sep 26 2019 04:28:02 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 150 status calls. 00000d93.0000027b 243bbbac Thu Sep 26 2019 04:28:03 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 151 status calls. 00000d93.0000027d 243bbbac Thu Sep 26 2019 04:28:04 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 152 status calls. 00000d93.0000027e 243bbbb6 Thu Sep 26 2019 04:28:05 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 153 status calls. 00000d93.0000027f 243bbbc0 Thu Sep 26 2019 04:28:06 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 154 status calls. 00000d93.00000280 243bbbca Thu Sep 26 2019 04:28:07 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 155 status calls. 00000d93.00000281 243bbbde Thu Sep 26 2019 04:28:08 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 156 status calls. 00000d93.00000283 243bbbde Thu Sep 26 2019 04:28:09 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 157 status calls. 00000d93.00000284 243bbbe8 Thu Sep 26 2019 04:28:10 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 158 status calls. 00000d93.00000285 243bbbf2 Thu Sep 26 2019 04:28:11 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 159 status calls. 00000d93.00000286 243bbbfc Thu Sep 26 2019 04:28:12 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 160 status calls. 00000d93.00000287 243bbc10 Thu Sep 26 2019 04:28:13 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 161 status calls. 00000d93.00000289 243bbc10 Thu Sep 26 2019 04:28:14 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 162 status calls. 00000d93.0000028a 243bbc1a Thu Sep 26 2019 04:28:15 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 163 status calls. 00000d93.0000028b 243bbc24 Thu Sep 26 2019 04:28:16 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 164 status calls. 00000d93.0000028c 243bbc2e Thu Sep 26 2019 04:28:17 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 165 status calls. 00000d93.0000028d 243bbc42 Thu Sep 26 2019 04:28:18 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 166 status calls. 00000d93.0000028f 243bbc42 Thu Sep 26 2019 04:28:19 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 167 status calls. 00000d93.00000290 243bbc4c Thu Sep 26 2019 04:28:20 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 168 status calls. 00000d93.00000291 243bbc56 Thu Sep 26 2019 04:28:21 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 169 status calls. 00000d93.00000292 243bbc60 Thu Sep 26 2019 04:28:22 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 170 status calls. 00000d93.00000293 243bbc74 Thu Sep 26 2019 04:28:23 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 171 status calls. 00000d93.00000295 243bbc74 Thu Sep 26 2019 04:28:24 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 172 status calls. 00000d93.00000296 243bbc7e Thu Sep 26 2019 04:28:25 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 173 status calls. 00000d93.00000297 243bbc88 Thu Sep 26 2019 04:28:26 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 174 status calls. 00000d93.00000298 243bbc9c Thu Sep 26 2019 04:28:27 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 175 status calls. 00000d93.0000029a 243bbc9c Thu Sep 26 2019 04:28:28 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 176 status calls. 00000d93.0000029b 243bbca6 Thu Sep 26 2019 04:28:29 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 177 status calls. 00000d93.0000029c 243bbcb0 Thu Sep 26 2019 04:28:30 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 178 status calls. 00000d93.0000029d 243bbcba Thu Sep 26 2019 04:28:31 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 179 status calls. 00000d93.0000029e 243bbcce Thu Sep 26 2019 04:28:32 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 180 status calls. 00000d93.000002a0 243bbcce Thu Sep 26 2019 04:28:33 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 181 status calls. 00000d93.000002a1 243bbcd8 Thu Sep 26 2019 04:28:34 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 182 status calls. 00000d93.000002a2 243bbce2 Thu Sep 26 2019 04:28:35 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 183 status calls. 00000d93.000002a3 243bbcec Thu Sep 26 2019 04:28:36 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 184 status calls. 00000d93.000002a4 243bbd00 Thu Sep 26 2019 04:28:37 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 185 status calls. 00000d93.000002a6 243bbd00 Thu Sep 26 2019 04:28:38 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 186 status calls. 00000d93.000002a7 243bbd0a Thu Sep 26 2019 04:28:39 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 187 status calls. 00000d93.000002a8 243bbd14 Thu Sep 26 2019 04:28:40 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 188 status calls. 00000d93.000002a9 243bbd1e Thu Sep 26 2019 04:28:41 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 189 status calls. 00000d93.000002aa 243bbd32 Thu Sep 26 2019 04:28:42 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 190 status calls. 00000d93.000002ac 243bbd32 Thu Sep 26 2019 04:28:43 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 191 status calls. 00000d93.000002ad 243bbd3c Thu Sep 26 2019 04:28:44 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 192 status calls. 00000d93.000002ae 243bbd46 Thu Sep 26 2019 04:28:45 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 193 status calls. 00000d93.000002af 243bbd50 Thu Sep 26 2019 04:28:46 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 194 status calls. 00000d93.000002b0 243bbd64 Thu Sep 26 2019 04:28:47 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 195 status calls. 00000d93.000002b2 243bbd64 Thu Sep 26 2019 04:28:48 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 196 status calls. 00000d93.000002b3 243bbd6e Thu Sep 26 2019 04:28:49 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 197 status calls. 00000d93.000002b4 243bbd78 Thu Sep 26 2019 04:28:50 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 198 status calls. 00000d93.000002b5 243bbd82 Thu Sep 26 2019 04:28:51 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 199 status calls. 00000d93.000002b6 243bbd96 Thu Sep 26 2019 04:28:52 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 200 status calls. 00000d93.000002b8 243bbd96 Thu Sep 26 2019 04:28:53 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 201 status calls. 00000d93.000002b9 243bbda0 Thu Sep 26 2019 04:28:54 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 202 status calls. 00000d93.000002ba 243bbdaa Thu Sep 26 2019 04:28:55 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 203 status calls. 00000d93.000002bb 243bbdb4 Thu Sep 26 2019 04:28:56 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 204 status calls. 00000d93.000002bc 243bbdc8 Thu Sep 26 2019 04:28:57 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 205 status calls. 00000d93.000002be 243bbdc8 Thu Sep 26 2019 04:28:58 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 206 status calls. 00000d93.000002bf 243bbdd2 Thu Sep 26 2019 04:28:59 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 207 status calls. 00000d93.000002c0 243bbddc Thu Sep 26 2019 04:29:00 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 208 status calls. 00000d93.000002c1 243bbde6 Thu Sep 26 2019 04:29:01 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 209 status calls. 00000d93.000002c2 243bbdfa Thu Sep 26 2019 04:29:02 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 210 status calls. 00000d93.000002c4 243bbdfa Thu Sep 26 2019 04:29:03 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 211 status calls. 00000d93.000002c5 243bbe04 Thu Sep 26 2019 04:29:04 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 212 status calls. 00000d93.000002c6 243bbe0e Thu Sep 26 2019 04:29:05 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 213 status calls. 00000d93.000002c7 243bbe18 Thu Sep 26 2019 04:29:06 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 214 status calls. 00000d93.000002c8 243bbe2c Thu Sep 26 2019 04:29:07 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 215 status calls. 00000d93.000002ca 243bbe2c Thu Sep 26 2019 04:29:08 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 216 status calls. 00000d93.000002cb 243bbe36 Thu Sep 26 2019 04:29:09 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 217 status calls. 00000d93.000002cc 243bbe40 Thu Sep 26 2019 04:29:10 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 218 status calls. 00000d93.000002cd 243bbe4a Thu Sep 26 2019 04:29:11 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 219 status calls. 00000d93.000002ce 243bbe5e Thu Sep 26 2019 04:29:12 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 220 status calls. 00000d93.000002d0 243bbe5e Thu Sep 26 2019 04:29:13 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 221 status calls. 00000d93.000002d1 243bbe68 Thu Sep 26 2019 04:29:14 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 222 status calls. 00000d93.000002d2 243bbe72 Thu Sep 26 2019 04:29:15 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 223 status calls. 00000d93.000002d3 243bbe7d Thu Sep 26 2019 04:29:16 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 224 status calls. 00000d93.000002d4 243bbe91 Thu Sep 26 2019 04:29:17 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 225 status calls. 00000d93.000002d6 243bbe91 Thu Sep 26 2019 04:29:18 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 226 status calls. 00000d93.000002d7 243bbe9b Thu Sep 26 2019 04:29:19 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 227 status calls. 00000d93.000002d8 243bbea5 Thu Sep 26 2019 04:29:20 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 228 status calls. 00000d93.000002d9 243bbeaf Thu Sep 26 2019 04:29:21 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 229 status calls. 00000d93.000002da 243bbec3 Thu Sep 26 2019 04:29:22 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 230 status calls. 00000d93.000002dc 243bbec3 Thu Sep 26 2019 04:29:23 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 231 status calls. 00000d93.000002dd 243bbecd Thu Sep 26 2019 04:29:24 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 232 status calls. 00000d93.000002de 243bbed7 Thu Sep 26 2019 04:29:25 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 233 status calls. 00000d93.000002df 243bbee1 Thu Sep 26 2019 04:29:26 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 234 status calls. 00000d93.000002e0 243bbef5 Thu Sep 26 2019 04:29:27 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 235 status calls. 00000d93.000002e2 243bbef5 Thu Sep 26 2019 04:29:28 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 236 status calls. 00000d93.000002e3 243bbeff Thu Sep 26 2019 04:29:29 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 237 status calls. 00000d93.000002e4 243bbf09 Thu Sep 26 2019 04:29:30 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 238 status calls. 00000d93.000002e5 243bbf13 Thu Sep 26 2019 04:29:31 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 239 status calls. 00000d93.000002e6 243bbf27 Thu Sep 26 2019 04:29:32 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 240 status calls. 00000d93.000002e8 243bbf27 Thu Sep 26 2019 04:29:33 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 241 status calls. 00000d93.000002e9 243bbf31 Thu Sep 26 2019 04:29:34 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 242 status calls. 00000d93.000002ea 243bbf3b Thu Sep 26 2019 04:29:35 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 243 status calls. 00000d93.000002eb 243bbf45 Thu Sep 26 2019 04:29:36 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 244 status calls. 00000d93.000002ec 243bbf59 Thu Sep 26 2019 04:29:37 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 245 status calls. 00000d93.000002ee 243bbf59 Thu Sep 26 2019 04:29:38 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 246 status calls. 00000d93.000002ef 243bbf63 Thu Sep 26 2019 04:29:39 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 247 status calls. 00000d93.000002f0 243bbf6d Thu Sep 26 2019 04:29:40 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 248 status calls. 00000d93.000002f1 243bbf77 Thu Sep 26 2019 04:29:41 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 249 status calls. 00000d93.000002f2 243bbf8b Thu Sep 26 2019 04:29:42 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 250 status calls. 00000d93.000002f4 243bbf8b Thu Sep 26 2019 04:29:43 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 251 status calls. 00000d93.000002f5 243bbf95 Thu Sep 26 2019 04:29:44 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 252 status calls. 00000d93.000002f6 243bbf9f Thu Sep 26 2019 04:29:45 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 253 status calls. 00000d93.000002f7 243bbfa9 Thu Sep 26 2019 04:29:46 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 254 status calls. 00000d93.000002f8 243bbfbd Thu Sep 26 2019 04:29:47 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 255 status calls. 00000d93.000002fa 243bbfbd Thu Sep 26 2019 04:29:48 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 256 status calls. 00000d93.000002fb 243bbfc7 Thu Sep 26 2019 04:29:49 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 257 status calls. 00000d93.000002fc 243bbfd1 Thu Sep 26 2019 04:29:50 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 258 status calls. 00000d93.000002fd 243bbfdb Thu Sep 26 2019 04:29:51 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 259 status calls. 00000d93.000002fe 243bbfef Thu Sep 26 2019 04:29:52 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 260 status calls. 00000d93.00000300 243bbfef Thu Sep 26 2019 04:29:53 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 261 status calls. 00000d93.00000301 243bbff9 Thu Sep 26 2019 04:29:54 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 262 status calls. 00000d93.00000302 243bc003 Thu Sep 26 2019 04:29:55 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 263 status calls. 00000d93.00000303 243bc00d Thu Sep 26 2019 04:29:56 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 264 status calls. 00000d93.00000304 243bc021 Thu Sep 26 2019 04:29:57 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 265 status calls. 00000d93.00000306 243bc021 Thu Sep 26 2019 04:29:58 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 266 status calls. 00000d93.00000307 243bc02b Thu Sep 26 2019 04:29:59 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 267 status calls. 00000d93.00000308 243bc035 Thu Sep 26 2019 04:30:00 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 268 status calls. 00000d93.00000309 243bc03f Thu Sep 26 2019 04:30:01 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 269 status calls. 00000d93.0000030a 243bc053 Thu Sep 26 2019 04:30:02 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 270 status calls. 00000d93.0000030c 243bc053 Thu Sep 26 2019 04:30:03 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 271 status calls. 00000d93.0000030d 243bc05d Thu Sep 26 2019 04:30:04 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 272 status calls. 00000d93.0000030e 243bc067 Thu Sep 26 2019 04:30:05 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 273 status calls. 00000d93.0000030f 243bc071 Thu Sep 26 2019 04:30:06 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 274 status calls. 00000d93.00000310 243bc085 Thu Sep 26 2019 04:30:07 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 275 status calls. 00000d93.00000312 243bc085 Thu Sep 26 2019 04:30:08 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 276 status calls. 00000d93.00000313 243bc08f Thu Sep 26 2019 04:30:09 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 277 status calls. 00000d93.00000314 243bc099 Thu Sep 26 2019 04:30:10 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 278 status calls. 00000d93.00000315 243bc0a3 Thu Sep 26 2019 04:30:11 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 279 status calls. 00000d93.00000316 243bc0b7 Thu Sep 26 2019 04:30:12 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 280 status calls. 00000d93.00000318 243bc0b7 Thu Sep 26 2019 04:30:13 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 281 status calls. 00000d93.00000319 243bc0c1 Thu Sep 26 2019 04:30:14 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 282 status calls. 00000d93.0000031a 243bc0cb Thu Sep 26 2019 04:30:15 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 283 status calls. 00000d93.0000031b 243bc0d5 Thu Sep 26 2019 04:30:16 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 284 status calls. 00000d93.0000031c 243bc0e9 Thu Sep 26 2019 04:30:17 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 285 status calls. 00000d93.0000031e 243bc0e9 Thu Sep 26 2019 04:30:18 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 286 status calls. 00000d93.0000031f 243bc0f3 Thu Sep 26 2019 04:30:19 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 287 status calls. 00000d93.00000320 243bc0fd Thu Sep 26 2019 04:30:20 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 288 status calls. 00000d93.00000321 243bc107 Thu Sep 26 2019 04:30:21 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 289 status calls. 00000d93.00000322 243bc11b Thu Sep 26 2019 04:30:22 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 290 status calls. 00000d93.00000324 243bc11b Thu Sep 26 2019 04:30:23 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 291 status calls. 00000d93.00000325 243bc125 Thu Sep 26 2019 04:30:24 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 292 status calls. 00000d93.00000326 243bc12f Thu Sep 26 2019 04:30:25 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 293 status calls. 00000d93.00000327 243bc139 Thu Sep 26 2019 04:30:26 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 294 status calls. 00000d93.00000328 243bc14d Thu Sep 26 2019 04:30:27 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 295 status calls. 00000d93.0000032a 243bc14d Thu Sep 26 2019 04:30:28 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 296 status calls. 00000d93.0000032b 243bc157 Thu Sep 26 2019 04:30:29 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 297 status calls. 00000d93.0000032c 243bc161 Thu Sep 26 2019 04:30:30 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 298 status calls. 00000d93.0000032d 243bc16b Thu Sep 26 2019 04:30:31 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 299 status calls. 00000d93.0000032e 243bc17f Thu Sep 26 2019 04:30:32 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 300 status calls. 00000d93.00000330 243bc17f Thu Sep 26 2019 04:30:33 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 301 status calls. 00000d93.00000331 243bc189 Thu Sep 26 2019 04:30:34 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 302 status calls. 00000d93.00000332 243bc193 Thu Sep 26 2019 04:30:35 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 303 status calls. 00000d93.00000333 243bc19d Thu Sep 26 2019 04:30:36 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 304 status calls. 00000d93.00000334 243bc1b1 Thu Sep 26 2019 04:30:37 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 305 status calls. 00000d93.00000336 243bc1b1 Thu Sep 26 2019 04:30:38 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 306 status calls. 00000d93.00000337 243bc1bb Thu Sep 26 2019 04:30:39 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 307 status calls. 00000d93.00000338 243bc1c5 Thu Sep 26 2019 04:30:40 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 308 status calls. 00000d93.00000339 243bc1cf Thu Sep 26 2019 04:30:41 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 309 status calls. 00000d93.0000033a 243bc1e3 Thu Sep 26 2019 04:30:42 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 310 status calls. 00000d93.0000033c 243bc1e3 Thu Sep 26 2019 04:30:43 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 311 status calls. 00000d93.0000033d 243bc1ed Thu Sep 26 2019 04:30:44 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 312 status calls. 00000d93.0000033e 243bc1f7 Thu Sep 26 2019 04:30:45 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 313 status calls. 00000d93.0000033f 243bc201 Thu Sep 26 2019 04:30:46 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 314 status calls. 00000d93.00000340 243bc215 Thu Sep 26 2019 04:30:47 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 315 status calls. 00000d93.00000342 243bc215 Thu Sep 26 2019 04:30:48 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 316 status calls. 00000d93.00000343 243bc21f Thu Sep 26 2019 04:30:49 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 317 status calls. 00000d93.00000344 243bc229 Thu Sep 26 2019 04:30:50 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 318 status calls. 00000d93.00000345 243bc233 Thu Sep 26 2019 04:30:51 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 319 status calls. 00000d93.00000346 243bc247 Thu Sep 26 2019 04:30:52 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 320 status calls. 00000d93.00000348 243bc247 Thu Sep 26 2019 04:30:53 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 321 status calls. 00000d93.00000349 243bc251 Thu Sep 26 2019 04:30:54 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 322 status calls. 00000d93.0000034a 243bc25b Thu Sep 26 2019 04:30:55 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 323 status calls. 00000d93.0000034b 243bc265 Thu Sep 26 2019 04:30:56 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 324 status calls. 00000d93.0000034c 243bc27a Thu Sep 26 2019 04:30:57 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 325 status calls. 00000d93.0000034e 243bc27a Thu Sep 26 2019 04:30:58 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 326 status calls. 00000d93.0000034f 243bc284 Thu Sep 26 2019 04:30:59 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 327 status calls. 00000d93.00000350 243bc28e Thu Sep 26 2019 04:31:00 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 328 status calls. 00000d93.00000351 243bc298 Thu Sep 26 2019 04:31:01 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 329 status calls. 00000d93.00000352 243bc2ac Thu Sep 26 2019 04:31:02 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 330 status calls. 00000d93.00000354 243bc2ac Thu Sep 26 2019 04:31:03 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 331 status calls. 00000d93.00000355 243bc2b6 Thu Sep 26 2019 04:31:04 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 332 status calls. 00000d93.00000356 243bc2c0 Thu Sep 26 2019 04:31:05 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 333 status calls. 00000d93.00000357 243bc2ca Thu Sep 26 2019 04:31:06 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 334 status calls. 00000d93.00000358 243bc2de Thu Sep 26 2019 04:31:07 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 335 status calls. 00000d93.0000035a 243bc2de Thu Sep 26 2019 04:31:08 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 336 status calls. 00000d93.0000035b 243bc2e8 Thu Sep 26 2019 04:31:09 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 337 status calls. 00000d93.0000035c 243bc2f2 Thu Sep 26 2019 04:31:10 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 338 status calls. 00000d93.0000035d 243bc2fc Thu Sep 26 2019 04:31:11 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 339 status calls. 00000d93.0000035e 243bc310 Thu Sep 26 2019 04:31:12 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 340 status calls. 00000d93.00000360 243bc310 Thu Sep 26 2019 04:31:13 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 341 status calls. 00000d93.00000361 243bc31a Thu Sep 26 2019 04:31:14 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 342 status calls. 00000d93.00000362 243bc324 Thu Sep 26 2019 04:31:15 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 343 status calls. 00000d93.00000363 243bc32e Thu Sep 26 2019 04:31:16 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 344 status calls. 00000d93.00000364 243bc342 Thu Sep 26 2019 04:31:17 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 345 status calls. 00000d93.00000366 243bc342 Thu Sep 26 2019 04:31:18 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 346 status calls. 00000d93.00000367 243bc34c Thu Sep 26 2019 04:31:19 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 347 status calls. 00000d93.00000368 243bc356 Thu Sep 26 2019 04:31:20 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 348 status calls. 00000d93.00000369 243bc360 Thu Sep 26 2019 04:31:21 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 349 status calls. 00000d93.0000036a 243bc374 Thu Sep 26 2019 04:31:22 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 350 status calls. 00000d93.0000036c 243bc374 Thu Sep 26 2019 04:31:23 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 351 status calls. 00000d93.0000036d 243bc37e Thu Sep 26 2019 04:31:24 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 352 status calls. 00000d93.0000036e 243bc388 Thu Sep 26 2019 04:31:25 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 353 status calls. 00000d93.0000036f 243bc392 Thu Sep 26 2019 04:31:26 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 354 status calls. 00000d93.00000370 243bc3a6 Thu Sep 26 2019 04:31:27 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 355 status calls. 00000d93.00000372 243bc3a6 Thu Sep 26 2019 04:31:28 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 356 status calls. 00000d93.00000373 243bc3b0 Thu Sep 26 2019 04:31:29 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 357 status calls. 00000d93.00000374 243bc3ba Thu Sep 26 2019 04:31:30 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 358 status calls. 00000d93.00000375 243bc3c4 Thu Sep 26 2019 04:31:31 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 359 status calls. 00000d93.00000376 243bc3d8 Thu Sep 26 2019 04:31:32 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 360 status calls. 00000d93.00000378 243bc3d8 Thu Sep 26 2019 04:31:33 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 361 status calls. 00000d93.00000379 243bc3e2 Thu Sep 26 2019 04:31:34 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 16 boot level is 1 after 362 status calls. 00000d93.0000037a 243bc40b Thu Sep 26 2019 04:31:35 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggr: boot state is 256 boot level is 1 after 363 status calls. 00000d93.0000037b 243bc40b Thu Sep 26 2019 04:31:35 -06:00 [kern_mgwd:info:2284] mhost_boot: ksmf boot state end BootLevel::_waiting 00000d93.0000037c 243bc40b Thu Sep 26 2019 04:31:35 -06:00 [kern_mgwd:info:2284] mhost_boot: ksmf root aggr start BootLevel::_waiting 00000d93.0000037d 243bc40b Thu Sep 26 2019 04:31:35 -06:00 [kern_mgwd:info:2284] mhost_boot: scan_local_netports start 00000d93.0000037e 243bc40b Thu Sep 26 2019 04:31:35 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: NOTICE: RpcConnectionCache: markFailure: markFailure: New Quarantine for 127.0.0.1:536873492(vifmgr)/1[tcp]{fffffffd|8} with timeout(5.000s) Reason:: RPC: Program not registered 00000d93.0000037f 243bc40b Thu Sep 26 2019 04:31:35 -06:00 [kern_mgwd:info:2284] [0x81a804200] [SkStackMgr::registerPorts] Found nblade port with name e0M, assigning ID 0 00000d93.00000380 243bc40b Thu Sep 26 2019 04:31:35 -06:00 [kern_mgwd:info:2284] [0x81a804200] [SkStackMgr::registerPorts] Found nblade port with name e0a, assigning ID 1 00000d93.00000381 243bc40b Thu Sep 26 2019 04:31:36 -06:00 [kern_mgwd:info:2284] [0x81a804200] [SkStackMgr::registerPorts] Found nblade port with name e0b, assigning ID 2 00000d93.00000382 243bc40b Thu Sep 26 2019 04:31:36 -06:00 [kern_mgwd:info:2284] [0x81a804200] [SkStackMgr::registerPorts] Found nblade port with name e0c, assigning ID 3 00000d93.00000383 243bc40b Thu Sep 26 2019 04:31:36 -06:00 [kern_mgwd:info:2284] [0x81a804200] [SkStackMgr::registerPorts] Found nblade port with name e0d, assigning ID 4 00000d93.00000384 243bc40b Thu Sep 26 2019 04:31:36 -06:00 [kern_mgwd:info:2284] [0x81a804200] [SkStackMgr::registerPorts] Found nblade port with name e0e, assigning ID 5 00000d93.00000385 243bc40b Thu Sep 26 2019 04:31:36 -06:00 [kern_mgwd:info:2284] [0x81a804200] [SkStackMgr::registerPorts] Found nblade port with name e0f, assigning ID 6 00000d93.00000386 243bc40b Thu Sep 26 2019 04:31:36 -06:00 [kern_mgwd:info:2284] [0x81a804200] [SkStackMgr::registerPorts] Found nblade port with name e0g, assigning ID 7 00000d93.00000387 243bc40b Thu Sep 26 2019 04:31:37 -06:00 [kern_mgwd:info:2284] [0x81a804200] [SkStackMgr::registerPorts] Found nblade port with name e0h, assigning ID 8 00000d93.00000388 243bc40b Thu Sep 26 2019 04:31:37 -06:00 [kern_mgwd:info:2284] [0x81a804200] [SkStackMgr::registerPorts] Found nblade port with name e0i, assigning ID 9 00000d93.00000389 243bc40b Thu Sep 26 2019 04:31:37 -06:00 [kern_mgwd:info:2284] [0x81a804200] [SkStackMgr::registerPorts] Found nblade port with name e0j, assigning ID 10 00000d93.0000038a 243bc40b Thu Sep 26 2019 04:31:37 -06:00 [kern_mgwd:info:2284] [0x81a804200] [SkStackMgr::registerPorts] Found nblade port with name e0k, assigning ID 11 00000d93.0000038b 243bc40b Thu Sep 26 2019 04:31:37 -06:00 [kern_mgwd:info:2284] [0x81a804200] [SkStackMgr::registerPorts] Found nblade port with name e0l, assigning ID 12 00000d93.0000038c 243bc40b Thu Sep 26 2019 04:31:37 -06:00 [kern_mgwd:info:2284] [0x81a804200] [SkStackMgr::registerPorts] Found nblade port with name e3a, assigning ID 13 00000d93.0000038d 243bc40b Thu Sep 26 2019 04:31:37 -06:00 [kern_mgwd:info:2284] [0x81a804200] [SkStackMgr::registerPorts] Found nblade port with name e3b, assigning ID 14 00000d93.0000038e 243bc40b Thu Sep 26 2019 04:31:37 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: NOTICE: net_ports: update_operational_fields:src/tables/net_ports.cc:497 installing new/updated mac based on driver state: mac=00:a0:98:57:30:66 00000d93.0000038f 243bc40b Thu Sep 26 2019 04:31:37 -06:00 [kern_mgwd:info:2284] Scan local netports got port e0M with link 3. 00000d93.000003b7 243bc40b Thu Sep 26 2019 04:31:38 -06:00 [kern_mgwd:info:2284] mhost_boot: ems_eventCatalogDB: loading start 00000d93.0000049a 243bc432 Thu Sep 26 2019 04:31:39 -06:00 [kern_mgwd:info:2284] mhost_boot: ems_eventCatalogDB: loading end 00000d93.0000052d 243bc432 Thu Sep 26 2019 04:31:41 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: NOTICE: net_ports: update_operational_fields:src/tables/net_ports.cc:497 installing new/updated mac based on driver state: mac=00:a0:98:57:30:52 00000d93.0000052e 243bc432 Thu Sep 26 2019 04:31:41 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: NOTICE: net_ports: update_operational_fields:src/tables/net_ports.cc:497 installing new/updated mac based on driver state: mac=00:a0:98:57:30:53 00000d93.0000052f 243bc432 Thu Sep 26 2019 04:31:41 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: NOTICE: net_ports: update_operational_fields:src/tables/net_ports.cc:497 installing new/updated mac based on driver state: mac=00:a0:98:57:30:54 00000d93.00000530 243bc432 Thu Sep 26 2019 04:31:41 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: NOTICE: net_ports: update_operational_fields:src/tables/net_ports.cc:497 installing new/updated mac based on driver state: mac=00:a0:98:57:30:55 00000d93.00000531 243bc432 Thu Sep 26 2019 04:31:41 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: NOTICE: net_ports: update_operational_fields:src/tables/net_ports.cc:497 installing new/updated mac based on driver state: mac=00:a0:98:57:30:56 00000d93.00000532 243bc432 Thu Sep 26 2019 04:31:41 -06:00 [kern_mgwd:info:2284] Scan local netports got port e0e with link 3. 00000d93.00000548 243bc48d Thu Sep 26 2019 04:31:51 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: NOTICE: net_ports: update_operational_fields:src/tables/net_ports.cc:497 installing new/updated mac based on driver state: mac=00:a0:98:57:30:57 00000d93.00000549 243bc48d Thu Sep 26 2019 04:31:51 -06:00 [kern_mgwd:info:2284] Scan local netports got port e0f with link 3. 00000d93.00000551 243bc4f1 Thu Sep 26 2019 04:32:01 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: NOTICE: net_ports: update_operational_fields:src/tables/net_ports.cc:497 installing new/updated mac based on driver state: mac=00:a0:98:57:30:58 00000d93.00000552 243bc4f1 Thu Sep 26 2019 04:32:01 -06:00 [kern_mgwd:info:2284] Scan local netports got port e0g with link 3. 00000d93.00000553 243bc555 Thu Sep 26 2019 04:32:11 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: NOTICE: net_ports: update_operational_fields:src/tables/net_ports.cc:497 installing new/updated mac based on driver state: mac=00:a0:98:57:30:59 00000d93.00000554 243bc555 Thu Sep 26 2019 04:32:11 -06:00 [kern_mgwd:info:2284] Scan local netports got port e0h with link 3. 00000d93.00000563 243bc57c Thu Sep 26 2019 04:32:15 -06:00 [kern_mgwd:info:2284] rdb_tran: set mtrace buffer size to 262400 bytes 00000d93.00000564 243bc57c Thu Sep 26 2019 04:32:15 -06:00 [kern_mgwd:info:2284] 0x81b244200: 0: ERR: rdb_tran_glue: create: tid=0x81b244200 failed to create transaction for label='SP: rdb-info-update lock': NULL runit::ptr 00000d93.00000565 243bc57c Thu Sep 26 2019 04:32:15 -06:00 [kern_mgwd:info:2284] 0x81b244200: 0: ERR: rdb_tran_glue: abort: tid=0x81b244200 missing thread context 00000d93.00000566 243bc57c Thu Sep 26 2019 04:32:15 -06:00 [kern_mgwd:info:2284] 0x81b244200: 0: ERR: MGWD::sp_netcfg: spcli_update_rdb: failed to obtain SP RDB transaction 00000d93.0000058a 243bc5ba Thu Sep 26 2019 04:32:21 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: NOTICE: net_ports: update_operational_fields:src/tables/net_ports.cc:497 installing new/updated mac based on driver state: mac=00:a0:98:57:30:62 00000d93.0000058b 243bc5ba Thu Sep 26 2019 04:32:21 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: NOTICE: net_ports: update_operational_fields:src/tables/net_ports.cc:497 installing new/updated mac based on driver state: mac=00:a0:98:57:30:63 00000d93.0000058c 243bc5ba Thu Sep 26 2019 04:32:21 -06:00 [kern_mgwd:info:2284] Scan local netports got port e0j with link 3. 00000d93.00000590 243bc61e Thu Sep 26 2019 04:32:31 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: NOTICE: net_ports: update_operational_fields:src/tables/net_ports.cc:497 installing new/updated mac based on driver state: mac=00:a0:98:57:30:64 00000d93.00000591 243bc61e Thu Sep 26 2019 04:32:31 -06:00 [kern_mgwd:info:2284] Scan local netports got port e0k with link 3. 00000d93.000005a2 243bc645 Thu Sep 26 2019 04:32:35 -06:00 [kern_mgwd:info:2284] 0x81b244c00: 0: ERR: MGWD::sp_mgmt: 00000d93.000005a3 243bc645 Thu Sep 26 2019 04:32:35 -06:00 [kern_mgwd:info:2284] PresentSystemAurora = 2 00000d93.000005ab 243bc645 Thu Sep 26 2019 04:32:35 -06:00 [kern_mgwd:info:2284] 0x81b244c00: 0: NOTICE: net_ports: update_operational_fields:src/tables/net_ports.cc:497 installing new/updated mac based on driver state: mac=00:a0:98:57:30:65 00000d93.000005ac 243bc645 Thu Sep 26 2019 04:32:35 -06:00 [kern_mgwd:info:2284] 0x81b244c00: 0: NOTICE: net_ports: update_operational_fields:src/tables/net_ports.cc:497 installing new/updated mac based on driver state: mac=90:e2:ba:2e:3d:ec 00000d93.000005ad 243bc645 Thu Sep 26 2019 04:32:35 -06:00 [kern_mgwd:info:2284] 0x81b244c00: 0: NOTICE: net_ports: update_operational_fields:src/tables/net_ports.cc:497 installing new/updated mac based on driver state: mac=90:e2:ba:2e:3d:ed 00000d93.000005ae 243bc645 Thu Sep 26 2019 04:32:35 -06:00 [kern_mgwd:info:2284] 0x81b245100: 0: NOTICE: RpcConnectionCache: markFailure: markFailure: New Quarantine for 127.0.0.1:536873514(cm)/1[tcp]{fffffffd|8} with timeout(5.000s) Reason:: RPC: Program not registered 00000d93.000005b6 243bc645 Thu Sep 26 2019 04:32:35 -06:00 [kern_mgwd:info:2284] 0x81b7bc900: 0: ERR: RpcConnectionCache: dup_suppress_log: Excessive Duplicate calls to doClntCreateTimed... 00000d93.000005bb 243bc645 Thu Sep 26 2019 04:32:35 -06:00 [kern_mgwd:info:2284] 0x81b7bc900: 0: NOTICE: net_ports: update_operational_fields:src/tables/net_ports.cc:497 installing new/updated mac based on driver state: mac=00:a0:98:57:30:65 00000d93.000005bc 243bc645 Thu Sep 26 2019 04:32:35 -06:00 [kern_mgwd:info:2284] 0x81b7bc900: 0: NOTICE: net_ports: update_operational_fields:src/tables/net_ports.cc:497 installing new/updated mac based on driver state: mac=90:e2:ba:2e:3d:ec 00000d93.000005bd 243bc645 Thu Sep 26 2019 04:32:35 -06:00 [kern_mgwd:info:2284] 0x81b7bc900: 0: NOTICE: net_ports: update_operational_fields:src/tables/net_ports.cc:497 installing new/updated mac based on driver state: mac=90:e2:ba:2e:3d:ed 00000d93.000005d6 243bc683 Thu Sep 26 2019 04:32:41 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: NOTICE: net_ports: update_operational_fields:src/tables/net_ports.cc:497 installing new/updated mac based on driver state: mac=00:a0:98:57:30:65 00000d93.000005d7 243bc683 Thu Sep 26 2019 04:32:41 -06:00 [kern_mgwd:info:2284] Scan local netports got port e0l with link 3. 00000d93.000005e2 243bc6ec Thu Sep 26 2019 04:32:51 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: NOTICE: net_ports: update_operational_fields:src/tables/net_ports.cc:497 installing new/updated mac based on driver state: mac=90:e2:ba:2e:3d:ec 00000d93.000005e3 243bc6ec Thu Sep 26 2019 04:32:51 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: NOTICE: net_ports: update_operational_fields:src/tables/net_ports.cc:497 installing new/updated mac based on driver state: mac=90:e2:ba:2e:3d:ed 00000d93.000005e4 243bc6ec Thu Sep 26 2019 04:32:52 -06:00 [kern_mgwd:info:2284] mgwd_cdb_netports: burt852016: Get on unset field aggr-node='nmg-fas8040-02 00000d93.000005e5 243bc6ec Thu Sep 26 2019 04:32:52 -06:00 [kern_mgwd:info:2284] Row: nmg-fas8040-02 a0a data - 1500 true - auto - auto - - true if-group - - ip multimode_lacp - - - - full - 4294967295 auto 00000d93.000005e6 243bc6ec Thu Sep 26 2019 04:32:52 -06:00 [kern_mgwd:info:2284] Backtrace: 00000d93.000005e7 243bc6ec Thu Sep 26 2019 04:32:52 -06:00 [kern_mgwd:info:2284] 0x4f7062: mgwd_cdb_netports_base_iterator::get_aggr_node() 00000d93.000005e8 243bc6ec Thu Sep 26 2019 04:32:52 -06:00 [kern_mgwd:info:2284] 0x4f6399: smdb_error::~smdb_error()+7049 00000d93.000005e9 243bc6ec Thu Sep 26 2019 04:32:52 -06:00 [kern_mgwd:info:2284] 0x4f6c1b: smdb_error::~smdb_error()+9227 00000d93.000005ea 243bc6ec Thu Sep 26 2019 04:32:52 -06:00 [kern_mgwd:info:2284] 0x50dca1: std::_Rb_tree, std::_Select1st >, std::less, std::allocator > >::_M_erase(std::_Rb_tree_node >*)+1585 00000d93.000005eb 243bc6ec Thu Sep 26 2019 04:32:52 -06:00 [kern_mgwd:info:2284] 0x8158ba3c7: operStartup::operator()(startupTask*)+359 00000d93.000005ec 243bc6ec Thu Sep 26 2019 04:32:52 -06:00 [kern_mgwd:info:2284] 0x8158b18af: void mhostsvc_dependencies::TTaskDependencies::execute(operStartup&, bool)+223 00000d93.000005ed 243bc6ec Thu Sep 26 2019 04:32:52 -06:00 [kern_mgwd:info:2284] 0x8158b1317: mhostsvc_dependencies::StartupTaskDependenciesImpl::executeStartupSerial()+23 00000d93.000005ee 243bc6ec Thu Sep 26 2019 04:32:52 -06:00 [kern_mgwd:info:2284] 0x504ef8: smdb_error::~smdb_error()+67304 00000d93.000005ef 243bc6ec Thu Sep 26 2019 04:32:52 -06:00 [kern_mgwd:info:2284] 0x4f433e: info.dli_sname NULL+5194558 00000d93.000005f0 243bc6ec Thu Sep 26 2019 04:32:52 -06:00 [kern_mgwd:info:2284] 0x800601000 00000d93.000005f1 243bc6ec Thu Sep 26 2019 04:32:52 -06:00 [kern_mgwd:info:2284] mhost_boot: scan_local_netports end 00000d93.000005f2 243bc6ec Thu Sep 26 2019 04:32:52 -06:00 [kern_mgwd:info:2284] mhost_boot: ksmf root aggr end BootLevel::_normal 00000d93.000005f3 243bc6ec Thu Sep 26 2019 04:32:52 -06:00 [kern_mgwd:info:2284] wait_for_dblade_root_aggrs: Waiting for root aggr false, failover state is 00000d93.000005f4 243bc6ec Thu Sep 26 2019 04:32:52 -06:00 [kern_mgwd:info:2284] mhost_boot: StartTask: mainDBladeTask end 00000d93.000005f5 243bc6ec Thu Sep 26 2019 04:32:52 -06:00 [kern_mgwd:info:2284] mhost_boot: StartTask: mainPostDBladeTask start 00000d93.000005f6 243bc6ec Thu Sep 26 2019 04:32:52 -06:00 [kern_mgwd:info:2284] mhost_boot: arp config CDB start 00000d93.000005f7 243bc6ec Thu Sep 26 2019 04:32:52 -06:00 [kern_mgwd:info:2284] mhost_boot: arp config CDB end 00000d93.000005f8 243bc6ec Thu Sep 26 2019 04:32:52 -06:00 [kern_mgwd:info:2284] mhost_boot: cluster interface NVRAM start 00000d93.000005f9 243bc6fc Thu Sep 26 2019 04:32:54 -06:00 [kern_mgwd:info:2284] [0x81a804200] [boot_config::addLocalInterface] Hosted CDB LIF: nmg-fas8040-02_clus1 (1007), address: 169.254.5.219, node: , port: e0a 00000d93.000005fa 243bc70c Thu Sep 26 2019 04:32:55 -06:00 [kern_mgwd:info:2284] [0x81a804200] [boot_config::addLocalInterface] Hosted CDB LIF: nmg-fas8040-02_clus2 (1008), address: 169.254.8.83, node: , port: e0b 00000d93.000005fb 243bc71b Thu Sep 26 2019 04:32:57 -06:00 [kern_mgwd:info:2284] [0x81a804200] [boot_config::addLocalInterface] Hosted CDB LIF: nmg-fas8040-02_clus3 (1009), address: 169.254.106.161, node: , port: e0c 00000d93.000005fc 243bc72b Thu Sep 26 2019 04:32:58 -06:00 [kern_mgwd:info:2284] [0x81a804200] [boot_config::addLocalInterface] Hosted CDB LIF: nmg-fas8040-02_clus4 (1010), address: 169.254.196.32, node: , port: e0d 00000d93.00000604 243bc77e Thu Sep 26 2019 04:33:07 -06:00 [kern_mgwd:info:2284] 0x81b245100: 0: NOTICE: net_ports: update_operational_fields:src/tables/net_ports.cc:497 installing new/updated mac based on driver state: mac=02:a0:98:57:30:67 00000d93.00000605 243bc77e Thu Sep 26 2019 04:33:07 -06:00 [kern_mgwd:info:2284] 0x81b7bce00: 0: NOTICE: net_ports: update_operational_fields:src/tables/net_ports.cc:497 installing new/updated mac based on driver state: mac=02:a0:98:57:30:67 00000d93.00000606 243bc77e Thu Sep 26 2019 04:33:07 -06:00 [kern_mgwd:info:2284] 0x81b7bce00: 0: ERR: RpcConnectionCache: dup_suppress_log: Excessive Duplicate calls to doClntCreateTimed... 00000d93.00000607 243bc79e Thu Sep 26 2019 04:33:08 -06:00 [kern_mgwd:info:2284] [0x81a804200] [boot_config::addLocalInterface] Hosted CDB LIF: nmg-fas8040-02_mgmt1 (1011), address: 10.0.128.182, node: , port: e0M 00000d93.00000608 243bc79e Thu Sep 26 2019 04:33:08 -06:00 [kern_mgwd:info:2284] mhost_boot: cluster interface NVRAM end 00000d93.00000609 243bc79e Thu Sep 26 2019 04:33:08 -06:00 [kern_mgwd:info:2284] mhost_boot: volume list zapi start 00000d93.0000060a 243bc79e Thu Sep 26 2019 04:33:08 -06:00 [kern_mgwd:info:2284] mhost_boot: volume list zapi end 00000d93.0000060b 243bc79e Thu Sep 26 2019 04:33:08 -06:00 [kern_mgwd:info:2284] mhost_boot: upgrade startup start 00000d93.0000060c 243bc79e Thu Sep 26 2019 04:33:08 -06:00 [kern_mgwd:info:2284] mhost_boot: upgrade startup end 00000d93.0000060d 243bc79e Thu Sep 26 2019 04:33:08 -06:00 [kern_mgwd:info:2284] mgwd RSS Limit=681574400 Bytes 00000d93.0000060e 243bc79e Thu Sep 26 2019 04:33:08 -06:00 [kern_mgwd:info:2284] mgwd MEMLOCK Limit=681574400 Bytes 00000d93.0000060f 243bc79e Thu Sep 26 2019 04:33:08 -06:00 [kern_mgwd:info:2284] mhost_boot: StartTask: mainPostDBladeTask end 00000d93.00000610 243bc79e Thu Sep 26 2019 04:33:08 -06:00 [kern_mgwd:info:2284] mhost_boot: StartTask: mainWaitForMroot start 00000d93.00000611 243bc79e Thu Sep 26 2019 04:33:08 -06:00 [kern_mgwd:info:2284] src/libnwd/unwd.cc:57 _call_sysctl : Wrote updated progress "mounting mroot start" to oid debug.nodewatchdog.progress.mgwd. 00000d93.00000612 243bc79e Thu Sep 26 2019 04:33:08 -06:00 [kern_mgwd:info:2284] mhost_boot: wait for mroot start 00000d93.00000613 243bc79e Thu Sep 26 2019 04:33:08 -06:00 [kern_mgwd:info:2284] wait_secs_mroot_mounted: Waited 0 seconds for /mroot. 00000d93.00000614 243bc79e Thu Sep 26 2019 04:33:08 -06:00 [kern_mgwd:info:2284] mhost_boot: wait for mroot end 00000d93.00000615 243bc79e Thu Sep 26 2019 04:33:08 -06:00 [kern_mgwd:info:2284] mhost_boot: StartTask: mainWaitForMroot end 00000d93.00000616 243bc79e Thu Sep 26 2019 04:33:08 -06:00 [kern_mgwd:info:2284] mhost_boot: StartTask: mainPostMroot start 00000d93.00000617 243bc79e Thu Sep 26 2019 04:33:08 -06:00 [kern_mgwd:info:2284] mhost_boot: check mroot for nvfail start 00000d93.00000618 243bc79e Thu Sep 26 2019 04:33:08 -06:00 [kern_mgwd:info:2284] mhost_boot: check mroot for nvfail end 00000d93.00000619 243bc79e Thu Sep 26 2019 04:33:08 -06:00 [kern_mgwd:info:2284] mhost_boot: set dhcp config start 00000d93.0000061a 243bc79e Thu Sep 26 2019 04:33:08 -06:00 [kern_mgwd:info:2284] mhost_boot: set dhcp config end 00000d93.0000061b 243bc79e Thu Sep 26 2019 04:33:08 -06:00 [kern_mgwd:info:2284] mhost_boot: precluster start 00000d93.0000061c 243bc79e Thu Sep 26 2019 04:33:08 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Starting module group: FILER_PRE_CLUSTER_APPS, sequence: smf_coredump_precluster_module 00000d93.0000061d 243bc79e Thu Sep 26 2019 04:33:08 -06:00 [kern_mgwd:info:2284] In coredump_precluster_module::module_init() 00000d93.0000061e 243bc79e Thu Sep 26 2019 04:33:08 -06:00 [kern_mgwd:info:2284] coredump_config: notify_imp, op=1 00000d93.0000061f 243bc79e Thu Sep 26 2019 04:33:08 -06:00 [kern_mgwd:info:2284] sparsecore enabled=^A 00000d93.00000620 243bc79e Thu Sep 26 2019 04:33:08 -06:00 [kern_mgwd:info:2284] min free=262144000 00000d93.00000621 243bc79e Thu Sep 26 2019 04:33:08 -06:00 [kern_mgwd:info:2284] coredump attempts=2 00000d93.00000622 243bc79e Thu Sep 26 2019 04:33:08 -06:00 [kern_mgwd:info:2284] save attempts=2 00000d93.00000623 243bc79e Thu Sep 26 2019 04:33:08 -06:00 [kern_mgwd:info:2284] save onstartup=^A 00000d93.00000624 243bc79e Thu Sep 26 2019 04:33:08 -06:00 [kern_mgwd:info:2284] upload location=ftp://ftp.netapp.com/to-ntap/ 00000d93.00000625 243bc79e Thu Sep 26 2019 04:33:08 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Finishing module group: FILER_PRE_CLUSTER_APPS, sequence: smf_coredump_precluster_module 00000d93.00000626 243bc79e Thu Sep 26 2019 04:33:08 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Starting module group: FILER_PRE_CLUSTER_APPS, sequence: smf_zapi_trace_level 00000d93.00000627 243bc79e Thu Sep 26 2019 04:33:08 -06:00 [kern_mgwd:info:2284] In zapi_trace_level_precluster_module::module_init() 00000d93.00000628 243bc79e Thu Sep 26 2019 04:33:08 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Finishing module group: FILER_PRE_CLUSTER_APPS, sequence: smf_zapi_trace_level 00000d93.00000629 243bc79e Thu Sep 26 2019 04:33:08 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Starting module group: FILER_PRE_CLUSTER_APPS, sequence: smf_system_health_registration_module 00000d93.0000062a 243bc79e Thu Sep 26 2019 04:33:08 -06:00 [kern_mgwd:info:2284] Info: check wait for RDB unit to be ONLINE 00000d93.0000062b 243bc79e Thu Sep 26 2019 04:33:08 -06:00 [kern_mgwd:info:2284] Info: HM asup thread create succeeded at mgwd init 00000d93.0000062c 243bc79e Thu Sep 26 2019 04:33:08 -06:00 [kern_mgwd:info:2284] Info: HM asup thread starting 00000d93.0000062d 243bc79e Thu Sep 26 2019 04:33:08 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Finishing module group: FILER_PRE_CLUSTER_APPS, sequence: smf_system_health_registration_module 00000d93.0000062e 243bc79e Thu Sep 26 2019 04:33:08 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Starting module group: FILER_PRE_CLUSTER_APPS, sequence: smf_dblade_users_refresh_registration_module 00000d93.0000062f 243bc79e Thu Sep 26 2019 04:33:08 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Finishing module group: FILER_PRE_CLUSTER_APPS, sequence: smf_dblade_users_refresh_registration_module 00000d93.00000630 243bc79e Thu Sep 26 2019 04:33:08 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Starting module group: FILER_PRE_CLUSTER_APPS, sequence: smf_config_backup_restore_pre_module 00000d93.00000631 243bc79e Thu Sep 26 2019 04:33:08 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Finishing module group: FILER_PRE_CLUSTER_APPS, sequence: smf_config_backup_restore_pre_module 00000d93.00000632 243bc79e Thu Sep 26 2019 04:33:08 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Exiting module startup for group: 9 00000d93.00000633 243bc79e Thu Sep 26 2019 04:33:08 -06:00 [kern_mgwd:info:2284] mhost_boot: precluster end 00000d93.00000634 243bc79e Thu Sep 26 2019 04:33:09 -06:00 [kern_mgwd:info:2284] src/libnwd/unwd.cc:57 _call_sysctl : Wrote updated progress "mroot phase1 start" to oid debug.nodewatchdog.progress.mgwd. 00000d93.00000635 243bc79e Thu Sep 26 2019 04:33:09 -06:00 [kern_mgwd:info:2284] mhost_boot: phase1 start 00000d93.00000636 243bc79e Thu Sep 26 2019 04:33:09 -06:00 [kern_mgwd:info:2284] mhost_boot: phase1 func ptrs start 00000d93.00000637 243bc79e Thu Sep 26 2019 04:33:09 -06:00 [kern_mgwd:info:2284] mhost_boot: phase1 func ptrs end 00000d93.00000638 243bc79e Thu Sep 26 2019 04:33:09 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Starting module group: LATENCY_STATS, sequence: smf_latency_stats 00000d93.00000639 243bc79e Thu Sep 26 2019 04:33:09 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Finishing module group: LATENCY_STATS, sequence: smf_latency_stats 00000d93.0000063a 243bc79e Thu Sep 26 2019 04:33:09 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Exiting module startup for group: 12 00000d93.0000063b 243bc79e Thu Sep 26 2019 04:33:09 -06:00 [kern_mgwd:info:2284] mhost_boot: phase1 jobmgr start 00000d93.0000063c 243bc79e Thu Sep 26 2019 04:33:09 -06:00 [kern_mgwd:info:2284] mhost_boot: phase1 jobmgr end 00000d93.0000063d 243bc79e Thu Sep 26 2019 04:33:09 -06:00 [kern_mgwd:info:2284] mhost_boot: phase1 cluster_setup start 00000d93.0000063e 243bc79e Thu Sep 26 2019 04:33:09 -06:00 [kern_mgwd:info:2284] mhost_boot: phase1 cluster_setup end 00000d93.0000063f 243bc79e Thu Sep 26 2019 04:33:09 -06:00 [kern_mgwd:info:2284] mhost_boot: mount mroot cluster_cache start 00000d93.00000640 243bc79e Thu Sep 26 2019 04:33:09 -06:00 [kern_mgwd:info:2284] mhost_boot: mount mroot cluster_cache end 00000d93.00000641 243bc79e Thu Sep 26 2019 04:33:09 -06:00 [kern_mgwd:info:2284] mhost_boot: phase1 mdb start 00000d93.00000642 243bc79e Thu Sep 26 2019 04:33:09 -06:00 [kern_mgwd:info:2284] src/libnwd/unwd.cc:57 _call_sysctl : Wrote updated progress "phase1 mdb start" to oid debug.nodewatchdog.progress.mgwd. 00000d93.00000643 243bc79e Thu Sep 26 2019 04:33:09 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Starting module group: MDB, sequence: smf_mdb 00000d93.00000644 243bc79e Thu Sep 26 2019 04:33:09 -06:00 [kern_mgwd:info:2284] MDB: module init 00000d93.00000645 243bc79e Thu Sep 26 2019 04:33:09 -06:00 [kern_mgwd:info:2284] mhost_boot: mdb_env::open start 00000d93.00000646 243bc79e Thu Sep 26 2019 04:33:09 -06:00 [kern_mgwd:info:2284] MDB: Opening databases. Cache size is 2097152 bytes. Prior recovery: no. Perform fatal recovery: no 00000d93.00000647 243bc79e Thu Sep 26 2019 04:33:09 -06:00 [kern_mgwd:info:2284] MDB: Data dir: /mroot/etc/cluster_config/mdb/mgwd. TMP dir: /mroot/etc/cluster_config/bdbtmp/mgwd. 00000d93.00000663 243bc79e Thu Sep 26 2019 04:33:09 -06:00 [kern_mgwd:info:2284] Starting a backup of the var filesystem. 00000d93.00000679 243bc7a9 Thu Sep 26 2019 04:33:11 -06:00 [kern_mgwd:info:2284] MDB: Opened 114 of 114 databases. 00000d93.0000067a 243bc7a9 Thu Sep 26 2019 04:33:11 -06:00 [kern_mgwd:info:2284] mhost_boot: mdb_env::open end okay 00000d93.0000067b 243bc7a9 Thu Sep 26 2019 04:33:11 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Finishing module group: MDB, sequence: smf_mdb 00000d93.0000067c 243bc7a9 Thu Sep 26 2019 04:33:11 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Starting module group: MDB, sequence: smf_coreseg_mdb_module 00000d93.0000067d 243bc7a9 Thu Sep 26 2019 04:33:11 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Finishing module group: MDB, sequence: smf_coreseg_mdb_module 00000d93.0000067e 243bc7a9 Thu Sep 26 2019 04:33:11 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Exiting module startup for group: 6 00000d93.0000067f 243bc7a9 Thu Sep 26 2019 04:33:11 -06:00 [kern_mgwd:info:2284] mhost_boot: phase1 mdb end 00000d93.00000680 243bc7a9 Thu Sep 26 2019 04:33:11 -06:00 [kern_mgwd:info:2284] mhost_boot: phase1 rdb start 00000d93.00000681 243bc7a9 Thu Sep 26 2019 04:33:11 -06:00 [kern_mgwd:info:2284] src/libnwd/unwd.cc:57 _call_sysctl : Wrote updated progress "phase1 rdb start" to oid debug.nodewatchdog.progress.mgwd. 00000d93.00000682 243bc7a9 Thu Sep 26 2019 04:33:11 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Starting module group: RDB, sequence: smf_rdb 00000d93.00000683 243bc7a9 Thu Sep 26 2019 04:33:11 -06:00 [kern_mgwd:info:2284] A [src/rdb/unit.cc 196 (0x81a804200)]: FAILOVER rdb: Opening local unit Management 00000d93.00000684 243bc7a9 Thu Sep 26 2019 04:33:11 -06:00 [kern_mgwd:info:2284] A [src/rdb/unit.cc 199 (0x81a804200)]: Opening unit Management (1). QM_FASTPATH true, QM_FAST_SECONDARY_OOQ false, DISABLE_MTRACE_DUMP false, flags 0x48. 00000d93.00000685 243bc7a9 Thu Sep 26 2019 04:33:11 -06:00 [user_mgwd:notice] SteadyUptime: ins/sec 2094998408); initialized at 60789290 (uptime at startup). 00000d93.00000686 243bc7a9 Thu Sep 26 2019 04:33:11 -06:00 [kern_mgwd:info:2284] A [src/rdb/cluster_events.cc 78 (0x81a804200)]: Cluster event: node-event, epoch 0, site 1001 [opening local replica]. 00000d93.00000687 243bc7b8 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] A [src/rdb/cluster_events.cc 78 (0x81a804200)]: Cluster event: node-event, epoch 5, site 1001 [opened local replica]. 00000d93.00000688 243bc7b8 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] A [src/rdb/SLM.cc 2638 (0x81a804200)]: Sitelist: startup: SLV <<1,2047>, 635fd3fb-b79d-11e7-9b07-00a09857ee8f> name epsilonId <0> attr <0x1> reserved <0> _building <0> _maxId <1001> numEligible <2> localIdLabel <1001> modifiable 0 modifyVersion <0,0> 00000d93.00000689 243bc7b8 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] *** Starting RDB Compression thread: thread 0x81cd1c800 (RDB:Compress) *** 00000d93.0000068a 243bc7b8 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] A [src/rdb/SLM.cc 2649 (0x81a804200)]: Sitelist: siteVec 1 of 2: 00000d93.0000068b 243bc7b8 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] A [src/rdb/SLM.cc 2649 (0x81a804200)]: Sitelist: siteVec 2 of 2: 00000d93.0000068c 243bc7b8 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] A [src/rdb/quorum/quorumimpl.cc 275 (0x81a804200)]: QuorumImpl::QuorumImpl _disableFastPathBootArg 0, _fastPathDefault 1, qVoteAvailAfter 60789294, fastSecondaryOOQ 0 00000d93.0000068d 243bc7b8 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] [src/rdb/quorum/dslqm.cc 113]: *** Starting QM worker thread: thread 0x81d246c00 (QM:QuorumImpl) *** 00000d93.0000068e 243bc7b8 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] A [src/rdb/quorum/qm_states/qm_out_initializing.cc 36 (0x81d246c00)]: InitializingState::state2: WS_Initializing -> WS_WaitingForStart 00000d93.0000068f 243bc7b8 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] A [src/rdb/quorum/qm_states/qm_state.cc 552 (0x81d246c00)]: stateTrans: WS_Initializing -> WS_WaitingForStart at: 60789752s 00000d93.00000690 243bc7b8 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] A [src/rdb/HAM.cc 807 (0x81a804200)]: Validating HA_ACTIVE state 00000d93.00000691 243bc7b8 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] A [src/rdb/HAM.cc 939 (0x81a804200)]: HAM 1001: Sitelist SLV <<1,2047>, 635fd3fb-b79d-11e7-9b07-00a09857ee8f> is HA-configuration involving local site. 00000d93.00000692 243bc7b8 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: INFO: librpcpool: bind_server: port=7815 VSID=-3 LIFROLE=-1 IPv6=no Listen=yes Realtime=no Added IPv4/IPv6 port=7815 vsid=-3 lifrole=-1 00000d93.00000693 243bc7b8 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: INFO: librpcpool: bind_server: port=7816 VSID=-3 LIFROLE=-1 IPv6=no Listen=yes Realtime=no Added IPv4/IPv6 port=7816 vsid=-3 lifrole=-1 00000d93.00000694 243bc7b8 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: INFO: librpcpool: bind_server: port=7817 VSID=-3 LIFROLE=-1 IPv6=no Listen=yes Realtime=no Added IPv4/IPv6 port=7817 vsid=-3 lifrole=-1 00000d93.00000695 243bc7b8 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: INFO: librpcpool: bind_server: port=7818 VSID=-3 LIFROLE=-1 IPv6=no Listen=yes Realtime=no Added IPv4/IPv6 port=7818 vsid=-3 lifrole=-1 00000d93.00000696 243bc7b8 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: INFO: librpcpool: bind_server: port=7819 VSID=-3 LIFROLE=-1 IPv6=no Listen=yes Realtime=yes Added IPv4/IPv6 port=7819 vsid=-3 lifrole=-1 00000d93.00000697 243bc7b8 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] A [src/rdb/quorum/qm_states/qm_out_waiting_for_start.cc 41 (0x81a804200)]: WaitingForStartState::state2: WS_WaitingForStart -> WS_WaitingForVotes 00000d93.00000698 243bc7b8 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] A [src/rdb/quorum/qm_states/qm_out_no_reply.cc 45 (0x81a804200)]: NoReplyState::stateUp2NoReply: WS_WaitingForStart -> WS_WaitingForVotes 00000d93.00000699 243bc7b8 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] A [src/rdb/quorum/qm_states/qm_state.cc 306 (0x81a804200)]: qmsPreferredCandidate_set till: 60789294s who: 1001. 00000d93.0000069a 243bc7b8 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] A [src/rdb/quorum/qm_states/qm_out_out_of_quorum.cc 61 (0x81a804200)]: OutOfQuorumState::stateUp2OutOfQuorum: WS_WaitingForStart -> WS_WaitingForVotes 00000d93.0000069b 243bc7b8 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] A [src/rdb/quorum/qm_states/qm_state.cc 552 (0x81a804200)]: stateTrans: WS_WaitingForStart -> WS_WaitingForVotes at: 60789752s 00000d93.0000069c 243bc7b8 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] A [src/rdb/unit.cc 354 (0x81a804200)]: Opened unit Management (1); incarnation 3 at epoch 5, version <5,981471> 00000d93.0000069d 243bc7b8 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] A [src/rdb/unit.cc 357 (0x81a804200)]: FAILOVER rdb: Opened local unit Management 00000d93.0000069e 243bc7b8 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] Notice: open_unit: Open RDB unit: status 0 (SUCCESS). 00000d93.0000069f 243bc7b8 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] A [src/rdb/unit.cc 2739 (0x81a804200)]: requestTxnCallback 00000d93.000006a0 243bc7b8 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] E [src/rdb/unit_local.cc 3585 (0x81a804200)]: LU: requestTxnCallback 00000d93.000006a1 243bc7b8 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81b13fb00: 0: INFO: SFO_Ready|Cluster_Apps: Startup stability test: management databases are open. 00000d93.000006a2 243bc7b8 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] mhost_boot: MHOST_SF: Stability has DBs 00000d93.000006a3 243bc7b8 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81b13fb00: 0: INFO: SFO_Ready|Cluster_Apps: Startup stability test: wait until management online. 00000d93.000006a4 243bc7b8 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] ASUP_CLUSTER_INFO: update_request(on_site_added:nmg-fas8040-02) 00000d93.000006a5 243bc7b8 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] ASUP_CLUSTER_INFO: update_request(on_site_added:nmg-fas8040-01) 00000d93.000006a6 243bc7b8 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] ASUP_CLUSTER_INFO: Cache update complete. Next timeout is 86400 seconds 00000d93.000006a7 243bc7b8 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] A [src/rdb/TM.cc 1459 (0x81a804200)]: TM 0: Report UNIT_IS_OFFLINE (epoch 0, master 0). Reason: process startup. 00000d93.000006a8 243bc7b8 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] A [src/rdb/TM.cc 1464 (0x81a804200)]: FAILOVER rdb: Local unit Management offline 00000d93.000006a9 243bc7b8 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Finishing module group: RDB, sequence: smf_rdb 00000d93.000006aa 243bc7b8 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Exiting module startup for group: 7 00000d93.000006ab 243bc7b8 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] mhost_boot: phase1 rdb end 00000d93.000006ac 243bc7b8 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] ASUP_CLUSTER_INFO: Cache update complete. Next timeout is 86400 seconds 00000d93.000006ad 243bc7b8 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81d248000: 0: NOTICE: rdb_sitelist: slpush: successfully pushed site information to _push_sitelist_id[10] nodelist_status non_authoritative 00000d93.000006ae 243bc7b8 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] mhost_boot: phase1 end 00000d93.000006af 243bc7b8 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] src/libnwd/unwd.cc:57 _call_sysctl : Wrote updated progress "mroot phase2 start" to oid debug.nodewatchdog.progress.mgwd. 00000d93.000006b0 243bc7b8 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] mhost_boot: phase2 start 00000d93.000006b1 243bc7b8 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] mhost_boot: phase2 ems start 00000d93.000006b2 243bc7b8 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81d248000: 0: NOTICE: rdb_sitelist: slpush: successfully pushed site information to _push_sitelist_id[11] nodelist_status non_authoritative 00000d93.000006b3 243bc7b8 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81d248000: 0: NOTICE: rdb_sitelist: slpush: successfully pushed site information to _push_sitelist_id[12] nodelist_status non_authoritative 00000d93.000006b4 243bc7b8 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] Notice: online_status_callback: RDB unit is offline 00000d93.000006b5 243bc7b8 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81d247b00: 0: INFO: RDB::callback::registrar: callback:src/rdb/rdb_online_registrar.cc:81 rdb_callbacks::ONLINE::BEGIN:: 60789764164 00000d93.000006b6 243bc7b8 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] check_defaults(): RDB Unit is offline. Returning without creating defaults. 00000d93.000006b7 243bc7b8 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] mhost_boot: phase2 ems end 00000d93.000006b8 243bc7b8 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] mhost_boot: phase2 end 00000d93.000006b9 243bc7b8 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] mhost_boot: StartTask: mainPostMroot end 00000d93.000006ba 243bc7b8 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] mhost_boot: StartTask: mainWaitForDatabases start 00000d93.000006bb 243bc7b8 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] mhost_boot: wait_for_databases start 00000d93.000006bc 243bc7b8 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] src/libnwd/unwd.cc:57 _call_sysctl : Wrote updated progress "wait_for_databases start" to oid debug.nodewatchdog.progress.mgwd. 00000d93.000006bd 243bc7b8 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] mhost_boot: wait_for_databases end 00000d93.000006be 243bc7b8 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81d247b00: 0: INFO: RDB::callback::registrar: callback:src/rdb/rdb_online_registrar.cc:106 rdb_callbacks::ONLINE::END:: 60789764166 00000d93.000006bf 243bc7b8 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81d248000: 0: ERR: rdb_sitelist: slpush: failed to push site information to _push_sitelist_id[13]: RPC: Program not registered [from mgwd on node "nmg-fas8040-02" (VSID: -3) to cm at 127.0.0.1]. 00000d93.000006c0 243bc7b8 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81d248000: 0: NOTICE: rdb_sitelist: slpush: successfully pushed site information to _push_sitelist_id[15] nodelist_status non_authoritative 00000d93.000006c1 243bc7b8 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81d248000: 0: NOTICE: RpcConnectionCache: markFailure: markFailure: New Quarantine for 127.0.0.1:536873522(mntsvc)/1[tcp]{fffffffd|8} with timeout(5.000s) Reason:: RPC: Program not registered 00000d93.000006c2 243bc7b8 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81d248000: 0: ERR: rdb_sitelist: slpush: failed to push site information to _push_sitelist_id[17]: RPC: Program not registered [from mgwd on node "nmg-fas8040-02" (VSID: -3) to mntsvc at 127.0.0.1]. 00000d93.000006c3 243bc7b8 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81d248000: 0: NOTICE: RpcConnectionCache: markFailure: markFailure: New Quarantine for 127.0.0.1:536873528(upgrademgr)/1[tcp]{fffffffd|8} with timeout(5.000s) Reason:: RPC: Program not registered 00000d93.000006c4 243bc7b8 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81d248000: 0: ERR: rdb_sitelist: slpush: failed to push site information to _push_sitelist_id[20]: RPC: Program not registered [from mgwd on node "nmg-fas8040-02" (VSID: -3) to upgrademgr at 127.0.0.1]. 00000d93.000006c5 243bc7b8 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81d248000: 0: NOTICE: rdb_sitelist: slpush: successfully pushed site information to _push_sitelist_id[25] nodelist_status non_authoritative 00000d93.000006c6 243bc7b8 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81d248000: 0: NOTICE: rdb_sitelist: slpush: successfully pushed site information to _push_sitelist_id[26] nodelist_status non_authoritative 00000d93.000006c7 243bc7b8 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] mhost_boot: StartTask: mainWaitForDatabases end 00000d93.000006c8 243bc7b8 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] mhost_boot: StartTask: mainPostDatabases start 00000d93.000006c9 243bc7b8 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] mhost_boot: start job manager start 00000d93.000006ca 243bc7b8 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] src/libnwd/unwd.cc:57 _call_sysctl : Wrote updated progress "start job manager start" to oid debug.nodewatchdog.progress.mgwd. 00000d93.000006cb 243bc7b9 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81d248000: 0: NOTICE: rdb_sitelist: slpush: successfully pushed site information to _push_sitelist_id[27] nodelist_status non_authoritative 00000d93.000006cd 243bc7b9 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81d248000: 0: NOTICE: rdb_sitelist: slpush: successfully pushed site information to _push_sitelist_id[33] nodelist_status non_authoritative 00000d93.000006ce 243bc7b9 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81d248000: 0: NOTICE: RpcConnectionCache: markFailure: markFailure: New Quarantine for 127.0.0.1:536873556(vserverdr)/1[tcp]{fffffffd|8} with timeout(5.000s) Reason:: RPC: Program not registered 00000d93.000006cf 243bc7b9 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81d248000: 0: ERR: rdb_sitelist: slpush: failed to push site information to _push_sitelist_id[34]: RPC: Program not registered [from mgwd on node "nmg-fas8040-02" (VSID: -3) to vserverdr at 127.0.0.1]. 00000d93.000006d0 243bc7b9 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81d248000: 0: NOTICE: RpcConnectionCache: markFailure: markFailure: New Quarantine for 127.0.0.1:536873562(No name for prog:536873562)/1[tcp]{fffffffd|8} with timeout(5.000s) Reason:: RPC: Program not registered 00000d93.000006d1 243bc7b9 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81d248000: 0: ERR: rdb_sitelist: slpush: failed to push site information to _push_sitelist_id[37]: RPC: Program not registered [from mgwd on node "nmg-fas8040-02" (VSID: -3) to Unknown Program:536873562 at 127.0.0.1]. 00000d93.000006d2 243bc7b9 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81d248000: 0: NOTICE: RpcConnectionCache: markFailure: markFailure: New Quarantine for 127.0.0.1:536873564(No name for prog:536873564)/1[tcp]{fffffffd|8} with timeout(5.000s) Reason:: RPC: Program not registered 00000d93.000006d3 243bc7b9 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81d248000: 0: ERR: rdb_sitelist: slpush: failed to push site information to _push_sitelist_id[38]: RPC: Program not registered [from mgwd on node "nmg-fas8040-02" (VSID: -3) to Unknown Program:536873564 at 127.0.0.1]. 00000d93.000006d5 243bc7b9 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81d248000: 0: NOTICE: rdb_sitelist: slpush: successfully pushed site information to _push_sitelist_id[41] nodelist_status non_authoritative 00000d93.000006d6 243bc7b9 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81d248000: 0: NOTICE: RpcConnectionCache: markFailure: markFailure: New Quarantine for 127.0.0.1:536873572(No name for prog:536873572)/1[tcp]{fffffffd|8} with timeout(5.000s) Reason:: RPC: Program not registered 00000d93.000006d7 243bc7b9 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81d248000: 0: ERR: rdb_sitelist: slpush: failed to push site information to _push_sitelist_id[42]: RPC: Program not registered [from mgwd on node "nmg-fas8040-02" (VSID: -3) to Unknown Program:536873572 at 127.0.0.1]. 00000d93.000006d8 243bc7b9 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81d248000: 0: NOTICE: RpcConnectionCache: markFailure: markFailure: New Quarantine for 127.0.0.1:536873574(No name for prog:536873574)/1[tcp]{fffffffd|8} with timeout(5.000s) Reason:: RPC: Program not registered 00000d93.000006d9 243bc7b9 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81d248000: 0: ERR: rdb_sitelist: slpush: failed to push site information to _push_sitelist_id[43]: RPC: Program not registered [from mgwd on node "nmg-fas8040-02" (VSID: -3) to Unknown Program:536873574 at 127.0.0.1]. 00000d93.000006da 243bc7b9 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81d248000: 0: NOTICE: RpcConnectionCache: markFailure: markFailure: New Quarantine for 127.0.0.1:536873576(No name for prog:536873576)/1[tcp]{fffffffd|8} with timeout(5.000s) Reason:: RPC: Program not registered 00000d93.000006db 243bc7b9 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81d248000: 0: ERR: rdb_sitelist: slpush: failed to push site information to _push_sitelist_id[44]: RPC: Program not registered [from mgwd on node "nmg-fas8040-02" (VSID: -3) to Unknown Program:536873576 at 127.0.0.1]. 00000d93.000006dc 243bc7b9 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81d248000: 0: NOTICE: RpcConnectionCache: markFailure: markFailure: New Quarantine for 127.0.0.1:536873578(No name for prog:536873578)/1[tcp]{fffffffd|8} with timeout(5.000s) Reason:: RPC: Program not registered 00000d93.000006dd 243bc7b9 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81d248000: 0: ERR: rdb_sitelist: slpush: failed to push site information to _push_sitelist_id[45]: RPC: Program not registered [from mgwd on node "nmg-fas8040-02" (VSID: -3) to Unknown Program:536873578 at 127.0.0.1]. 00000d93.000006de 243bc7b9 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81d248000: 0: NOTICE: RpcConnectionCache: markFailure: markFailure: New Quarantine for 127.0.0.1:536873580(No name for prog:536873580)/1[tcp]{fffffffd|8} with timeout(5.000s) Reason:: RPC: Program not registered 00000d93.000006df 243bc7b9 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81d248000: 0: ERR: rdb_sitelist: slpush: failed to push site information to _push_sitelist_id[46]: RPC: Program not registered [from mgwd on node "nmg-fas8040-02" (VSID: -3) to Unknown Program:536873580 at 127.0.0.1]. 00000d93.000006e0 243bc7b9 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: WARNING: job_manager::job_store: load_record_list:src/job_store.cc:382 ls_db::load_record_list: skipping file /mroot/etc/cluster_config/jm_cache/job_rdb_cache/update.489c38af-b80c-11e7-964b-00a098573067 00000d93.000006e1 243bc7b9 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: WARNING: job_manager::job_store: load_record_list:src/job_store.cc:382 ls_db::load_record_list: skipping file /mroot/etc/cluster_config/jm_cache/job_rdb_cache/bkup.489c38af-b80c-11e7-964b-00a098573067 00000d93.000006e2 243bc7b9 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: WARNING: job_manager::job_q: initialize:src/job_q.cc:141 starting recovery manager 00000d93.000006e3 243bc7b9 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: WARNING: job_manager::uijobid_db: create_instance:src/incr_db.cc:40 creating SMF incr_db 00000d93.000006e4 243bc7b9 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: WARNING: job_manager::uijobid_db: initialize:src/incr_db.cc:68 incr_db: initialize() 00000d93.000006e5 243bc7b9 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: WARNING: job_manager::rdb_util: create_instance:src/job_rec_int.cc:1348 creating SMF job rec interface 00000d93.000006e6 243bc7b9 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: WARNING: job_manager::sched_db: create_instance:src/sched_db.cc:604 creating SMF sched_db 00000d93.000006e7 243bc7b9 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: WARNING: job_manager::sched_db: create_instance:src/sched_interface.cc:37 creating SMF sched_interface 00000d93.000006e8 243bc7b9 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: WARNING: job_manager::job_sched: process_add:src/job_sched.cc:373 UIID:14 UUID:179bd90b-b79e-11e7-9b07-00a09857ee8f name:Vol Reaper rc:2 Cannot process cluster job as RDB is offline last_owner=00000000-0000-0000-0000-000000000000 00000d93.000006e9 243bc7b9 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: WARNING: job_manager::job_sched: process_add:src/job_sched.cc:373 UIID:15 UUID:179be0d8-b79e-11e7-9b07-00a09857ee8f name:FabricPool Space Job rc:2 Cannot process cluster job as RDB is offline last_owner=00000000-0000-0000-0000-000000000000 00000d93.000006ea 243bc7b9 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: WARNING: job_manager::job_sched: process_add:src/job_sched.cc:373 UIID:4 UUID:656fa976-b79d-11e7-9b07-00a09857ee8f name:SP log collection job rc:2 Cannot process cluster job as RDB is offline last_owner=00000000-0000-0000-0000-000000000000 00000d93.000006eb 243bc7b9 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: WARNING: job_manager::job_sched: process_add:src/job_sched.cc:373 UIID:3 UUID:65701994-b79d-11e7-9b07-00a09857ee8f name:SP Certificate Expiry Check Job rc:2 Cannot process cluster job as RDB is offline last_owner=00000000-0000-0000-0000-000000000000 00000d93.000006ec 243bc7b9 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81a9f1d00: 0: WARNING: job_manager::job_ext: on_online_status_cb:src/job_ext.cc:512 on_online_status_cb 00000d93.000006ed 243bc7b9 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81a9f1d00: 0: WARNING: job_manager::job_q: on_online_status_cb:src/job_q.cc:702 on_online_status_cb 00000d93.000006ee 243bc7b9 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81a9f1d00: 0: WARNING: job_manager::job_sched: on_online_status_cb:src/job_sched.cc:1203 on_online_status_cb (2) 00000d93.000006ef 243bc7b9 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] mhost_boot: start job manager end 00000d93.000006f0 243bc7b9 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] Warning: Unable to list entries on node nmg-fas8040-02. Internal error. Cannot retrieve information for LIFs with role "intercluster" while vifmgr is offline. Wait a few minutes, then try the command again. For further assistance, contact technical support. 00000d93.000006f1 243bc7b9 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] mhost_boot: service manager start 00000d93.000006f2 243bc7b9 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] src/libnwd/unwd.cc:57 _call_sysctl : Wrote updated progress "service manager start" to oid debug.nodewatchdog.progress.mgwd. 00000d93.000006f3 243bc7b9 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81d24d500: 86ffff000000000f: NOTICE: RpcConnectionCache: markFailure: markFailure: New Quarantine for 127.0.0.1:536873490(vldb)/1[tcp]{fffffffd|8} with timeout(5.000s) Reason:: RPC: Program not registered 00000d93.000006f4 243bc7b9 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81d24d500: 86ffff000000000f: ERR: VOPL::autoSnapshot: JobId[4] sched=hourly prefix=hourly createScheduledSnapshot:266 get policy=default error=RPC: Program not registered [from mgwd on node "nmg-fas8040-02" (VSID: -3) to vldb at 127.0.0.1] 00000d93.000006f5 243bc7b9 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] mhost_boot: service manager end 00000d93.000006f6 243bc7b9 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: INFO: librpcpool: bind_server: port=910 VSID=-2 LIFROLE=-1 IPv6=no Listen=yes Realtime=no ipspace/port already setup: port=910 vsid=-2 lifrole=-1 00000d93.000006f7 243bc7b9 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81d24d500: 86ffff000000000f: WARNING: VOPL::autoSnapshot: JobId[4] Couldn't connect to VLDB 00000d93.000006f8 243bc7b9 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Starting module group: FILER_APPS, sequence: smf_licensing_v2_filer_app_module 00000d93.000006f9 243bc7b9 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: INFO: license_v2: module init with V2 policy engine 00000d93.000006fa 243bc7b9 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: INFO: license_v2: register update callback 00000d93.000006fb 243bc7b9 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: INFO: license_v2: register online callback for module init 00000d93.000006fc 243bc7b9 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: INFO: license_v2: module_init register online callback 00000d93.000006fd 243bc7b9 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Finishing module group: FILER_APPS, sequence: smf_licensing_v2_filer_app_module 00000d93.000006fe 243bc7b9 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Starting module group: FILER_APPS, sequence: smf_certificate_filer_app_module 00000d93.000006ff 243bc7b9 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Finishing module group: FILER_APPS, sequence: smf_certificate_filer_app_module 00000d93.00000700 243bc7b9 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Starting module group: FILER_APPS, sequence: smf_vserver_template_registration_module 00000d93.00000701 243bc7b9 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Finishing module group: FILER_APPS, sequence: smf_vserver_template_registration_module 00000d93.00000702 243bc7b9 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Starting module group: FILER_APPS, sequence: smf_rdb_registration 00000d93.00000703 243bc7b9 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81a9f1d00: 0: INFO: license_v2: onlineInitCallback: RDB online callback, status= 2 00000d93.00000704 243bc7b9 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81a9f1d00: 0: INFO: license_v2: onlineInitCallback: RDB is offline [status= 2]. Default licenses are not created. 00000d93.00000705 243bc7b9 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81a9f1d00: 0: INFO: license_v2: RDB online callback. status=2 00000d93.00000706 243bc7b9 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Finishing module group: FILER_APPS, sequence: smf_rdb_registration 00000d93.00000707 243bc7b9 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Starting module group: FILER_APPS, sequence: smf_snapshot_policy_module 00000d93.00000708 243bc7b9 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81a9f1d00: 0: DEBUG: function_start_end_time_log: Start online_callback: 00000d93.00000709 243bc7b9 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81a9f1d00: 0: DEBUG: function_start_end_time_log: End online_callback: total elapsed time in milliseconds is 0 00000d93.0000070a 243bc7b9 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Finishing module group: FILER_APPS, sequence: smf_snapshot_policy_module 00000d93.0000070b 243bc7b9 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Starting module group: FILER_APPS, sequence: smf_qos_module 00000d93.0000070c 243bc7b9 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Finishing module group: FILER_APPS, sequence: smf_qos_module 00000d93.0000070d 243bc7b9 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Starting module group: FILER_APPS, sequence: smf_ec_qos_module 00000d93.0000070e 243bc7b9 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: ERR: EC: external_cache_module::module_init() 00000d93.0000070f 243bc7b9 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Finishing module group: FILER_APPS, sequence: smf_ec_qos_module 00000d93.00000710 243bc7b9 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Starting module group: FILER_APPS, sequence: smf_resource_group_module 00000d93.00000711 243bc7b9 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Finishing module group: FILER_APPS, sequence: smf_resource_group_module 00000d93.00000712 243bc7b9 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Starting module group: FILER_APPS, sequence: smf_config_backup_restore_app_module 00000d93.00000713 243bc7b9 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Finishing module group: FILER_APPS, sequence: smf_config_backup_restore_app_module 00000d93.00000714 243bc7b9 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Starting module group: FILER_APPS, sequence: smf_coredump_module 00000d93.00000715 243bc7b9 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] In coredump_module::module_init() 00000d93.00000716 243bc7b9 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] In coredump_module::module_init(): kicking off coredump job thread 00000d93.00000717 243bc7b9 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] In coredump_module::module_init() exit 00000d93.00000718 243bc7b9 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Finishing module group: FILER_APPS, sequence: smf_coredump_module 00000d93.00000719 243bc7b9 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] In coredump_module::queue_coredump_job_thread() 00000d93.0000071a 243bc7b9 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Starting module group: FILER_APPS, sequence: smf_cluster_ntp_module 00000d93.0000071b 243bc7b9 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81b512b00: 0: INFO: SFO_Ready|Cluster_Apps: wait_for_stable_mhost (waiter: Coredump), maxTime 180 sec: starting 00000d93.0000071c 243bc7b9 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Finishing module group: FILER_APPS, sequence: smf_cluster_ntp_module 00000d93.0000071d 243bc7b9 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81b512b00: 0: INFO: SFO_Ready|Cluster_Apps: wait_for_in_cluster (waiter: Coredump), maxTime 180 sec: starting 00000d93.0000071e 243bc7b9 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Starting module group: FILER_APPS, sequence: smf_snapmirror_policy_module 00000d93.0000071f 243bc7b9 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81b512b00: 0: INFO: SFO_Ready|Cluster_Apps: wait_for_in_cluster (waiter: Coredump), maxTime 180 sec: in cluster after 0 sec. 00000d93.00000720 243bc7b9 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Finishing module group: FILER_APPS, sequence: smf_snapmirror_policy_module 00000d93.00000721 243bc7b9 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Starting module group: FILER_APPS, sequence: smf_zapi_smf_trace_module 00000d93.00000722 243bc7b9 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Finishing module group: FILER_APPS, sequence: smf_zapi_smf_trace_module 00000d93.00000723 243bc7b9 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Starting module group: FILER_APPS, sequence: smf_PipCapabilityModule 00000d93.00000724 243bc7b9 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Finishing module group: FILER_APPS, sequence: smf_PipCapabilityModule 00000d93.00000725 243bc7b9 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Starting module group: FILER_APPS, sequence: smf_PipTopoMgwdModule 00000d93.00000726 243bc7b9 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Finishing module group: FILER_APPS, sequence: smf_PipTopoMgwdModule 00000d93.00000727 243bc7b9 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Starting module group: FILER_APPS, sequence: smf_PipStateModule 00000d93.00000728 243bc7b9 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Finishing module group: FILER_APPS, sequence: smf_PipStateModule 00000d93.00000729 243bc7b9 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Starting module group: FILER_APPS, sequence: smf_repos_registration_module 00000d93.0000072a 243bc7b9 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Finishing module group: FILER_APPS, sequence: smf_repos_registration_module 00000d93.0000072b 243bc7b9 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Starting module group: FILER_APPS, sequence: smf_vsm_ics_module 00000d93.0000072c 243bc7b9 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Finishing module group: FILER_APPS, sequence: smf_vsm_ics_module 00000d93.0000072d 243bc7b9 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Starting module group: FILER_APPS, sequence: smf_vserverdr_failover_module 00000d93.0000072e 243bc7b9 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Finishing module group: FILER_APPS, sequence: smf_vserverdr_failover_module 00000d93.0000072f 243bc7b9 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Starting module group: FILER_APPS, sequence: smf_vserver_info_kernel_registration 00000d93.00000730 243bc7b9 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Finishing module group: FILER_APPS, sequence: smf_vserver_info_kernel_registration 00000d93.00000731 243bc7b9 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Starting module group: FILER_APPS, sequence: smf_vserver_audit_config_reply_reg_module 00000d93.00000732 243bc7b9 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Finishing module group: FILER_APPS, sequence: smf_vserver_audit_config_reply_reg_module 00000d93.00000733 243bc7b9 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Starting module group: FILER_APPS, sequence: smf_vscan_replay_registration_module 00000d93.00000734 243bc7b9 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Finishing module group: FILER_APPS, sequence: smf_vscan_replay_registration_module 00000d93.00000735 243bc7b9 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Starting module group: FILER_APPS, sequence: smf_storage_switch_module 00000d93.00000736 243bc7b9 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Finishing module group: FILER_APPS, sequence: smf_storage_switch_module 00000d93.00000737 243bc7b9 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Starting module group: FILER_APPS, sequence: smf_storage_bridge_module 00000d93.00000738 243bc7b9 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Finishing module group: FILER_APPS, sequence: smf_storage_bridge_module 00000d93.00000739 243bc7b9 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Starting module group: FILER_APPS, sequence: smf_sfod_registration_module 00000d93.0000073a 243bc7b9 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Finishing module group: FILER_APPS, sequence: smf_sfod_registration_module 00000d93.0000073b 243bc7b9 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Starting module group: FILER_APPS, sequence: smf_Collector 00000d93.0000073c 243bc7b9 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Finishing module group: FILER_APPS, sequence: smf_Collector 00000d93.0000073d 243bc7b9 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Starting module group: FILER_APPS, sequence: smf_san_aix_support_registration 00000d93.0000073e 243bc7b9 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Finishing module group: FILER_APPS, sequence: smf_san_aix_support_registration 00000d93.0000073f 243bc7b9 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Starting module group: FILER_APPS, sequence: smf_nwd_kernel_update 00000d93.00000740 243bc7b9 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Finishing module group: FILER_APPS, sequence: smf_nwd_kernel_update 00000d93.00000741 243bc7b9 Thu Sep 26 2019 04:33:13 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Starting module group: FILER_APPS, sequence: smf_ypbind_registration_module 00000d93.00000742 243bc7b9 Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Finishing module group: FILER_APPS, sequence: smf_ypbind_registration_module 00000d93.00000743 243bc7b9 Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Starting module group: FILER_APPS, sequence: smf_cfg_replay_registration_module 00000d93.00000744 243bc7b9 Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] Could not update schema validator config Reason:RDB is offline 00000d93.00000745 243bc7b9 Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 0x81a9f1d00: 0: ERR: tables::exports_cache: online_callback:src/tables/exports_cache.cc:2273 Unit is offline 00000d93.00000746 243bc7b9 Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 0x81a9f1d00: 0: NOTICE: MGWD::cluster_id: online_callback reported UNIT_IS_OFFLINE 00000d93.00000747 243bc7b9 Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] Could not update default values for ontapi limits Reason:RDB is offline 00000d93.00000748 243bc7b9 Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 0x81a9f1d00: 0: NOTICE: Tables:web: Synchronizing the web server configuration from the cluster to the nodes 00000d93.00000749 243bc7b9 Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 0x81b514900: 0: NOTICE: RpcConnectionCache: markFailure: markFailure: New Quarantine for localhost:536873477(secd)/1[tcp]{fffffffd|ffffffff} with timeout(3.000s) Reason:: RPC: Port mapper failure - RPC: Unable to send 00000d93.0000074a 243bc7b9 Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] A [src/rdb/unit.cc 2739 (0x81a804200)]: requestTxnCallback 00000d93.0000074b 243bc7b9 Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] E [src/rdb/unit_local.cc 3585 (0x81a804200)]: LU: requestTxnCallback 00000d93.0000074c 243bc7b9 Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] A [src/rdb/unit.cc 2739 (0x81a804200)]: requestTxnCallback 00000d93.0000074d 243bc7b9 Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] E [src/rdb/unit_local.cc 3585 (0x81a804200)]: LU: requestTxnCallback 00000d93.0000074e 243bc7b9 Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] A [src/rdb/unit.cc 2739 (0x81a804200)]: requestTxnCallback 00000d93.0000074f 243bc7b9 Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] E [src/rdb/unit_local.cc 3585 (0x81a804200)]: LU: requestTxnCallback 00000d93.00000750 243bc7b9 Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] A [src/rdb/unit.cc 2739 (0x81a804200)]: requestTxnCallback 00000d93.00000751 243bc7b9 Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] E [src/rdb/unit_local.cc 3585 (0x81a804200)]: LU: requestTxnCallback 00000d93.00000752 243bc7b9 Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] A [src/rdb/unit.cc 2739 (0x81a804200)]: requestTxnCallback 00000d93.00000753 243bc7b9 Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] E [src/rdb/unit_local.cc 3585 (0x81a804200)]: LU: requestTxnCallback 00000d93.00000754 243bc7b9 Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] A [src/rdb/unit.cc 2739 (0x81a804200)]: requestTxnCallback 00000d93.00000755 243bc7b9 Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] E [src/rdb/unit_local.cc 3585 (0x81a804200)]: LU: requestTxnCallback 00000d93.00000756 243bc7b9 Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] A [src/rdb/unit.cc 2739 (0x81a804200)]: requestTxnCallback 00000d93.00000757 243bc7b9 Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] E [src/rdb/unit_local.cc 3585 (0x81a804200)]: LU: requestTxnCallback 00000d93.00000758 243bc7b9 Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] A [src/rdb/unit.cc 2739 (0x81a804200)]: requestTxnCallback 00000d93.00000759 243bc7b9 Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] E [src/rdb/unit_local.cc 3585 (0x81a804200)]: LU: requestTxnCallback 00000d93.0000075a 243bc7b9 Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] A [src/rdb/unit.cc 2739 (0x81a804200)]: requestTxnCallback 00000d93.0000075b 243bc7b9 Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] E [src/rdb/unit_local.cc 3585 (0x81a804200)]: LU: requestTxnCallback 00000d93.0000075c 243bc7b9 Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] A [src/rdb/unit.cc 2739 (0x81a804200)]: requestTxnCallback 00000d93.0000075d 243bc7b9 Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] E [src/rdb/unit_local.cc 3585 (0x81a804200)]: LU: requestTxnCallback 00000d93.0000075e 243bc7b9 Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] A [src/rdb/unit.cc 2739 (0x81a804200)]: requestTxnCallback 00000d93.0000075f 243bc7b9 Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] E [src/rdb/unit_local.cc 3585 (0x81a804200)]: LU: requestTxnCallback 00000d93.00000760 243bc7b9 Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] A [src/rdb/unit.cc 2739 (0x81a804200)]: requestTxnCallback 00000d93.00000761 243bc7b9 Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] E [src/rdb/unit_local.cc 3585 (0x81a804200)]: LU: requestTxnCallback 00000d93.00000762 243bc7b9 Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] A [src/rdb/unit.cc 2739 (0x81a804200)]: requestTxnCallback 00000d93.00000763 243bc7b9 Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] E [src/rdb/unit_local.cc 3585 (0x81a804200)]: LU: requestTxnCallback 00000d93.00000764 243bc7b9 Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] A [src/rdb/unit.cc 2739 (0x81a804200)]: requestTxnCallback 00000d93.00000765 243bc7b9 Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] E [src/rdb/unit_local.cc 3585 (0x81a804200)]: LU: requestTxnCallback 00000d93.00000766 243bc7b9 Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] A [src/rdb/unit.cc 2739 (0x81a804200)]: requestTxnCallback 00000d93.00000767 243bc7b9 Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] E [src/rdb/unit_local.cc 3585 (0x81a804200)]: LU: requestTxnCallback 00000d93.00000768 243bc7b9 Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] A [src/rdb/unit.cc 2739 (0x81a804200)]: requestTxnCallback 00000d93.00000769 243bc7b9 Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] E [src/rdb/unit_local.cc 3585 (0x81a804200)]: LU: requestTxnCallback 00000d93.0000076a 243bc7b9 Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] A [src/rdb/unit.cc 2739 (0x81a804200)]: requestTxnCallback 00000d93.0000076b 243bc7b9 Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] E [src/rdb/unit_local.cc 3585 (0x81a804200)]: LU: requestTxnCallback 00000d93.0000076c 243bc7b9 Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] A [src/rdb/unit.cc 2739 (0x81a804200)]: requestTxnCallback 00000d93.0000076d 243bc7b9 Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] E [src/rdb/unit_local.cc 3585 (0x81a804200)]: LU: requestTxnCallback 00000d93.0000076e 243bc7b9 Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] A [src/rdb/unit.cc 2739 (0x81a804200)]: requestTxnCallback 00000d93.0000076f 243bc7b9 Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] E [src/rdb/unit_local.cc 3585 (0x81a804200)]: LU: requestTxnCallback 00000d93.00000770 243bc7b9 Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] A [src/rdb/unit.cc 2739 (0x81a804200)]: requestTxnCallback 00000d93.00000771 243bc7b9 Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] E [src/rdb/unit_local.cc 3585 (0x81a804200)]: LU: requestTxnCallback 00000d93.00000772 243bc7b9 Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] A [src/rdb/unit.cc 2739 (0x81a804200)]: requestTxnCallback 00000d93.00000773 243bc7b9 Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] E [src/rdb/unit_local.cc 3585 (0x81a804200)]: LU: requestTxnCallback 00000d93.00000774 243bc7b9 Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] A [src/rdb/unit.cc 2739 (0x81a804200)]: requestTxnCallback 00000d93.00000775 243bc7b9 Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] E [src/rdb/unit_local.cc 3585 (0x81a804200)]: LU: requestTxnCallback 00000d93.00000776 243bc7b9 Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] A [src/rdb/unit.cc 2739 (0x81a804200)]: requestTxnCallback 00000d93.00000777 243bc7b9 Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] E [src/rdb/unit_local.cc 3585 (0x81a804200)]: LU: requestTxnCallback 00000d93.00000778 243bc7b9 Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] A [src/rdb/unit.cc 2739 (0x81a804200)]: requestTxnCallback 00000d93.00000779 243bc7b9 Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] E [src/rdb/unit_local.cc 3585 (0x81a804200)]: LU: requestTxnCallback 00000d93.0000077a 243bc7b9 Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] A [src/rdb/unit.cc 2739 (0x81a804200)]: requestTxnCallback 00000d93.0000077b 243bc7b9 Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] E [src/rdb/unit_local.cc 3585 (0x81a804200)]: LU: requestTxnCallback 00000d93.0000077c 243bc7b9 Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] A [src/rdb/unit.cc 2739 (0x81a804200)]: requestTxnCallback 00000d93.0000077d 243bc7b9 Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] E [src/rdb/unit_local.cc 3585 (0x81a804200)]: LU: requestTxnCallback 00000d93.0000077e 243bc7b9 Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] A [src/rdb/unit.cc 2739 (0x81a804200)]: requestTxnCallback 00000d93.0000077f 243bc7b9 Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] E [src/rdb/unit_local.cc 3585 (0x81a804200)]: LU: requestTxnCallback 00000d93.00000780 243bc7b9 Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] A [src/rdb/unit.cc 2739 (0x81a804200)]: requestTxnCallback 00000d93.00000781 243bc7b9 Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] E [src/rdb/unit_local.cc 3585 (0x81a804200)]: LU: requestTxnCallback 00000d93.00000782 243bc7b9 Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] A [src/rdb/unit.cc 2739 (0x81a804200)]: requestTxnCallback 00000d93.00000783 243bc7b9 Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] E [src/rdb/unit_local.cc 3585 (0x81a804200)]: LU: requestTxnCallback 00000d93.00000784 243bc7b9 Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] A [src/rdb/unit.cc 2739 (0x81a804200)]: requestTxnCallback 00000d93.00000785 243bc7b9 Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] E [src/rdb/unit_local.cc 3585 (0x81a804200)]: LU: requestTxnCallback 00000d93.00000786 243bc7b9 Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] A [src/rdb/unit.cc 2739 (0x81a804200)]: requestTxnCallback 00000d93.00000787 243bc7b9 Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] E [src/rdb/unit_local.cc 3585 (0x81a804200)]: LU: requestTxnCallback 00000d93.00000788 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] A [src/rdb/unit.cc 2739 (0x81a804200)]: requestTxnCallback 00000d93.00000789 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] E [src/rdb/unit_local.cc 3585 (0x81a804200)]: LU: requestTxnCallback 00000d93.0000078a 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] A [src/rdb/unit.cc 2739 (0x81a804200)]: requestTxnCallback 00000d93.0000078b 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] E [src/rdb/unit_local.cc 3585 (0x81a804200)]: LU: requestTxnCallback 00000d93.0000078c 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] A [src/rdb/unit.cc 2739 (0x81a804200)]: requestTxnCallback 00000d93.0000078d 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] E [src/rdb/unit_local.cc 3585 (0x81a804200)]: LU: requestTxnCallback 00000d93.0000078e 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] A [src/rdb/unit.cc 2739 (0x81a804200)]: requestTxnCallback 00000d93.0000078f 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] E [src/rdb/unit_local.cc 3585 (0x81a804200)]: LU: requestTxnCallback 00000d93.00000790 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] A [src/rdb/unit.cc 2739 (0x81a804200)]: requestTxnCallback 00000d93.00000791 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] E [src/rdb/unit_local.cc 3585 (0x81a804200)]: LU: requestTxnCallback 00000d93.00000792 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] A [src/rdb/unit.cc 2739 (0x81a804200)]: requestTxnCallback 00000d93.00000793 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] E [src/rdb/unit_local.cc 3585 (0x81a804200)]: LU: requestTxnCallback 00000d93.00000794 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] A [src/rdb/unit.cc 2739 (0x81a804200)]: requestTxnCallback 00000d93.00000795 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] E [src/rdb/unit_local.cc 3585 (0x81a804200)]: LU: requestTxnCallback 00000d93.00000796 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] A [src/rdb/unit.cc 2739 (0x81a804200)]: requestTxnCallback 00000d93.00000797 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] E [src/rdb/unit_local.cc 3585 (0x81a804200)]: LU: requestTxnCallback 00000d93.00000798 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] A [src/rdb/unit.cc 2739 (0x81a804200)]: requestTxnCallback 00000d93.00000799 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] E [src/rdb/unit_local.cc 3585 (0x81a804200)]: LU: requestTxnCallback 00000d93.0000079a 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] A [src/rdb/unit.cc 2739 (0x81a804200)]: requestTxnCallback 00000d93.0000079b 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] E [src/rdb/unit_local.cc 3585 (0x81a804200)]: LU: requestTxnCallback 00000d93.0000079c 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] A [src/rdb/unit.cc 2739 (0x81a804200)]: requestTxnCallback 00000d93.0000079d 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] E [src/rdb/unit_local.cc 3585 (0x81a804200)]: LU: requestTxnCallback 00000d93.0000079e 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] A [src/rdb/unit.cc 2739 (0x81a804200)]: requestTxnCallback 00000d93.0000079f 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] E [src/rdb/unit_local.cc 3585 (0x81a804200)]: LU: requestTxnCallback 00000d93.000007a0 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Finishing module group: FILER_APPS, sequence: smf_cfg_replay_registration_module 00000d93.000007a1 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Starting module group: FILER_APPS, sequence: smf_nblade_replay_registration_modu 00000d93.000007a2 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Finishing module group: FILER_APPS, sequence: smf_nblade_replay_registration_modu 00000d93.000007a3 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Starting module group: FILER_APPS, sequence: smf_mcbp_module 00000d93.000007a4 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Finishing module group: FILER_APPS, sequence: smf_mcbp_module 00000d93.000007a5 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Starting module group: FILER_APPS, sequence: smf_hashd_replay_registration_module 00000d93.000007a6 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Finishing module group: FILER_APPS, sequence: smf_hashd_replay_registration_module 00000d93.000007a7 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Starting module group: FILER_APPS, sequence: smf_fsm_replay_registration_module 00000d93.000007a8 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Finishing module group: FILER_APPS, sequence: smf_fsm_replay_registration_module 00000d93.000007a9 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Starting module group: FILER_APPS, sequence: smf_flexcache_registration_module 00000d93.000007aa 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Finishing module group: FILER_APPS, sequence: smf_flexcache_registration_module 00000d93.000007ab 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Starting module group: FILER_APPS, sequence: smf_flexclone_registration_module 00000d93.000007ac 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Finishing module group: FILER_APPS, sequence: smf_flexclone_registration_module 00000d93.000007ad 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Starting module group: FILER_APPS, sequence: smf_filer_kernel_registration 00000d93.000007ae 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Finishing module group: FILER_APPS, sequence: smf_filer_kernel_registration 00000d93.000007af 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Starting module group: FILER_APPS, sequence: smf_DataPolicyRegistrationModule 00000d93.000007b0 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Finishing module group: FILER_APPS, sequence: smf_DataPolicyRegistrationModule 00000d93.000007b1 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Starting module group: FILER_APPS, sequence: smf_col_kernel_registration 00000d93.000007b2 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Finishing module group: FILER_APPS, sequence: smf_col_kernel_registration 00000d93.000007b3 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Starting module group: FILER_APPS, sequence: smf_cm_preset_module 00000d93.000007b4 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Finishing module group: FILER_APPS, sequence: smf_cm_preset_module 00000d93.000007b5 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Starting module group: FILER_APPS, sequence: smf_upgrademgr_module 00000d93.000007b6 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:debug:2284] upgrademgr_module::module_init - invoked; [ANDU_LOG at src/tables/cluster_image.cc:6214 in 'module_init'] 00000d93.000007b7 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:debug:2284] upgrademgr_event_callback::init - invoked; [ANDU_LOG at src/tables/cluster_image.cc:6598 in 'init'] 00000d93.000007b8 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:debug:2284] upgrademgr_module::registerUpdateCallback - invoked on table non_disruptive_update_task_rdb; [ANDU_LOG at src/tables/cluster_image.cc:6238 in 'registerUpdateCallback'] 00000d93.000007b9 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:debug:2284] upgrademgr_event_callback::run - invoked; [ANDU_LOG at src/tables/cluster_image.cc:6301 in 'run'] 00000d93.000007ba 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:debug:2284] upgrademgr_module::registerUpdateCallback - invoked on table upgrade_manager_rdb; [ANDU_LOG at src/tables/cluster_image.cc:6238 in 'registerUpdateCallback'] 00000d93.000007bb 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Finishing module group: FILER_APPS, sequence: smf_upgrademgr_module 00000d93.000007bc 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Starting module group: FILER_APPS, sequence: smf_secd_registration_module 00000d93.000007bd 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Finishing module group: FILER_APPS, sequence: smf_secd_registration_module 00000d93.000007be 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Starting module group: FILER_APPS, sequence: smf_cifs_registration 00000d93.000007bf 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Finishing module group: FILER_APPS, sequence: smf_cifs_registration 00000d93.000007c0 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Starting module group: FILER_APPS, sequence: smf_Cifs_config_replay_registration_module 00000d93.000007c1 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Finishing module group: FILER_APPS, sequence: smf_Cifs_config_replay_registration_module 00000d93.000007c2 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Starting module group: FILER_APPS, sequence: smf_ccma_config_module 00000d93.000007c3 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Finishing module group: FILER_APPS, sequence: smf_ccma_config_module 00000d93.000007c4 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Starting module group: FILER_APPS, sequence: smf_capability_kernel_registration 00000d93.000007c5 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: INFO: capability_kernel: Registered update callback - 0 00000d93.000007c6 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Finishing module group: FILER_APPS, sequence: smf_capability_kernel_registration 00000d93.000007c7 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Starting module group: FILER_APPS, sequence: smf_ECV_Monitor 00000d93.000007c8 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 0x81b515d00: 0: INFO: capability_kernel: capability_push: worker thread created 00000d93.000007c9 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Finishing module group: FILER_APPS, sequence: smf_ECV_Monitor 00000d93.000007ca 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Starting module group: FILER_APPS, sequence: smf_antivirus_registration_module 00000d93.000007cb 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: ERR: rdb_tran_glue: create: tid=0x81a804200 failed to create transaction for label='antivirus transaction reset': UNIT_OFFLINE 00000d93.000007cc 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: ERR: rdb_tran_glue: commit: commit attempted on an aborted transaction: tid=0x81a804200 label=antivirus transaction reset 00000d93.000007cd 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Finishing module group: FILER_APPS, sequence: smf_antivirus_registration_module 00000d93.000007ce 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Starting module group: FILER_APPS, sequence: smf_adb_job_init 00000d93.000007cf 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Finishing module group: FILER_APPS, sequence: smf_adb_job_init 00000d93.000007d0 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Starting module group: FILER_APPS, sequence: smf_CM_RDBInitializer 00000d93.000007d1 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Finishing module group: FILER_APPS, sequence: smf_CM_RDBInitializer 00000d93.000007d2 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Starting module group: FILER_APPS, sequence: smf_vserver_registration 00000d93.000007d3 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Finishing module group: FILER_APPS, sequence: smf_vserver_registration 00000d93.000007d4 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Starting module group: FILER_APPS, sequence: smf_volume_transition_registration 00000d93.000007d5 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Finishing module group: FILER_APPS, sequence: smf_volume_transition_registration 00000d93.000007d6 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Starting module group: FILER_APPS, sequence: smf_ndo_manager 00000d93.000007d7 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: INFO: MGWD::NDO: NDO Manager module has been initialized. 00000d93.000007d8 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: INFO: MGWD::NDO: Registering task Cluster Quorum Check in the category Precheck. 00000d93.000007d9 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: INFO: MGWD::NDO: Registering task Image Version Check in the category Precheck. 00000d93.000007da 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: INFO: MGWD::NDO: Registering task HA Status Check in the category Precheck. 00000d93.000007db 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: INFO: MGWD::NDO: Registering task Aggr Status Check in the category Precheck. 00000d93.000007dc 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: INFO: MGWD::NDO: Registering task Disk Status Check in the category Precheck. 00000d93.000007dd 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: INFO: MGWD::NDO: Registering task Data LIF Status Check in the category Precheck. 00000d93.000007de 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: INFO: MGWD::NDO: Registering task Cluster LIF Status Check in the category Precheck. 00000d93.000007df 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: INFO: MGWD::NDO: Registering task ASUP Status Check in the category Precheck. 00000d93.000007e0 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: INFO: MGWD::NDO: Registering task Aggr Relocation in the category Resource-release. 00000d93.000007e1 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: INFO: MGWD::NDO: Registering task LIF Migration in the category Resource-release. 00000d93.000007e2 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: INFO: MGWD::NDO: Registering task Disable Cluster HA in the category Resource-release. 00000d93.000007e3 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: INFO: MGWD::NDO: Registering task Epsilon Migration in the category Resource-release. 00000d93.000007e4 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: INFO: MGWD::NDO: Registering task Halt in the category Resource-release. 00000d93.000007e5 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: INFO: MGWD::NDO: Registering task LIF Migration in the category Resource-regain. 00000d93.000007e6 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: INFO: MGWD::NDO: Registering task Aggr Relocation in the category Resource-regain. 00000d93.000007e7 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: INFO: MGWD::NDO: Registering task Mark Completion in the category Cleanup. 00000d93.000007e8 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Finishing module group: FILER_APPS, sequence: smf_ndo_manager 00000d93.000007e9 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Starting module group: FILER_APPS, sequence: smf_cps_notification_thread_module 00000d93.000007ea 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Finishing module group: FILER_APPS, sequence: smf_cps_notification_thread_module 00000d93.000007eb 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Starting module group: FILER_APPS, sequence: smf_MiscNodeInfoRegistration 00000d93.000007ec 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Finishing module group: FILER_APPS, sequence: smf_MiscNodeInfoRegistration 00000d93.000007ed 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Starting module group: FILER_APPS, sequence: smf_SLMSupport 00000d93.000007ee 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Finishing module group: FILER_APPS, sequence: smf_SLMSupport 00000d93.000007ef 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Starting module group: FILER_APPS, sequence: smf_keymanager_rdb_registration 00000d93.000007f0 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Finishing module group: FILER_APPS, sequence: smf_keymanager_rdb_registration 00000d93.000007f1 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Starting module group: FILER_APPS, sequence: smf_cluster_kdb_rdb_registration 00000d93.000007f2 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Finishing module group: FILER_APPS, sequence: smf_cluster_kdb_rdb_registration 00000d93.000007f3 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Starting module group: FILER_APPS, sequence: smf_km_wrapped_kdb_rdb_registration 00000d93.000007f4 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Finishing module group: FILER_APPS, sequence: smf_km_wrapped_kdb_rdb_registration 00000d93.000007f5 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Starting module group: FILER_APPS, sequence: smf_svm_kdb_rdb_registration 00000d93.000007f6 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Finishing module group: FILER_APPS, sequence: smf_svm_kdb_rdb_registration 00000d93.000007f7 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Starting module group: FILER_APPS, sequence: smf_migrate_failover_module 00000d93.000007f8 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Finishing module group: FILER_APPS, sequence: smf_migrate_failover_module 00000d93.000007f9 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Starting module group: FILER_APPS, sequence: smf_security_protocol_ssh_registration 00000d93.000007fa 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Finishing module group: FILER_APPS, sequence: smf_security_protocol_ssh_registration 00000d93.000007fb 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Starting module group: FILER_APPS, sequence: smf_kmip_keyserver_smdb_resync_registration 00000d93.000007fc 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Finishing module group: FILER_APPS, sequence: smf_kmip_keyserver_smdb_resync_registration 00000d93.000007fd 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Starting module group: FILER_APPS, sequence: smf_certificate_registration 00000d93.000007fe 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Finishing module group: FILER_APPS, sequence: smf_certificate_registration 00000d93.000007ff 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: DEBUG: module_name: Exiting module startup for group: 8 00000d93.00000800 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] mhost_boot: start_mroot_and_rdb end 00000d93.00000801 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] mhost_boot: rdb/mhost apps start 00000d93.00000802 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: INFO: SFO_Ready|Start_Stop_Rdb_Clients: start_rdb_clients:41 here 00000d93.00000803 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] mhost_boot: cap check wait for unit online start 00000d93.00000804 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 0x81b517100: 0: INFO: SFO_Ready|inform_dblade: inform_dblade_when_apps_online_pthread_func:51 started 00000d93.00000805 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: INFO: SFO_Ready|Start_Stop_Rdb_Clients: start_rdb_clients:47 handle VIFs 00000d93.00000806 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] mhost_boot: MHOST_SF: inform dblade start 00000d93.00000807 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 0x81b517100: 0: INFO: SFO_Ready|Cluster_Apps: wait_for_stable_mhost (waiter: Setup), maxTime 900 sec: starting 00000d93.00000808 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] mhost_boot: MHOST_SF: CFO stable site start 00000d93.00000809 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 0x81b517100: 0: INFO: SFO_Ready|Cluster_Apps: wait_for_in_cluster (waiter: Setup), maxTime 900 sec: starting 00000d93.0000080a 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 0x81b517100: 0: INFO: SFO_Ready|Cluster_Apps: wait_for_in_cluster (waiter: Setup), maxTime 900 sec: in cluster after 0 sec. 00000d93.0000080b 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] mhost_boot: MHOST_SF: CFO stable site OK/end 00000d93.0000080c 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] mhost_boot: MHOST_SF: CFO stable apps start 00000d93.0000080d 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: INFO: SFO_Ready|Start_Stop_Rdb_Clients: start_rdb_clients:87 start mhost apps 00000d93.0000080e 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 0x81a9f1d00: 0: NOTICE: Tables:web: The web server configuration has been synchronized from the cluster to the nodes 00000d93.0000080f 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 0x81a9f1d00: 0: NOTICE: Tables:web: reconf_cluster_proxy_async (local): Reconfiguration is now in progress 00000d93.00000810 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 0x81a9f1d00: 0: NOTICE: Tables:web: web_server_config online callback: Updating the proxy server 00000d93.00000811 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 0x81b518500: 0: NOTICE: Tables:web: Starting cluster proxy reconfig thread is_all_nodes=0 00000d93.00000812 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] rebaseline_stream:Thu Sep 26 04:33:14 2019:src/tables/vserverdr/common/synthetic_method_handling.cc:76 (thr_id:0x81a9f1d00) 0x81a9f1d00: online_callback, Entry 00000d93.00000813 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] rebaseline_stream:Thu Sep 26 04:33:14 2019:src/tables/vserverdr/common/synthetic_method_handling.cc:76 (thr_id:0x81a9f1d00) 0x81a9f1d00: checkMasterAndSpawnThread, Entry 00000d93.00000814 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] rebaseline_stream:Thu Sep 26 04:33:14 2019:src/tables/vserverdr/common/synthetic_method_handling.cc:76 (thr_id:0x81a9f1d00) 0x81a9f1d00: checkMasterAndSpawnThread, No Entries in the trigger_baseline_generation RDB 00000d93.00000815 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 0x81a9f1d00: 0: ERR: RBAC::login: src/tables/vs_userprofile.cc:2843 calling online_callback status 2 00000d93.0000081d 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] SSTEST: got diagnosis callback for process schmd state stop 00000d93.00000825 243bc7bd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] SSTEST: got diagnosis callback for process cphmd state stop 00000d93.00000833 243bc7be Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] SSTEST: got diagnosis callback for process shmd state stop 00000d93.0000083b 243bc7be Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] SSTEST: got diagnosis callback for process cshmd state stop 00000d93.00000892 243bc7cd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 0x81b7bc900: 0: NOTICE: librpcpool: MAX DISPATCHED delta:6 prio:0 old_max_dispatch:2 00000d93.00000893 243bc7cd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 0x81b244700: 0: ERR: librpcpool: ******************************************************************************** 00000d93.00000894 243bc7cd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 0x81b244700: 0: ERR: librpcpool: * 00000d93.00000895 243bc7cd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 0x81b244700: 0: ERR: librpcpool: * 00000d93.00000896 243bc7cd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 0x81b244700: 0: ERR: librpcpool: RPC Server Data Dump. 0 previous dumps skipped 00000d93.00000897 243bc7cd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] Thread Stats: 00000d93.00000898 243bc7cd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] Priority Thread Program Version Exec Time To Running Function 00000d93.00000899 243bc7cd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] Start Time 00000d93.0000089a 243bc7cd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 0 0 0x20000A10 1 2 0 false DSMDB_GET_IMP - node 00000d93.0000089b 243bc7cd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 0 1 0x20000A10 1 2 0 false DSMDB_GET_IMP - node 00000d93.0000089c 243bc7cd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 0 2 0x20000A01 1 3 0 false mgwd_vservers_with_active_nis 00000d93.0000089d 243bc7cd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 1 0 0x20000A10 1 0 0 false DSMDB_NEXT_IMP - sharedStoragePoolNode 00000d93.0000089e 243bc7cd Thu Sep 26 2019 04:33:14 -06:00 [kern_mgwd:info:2284] 1 1 0x20000A01 1 0 0 false mgwd_get_blade_info 00000d93.0000089f 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] Queue Stats: 00000d93.000008a0 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] Priority Threads Active High Low Misses Spliced Spliced Depth Depth 00000d93.000008a1 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] Pri Pri MAX MAX 00000d93.000008a2 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] 0 3 1 0 0 521 434 1 1 1 00000d93.000008a3 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] 1 2 0 70 0 1093 21 1 0 1 00000d93.000008a4 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] 2 0 0 0 0 0 0 0 0 0 00000d93.000008a5 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] 3 0 0 0 0 0 0 0 0 0 00000d93.000008a6 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] 4 0 0 0 0 0 0 0 0 0 00000d93.000008a7 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] Program Stats: 00000d93.000008a8 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] Max Count Saturated PROG Recursive Active Max Sat Queue 00000d93.000008a9 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] Time Active Length 00000d93.000008aa 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] 3 23 0 0x20000A01 0 0 1 0 00000d93.000008ab 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] 0 0 0 0x20000A03 0 0 0 0 00000d93.000008ac 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] 0 0 0 0x20000A07 0 0 0 0 00000d93.000008ad 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] 0 0 0 0x20000A0E 0 0 0 0 00000d93.000008ae 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] 1332 394 0 0x20000A10 0 0 2 0 00000d93.000008af 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] 0 0 0 0x20000A11 0 0 0 0 00000d93.000008b0 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] 0 0 0 0x20000B20 0 0 0 0 00000d93.000008b1 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] 0 0 0 0x20000C00 0 0 0 0 00000d93.000008b2 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] 0 0 0 0x20000D20 0 0 0 0 00000d93.000008b3 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] 0 0 0 0x20000D21 0 0 0 0 00000d93.000008b4 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] 0 0 0 0x20000D22 0 0 0 0 00000d93.000008b5 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] 0 0 0 0x20000D23 0 0 0 0 00000d93.000008b6 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] 0 0 0 0x20000D24 0 0 0 0 00000d93.000008b7 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] Reused Transports: 372 00000d93.000008b8 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] Recursive Calls: 0 00000d93.000008b9 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] Completed: 417 00000d93.000008ba 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] Headers Read: 432 00000d93.000008bb 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] Arguments Fetched: 432 00000d93.000008bc 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] Dispatches Called: 417 00000d93.000008bd 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] Transport Rollover: 47 00000d93.000008be 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] Saturation Count: 0 00000d93.000008bf 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] Select Terminations: 39 00000d93.000008c0 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] Direct Dispatches: 0 00000d93.000008c1 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] Current Transports: 7 00000d93.000008c2 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] Longest Time To Dispatch: 6 00000d93.000008c3 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] Number of NULL RPCs: 15 00000d93.000008c4 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] Header ONLY Dispatches 0 00000d93.000008c5 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] Max Time All Threads in Use (ms): 0 00000d93.000008c6 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] 0x81b244700: 0: ERR: librpcpool: Dumping librpcpool non-RT mtrace buffer 00000d93.000008c7 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] Thu Sep 26 04:33:13 2019:src/mtsvc_dispatch_pool.cc:1610 (thr_id:0x81a804200) register_disp: program: 536874272 version: 1 logical_prog: 0 port: 7815 VSID=-3 LIFROLE=-1 IPv6=no entered 00000d93.000008c8 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] Thu Sep 26 04:33:13 2019:src/mtsvc_dispatch_pool.cc:921 (thr_id:0x81a804200) bind_server: port=7815 VSID=-3 LIFROLE=-1 IPv6=no Listen=yes Realtime=no called 00000d93.000008c9 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] Thu Sep 26 04:33:13 2019:src/mtsvc_dispatch_pool.cc:960 (thr_id:0x81a804200) bind_server: port=7815 VSID=-3 LIFROLE=-1 IPv6=no Listen=yes Realtime=no ipspace/port not setup, adding 00000d93.000008ca 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] Thu Sep 26 04:33:13 2019:src/mtsvc_dispatch_pool.cc:1034 (thr_id:0x81a804200) bind_server: port=7815 VSID=-3 LIFROLE=-1 IPv6=no Listen=yes Realtime=no Bound IPV4 address INADDR_ANY port 7815 00000d93.000008cb 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] Thu Sep 26 04:33:13 2019:src/mtsvc_dispatch_pool.cc:1056 (thr_id:0x81a804200) bind_server: port=7815 VSID=-3 LIFROLE=-1 IPv6=no Listen=yes Realtime=no Added IPv4/IPv6 port=7815 vsid=-3 lifrole=-1 00000d93.000008cc 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] Thu Sep 26 04:33:13 2019:src/mtsvc_dispatch_pool.cc:1610 (thr_id:0x81a804200) register_disp: program: 536874273 version: 1 logical_prog: 0 port: 7816 VSID=-3 LIFROLE=-1 IPv6=no entered 00000d93.000008cd 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] Thu Sep 26 04:33:13 2019:src/mtsvc_dispatch_pool.cc:921 (thr_id:0x81a804200) bind_server: port=7816 VSID=-3 LIFROLE=-1 IPv6=no Listen=yes Realtime=no called 00000d93.000008ce 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] Thu Sep 26 04:33:13 2019:src/mtsvc_dispatch_pool.cc:960 (thr_id:0x81a804200) bind_server: port=7816 VSID=-3 LIFROLE=-1 IPv6=no Listen=yes Realtime=no ipspace/port not setup, adding 00000d93.000008cf 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] Thu Sep 26 04:33:13 2019:src/mtsvc_dispatch_pool.cc:1034 (thr_id:0x81a804200) bind_server: port=7816 VSID=-3 LIFROLE=-1 IPv6=no Listen=yes Realtime=no Bound IPV4 address INADDR_ANY port 7816 00000d93.000008d0 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] Thu Sep 26 04:33:13 2019:src/mtsvc_dispatch_pool.cc:1056 (thr_id:0x81a804200) bind_server: port=7816 VSID=-3 LIFROLE=-1 IPv6=no Listen=yes Realtime=no Added IPv4/IPv6 port=7816 vsid=-3 lifrole=-1 00000d93.000008d1 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] Thu Sep 26 04:33:13 2019:src/mtsvc_dispatch_pool.cc:1610 (thr_id:0x81a804200) register_disp: program: 536874274 version: 1 logical_prog: 0 port: 7817 VSID=-3 LIFROLE=-1 IPv6=no entered 00000d93.000008d2 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] Thu Sep 26 04:33:13 2019:src/mtsvc_dispatch_pool.cc:921 (thr_id:0x81a804200) bind_server: port=7817 VSID=-3 LIFROLE=-1 IPv6=no Listen=yes Realtime=no called 00000d93.000008d3 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] Thu Sep 26 04:33:13 2019:src/mtsvc_dispatch_pool.cc:960 (thr_id:0x81a804200) bind_server: port=7817 VSID=-3 LIFROLE=-1 IPv6=no Listen=yes Realtime=no ipspace/port not setup, adding 00000d93.000008d4 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] Thu Sep 26 04:33:13 2019:src/mtsvc_dispatch_pool.cc:1034 (thr_id:0x81a804200) bind_server: port=7817 VSID=-3 LIFROLE=-1 IPv6=no Listen=yes Realtime=no Bound IPV4 address INADDR_ANY port 7817 00000d93.000008d5 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] Thu Sep 26 04:33:13 2019:src/mtsvc_dispatch_pool.cc:1056 (thr_id:0x81a804200) bind_server: port=7817 VSID=-3 LIFROLE=-1 IPv6=no Listen=yes Realtime=no Added IPv4/IPv6 port=7817 vsid=-3 lifrole=-1 00000d93.000008d6 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] Thu Sep 26 04:33:13 2019:src/mtsvc_dispatch_pool.cc:1610 (thr_id:0x81a804200) register_disp: program: 536874275 version: 1 logical_prog: 0 port: 7818 VSID=-3 LIFROLE=-1 IPv6=no entered 00000d93.000008d7 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] Thu Sep 26 04:33:13 2019:src/mtsvc_dispatch_pool.cc:921 (thr_id:0x81a804200) bind_server: port=7818 VSID=-3 LIFROLE=-1 IPv6=no Listen=yes Realtime=no called 00000d93.000008d8 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] Thu Sep 26 04:33:13 2019:src/mtsvc_dispatch_pool.cc:960 (thr_id:0x81a804200) bind_server: port=7818 VSID=-3 LIFROLE=-1 IPv6=no Listen=yes Realtime=no ipspace/port not setup, adding 00000d93.000008d9 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] Thu Sep 26 04:33:13 2019:src/mtsvc_dispatch_pool.cc:1034 (thr_id:0x81a804200) bind_server: port=7818 VSID=-3 LIFROLE=-1 IPv6=no Listen=yes Realtime=no Bound IPV4 address INADDR_ANY port 7818 00000d93.000008da 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] Thu Sep 26 04:33:13 2019:src/mtsvc_dispatch_pool.cc:1056 (thr_id:0x81a804200) bind_server: port=7818 VSID=-3 LIFROLE=-1 IPv6=no Listen=yes Realtime=no Added IPv4/IPv6 port=7818 vsid=-3 lifrole=-1 00000d93.000008db 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] Thu Sep 26 04:33:13 2019:src/mtsvc_dispatch_pool.cc:1610 (thr_id:0x81a804200) register_disp: program: 536874276 version: 1 logical_prog: 0 port: 7819 VSID=-3 LIFROLE=-1 IPv6=no entered 00000d93.000008dc 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] Thu Sep 26 04:33:13 2019:src/mtsvc_dispatch_pool.cc:921 (thr_id:0x81a804200) bind_server: port=7819 VSID=-3 LIFROLE=-1 IPv6=no Listen=yes Realtime=yes called 00000d93.000008dd 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] Thu Sep 26 04:33:13 2019:src/mtsvc_dispatch_pool.cc:960 (thr_id:0x81a804200) bind_server: port=7819 VSID=-3 LIFROLE=-1 IPv6=no Listen=yes Realtime=yes ipspace/port not setup, adding 00000d93.000008de 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] Thu Sep 26 04:33:13 2019:src/mtsvc_dispatch_pool.cc:1034 (thr_id:0x81a804200) bind_server: port=7819 VSID=-3 LIFROLE=-1 IPv6=no Listen=yes Realtime=yes Bound IPV4 address INADDR_ANY port 7819 00000d93.000008df 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] Thu Sep 26 04:33:13 2019:src/mtsvc_dispatch_pool.cc:1056 (thr_id:0x81a804200) bind_server: port=7819 VSID=-3 LIFROLE=-1 IPv6=no Listen=yes Realtime=yes Added IPv4/IPv6 port=7819 vsid=-3 lifrole=-1 00000d93.000008e0 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] Thu Sep 26 04:33:13 2019:src/mtsvc_dispatch_pool.cc:1610 (thr_id:0x81a804200) register_disp: program: 536873984 version: 1 logical_prog: 0 port: 910 VSID=-2 LIFROLE=-1 IPv6=no entered 00000d93.000008e1 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] Thu Sep 26 04:33:13 2019:src/mtsvc_dispatch_pool.cc:921 (thr_id:0x81a804200) bind_server: port=910 VSID=-2 LIFROLE=-1 IPv6=no Listen=yes Realtime=no called 00000d93.000008e2 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] Thu Sep 26 04:33:13 2019:src/mtsvc_dispatch_pool.cc:950 (thr_id:0x81a804200) bind_server: port=910 VSID=-2 LIFROLE=-1 IPv6=no Listen=yes Realtime=no ipspace/port already setup: port=910 vsid=-2 lifrole=-1 00000d93.000008e3 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] Thu Sep 26 04:33:14 2019:src/disp_processor.cc:441 (thr_id:0x81b7bc900) MAX DISPATCHED delta:6 prio:0 old_max_dispatch:2 00000d93.000008e4 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] 0x81b244700: 0: ERR: librpcpool: Dumping librpcpool RT mtrace buffer 00000d93.000008e5 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] 0x81b244700: 0: ERR: librpcpool: * 00000d93.000008e6 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:debug:2284] upgrademgr_event_callback::database_online_callback - invoked; [ANDU_LOG at src/tables/cluster_image.cc:6583 in 'database_online_callback'] 00000d93.000008e7 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] 0x81b244700: 0: ERR: librpcpool: * 00000d93.000008e8 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] 0x81b244700: 0: ERR: librpcpool: ******************************************************************************** 00000d93.000008e9 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: INFO: SFO_Ready|Start_Stop_Rdb_Clients: start_rdb_clients:104 start mhost apps worked, output Starting mhost apps 00000d93.000008ea 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] Starting rdb apps 00000d93.000008eb 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] Starting mhost apps 00000d93.000008ec 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] Starting rdb apps 00000d93.000008ed 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: INFO: SFO_Ready|Start_Stop_Rdb_Clients: start_rdb_clients:108 finish mhost apps 00000d93.000008ee 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] mhost_boot: rdb/mhost apps end 00000d93.000008ef 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] mhost_boot: mlog reader server start 00000d93.000008f0 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] mhost_boot: mlog reader server end 00000d93.000008f1 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] 0x81a9f1d00: 0: INFO: nonshared_clus_online_cb: join_unjoin_work:src/cluster_setup/nonshared_clus_online_cb.cc:233 UNIT_IS_OFFLINE, ignoring callback 00000d93.000008f2 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] 0x81a9f1d00: 0: INFO: nonshared_clus_online_cb: eligibility_change_work:src/cluster_setup/nonshared_clus_online_cb.cc:265 UNIT_IS_OFFLINE, ignoring eligibility work callback 00000d93.000008f3 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] snapshot_policy: online status callback status=2 00000d93.000008f4 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] 0x81a9f1d00: 0: INFO: vserverdr_failover: src/tables/vserverdr/svmdr/vserverdr_failover.cc:87 on_online_status_cb In online callback 00000d93.000008f5 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] 0x81a9f1d00: 0: ERR: FlexCache: **************online status callback status=2********** 00000d93.000008f6 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] 0x81a9f1d00: 0: ERR: TABLES::COPY_OFFLOAD: src/tables/copy_offload.cc:onlineEvent:503 Unit is still offline 00000d93.000008f7 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] 0x81b517b00: 0: ERR: rdb_tran_glue: create: tid=0x81b517b00 failed to create transaction for label='Upgrade EC QoS task : external_cache_create_preset_cache_rows': UNIT_OFFLINE 00000d93.000008f8 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] 0x81b517b00: 0: ERR: EC: EC QoS upgrade task failed to create a transaction. Error: Node "nmg-fas8040-02" on ring "Management" is offline. Check the health of the cluster using the "cluster show" command. For further assistance, contact technical support. 00000d93.000008f9 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] 0x81b518000: 0: NOTICE: RpcConnectionCache: markFailure: markFailure: New Quarantine for 127.0.0.1:536873548(pipd)/1[tcp]{fffffffd|8} with timeout(5.000s) Reason:: RPC: Program not registered 00000d93.000008fa 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] A [src/rdb/HAM.cc 1233 (0x81a804200)]: checkHASFOConfigVerbose: haConfigured: True, result HA_AGENT_CONFIGURED. 00000d93.000008fb 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] mhost_boot: web services database start 00000d93.000008fc 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] mhost_boot: web services database end 00000d93.000008fd 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] mhost_boot: cm cm_init start 00000d93.000008fe 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] 0x81b7bbf00: 0: INFO: SFO_Ready|Cluster_Apps: wait_for_apps_online (waiter: Configure mdnsd) apps: vifmgr, maxTime 0 sec: starting 00000d93.000008ff 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: INFO: librpcpool: bind_server: port=910 VSID=-2 LIFROLE=-1 IPv6=no Listen=yes Realtime=no ipspace/port already setup: port=910 vsid=-2 lifrole=-1 00000d93.00000900 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] mhost_boot: cm cm_init end 00000d93.00000901 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] mhost_boot: query start 00000d93.00000902 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] mhost_boot: query end 00000d93.00000903 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: INFO: MGWD::PRECLUSTER_CERT_MGMT: create_ssl_cert:src/precluster_cert_mgmt/precluster_cert_mgmt.cc:172 create_ssl_cert: SSL certificates exist, no need to create more. 00000d93.00000904 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] mhost_boot: start initialize auto cluster lifs 00000d93.00000905 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] mhost_boot: end initialize auto cluster lifs 00000d93.00000906 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] 0x81a9f1d00: 0: INFO: capability_kernel: online status callback capability_kernel status=2 00000d93.00000907 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] 0x81a9f1d00: 0: INFO: MGWD::NDO: Processing RDB online callback. Online status is Offline. 00000d93.00000908 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] ************trigger_lif_modify_update 2761LIF PLACEMENT THREAD: LIF MODIFY UPDATE************* 00000d93.00000909 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] 0x81b518000: 0: INFO: SFO_Ready|Cluster_Apps: wait_for_apps_online (waiter: LIF_PLACEMENT_THREAD) apps: vifmgr bcomd, maxTime 0 sec: starting 00000d93.0000090a 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] ************handle_lif_placement 2669LIF PLACEMENT THREAD: HANDLE LIF PLACEMENT************* 00000d93.0000090b 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] 0x81b51a300: 0: INFO: SFO_Ready|Cluster_Apps: wait_for_apps_online (waiter: LIF_PLACEMENT_THREAD) apps: vifmgr bcomd, maxTime 0 sec: starting 00000d93.0000090c 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] 0x81b518000: 0: NOTICE: RpcConnectionCache: markFailure: markFailure: New Quarantine for 127.0.0.1:536873504(bcomd)/1[tcp]{fffffffd|8} with timeout(5.000s) Reason:: RPC: Program not registered 00000d93.0000090d 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] upgrademgr_event_callback::run - UNIT_IS_OFFLINE; [ANDU_LOG at src/tables/cluster_image.cc:6445 in 'run'] 00000d93.0000090f 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] 0x81a804200: 0: INFO: fbsd_stack_options: Finished initialize optionslist with 34 entries. 00000d93.00000910 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] mgwd IS READY FOR ACTION! 00000d93.00000911 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] mhost_boot: main end 00000d93.00000912 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] src/libnwd/unwd.cc:57 _call_sysctl : Wrote updated progress "" to oid debug.nodewatchdog.progress.mgwd. 00000d93.00000913 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] mhost_boot: StartTask: mainPostDatabases end 00000d93.00000914 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] mhost_boot: startup order execute end 00000d93.00000915 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] log_consolidated_times: :wall 0.030677 user 0.015625 system 0.023438 total 0.039062, entry: 0, consolidation method: main_pre_dblade 00000d93.00000916 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] log_consolidated_times: :wall 0.019351 user 0.031250 system 0.000000 total 0.031250, entry: 1, consolidation method: main_pre_dblade 00000d93.00000917 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] log_consolidated_times: :wall 0.550352 user 0.546875 system 0.000000 total 0.546875, entry: 2, consolidation method: main_pre_dblade 00000d93.00000918 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] log_consolidated_times: :wall 0.007881 user 0.007812 system 0.000000 total 0.007812, entry: 3, consolidation method: main_pre_dblade 00000d93.00000919 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] log_consolidated_times: :wall 0.001540 user 0.007812 system 0.000000 total 0.007812, entry: 4, consolidation method: main_pre_dblade 00000d93.0000091a 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] log_consolidated_times: :wall 439.485099 user 1.507812 system 0.500000 total 2.007812, entry: 0, consolidation method: main_dblade 00000d93.0000091b 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] log_consolidated_times: :wall 16.335083 user 1.296875 system 0.078125 total 1.375000, entry: 0, consolidation method: main_post_dblade 00000d93.0000091c 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] log_consolidated_times: :wall 0.000763 user 0.000000 system 0.000000 total 0.000000, entry: 1, consolidation method: main_post_dblade 00000d93.0000091d 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] log_consolidated_times: :wall 0.000019 user 0.000000 system 0.000000 total 0.000000, entry: 2, consolidation method: main_post_dblade 00000d93.0000091e 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] log_consolidated_times: :wall 0.000757 user 0.000000 system 0.000000 total 0.000000, entry: 3, consolidation method: main_post_dblade 00000d93.0000091f 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] log_consolidated_times: :wall 0.000906 user 0.000000 system 0.000000 total 0.000000, entry: 4, consolidation method: main_post_dblade 00000d93.00000920 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] log_consolidated_times: :wall 0.000343 user 0.000000 system 0.000000 total 0.000000, entry: 5, consolidation method: main_post_dblade 00000d93.00000921 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] log_consolidated_times: :wall 0.000006 user 0.000000 system 0.000000 total 0.000000, entry: 6, consolidation method: main_post_dblade 00000d93.00000922 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] log_consolidated_times: :wall 0.000148 user 0.000000 system 0.000000 total 0.000000, entry: 7, consolidation method: main_post_dblade 00000d93.00000923 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] log_consolidated_times: :wall 0.013836 user 0.000000 system 0.000000 total 0.000000, entry: 0, consolidation method: main_post_mroot 00000d93.00000924 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] log_consolidated_times: :wall 0.001592 user 0.000000 system 0.000000 total 0.000000, entry: 1, consolidation method: main_post_mroot 00000d93.00000925 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] log_consolidated_times: :wall 0.029382 user 0.000000 system 0.007812 total 0.007812, entry: 2, consolidation method: main_post_mroot 00000d93.00000926 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] log_consolidated_times: :wall 0.001755 user 0.000000 system 0.000000 total 0.000000, entry: 3, consolidation method: main_post_mroot 00000d93.00000927 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] log_consolidated_times: :wall 0.005464 user 0.000000 system 0.000000 total 0.000000, entry: 4, consolidation method: main_post_mroot 00000d93.00000928 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] log_consolidated_times: :wall 0.000141 user 0.000000 system 0.000000 total 0.000000, entry: 5, consolidation method: main_post_mroot 00000d93.00000929 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] log_consolidated_times: :wall 4.705342 user 0.875000 system 0.335938 total 1.210938, entry: 6, consolidation method: main_post_mroot 00000d93.0000092a 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] log_consolidated_times: :wall 0.000011 user 0.000000 system 0.000000 total 0.000000, entry: 7, consolidation method: main_post_mroot 00000d93.0000092b 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] log_consolidated_times: :wall 0.007578 user 0.007812 system 0.000000 total 0.007812, entry: 8, consolidation method: main_post_mroot 00000d93.0000092c 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] log_consolidated_times: :wall 0.000023 user 0.000000 system 0.000000 total 0.000000, entry: 9, consolidation method: main_post_mroot 00000d93.0000092d 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] log_consolidated_times: :wall 0.166935 user 0.078125 system 0.000000 total 0.078125, entry: 0, consolidation method: main_post_databases 00000d93.0000092e 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] log_consolidated_times: :wall 0.059621 user 0.062500 system 0.007812 total 0.070312, entry: 1, consolidation method: main_post_databases 00000d93.0000092f 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] log_consolidated_times: :wall 0.108998 user 0.140625 system 0.046875 total 0.187500, entry: 2, consolidation method: main_post_databases 00000d93.00000930 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] log_consolidated_times: :wall 1.016767 user 0.179688 system 0.234375 total 0.414062, entry: 3, consolidation method: main_post_databases 00000d93.00000931 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] log_consolidated_times: :wall 0.000022 user 0.000000 system 0.000000 total 0.000000, entry: 4, consolidation method: main_post_databases 00000d93.00000932 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] log_consolidated_times: :wall 0.000133 user 0.000000 system 0.000000 total 0.000000, entry: 5, consolidation method: main_post_databases 00000d93.00000933 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] log_consolidated_times: :wall 0.012130 user 0.000000 system 0.000000 total 0.000000, entry: 6, consolidation method: main_post_databases 00000d93.00000934 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] log_consolidated_times: :wall 0.115110 user 0.039062 system 0.000000 total 0.039062, entry: 7, consolidation method: main_post_databases 00000d93.00000935 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] log_consolidated_times: :wall 0.020328 user 0.007812 system 0.007812 total 0.015625, entry: 8, consolidation method: main_post_databases 00000d93.00000936 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] log_consolidated_times: :wall 0.000031 user 0.000000 system 0.000000 total 0.000000, entry: 9, consolidation method: main_post_databases 00000d93.00000937 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] log_consolidated_times: :wall 0.000012 user 0.000000 system 0.000000 total 0.000000, entry: 10, consolidation method: main_post_databases 00000d93.00000938 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] log_consolidated_times: :wall 0.000011 user 0.000000 system 0.000000 total 0.000000, entry: 11, consolidation method: main_post_databases 00000d93.00000939 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] log_consolidated_times: :wall 0.000011 user 0.000000 system 0.000000 total 0.000000, entry: 12, consolidation method: main_post_databases 00000d93.0000093a 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] log_consolidated_times: :wall 0.000099 user 0.000000 system 0.000000 total 0.000000, entry: 13, consolidation method: main_post_databases 00000d93.0000093b 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] log_consolidated_times: :wall 0.000073 user 0.000000 system 0.000000 total 0.000000, entry: 14, consolidation method: main_post_databases 00000d93.0000093c 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] log_consolidated_times: :wall 0.033352 user 0.007812 system 0.000000 total 0.007812, entry: 15, consolidation method: main_post_databases 00000d93.0000093d 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] log_consolidated_times: :wall 0.017084 user 0.007812 system 0.000000 total 0.007812, entry: 16, consolidation method: main_post_databases 00000d93.0000093e 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] log_consolidated_times: :wall 0.000040 user 0.000000 system 0.000000 total 0.000000, entry: 17, consolidation method: main_post_databases 00000d93.0000093f 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] log_consolidated_times: :wall 0.000052 user 0.000000 system 0.000000 total 0.000000, entry: 18, consolidation method: main_post_databases 00000d93.00000940 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] log_consolidated_times: :wall 0.054529 user 0.015625 system 0.007812 total 0.023438, entry: 19, consolidation method: main_post_databases 00000d93.00000941 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] log_consolidated_times: :wall 0.000388 user 0.000000 system 0.000000 total 0.000000, entry: 20, consolidation method: main_post_databases 00000d93.00000942 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] log_consolidated_times: :wall 0.000716 user 0.000000 system 0.000000 total 0.000000, entry: 21, consolidation method: main_post_databases 00000d93.00000943 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] log_consolidated_times: :wall 0.000032 user 0.000000 system 0.000000 total 0.000000, entry: 22, consolidation method: main_post_databases 00000d93.00000944 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] log_consolidated_times: :wall 0.030582 user 0.007812 system 0.000000 total 0.007812, entry: 23, consolidation method: main_post_databases 00000d93.00000945 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] log_consolidated_times: :wall 0.000055 user 0.000000 system 0.000000 total 0.000000, entry: 24, consolidation method: main_post_databases 00000d93.00000946 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] log_consolidated_times: :wall 0.000012 user 0.000000 system 0.000000 total 0.000000, entry: 25, consolidation method: main_post_databases 00000d93.00000947 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] log_consolidated_times: :wall 0.128872 user 0.015625 system 0.007812 total 0.023438, entry: 26, consolidation method: main_post_databases 00000d93.00000948 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] log_consolidated_times: :wall 0.000289 user 0.000000 system 0.000000 total 0.000000, entry: 27, consolidation method: main_post_databases 00000d93.00000949 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] log_consolidated_times: :wall 0.010870 user 0.007812 system 0.000000 total 0.007812, entry: 28, consolidation method: main_post_databases 00000d93.0000094a 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] log_consolidated_times: :wall 0.000618 user 0.000000 system 0.000000 total 0.000000, entry: 29, consolidation method: main_post_databases 00000d93.0000094b 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] log_consolidated_times: :wall 0.000140 user 0.000000 system 0.000000 total 0.000000, entry: 30, consolidation method: main_post_databases 00000d93.0000094c 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] log_consolidated_times: :wall 0.005067 user 0.000000 system 0.000000 total 0.000000, entry: 31, consolidation method: main_post_databases 00000d93.0000094d 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] log_consolidated_times: :wall 0.000013 user 0.000000 system 0.000000 total 0.000000, entry: 32, consolidation method: main_post_databases 00000d93.0000094e 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] log_consolidated_times: :wall 0.002623 user 0.000000 system 0.000000 total 0.000000, entry: 33, consolidation method: main_post_databases 00000d93.0000094f 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] mhost_boot: startup SIGTERM enabled 00000d93.00000950 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] A [src/rdb/quorum/quorumimpl.cc 339 (0x81b7bc900)]: ensureBeaconOrder: sender 1000, voter incarnation 0 -> 4, voter beacon epoch 0 -> 6, voter beacon count 0 -> 7415 00000d93.00000951 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] A [src/rdb/quorum/qm_states/qm_out_waiting_for_votes.cc 81 (0x81b7bc900)]: WaitingForVotesState::state2: WS_WaitingForVotes -> WS_BecomingMember 00000d93.00000952 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] A [src/rdb/quorum/qm_states/qm_out_replying_out_of_quorum.cc 222 (0x81b7bc900)]: ReplyingOutOfQuorumState::stateUp2ReplyingOutOfQuorum: WS_WaitingForVotes -> WS_BecomingMember 00000d93.00000953 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] A [src/rdb/quorum/qm_states/qm_out_out_of_quorum.cc 61 (0x81b7bc900)]: OutOfQuorumState::stateUp2OutOfQuorum: WS_WaitingForVotes -> WS_BecomingMember 00000d93.00000954 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] A [src/rdb/quorum/qm_states/qm_state.cc 552 (0x81b7bc900)]: stateTrans: WS_WaitingForVotes -> WS_BecomingMember at: 60789754s 00000d93.00000955 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] A [src/rdb/quorum/quorumimpl.cc 1771 (0x81b7bc900)]: inContactUpcall QM Upcall status: Offline ==> Secondary Epoch: 0 => 6 isFastPath 1 isFastPathOverride 0 membershipDisabled: 0 00000d93.00000956 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] A [src/rdb/unit_local.cc 1547 (0x81b7bc900)]: setEpoch: unit file epoch 5 => 6 00000d93.00000957 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] A [src/rdb/quorum/qm_states/qm_in_becoming_secondary.cc 75 (0x81b7bc900)]: BecomingSecondaryState::state2: WS_BecomingMember -> WS_QuorumMember 00000d93.00000958 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] A [src/rdb/quorum/qm_states/qm_state.cc 552 (0x81b7bc900)]: stateTrans: WS_BecomingMember -> WS_QuorumMember at: 60789754s 00000d93.00000959 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] A [src/rdb/unit_local.cc 1753 (0x81b244c00)]: Begin local reconciliation from <5,981471> to <6,58>. 00000d93.0000095b 243bc7cd Thu Sep 26 2019 04:33:15 -06:00 [kern_mgwd:info:2284] 0x81b244c00: 0: NOTICE: librpcpool: MAX DISPATCHED delta:13 prio:0 old_max_dispatch:6 00000d93.00000963 243bc7cd Thu Sep 26 2019 04:33:16 -06:00 [kern_mgwd:info:2284] 0x81b7bce00: 0: NOTICE: librpcpool: MAX DISPATCHED delta:24 prio:1 old_max_dispatch:13 00000d93.000009d8 243bc7d4 Thu Sep 26 2019 04:33:16 -06:00 [kern_mgwd:info:2284] 0x81b24ab00: 0: NOTICE: mgwd: Starting replay for hashd-app 00000d93.000009d9 243bc7d4 Thu Sep 26 2019 04:33:16 -06:00 [kern_mgwd:info:2284] 0x81b24ab00: 0: NOTICE: mgwd: MGWD not ready for replay. Replay will be triggered automatically once it is. No actiontaken for now... 00000d93.000009ee 243bc7d4 Thu Sep 26 2019 04:33:16 -06:00 [kern_mgwd:info:2284] Error: Loop detected in next() for table kern_hotplug. Next on "nmg-fas8040-02 1" returned "nmg-fas8040-02 1". 00000d93.00000a2e 243bc7d4 Thu Sep 26 2019 04:33:16 -06:00 [kern_mgwd:info:2284] A [src/rdb/unit_local.cc 2181 (0x81b244c00)]: Completed local reconciliation to <6,58> (canRollback: true). 00000d93.00000a2f 243bc7d4 Thu Sep 26 2019 04:33:16 -06:00 [kern_mgwd:info:2284] A [src/rdb/TM.cc 1721 (0x81b244200)]: TM 1001: change role at epoch 0 to 0x40 recovery transaction number 60 00000d93.00000a30 243bc7d4 Thu Sep 26 2019 04:33:16 -06:00 [kern_mgwd:info:2284] A [src/rdb/TM.cc 1721 (0x81b244200)]: TM 1001: change role at epoch 6 to 0x220 recovery transaction number 60 00000d93.00000a31 243bc7d4 Thu Sep 26 2019 04:33:16 -06:00 [kern_mgwd:info:2284] A [src/rdb/TM.cc 1486 (0x81b244200)]: TM 1001: Report UNIT_IS_SECONDARY (epoch 6, master 1000). Reason: process startup. 00000d93.00000a32 243bc7d4 Thu Sep 26 2019 04:33:16 -06:00 [kern_mgwd:info:2284] A [src/rdb/TM.cc 1491 (0x81b244200)]: FAILOVER rdb: Local unit Management online 00000d93.00000a33 243bc7d4 Thu Sep 26 2019 04:33:16 -06:00 [kern_mgwd:info:2284] A [src/rdb/cluster_events.cc 78 (0x81b244200)]: Cluster event: node-event, epoch 6, site 1001 [local node online (secondary): process startup]. 00000d93.00000a34 243bc7d4 Thu Sep 26 2019 04:33:16 -06:00 [kern_mgwd:info:2284] Notice: online_status_callback: RDB unit is online as Secondary 00000d93.00000a35 243bc7d4 Thu Sep 26 2019 04:33:16 -06:00 [kern_mgwd:info:2284] 0x81d247b00: 0: INFO: RDB::callback::registrar: callback:src/rdb/rdb_online_registrar.cc:81 rdb_callbacks::ONLINE::BEGIN:: 60789766991 00000d93.00000a36 243bc7d4 Thu Sep 26 2019 04:33:16 -06:00 [kern_mgwd:info:2284] Info: check wait for RDB unit to be OFFLINE0x81d247b00: 0: INFO: RDB::callback::registrar: callback:src/rdb/rdb_online_registrar.cc:106 rdb_callbacks::ONLINE::END:: 60789766991 00000d93.00000a37 243bc7d4 Thu Sep 26 2019 04:33:16 -06:00 [kern_mgwd:info:2284] 0x81b13fb00: 0: INFO: SFO_Ready|Cluster_Apps: Startup stability test: management online. 00000d93.00000a38 243bc7d4 Thu Sep 26 2019 04:33:16 -06:00 [kern_mgwd:info:2284] mhost_boot: MHOST_SF: Stability MGWD online 00000d93.00000a39 243bc7d4 Thu Sep 26 2019 04:33:16 -06:00 [kern_mgwd:info:2284] 0x81b13fb00: 0: INFO: SFO_Ready|Cluster_Apps: Startup stability test: wait until CAPM initialized. 00000d93.00000a3a 243bc7d4 Thu Sep 26 2019 04:33:16 -06:00 [kern_mgwd:info:2284] mhost_boot: cap check wait for unit online end 00000d93.00000a3b 243bc7d4 Thu Sep 26 2019 04:33:16 -06:00 [kern_mgwd:info:2284] mhost_boot: cap check start 00000d93.00000a3c 243bc7d4 Thu Sep 26 2019 04:33:16 -06:00 [kern_mgwd:info:2284] A [src/rdb/HAM.cc 1571 (0x81d247100)]: HAM: Entering HA-normal. 00000d93.00000a3d 243bc7d4 Thu Sep 26 2019 04:33:16 -06:00 [kern_mgwd:info:2284] Notice: _initLicenseJob: Ensure license daemon is active: succeeded. 00000d93.00000a3e 243bc7d4 Thu Sep 26 2019 04:33:16 -06:00 [kern_mgwd:info:2284] 0x81a9f1d00: 0: INFO: license_v2: onlineInitCallback: RDB online callback, status= 1 00000d93.00000a3f 243bc7d4 Thu Sep 26 2019 04:33:16 -06:00 [kern_mgwd:info:2284] Notice: _initUnjunctionedCleanerJob: Ensure UnjunctionedCleaner daemon is active: succeeded. 00000d93.00000a40 243bc7d4 Thu Sep 26 2019 04:33:16 -06:00 [kern_mgwd:info:2284] 0x81b245100: 0: ERR: RPCServer: mgwd_ready_for_replay: vifmgr is not ready (state: unknown) 00000d93.00000a48 243bc7d4 Thu Sep 26 2019 04:33:16 -06:00 [kern_mgwd:info:2284] Notice: _initLicMonitorJob: License Monitor job successfully queued. 00000d93.00000a49 243bc7d4 Thu Sep 26 2019 04:33:16 -06:00 [kern_mgwd:info:2284] 0x81b141e00: 0: NOTICE: ECV: cluster_version_replicated_createRow:src/tables/ecv/ecv.cc:347 Modifying node record for ECV 00000d93.00000a4a 243bc7d4 Thu Sep 26 2019 04:33:16 -06:00 [kern_mgwd:info:2284] ASUP_CLUSTER_INFO: update_request(online callback: status:1) 00000d93.00000a4b 243bc7d4 Thu Sep 26 2019 04:33:16 -06:00 [kern_mgwd:info:2284] 0x81a9f1d00: 0: INFO: nonshared_clus_online_cb: join_unjoin_work:src/cluster_setup/nonshared_clus_online_cb.cc:238 Not a nonshared ha platform, nothing to do 00000d93.00000a4c 243bc7d4 Thu Sep 26 2019 04:33:16 -06:00 [kern_mgwd:info:2284] Info: _queueMFUsageJob: Successfully queued managed feature usage collection job. Job id = fac3c78b-b80c-11e7-964b-00a098573067 00000d93.00000a4d 243bc7d4 Thu Sep 26 2019 04:33:16 -06:00 [kern_mgwd:info:2284] Notice: _initMFUsageJob: Managed Feature Usage collection job successfully queued. 00000d93.00000a4e 243bc7d4 Thu Sep 26 2019 04:33:16 -06:00 [kern_mgwd:info:2284] 0x81a9f1d00: 0: INFO: nonshared_clus_online_cb: eligibility_change_work:src/cluster_setup/nonshared_clus_online_cb.cc:271 Eligibility work: Not a nonshared ha platform, nothing to do 00000d93.00000a50 243bc7d4 Thu Sep 26 2019 04:33:16 -06:00 [kern_mgwd:info:2284] Info: _queueMFUsageCleanupJob: Successfully queued managed feature usage cleanup job. Job id = 176c80e9-b79e-11e7-9b07-00a09857ee8f 00000d93.00000a52 243bc7d4 Thu Sep 26 2019 04:33:16 -06:00 [kern_mgwd:info:2284] Notice: _initMFUsageCleanupJob: Managed Feature Usage cleanup job successfully queued. 00000d93.00000a53 243bc7d4 Thu Sep 26 2019 04:33:16 -06:00 [kern_mgwd:info:2284] Notice: _initUnjunctionedCleanerJob: Ensure UnjunctionedCleaner daemon is active: succeeded. 00000d93.00000a60 243bc7d4 Thu Sep 26 2019 04:33:16 -06:00 [kern_mgwd:info:2284] 0x81b0a4d00: 0: NOTICE: ECVCache: ECV set to current ONTAP version: true 00000d93.00000a63 243bc7d4 Thu Sep 26 2019 04:33:16 -06:00 [kern_mgwd:info:2284] ASUP_CLUSTER_INFO: Cache update complete. Next timeout is 86400 seconds 00000d93.00000a67 243bc7d5 Thu Sep 26 2019 04:33:16 -06:00 [kern_mgwd:info:2284] Debug: sync_rdb_and_cdb_iters: Calling next() on rdb and cdb iterators 00000d93.00000a68 243bc7d5 Thu Sep 26 2019 04:33:16 -06:00 [kern_mgwd:info:2284] 0x81b0a4d00: 0: WARNING: ECVCache::MONITOR: src/tables/ECV_Monitor.cc:publishMain:248 Failed to update bcomd with ECV (will retry): RPC: Program not registered [from mgwd on node "nmg-fas8040-02" (VSID: -3) to bcomd at 127.0.0.1] 00000d93.00000a69 243bc7d5 Thu Sep 26 2019 04:33:16 -06:00 [kern_mgwd:info:2284] Debug: sync_rdb_and_cdb_iters: Empty_rdb=false rdb_cmp_cdb=0 00000d93.00000a6a 243bc7d5 Thu Sep 26 2019 04:33:16 -06:00 [kern_mgwd:info:2284] Debug: sync_rdb_and_cdb_iters: Calling next() on rdb and cdb iterators 00000d93.00000a6b 243bc7d5 Thu Sep 26 2019 04:33:16 -06:00 [kern_mgwd:info:2284] Debug: sync_rdb_and_cdb_iters: Empty_rdb=false rdb_cmp_cdb=0 00000d93.00000a6c 243bc7d5 Thu Sep 26 2019 04:33:16 -06:00 [kern_mgwd:info:2284] Debug: sync_rdb_and_cdb_iters: Calling next() on rdb and cdb iterators 00000d93.00000a6d 243bc7d5 Thu Sep 26 2019 04:33:16 -06:00 [kern_mgwd:info:2284] Debug: sync_rdb_and_cdb_iters: Empty_rdb=false rdb_cmp_cdb=0 00000d93.00000a6e 243bc7d5 Thu Sep 26 2019 04:33:16 -06:00 [kern_mgwd:info:2284] Debug: sync_rdb_and_cdb_iters: Calling next() on rdb and cdb iterators 00000d93.00000a70 243bc7d5 Thu Sep 26 2019 04:33:16 -06:00 [kern_mgwd:info:2284] 0x81a9f1d00: 0: ERR: MGWD::DNS: src/tables/name_services/dns.cc:1756:update_disablesoa_bootarg ERROR: failed to set kenv "bootarg.nameservices.DNSLB_Disable_SOA" to NULL due to No such file or directory 00000d93.00000a72 243bc7d5 Thu Sep 26 2019 04:33:16 -06:00 [kern_mgwd:info:2284] 0x81a9f1d00: 0: INFO: Autosupport::Config: Successfully updated ASUP cert-directory to (/mroot/etc/vserver_4294967295/certificates/ssl/serverCa) for the node nmg-fas8040-02 00000d93.00000a73 243bc7d5 Thu Sep 26 2019 04:33:16 -06:00 [kern_mgwd:info:2284] 0x81a9f1d00: 0: INFO: capability_kernel: online status callback capability_kernel status=1 00000d93.00000a74 243bc7d5 Thu Sep 26 2019 04:33:16 -06:00 [kern_mgwd:info:2284] 0x81a9f1d00: 0: INFO: capability_kernel: Updating kernel with available capabilities 00000d93.00000a75 243bc7d5 Thu Sep 26 2019 04:33:16 -06:00 [kern_mgwd:info:2284] 0x81d248000: 0: NOTICE: RpcConnectionCache: clearFailure: called for 127.0.0.1:536873522(mntsvc)/1[tcp]{fffffffd|8} 00000d93.00000a77 243bc7d5 Thu Sep 26 2019 04:33:16 -06:00 [kern_mgwd:info:2284] 0x81d248000: 0: NOTICE: rdb_sitelist: slpush: successfully pushed site information to _push_sitelist_id[17] nodelist_status non_authoritative 00000d93.00000a78 243bc7d5 Thu Sep 26 2019 04:33:16 -06:00 [kern_mgwd:info:2284] 0x81b515d00: 0: INFO: capability_kernel: Updating DATA_ONTAP.8.2.0 state 00000d93.00000a79 243bc7d5 Thu Sep 26 2019 04:33:16 -06:00 [kern_mgwd:info:2284] 0x81b515d00: 0: INFO: capability_kernel: Successfully updated DATA_ONTAP.8.2.0 state 00000d93.00000a7a 243bc7d5 Thu Sep 26 2019 04:33:16 -06:00 [kern_mgwd:info:2284] 0x81b515d00: 0: INFO: capability_kernel: Updating DATA_ONTAP.8.2.1 state 00000d93.00000a7b 243bc7d5 Thu Sep 26 2019 04:33:16 -06:00 [kern_mgwd:info:2284] 0x81b515d00: 0: INFO: capability_kernel: Successfully updated DATA_ONTAP.8.2.1 state 00000d93.00000a7c 243bc7d5 Thu Sep 26 2019 04:33:16 -06:00 [kern_mgwd:info:2284] 0x81b515d00: 0: INFO: capability_kernel: Updating DATA_ONTAP.8.2.2 state 00000d93.00000a7d 243bc7d5 Thu Sep 26 2019 04:33:16 -06:00 [kern_mgwd:info:2284] 0x81b515d00: 0: INFO: capability_kernel: Successfully updated DATA_ONTAP.8.2.2 state 00000d93.00000a7e 243bc7d5 Thu Sep 26 2019 04:33:16 -06:00 [kern_mgwd:info:2284] 0x81b515d00: 0: INFO: capability_kernel: Updating DATA_ONTAP.8.2.3 state 00000d93.00000a7f 243bc7d5 Thu Sep 26 2019 04:33:16 -06:00 [kern_mgwd:info:2284] 0x81b515d00: 0: INFO: capability_kernel: Successfully updated DATA_ONTAP.8.2.3 state 00000d93.00000a80 243bc7d5 Thu Sep 26 2019 04:33:16 -06:00 [kern_mgwd:info:2284] 0x81b515d00: 0: INFO: capability_kernel: Updating DATA_ONTAP.8.2.4 state 00000d93.00000a81 243bc7d5 Thu Sep 26 2019 04:33:16 -06:00 [kern_mgwd:info:2284] 0x81b515d00: 0: INFO: capability_kernel: Successfully updated DATA_ONTAP.8.2.4 state 00000d93.00000a82 243bc7d5 Thu Sep 26 2019 04:33:16 -06:00 [kern_mgwd:info:2284] 0x81b515d00: 0: INFO: capability_kernel: Updating DATA_ONTAP.8.3.0 state 00000d93.00000a83 243bc7d5 Thu Sep 26 2019 04:33:16 -06:00 [kern_mgwd:info:2284] 0x81b515d00: 0: INFO: capability_kernel: Successfully updated DATA_ONTAP.8.3.0 state 00000d93.00000a84 243bc7d5 Thu Sep 26 2019 04:33:16 -06:00 [kern_mgwd:info:2284] 0x81b515d00: 0: INFO: capability_kernel: Updating DATA_ONTAP.8.3.0I1 state 00000d93.00000a85 243bc7d5 Thu Sep 26 2019 04:33:16 -06:00 [kern_mgwd:info:2284] A [src/rdb/HAM.cc 1233 (0x81a9f1d00)]: checkHASFOConfigVerbose: haConfigured: True, result HA_AGENT_CONFIGURED. 00000d93.00000a86 243bc7d5 Thu Sep 26 2019 04:33:16 -06:00 [kern_mgwd:info:2284] 0x81b515d00: 0: INFO: capability_kernel: Successfully updated DATA_ONTAP.8.3.0I1 state 00000d93.00000a87 243bc7d5 Thu Sep 26 2019 04:33:16 -06:00 [kern_mgwd:info:2284] 0x81b515d00: 0: INFO: capability_kernel: Updating DATA_ONTAP.8.3.1 state 00000d93.00000a88 243bc7d5 Thu Sep 26 2019 04:33:16 -06:00 [kern_mgwd:debug:2284] upgrademgr_event_callback::database_online_callback - invoked; [ANDU_LOG at src/tables/cluster_image.cc:6583 in 'database_online_callback'] 00000d93.00000a89 243bc7d5 Thu Sep 26 2019 04:33:16 -06:00 [kern_mgwd:info:2284] upgrademgr_event_callback::run - UNIT_IS_SECONDARY; [ANDU_LOG at src/tables/cluster_image.cc:6443 in 'run'] 00000d93.00000a8a 243bc7d5 Thu Sep 26 2019 04:33:16 -06:00 [kern_mgwd:info:2284] 0x81dac9c00: 0: ALERT: Table::vserver: flush_clusfs_cache 2404 mgwd online notification sent to clusfs. Action: 2, vsID: -1, name: nmg-fas8040, length: 14 00000d93.00000a8b 243bc7e3 Thu Sep 26 2019 04:33:16 -06:00 [kern_mgwd:info:2284] 0x81b515d00: 0: INFO: capability_kernel: Successfully updated DATA_ONTAP.8.3.1 state 00000d93.00000a8c 243bc7e3 Thu Sep 26 2019 04:33:16 -06:00 [kern_mgwd:info:2284] 0x81b515d00: 0: INFO: capability_kernel: Updating DATA_ONTAP.8.3.2 state 00000d93.00000a8d 243bc7e3 Thu Sep 26 2019 04:33:16 -06:00 [kern_mgwd:info:2284] 0x81b515d00: 0: INFO: capability_kernel: Successfully updated DATA_ONTAP.8.3.2 state 00000d93.00000a8e 243bc7e3 Thu Sep 26 2019 04:33:16 -06:00 [kern_mgwd:info:2284] 0x81b515d00: 0: INFO: capability_kernel: Updating DATA_ONTAP.8.3.3 state 00000d93.00000a8f 243bc7e3 Thu Sep 26 2019 04:33:16 -06:00 [kern_mgwd:info:2284] 0x81b515d00: 0: INFO: capability_kernel: Successfully updated DATA_ONTAP.8.3.3 state 00000d93.00000a90 243bc7e3 Thu Sep 26 2019 04:33:16 -06:00 [kern_mgwd:info:2284] 0x81b515d00: 0: INFO: capability_kernel: Updating DATA_ONTAP.8.4.0 state 00000d93.00000a91 243bc7e3 Thu Sep 26 2019 04:33:16 -06:00 [kern_mgwd:info:2284] 0x81b515d00: 0: INFO: capability_kernel: Successfully updated DATA_ONTAP.8.4.0 state 00000d93.00000a92 243bc7e3 Thu Sep 26 2019 04:33:16 -06:00 [kern_mgwd:info:2284] 0x81b515d00: 0: INFO: capability_kernel: Updating DATA_ONTAP.8.4.1 state 00000d93.00000a93 243bc7e3 Thu Sep 26 2019 04:33:16 -06:00 [kern_mgwd:info:2284] 0x81b515d00: 0: INFO: capability_kernel: Successfully updated DATA_ONTAP.8.4.1 state 00000d93.00000a94 243bc7e3 Thu Sep 26 2019 04:33:16 -06:00 [kern_mgwd:info:2284] 0x81b515d00: 0: INFO: capability_kernel: Updating DATA_ONTAP.9.0.0 state 00000d93.00000a95 243bc7e3 Thu Sep 26 2019 04:33:16 -06:00 [kern_mgwd:info:2284] 0x81b515d00: 0: INFO: capability_kernel: Successfully updated DATA_ONTAP.9.0.0 state 00000d93.00000a96 243bc7e3 Thu Sep 26 2019 04:33:16 -06:00 [kern_mgwd:info:2284] 0x81b515d00: 0: INFO: capability_kernel: Updating DATA_ONTAP.9.1.0.RC2 state 00000d93.00000a97 243bc7e3 Thu Sep 26 2019 04:33:16 -06:00 [kern_mgwd:info:2284] 0x81b515d00: 0: INFO: capability_kernel: Successfully updated DATA_ONTAP.9.1.0.RC2 state 00000d93.00000a98 243bc7e3 Thu Sep 26 2019 04:33:16 -06:00 [kern_mgwd:info:2284] 0x81b515d00: 0: INFO: capability_kernel: Updating DATA_ONTAP.9.2.0 state 00000d93.00000a99 243bc7e3 Thu Sep 26 2019 04:33:16 -06:00 [kern_mgwd:info:2284] 0x81b515d00: 0: INFO: capability_kernel: Successfully updated DATA_ONTAP.9.2.0 state 00000d93.00000aed 243bc7e3 Thu Sep 26 2019 04:33:17 -06:00 [kern_mgwd:info:2284] 0x81b13fb00: 0: INFO: SFO_Ready|Cluster_Apps: Startup stability test: CAPM initialized. 00000d93.00000aee 243bc7e3 Thu Sep 26 2019 04:33:17 -06:00 [kern_mgwd:info:2284] 0x81b13fb00: 0: INFO: SFO_Ready|Cluster_Apps: Startup stability test: wait until BCOMD initialized. 00000d93.00000aef 243bc7e3 Thu Sep 26 2019 04:33:17 -06:00 [kern_mgwd:info:2284] 0x81b518500: 0: INFO: SFO_Ready|Cluster_Apps: wait_for_apps_online (waiter: RECONF_CLUSTER_PROXY_THREAD) apps: mgwd vldb vifmgr bcomd crs, maxTime 60 sec: starting 00000d93.00000af0 243bc7e3 Thu Sep 26 2019 04:33:17 -06:00 [kern_mgwd:info:2284] 0x81b518500: 0: INFO: SFO_Ready|Cluster_Apps: wait_for_apps_online (RECONF_CLUSTER_PROXY_THREAD): mgwd is online. Number of tries: 1. 00000d93.00000af1 243bc7e3 Thu Sep 26 2019 04:33:17 -06:00 [kern_mgwd:info:2284] 0x81b13fb00: 0: INFO: SFO_Ready|Cluster_Apps: Startup stability test: BCOMD initialized. 00000d93.00000af2 243bc7e3 Thu Sep 26 2019 04:33:17 -06:00 [kern_mgwd:info:2284] 0x81b13fb00: 0: INFO: SFO_Ready|Cluster_Apps: Startup stability test: wait until CRS initialized. 00000d93.00000af3 243bc7e3 Thu Sep 26 2019 04:33:17 -06:00 [kern_mgwd:info:2284] 0x81b516c00: 0: ERR: MGWD:composite_aggr_upgrade_revert: FabricPool capability already enabled. 00000d93.00000af4 243bc7e3 Thu Sep 26 2019 04:33:17 -06:00 [kern_mgwd:info:2284] mhost_boot: cap check end 00000d93.00000af5 243bc7e3 Thu Sep 26 2019 04:33:17 -06:00 [kern_mgwd:info:2284] 0x81b13fb00: 0: INFO: SFO_Ready|Cluster_Apps: Startup stability test: CRS initialized. 00000d93.00000af6 243bc7e3 Thu Sep 26 2019 04:33:17 -06:00 [kern_mgwd:info:2284] 0x81b13fb00: 0: INFO: SFO_Ready|Cluster_Apps: Startup stability test: waiting for mhost application stability. 00000d93.00000af7 243bc7e3 Thu Sep 26 2019 04:33:17 -06:00 [kern_mgwd:info:2284] mhost_boot: MHOST_SF: Stability Apps start 00000d93.00000af8 243bc7e3 Thu Sep 26 2019 04:33:17 -06:00 [kern_mgwd:info:2284] 0x81b13fb00: 0: INFO: SFO_Ready|Cluster_Apps: wait_for_apps_online (waiter: Startup stability test) apps: mgwd vldb vifmgr bcomd crs, maxTime 0 sec: starting 00000d93.00000af9 243bc7e3 Thu Sep 26 2019 04:33:17 -06:00 [kern_mgwd:info:2284] 0x81b13fb00: 0: INFO: SFO_Ready|Cluster_Apps: wait_for_apps_online (Startup stability test): mgwd is online. Number of tries: 1. 00000d93.00000afa 243bc7e3 Thu Sep 26 2019 04:33:17 -06:00 [kern_mgwd:info:2284] mhost_boot: MHOST_SF: Stable MGWD online 00000d93.00000afb 243bc7e3 Thu Sep 26 2019 04:33:17 -06:00 [kern_mgwd:info:2284] 0x81b518500: 0: NOTICE: RpcConnectionCache: markFailure: markFailure: New Quarantine for 127.0.0.1:536873552(crs)/1[tcp]{fffffffd|8} with timeout(5.000s) Reason:: RPC: Program not registered 00000d93.00000b0e 243bc7e3 Thu Sep 26 2019 04:33:17 -06:00 [kern_mgwd:info:2284] 0x81a9f1d00: 0: DEBUG: ntpd: signal_ntp_peer_update:1356 00000d93.00000b0f 243bc7e3 Thu Sep 26 2019 04:33:17 -06:00 [kern_mgwd:info:2284] 0x81a9f1d00: 0: DEBUG: ntpd: signal_ntp_update:1328 signal_ntp_update 00000d93.00000b10 243bc7e3 Thu Sep 26 2019 04:33:17 -06:00 [kern_mgwd:info:2284] 0x81b51b700: 0: DEBUG: ntpd: ntp_process_callback:1388 here 00000d93.00000b11 243bc7e3 Thu Sep 26 2019 04:33:17 -06:00 [kern_mgwd:info:2284] 0x81b51c600: 0: DEBUG: ntpd: look_for_ip_space_change:1225 00000d93.00000b12 243bc7e3 Thu Sep 26 2019 04:33:17 -06:00 [kern_mgwd:info:2284] 0x81b51b700: 0: DEBUG: ntpd: updateNtpConf:922 called 00000d93.00000b13 243bc7e3 Thu Sep 26 2019 04:33:17 -06:00 [kern_mgwd:info:2284] 0x81b51b700: 0: DEBUG: ntpd: updateNtpConf:932 evaluate_ntp_address: 0 00000d93.00000b14 243bc7e3 Thu Sep 26 2019 04:33:17 -06:00 [kern_mgwd:info:2284] 0x81b51b700: 0: DEBUG: ntpd_support: _find_ntp_address:426 looking up '10.0.0.2' (IS_CONFIG_FILE) 00000d93.00000b15 243bc7e3 Thu Sep 26 2019 04:33:17 -06:00 [kern_mgwd:info:2284] 0x81b51b700: 0: DEBUG: ntpd_support: _find_ntp_address:516 is '10.0.0.2' an IP address? 00000d93.00000b16 243bc7e3 Thu Sep 26 2019 04:33:17 -06:00 [kern_mgwd:info:2284] 0x81b51b700: 0: DEBUG: ntpd_support: _find_ntp_address:535 10.0.0.2 IS an IP address 00000d93.00000b17 243bc7e3 Thu Sep 26 2019 04:33:17 -06:00 [kern_mgwd:info:2284] 0x81b51b700: 0: INFO: ntpd_support: _find_ntp_address:701 ADDRESS 10.0.0.2 => IPv4 10.0.0.2 (10.0.0.2) OK 00000d93.00000b18 243bc7e3 Thu Sep 26 2019 04:33:17 -06:00 [kern_mgwd:info:2284] 0x81b51b700: 0: DEBUG: ntpd_support: find_ntp_address:1010 10.0.0.2 => 10.0.0.2: OKAY 00000d93.00000b19 243bc7e3 Thu Sep 26 2019 04:33:17 -06:00 [kern_mgwd:info:2284] 0x81b51b700: 0: DEBUG: ntpd: find_desired_config:739 mapping 10.0.0.2 to 10.0.0.2 00000d93.00000b1a 243bc7e3 Thu Sep 26 2019 04:33:17 -06:00 [kern_mgwd:info:2284] 0x81b51b700: 0: INFO: ntpd: updateNtpConf:995 writing new file 00000d93.00000b1b 243bc7e3 Thu Sep 26 2019 04:33:17 -06:00 [kern_mgwd:info:2284] 0x81a9f1d00: 0: ERR: tables::exports_cache: online_callback:src/tables/exports_cache.cc:1396 Failed to get mgwd exports ngbyhost rdb iter 00000d93.00000b1c 243bc7e3 Thu Sep 26 2019 04:33:17 -06:00 [kern_mgwd:info:2284] 0x81a9f1d00: 0: ERR: tables::exports_cache: online_callback:src/tables/exports_cache.cc:2580 Failed to get exports journal setting rdb iter 00000d93.00000b1d 243bc7e3 Thu Sep 26 2019 04:33:17 -06:00 [kern_mgwd:info:2284] 0x81a9f1d00: 0: ERR: FlexCache: **************online status callback status=1********** 00000d93.00000b1e 243bc7e3 Thu Sep 26 2019 04:33:17 -06:00 [kern_mgwd:info:2284] 0x81b516c00: 0: NOTICE: tables::fpolicy_replay: [fpolicyReplayThread] @@@@@@@@@ FPOLICY REPLAY THREAD INIT @@@@@@@@@ 00000d93.00000b1f 243bc7e3 Thu Sep 26 2019 04:33:17 -06:00 [kern_mgwd:info:2284] 0x81daca100: 0: NOTICE: mgwd: [hashdReplayThread] @@@@@@@@@ hashd REPLAY THREAD INIT @@@@@@@@@ 00000d93.00000b20 243bc7e3 Thu Sep 26 2019 04:33:17 -06:00 [kern_mgwd:info:2284] 0x81daca100: 0: NOTICE: mgwd: [hashdReplayInternal][entry][no-retry-run] index-mask: [0001] 00000d93.00000b21 243bc7e3 Thu Sep 26 2019 04:33:17 -06:00 [kern_mgwd:info:2284] 0x81b51b700: 0: DEBUG: ntpd: writeFile:1024 NEW Writing new ntp.conf file 00000d93.00000b22 243bc7e3 Thu Sep 26 2019 04:33:17 -06:00 [kern_mgwd:info:2284] 0x81b51b700: 0: DEBUG: ntpd: update_current_config:694 called 00000d93.00000b23 243bc7e3 Thu Sep 26 2019 04:33:17 -06:00 [kern_mgwd:info:2284] 0x81b516c00: 0: NOTICE: tables::fpolicy_replay: [virtual bool FpolicyReplayInfo::replayAtBoot()] destination is latest,refreshing mirror only 00000d93.00000b24 243bc7e3 Thu Sep 26 2019 04:33:17 -06:00 [kern_mgwd:info:2284] 0x81b51b700: 0: INFO: ntpd: updateNtpConf:1000 restarting ntpd 00000d93.00000b25 243bc7e3 Thu Sep 26 2019 04:33:17 -06:00 [kern_mgwd:info:2284] 0x81b51b700: 0: INFO: ntpd: reStartNtpDaemon:609 Starting/Restarting NTPd 00000d93.00000b26 243bc7e3 Thu Sep 26 2019 04:33:17 -06:00 [kern_mgwd:info:2284] 0x81b51b700: 0: DEBUG: ntpd: startNtpDaemon:295 Checking/starting NTPd 00000d93.00000b27 243bc7e3 Thu Sep 26 2019 04:33:17 -06:00 [kern_mgwd:info:2284] 0x81b51b700: 0: DEBUG: ntpd: startNtpDaemon:388 spm fetch failed: entry doesn't exist 00000d93.00000b28 243bc7e3 Thu Sep 26 2019 04:33:17 -06:00 [kern_mgwd:info:2284] 0x81daca100: 0: NOTICE: mgwd: [virtual bool HashdReplayInfo::replayAtBoot()] destination is latest; no operation to do 00000d93.00000b29 243bc7e3 Thu Sep 26 2019 04:33:17 -06:00 [kern_mgwd:info:2284] 0x81daca100: 0: NOTICE: mgwd: [hashdReplayInternal][ exit][no-retry-run] index-mask: [0000], result: [success] 00000d93.00000b2a 243bc7e3 Thu Sep 26 2019 04:33:17 -06:00 [kern_mgwd:info:2284] 0x81daca100: 0: NOTICE: mgwd: [hashdReplayInternal][entry][retry-run] index-mask: [0000] 00000d93.00000b2b 243bc7e3 Thu Sep 26 2019 04:33:17 -06:00 [kern_mgwd:info:2284] 0x81daca100: 0: NOTICE: mgwd: [hashdReplayInternal][ exit][retry-run] index-mask: [0000], result: [success] 00000d93.00000b2e 243bc7e3 Thu Sep 26 2019 04:33:17 -06:00 [kern_mgwd:info:2284] 0x81b516c00: 0: NOTICE: tables::fpolicy_replay: [virtual bool FpolicyReplayInfo::replayAtBoot()] destination is latest,refreshing mirror only 00000d93.00000b31 243bc7e3 Thu Sep 26 2019 04:33:17 -06:00 [kern_mgwd:info:2284] 0x81b51b700: 0: DEBUG: ntpd: startNtpDaemon:432 NTPd execute, OKAY 00000d93.00000b32 243bc7e3 Thu Sep 26 2019 04:33:17 -06:00 [kern_mgwd:info:2284] 0x81b516c00: 0: NOTICE: tables::fpolicy_replay: [virtual bool FpolicyReplayInfo::replayAtBoot()] destination is latest,refreshing mirror only 00000d93.00000b33 243bc7e3 Thu Sep 26 2019 04:33:17 -06:00 [kern_mgwd:info:2284] 0x81b516c00: 0: NOTICE: tables::fpolicy_replay: [virtual bool FpolicyReplayInfo::replayAtBoot()] destination is latest,refreshing mirror only 00000d93.00000b34 243bc7e3 Thu Sep 26 2019 04:33:17 -06:00 [kern_mgwd:info:2284] 0x81b516c00: 0: NOTICE: tables::fpolicy_replay: [virtual bool FpolicyReplayInfo::replayAtBoot()] destination is latest,refreshing mirror only 00000d93.00000b35 243bc7e3 Thu Sep 26 2019 04:33:17 -06:00 [kern_mgwd:info:2284] 0x81b516c00: 0: NOTICE: tables::fpolicy_replay: [virtual bool FPolicyKSMFReplayInfo::replayAtBoot()] destination is latest; no operation to do 00000d93.00000b36 243bc7e3 Thu Sep 26 2019 04:33:17 -06:00 [kern_mgwd:info:2284] 0x81b516c00: 0: NOTICE: tables::fpolicy_replay: [virtual bool FPolicyKSMFReplayInfo::replayAtBoot()] destination is latest; no operation to do 00000d93.00000b37 243bc7e3 Thu Sep 26 2019 04:33:17 -06:00 [kern_mgwd:info:2284] 0x81b516c00: 0: NOTICE: tables::fpolicy_replay: [virtual bool FPolicyKSMFReplayInfo::replayAtBoot()] destination is latest; no operation to do 00000d93.00000b38 243bc7e3 Thu Sep 26 2019 04:33:17 -06:00 [kern_mgwd:info:2284] 0x81b516c00: 0: NOTICE: tables::fpolicy_replay: [virtual bool FPolicyKSMFReplayInfo::replayAtBoot()] destination is latest; no operation to do 00000d93.00000b39 243bc7e3 Thu Sep 26 2019 04:33:17 -06:00 [kern_mgwd:info:2284] 0x81b516c00: 0: NOTICE: tables::fpolicy_replay: [virtual bool FPolicyKSMFReplayInfo::replayAtBoot()] destination is latest; no operation to do 00000d93.00000b3a 243bc7e3 Thu Sep 26 2019 04:33:17 -06:00 [kern_mgwd:info:2284] 0x81a9f1d00: 0: NOTICE: misc_node_info: src/tables/misc_node_info_view.cc:onlineEvent:167 misc_node_info onlineCallback 00000d93.00000b3b 243bc7e3 Thu Sep 26 2019 04:33:17 -06:00 [kern_mgwd:info:2284] 0x81a9f1d00: 0: INFO: Net::: switchless cluster online callback: unit is secondary 00000d93.00000b3c 243bc7e3 Thu Sep 26 2019 04:33:17 -06:00 [kern_mgwd:info:2284] 0x81b516c00: 0: NOTICE: tables::fpolicy_replay: [virtual bool FPolicyKSMFReplayInfo::replayAtBoot()] destination is latest; no operation to do 00000d93.00000b3d 243bc7e3 Thu Sep 26 2019 04:33:17 -06:00 [kern_mgwd:info:2284] 0x81dacb000: 0: WARNING: job_manager::job_store: flush_caches:src/job_store_support.cc:428 rdb_transaction: flush_caches discarded as job_manager is_waiting_for_online_cb 00000d93.00000b3e 243bc7e3 Thu Sep 26 2019 04:33:17 -06:00 [kern_mgwd:info:2284] 0x81b516c00: 0: NOTICE: tables::fpolicy_replay: [virtual bool FPolicyKSMFReplayInfo::replayAtBoot()] destination is latest; no operation to do 00000d93.00000b3f 243bc7e3 Thu Sep 26 2019 04:33:17 -06:00 [kern_mgwd:info:2284] 0x81d24e400: 86ffff0000000039: WARNING: tables::nblade_iter: Postponed resync since the vifmgr is not running 00000d93.00000b40 243bc7e3 Thu Sep 26 2019 04:33:17 -06:00 [kern_mgwd:info:2284] 0x81b516c00: 0: NOTICE: tables::fpolicy_replay: [virtual bool FPolicyKSMFReplayInfo::replayAtBoot()] destination is latest; no operation to do 00000d93.00000b41 243bc7e3 Thu Sep 26 2019 04:33:17 -06:00 [kern_mgwd:info:2284] 0x81b516c00: 0: NOTICE: tables::fpolicy_replay: [virtual bool FPolicyKSMFReplayInfo::replayAtBoot()] Normal reboot, replay whole Db to mirror 00000d93.00000b42 243bc7e3 Thu Sep 26 2019 04:33:17 -06:00 [kern_mgwd:info:2284] 0x81a9f1d00: 0: NOTICE: TABLES::qos: Resetting QoS configuration state 00000d93.00000b43 243bc7e3 Thu Sep 26 2019 04:33:17 -06:00 [kern_mgwd:info:2284] 0x81a9f1d00: 0: ERR: SAN::AIX_SUPPORT: src/tables/san/san_aix_support_capability.cc:Check:218 capability not enabled on every node of the cluster 00000d93.00000b44 243bc7e3 Thu Sep 26 2019 04:33:17 -06:00 [kern_mgwd:info:2284] 0x81a9f1d00: 0: ERR: tables:snapmirror_policy: snapmirror_policy: register callback function with license usage infrastructure 00000d93.00000b45 243bc7e3 Thu Sep 26 2019 04:33:17 -06:00 [kern_mgwd:info:2284] snapshot_policy: online status callback status=1 00000d93.00000b49 243bc7e3 Thu Sep 26 2019 04:33:17 -06:00 [kern_mgwd:info:2284] 0x81dacd800: 0: ERR: tables:snapmirror: snapmirror_utils::startServiceJobOnRdbOnline: queued and kicked SnapMirror Service Job. 00000d93.00000b4b 243bc7e3 Thu Sep 26 2019 04:33:17 -06:00 [kern_mgwd:info:2284] 0x81dacba00: 0: NOTICE: QOS::rpc: QoS reset: processing: 6 policy groups 00000d93.00000b4d 243bc7e3 Thu Sep 26 2019 04:33:17 -06:00 [kern_mgwd:info:2284] 0x81dacf100: 0: WARNING: job_manager::job_store: flush_caches:src/job_store_support.cc:428 rdb_transaction: flush_caches discarded as job_manager is_waiting_for_online_cb 00000d93.00000b50 243bc7e3 Thu Sep 26 2019 04:33:17 -06:00 [kern_mgwd:info:2284] 0x81dacba00: 0: NOTICE: QOS::rpc: QoS ksmf - 6 policy groups were sent (trans 0) to kernel 00000d93.00000b52 243bc7e3 Thu Sep 26 2019 04:33:17 -06:00 [kern_mgwd:info:2284] 0x81dacba00: 0: NOTICE: QOS::rpc: QoS reset: processing: 36 workloads 00000d93.00000b54 243bc7e3 Thu Sep 26 2019 04:33:17 -06:00 [kern_mgwd:info:2284] rebaseline_stream:Thu Sep 26 04:33:17 2019:src/tables/vserverdr/common/synthetic_method_handling.cc:76 (thr_id:0x81a9f1d00) 0x81a9f1d00: online_callback, Entry 00000d93.00000b55 243bc7e3 Thu Sep 26 2019 04:33:17 -06:00 [kern_mgwd:info:2284] rebaseline_stream:Thu Sep 26 04:33:17 2019:src/tables/vserverdr/common/synthetic_method_handling.cc:76 (thr_id:0x81a9f1d00) 0x81a9f1d00: checkMasterAndSpawnThread, Entry 00000d93.00000b56 243bc7e3 Thu Sep 26 2019 04:33:17 -06:00 [kern_mgwd:info:2284] rebaseline_stream:Thu Sep 26 04:33:17 2019:src/tables/vserverdr/common/synthetic_method_handling.cc:76 (thr_id:0x81a9f1d00) 0x81a9f1d00: checkMasterAndSpawnThread, No Entries in the trigger_baseline_generation RDB 00000d93.00000b57 243bc7e3 Thu Sep 26 2019 04:33:17 -06:00 [kern_mgwd:info:2284] 0x81dacb500: 0: ERR: fabricdatacollector::storage_switch: not creating fdc job, ishaoscmcc: false, ishaoscmcc check err = success {in onlineCallbackWorkerThread() at src/tables/storage_switch.cc:9264} 00000d93.00000b58 243bc7e3 Thu Sep 26 2019 04:33:17 -06:00 [kern_mgwd:info:2284] 0x81dacf100: 0: WARNING: tables::job_proxy: failed to queue cluster-affinity SP Certificate Expiry Check Job job: Job exists 00000d93.00000b59 243bc7e3 Thu Sep 26 2019 04:33:17 -06:00 [kern_mgwd:info:2284] 0x81dacf100: 0: ERR: MGWD:sp_mgmt: queue_spCertVfy_thread: sp_cert: Cannot queue sp certificate expiry verification job. Error : Job exists(6). 00000d93.00000b5a 243bc7e4 Thu Sep 26 2019 04:33:17 -06:00 [kern_mgwd:info:2284] 0x81dacba00: 0: NOTICE: QOS::rpc: QoS ksmf - 36 workloads were sent (trans 0) to kernel 00000d93.00000b5d 243bc7e4 Thu Sep 26 2019 04:33:17 -06:00 [kern_mgwd:info:2284] 0x81dacdd00: 0: WARNING: job_manager::sched_db: get_next_scheduler:src/sched_db.cc:442 Unable to retrieve next scheduler (by name) weekly with vserver nmg-fas8040, owner nmg-fas8040, clusterid: 635f71dd-b79d-11e7-9b07-00a09857ee8f: entry doesn't exist 00000d93.00000b61 243bc7e4 Thu Sep 26 2019 04:33:17 -06:00 [kern_mgwd:info:2284] 0x81d24e400: 86ffff000000000d: ERR: RpcConnectionCache: dup_suppress_log: Excessive Duplicate calls to doClntCreateTimed... 00000d93.00000b82 243bc7e4 Thu Sep 26 2019 04:33:18 -06:00 [kern_mgwd:debug:2284] online_callback: Failed: VMHOSTINFO failed with error - could not read sysctl 'hw.smbios.uuid'. 00000d93.00000b83 243bc7e4 Thu Sep 26 2019 04:33:18 -06:00 [kern_mgwd:info:2284] 0x81a9f1d00: 0: ERR: RBAC::login: src/tables/vs_userprofile.cc:2843 calling online_callback status 1 00000d93.00000b86 243bc7e4 Thu Sep 26 2019 04:33:18 -06:00 [kern_mgwd:info:2284] 0x81a9f1d00: 0: ERR: RBAC::login: killall xinetd happened 00000d93.00000b95 243bc807 Thu Sep 26 2019 04:33:18 -06:00 [kern_mgwd:info:2284] 0x81a9f1d00: 0: ERR: RBAC::login: killall xinetd happened 00000d93.00000b97 243bc807 Thu Sep 26 2019 04:33:18 -06:00 [kern_mgwd:info:2284] 0x81a9f1d00: 0: DEBUG: function_start_end_time_log: Start online_callback: 00000d93.00000b9d 243bc807 Thu Sep 26 2019 04:33:18 -06:00 [kern_mgwd:info:2284] 0x81a9f1d00: 0: INFO: VserverStackConfig: void (anonymous namespace)::get_kernel_vsid_set(std::set &): found vsid: 4294967280, 4294967289, 4294967292, 4294967293, 4294967294, 4294967295, 00000d93.00000b9e 243bc807 Thu Sep 26 2019 04:33:18 -06:00 [kern_mgwd:info:2284] 0x81a9f1d00: 0: INFO: VserverStackConfig: void (anonymous namespace)::add_stack_vsid(uint32_t, uint32_t, smdb_type_vstype): successfully added vsid 4294967293, ipsid 4294967294 to stack 00000d93.00000ba3 243bc807 Thu Sep 26 2019 04:33:18 -06:00 [kern_mgwd:info:2284] 0x81a9f1d00: 0: INFO: VserverStackConfig: void (anonymous namespace)::add_stack_vsid(uint32_t, uint32_t, smdb_type_vstype): successfully added vsid 4, ipsid 4294967295 to stack 00000d93.00000ba4 243bc807 Thu Sep 26 2019 04:33:18 -06:00 [kern_mgwd:info:2284] 0x81a9f1d00: 0: DEBUG: function_start_end_time_log: End online_callback: total elapsed time in milliseconds is 9 00000d93.00000ba5 243bc807 Thu Sep 26 2019 04:33:18 -06:00 [kern_mgwd:info:2284] 0x81dac9c00: 0: ALERT: Table::vserver: flush_clusfs_cache 2399 mgwd online notification sent to clusfs. Action: 4 00000d93.00000ba6 243bc807 Thu Sep 26 2019 04:33:18 -06:00 [kern_mgwd:info:2284] 0x81a9f1d00: 0: INFO: vserverdr_failover: src/tables/vserverdr/svmdr/vserverdr_failover.cc:87 on_online_status_cb In online callback 00000d93.00000ba7 243bc807 Thu Sep 26 2019 04:33:18 -06:00 [kern_mgwd:info:2284] 0x81a9f1d00: 0: NOTICE: Tables:web: Synchronizing the web server configuration from the cluster to the nodes 00000d93.00000ba8 243bc807 Thu Sep 26 2019 04:33:18 -06:00 [kern_mgwd:info:2284] 0x81a9f1d00: 0: NOTICE: Tables:web: The web server configuration has been synchronized from the cluster to the nodes 00000d93.00000ba9 243bc807 Thu Sep 26 2019 04:33:18 -06:00 [kern_mgwd:info:2284] 0x81a9f1d00: 0: NOTICE: Tables:web: reconf_cluster_proxy_async (all nodes): Reconfiguration is now in progress 00000d93.00000baa 243bc807 Thu Sep 26 2019 04:33:18 -06:00 [kern_mgwd:info:2284] 0x81dace200: 0: NOTICE: Tables:web: Starting cluster proxy reconfig thread is_all_nodes=1 00000d93.00000bab 243bc807 Thu Sep 26 2019 04:33:18 -06:00 [kern_mgwd:info:2284] 0x81a9f1d00: 0: NOTICE: Tables:web: web_server_config online callback: Updating the proxy server 00000d93.00000bac 243bc807 Thu Sep 26 2019 04:33:18 -06:00 [kern_mgwd:info:2284] 0x81dace200: 0: INFO: SFO_Ready|Cluster_Apps: wait_for_apps_online (waiter: RECONF_CLUSTER_PROXY_THREAD) apps: mgwd vldb vifmgr bcomd crs, maxTime 60 sec: starting 00000d93.00000bad 243bc807 Thu Sep 26 2019 04:33:18 -06:00 [kern_mgwd:info:2284] 0x81dace200: 0: INFO: SFO_Ready|Cluster_Apps: wait_for_apps_online (RECONF_CLUSTER_PROXY_THREAD): mgwd is online. Number of tries: 1. 00000d93.00000bae 243bc807 Thu Sep 26 2019 04:33:18 -06:00 [kern_mgwd:info:2284] 0x81a9f1d00: 0: INFO: license_v2: RDB online callback. status=1 00000d93.00000baf 243bc807 Thu Sep 26 2019 04:33:18 -06:00 [kern_mgwd:info:2284] 0x81dacf100: 0: INFO: license_v2: License online handler thread started 00000d93.00000bb0 243bc807 Thu Sep 26 2019 04:33:18 -06:00 [kern_mgwd:info:2284] 0x81a9f1d00: 0: INFO: license_v2: onlineHandlerThread created to do work 00000d93.00000bb1 243bc807 Thu Sep 26 2019 04:33:18 -06:00 [kern_mgwd:info:2284] 0x81dacb500: 0: NOTICE: Tables:web: Committed WSDB changes to web service ports 00000d93.00000bb2 243bc807 Thu Sep 26 2019 04:33:18 -06:00 [kern_mgwd:info:2284] Setting has been changed. 00000d93.00000bb3 243bc807 Thu Sep 26 2019 04:33:18 -06:00 [kern_mgwd:info:2284] Setting has been changed. 00000d93.00000bb4 243bc807 Thu Sep 26 2019 04:33:18 -06:00 [kern_mgwd:info:2284] imageVersion: 9.2, imageGeneration: 9, imageMajor: 2, imageMinor: 0; [ANDU_LOG at src/tables/cluster_image_utils.cc:399 in 'extractVersion'] 00000d93.00000bb5 243bc807 Thu Sep 26 2019 04:33:18 -06:00 [kern_mgwd:info:2284] 0x81a9f1d00: 0: INFO: MGWD::NDO: Processing RDB online callback. Online status is Secondary. 00000d93.00000bb6 243bc807 Thu Sep 26 2019 04:33:18 -06:00 [kern_mgwd:info:2284] 0x81a9f1d00: 0: WARNING: job_manager::job_ext: on_online_status_cb:src/job_ext.cc:512 on_online_status_cb 00000d93.00000bb7 243bc807 Thu Sep 26 2019 04:33:18 -06:00 [kern_mgwd:info:2284] 0x81a9f1d00: 0: WARNING: job_manager::job_q: on_online_status_cb:src/job_q.cc:702 on_online_status_cb 00000d93.00000bb8 243bc807 Thu Sep 26 2019 04:33:18 -06:00 [kern_mgwd:info:2284] 0x81a9f1d00: 0: WARNING: job_manager::job_sched: on_online_status_cb:src/job_sched.cc:1203 on_online_status_cb (1) 00000d93.00000bb9 243bc807 Thu Sep 26 2019 04:33:18 -06:00 [kern_mgwd:info:2284] 0x81a9f1d00: 0: WARNING: job_manager::job_sched: on_online_status_cb:src/job_sched.cc:1241 found 12 pre-existing cluster jobs locally 00000d93.00000bba 243bc807 Thu Sep 26 2019 04:33:18 -06:00 [kern_mgwd:info:2284] 0x81a9f1d00: 0: WARNING: job_manager::job_sched: on_online_status_cb:src/job_sched.cc:1289 skipping job 0db07695-e049-11e9-a00d-00a098573067 since we still own it 00000d93.00000bbb 243bc807 Thu Sep 26 2019 04:33:18 -06:00 [kern_mgwd:info:2284] 0x81a9f1d00: 0: WARNING: job_manager::job_sched: on_online_status_cb:src/job_sched.cc:1289 skipping job 489c38af-b80c-11e7-964b-00a098573067 since we still own it 00000d93.00000bbc 243bc807 Thu Sep 26 2019 04:33:18 -06:00 [kern_mgwd:info:2284] 0x81a9f1d00: 0: WARNING: job_manager::job_sched: on_online_status_cb:src/job_sched.cc:1289 skipping job 48c2382b-b80c-11e7-964b-00a098573067 since we still own it 00000d93.00000bbd 243bc807 Thu Sep 26 2019 04:33:18 -06:00 [kern_mgwd:info:2284] 0x81a9f1d00: 0: WARNING: job_manager::job_sched: on_online_status_cb:src/job_sched.cc:1289 skipping job fac3c78b-b80c-11e7-964b-00a098573067 since we still own it 00000d93.00000bbe 243bc807 Thu Sep 26 2019 04:33:18 -06:00 [kern_mgwd:info:2284] 0x81a9f1d00: 0: WARNING: job_manager::job_sched: on_online_status_cb:src/job_sched.cc:1325 potentially quiescing 0 jobs 00000d93.00000bbf 243bc807 Thu Sep 26 2019 04:33:18 -06:00 [kern_mgwd:info:2284] 0x81a9f1d00: 0: WARNING: job_manager::job_sched: on_online_status_cb:src/job_sched.cc:1335 potentially deleting 0 jobs 00000d93.00000bc0 243bc807 Thu Sep 26 2019 04:33:18 -06:00 [kern_mgwd:info:2284] 0x81a9f1d00: 0: WARNING: job_manager::job_sched: on_online_status_cb:src/job_sched.cc:1350 modifying 8 jobs 00000d93.00000bc1 243bc807 Thu Sep 26 2019 04:33:18 -06:00 [kern_mgwd:info:2284] 0x81a9f1d00: 0: WARNING: job_manager::job_sched: on_online_status_cb:src/job_sched.cc:1355 modifying job 176baaab-b79e-11e7-9b07-00a09857ee8f 00000d93.00000bc2 243bc807 Thu Sep 26 2019 04:33:18 -06:00 [kern_mgwd:info:2284] 0x81a9f1d00: 0: WARNING: job_manager::job_sched: on_online_status_cb:src/job_sched.cc:1355 modifying job 176c80e9-b79e-11e7-9b07-00a09857ee8f 00000d93.00000bc3 243bc807 Thu Sep 26 2019 04:33:18 -06:00 [kern_mgwd:info:2284] 0x81a9f1d00: 0: WARNING: job_manager::job_sched: on_online_status_cb:src/job_sched.cc:1355 modifying job 489d5bff-b80c-11e7-964b-00a098573067 00000d93.00000bc4 243bc807 Thu Sep 26 2019 04:33:18 -06:00 [kern_mgwd:info:2284] 0x81a9f1d00: 0: WARNING: job_manager::job_sched: on_online_status_cb:src/job_sched.cc:1355 modifying job 659699fb-b79d-11e7-9b07-00a09857ee8f 00000d93.00000bc5 243bc807 Thu Sep 26 2019 04:33:18 -06:00 [kern_mgwd:info:2284] 0x81a9f1d00: 0: WARNING: job_manager::job_sched: on_online_status_cb:src/job_sched.cc:1355 modifying job adade7cf-b79d-11e7-9b07-00a09857ee8f 00000d93.00000bc6 243bc807 Thu Sep 26 2019 04:33:18 -06:00 [kern_mgwd:info:2284] 0x81a9f1d00: 0: WARNING: job_manager::job_sched: on_online_status_cb:src/job_sched.cc:1355 modifying job adaed760-b79d-11e7-9b07-00a09857ee8f 00000d93.00000bc7 243bc807 Thu Sep 26 2019 04:33:18 -06:00 [kern_mgwd:info:2284] 0x81a9f1d00: 0: WARNING: job_manager::job_sched: on_online_status_cb:src/job_sched.cc:1355 modifying job adaf71a9-b79d-11e7-9b07-00a09857ee8f 00000d93.00000bc8 243bc807 Thu Sep 26 2019 04:33:18 -06:00 [kern_mgwd:info:2284] 0x81a9f1d00: 0: WARNING: job_manager::job_sched: on_online_status_cb:src/job_sched.cc:1355 modifying job d9549764-b91b-11e7-8fa8-00a098573067 00000d93.00000bc9 243bc807 Thu Sep 26 2019 04:33:18 -06:00 [kern_mgwd:info:2284] 0x81a9f1d00: 0: WARNING: job_manager::job_sched: on_online_status_cb:src/job_sched.cc:1365 adding 10 jobs 00000d93.00000bca 243bc807 Thu Sep 26 2019 04:33:18 -06:00 [kern_mgwd:info:2284] 0x81dacd300: 0: ERR: perf::statistics: fail open on /mroot/etc/log/stats/samples/. 00000d93.00000bcb 243bc807 Thu Sep 26 2019 04:33:18 -06:00 [kern_mgwd:info:2284] 0x81a9f1d00: 0: WARNING: job_manager::job_sched: on_online_status_cb:src/job_sched.cc:1390 finished processing online cb 00000d93.00000bcc 243bc807 Thu Sep 26 2019 04:33:18 -06:00 [kern_mgwd:info:2284] 0x81dacd300: 0: NOTICE: RpcConnectionCache: clearFailure: called for 127.0.0.1:536873514(cm)/1[tcp]{fffffffd|8} 00000d93.00000bd8 243bc807 Thu Sep 26 2019 04:33:18 -06:00 [kern_mgwd:info:2284] 0x81dad0000: 0: ERR: CR::Policy: Successfully initialized constituent scanner configuration 00000d93.00000bd9 243bc807 Thu Sep 26 2019 04:33:18 -06:00 [kern_mgwd:info:2284] 0x81dacf600: 0: ERR: CR::Policy: Successfully pushed notification logger configuration to Nblade 00000d93.00000bda 243bc807 Thu Sep 26 2019 04:33:18 -06:00 [kern_mgwd:info:2284] Info: online_status_callback: sp user refresh - online status = 1 00000d93.00000bdb 243bc807 Thu Sep 26 2019 04:33:18 -06:00 [kern_mgwd:info:2284] Info: online_status_callback: sp user refresh - db versions different 00000d93.00000bdd 243bc807 Thu Sep 26 2019 04:33:18 -06:00 [kern_mgwd:info:2284] 0x81dad0500: 0: NOTICE: RpcConnectionCache: clearFailure: called for 127.0.0.1:536873548(pipd)/1[tcp]{fffffffd|8} 00000d93.00000bde 243bc807 Thu Sep 26 2019 04:33:18 -06:00 [kern_mgwd:info:2284] Error: sp_user_refresh: sp_itr get error - invalid field. 00000d93.00000bdf 243bc807 Thu Sep 26 2019 04:33:18 -06:00 [kern_mgwd:info:2284] Notice: _initVolReaperJob: Ensure VolReaper daemon is active: succeeded. 00000d93.00000be7 243bc807 Thu Sep 26 2019 04:33:18 -06:00 [kern_mgwd:info:2284] 0x81b0a4d00: 0: WARNING: ECVCache::MONITOR: src/tables/ECV_Monitor.cc:publishMain:248 Failed to update bcomd with ECV (will retry): RPC: Program not registered [from mgwd on node "nmg-fas8040-02" (VSID: -3) to bcomd at 127.0.0.1] 00000d93.00000beb 243bc807 Thu Sep 26 2019 04:33:19 -06:00 [kern_mgwd:info:2284] 0x81b514900: 0: NOTICE: RpcConnectionCache: clearFailure: called for localhost:536873477(secd)/1[tcp]{fffffffd|ffffffff} 00000d93.00000bec 243bc807 Thu Sep 26 2019 04:33:19 -06:00 [kern_mgwd:info:2284] 0x81b514900: 0: ERR: RpcConnectionCache: dup_suppress_log: Excessive Duplicate calls to doClntCreateTimed... 00000d93.00000bed 243bc807 Thu Sep 26 2019 04:33:19 -06:00 [kern_mgwd:info:2284] Warning: "vserver services name-service unix-group file show" will not be accessible. 00000d93.00000bee 243bc807 Thu Sep 26 2019 04:33:19 -06:00 [kern_mgwd:info:2284] Set the access to "all" if you want to allow it. 00000d93.00000bef 243bc807 Thu Sep 26 2019 04:33:19 -06:00 [kern_mgwd:info:2284] Warning: "vserver services name-service unix-user file show" will not be accessible. 00000d93.00000bf0 243bc807 Thu Sep 26 2019 04:33:19 -06:00 [kern_mgwd:info:2284] Set the access to "all" if you want to allow it. 00000d93.00000bf1 243bc807 Thu Sep 26 2019 04:33:19 -06:00 [kern_mgwd:info:2284] 0x81b514900: 0: ERR: RpcConnectionCache: dup_suppress_log: Excessive Duplicate calls to doClntCreateTimed... 00000d93.00000bf2 243bc807 Thu Sep 26 2019 04:33:19 -06:00 [kern_mgwd:info:2284] 0x81b514900: 0: ERR: RpcConnectionCache: dup_suppress_log: Excessive Duplicate calls to doClntCreateTimed... 00000d93.00000bf3 243bc807 Thu Sep 26 2019 04:33:19 -06:00 [kern_mgwd:info:2284] 0x81b514900: 0: ERR: RpcConnectionCache: dup_suppress_log: Excessive Duplicate calls to doClntCreateTimed... 00000d93.00000bf4 243bc807 Thu Sep 26 2019 04:33:19 -06:00 [kern_mgwd:info:2284] 0x81d24d000: 86ffff0000000083: WARNING: job_manager::job_sched: modify_schedule:src/job_sched.cc:800 job_manager::job_sched::modify_schedule: Calling set_scheduler on the record: Current schedule = 0x81b615bc0; New schedule = 0x81c4b5640 00000d93.00000bf5 243bc807 Thu Sep 26 2019 04:33:19 -06:00 [kern_mgwd:info:2284] 0x81b514900: 0: ERR: RpcConnectionCache: dup_suppress_log: Excessive Duplicate calls to doClntCreateTimed... 00000d93.00000bf6 243bc807 Thu Sep 26 2019 04:33:19 -06:00 [kern_mgwd:info:2284] publickey_rdb: online status callback status=1 00000d93.00000bf7 243bc807 Thu Sep 26 2019 04:33:19 -06:00 [kern_mgwd:info:2284] 0x81dacb000: 0: ERR: RBAC::Roles: Stored version 9_2_0 : Current version 9_2_0. File : src/tables/vserverrbac/vs_profile.cc 00000d93.00000bf8 243bc807 Thu Sep 26 2019 04:33:19 -06:00 [kern_mgwd:info:2284] 0x81d248000: 0: NOTICE: rdb_sitelist: slpush: successfully pushed site information to _push_sitelist_id[13] nodelist_status non_authoritative 00000d93.00000bf9 243bc807 Thu Sep 26 2019 04:33:19 -06:00 [kern_mgwd:info:2284] 0x81b514900: 0: ERR: RpcConnectionCache: dup_suppress_log: Excessive Duplicate calls to doClntCreateTimed... 00000d93.00000bfa 243bc807 Thu Sep 26 2019 04:33:20 -06:00 [kern_mgwd:info:2284] 0x81b514900: 0: ERR: RpcConnectionCache: dup_suppress_log: Excessive Duplicate calls to doClntCreateTimed... 00000d93.00000bfc 243bc807 Thu Sep 26 2019 04:33:20 -06:00 [kern_mgwd:info:2284] 0x81b0a4d00: 0: WARNING: ECVCache::MONITOR: src/tables/ECV_Monitor.cc:publishMain:248 Failed to update bcomd with ECV (will retry): RPC: Program not registered [from mgwd on node "nmg-fas8040-02" (VSID: -3) to bcomd at 127.0.0.1] 00000d93.00000bfd 243bc807 Thu Sep 26 2019 04:33:20 -06:00 [kern_mgwd:info:2284] 0x81dacba00: 0: NOTICE: QOS::rpc: set_workloads_ksmf pg/workload reset end.Workload maintainer signaled. 00000d93.00000bfe 243bc807 Thu Sep 26 2019 04:33:20 -06:00 [kern_mgwd:info:2284] 0x81dacba00: 0: NOTICE: QOS::rpc: QoS workload maintainer thread started 00000d93.00000bff 243bc807 Thu Sep 26 2019 04:33:20 -06:00 [kern_mgwd:info:2284] 0x81dad0a00: 0: NOTICE: QOS::rpc: Syncing user/auto qos workload data attributes 00000d93.00000c00 243bc807 Thu Sep 26 2019 04:33:20 -06:00 [kern_mgwd:info:2284] 0x81dad0a00: 0: ERR: QOS::rpc: qos_workload_data_object_sync_driver: Unable to sync workload data object configuration. Reason: VLDB OFFLINE 00000d93.00000c02 243bc818 Thu Sep 26 2019 04:33:22 -06:00 [kern_mgwd:info:2284] 0x81b0a4d00: 0: WARNING: ECVCache::MONITOR: src/tables/ECV_Monitor.cc:publishMain:248 Failed to update bcomd with ECV (will retry): RPC: Program not registered [from mgwd on node "nmg-fas8040-02" (VSID: -3) to bcomd at 127.0.0.1] 00000d93.00000c03 243bc82c Thu Sep 26 2019 04:33:24 -06:00 [kern_mgwd:info:2284] 0x81b0a4d00: 0: WARNING: ECVCache::MONITOR: src/tables/ECV_Monitor.cc:publishMain:248 Failed to update bcomd with ECV (will retry): RPC: Program not registered [from mgwd on node "nmg-fas8040-02" (VSID: -3) to bcomd at 127.0.0.1] 00000d93.00000c04 243bc840 Thu Sep 26 2019 04:33:26 -06:00 [kern_mgwd:info:2284] 0x81b7bce00: 0: ERR: RPCServer: mgwd_ready_for_replay: vifmgr is not ready (state: unknown) 00000d93.00000c06 243bc840 Thu Sep 26 2019 04:33:26 -06:00 [kern_mgwd:info:2284] 0x81b0a4d00: 0: WARNING: ECVCache::MONITOR: src/tables/ECV_Monitor.cc:publishMain:248 Failed to update bcomd with ECV (will retry): RPC: Program not registered [from mgwd on node "nmg-fas8040-02" (VSID: -3) to bcomd at 127.0.0.1] 00000d93.00000c07 243bc854 Thu Sep 26 2019 04:33:28 -06:00 [kern_mgwd:info:2284] 0x81b0a4d00: 0: WARNING: ECVCache::MONITOR: src/tables/ECV_Monitor.cc:publishMain:248 Failed to update bcomd with ECV (will retry): RPC: Program not registered [from mgwd on node "nmg-fas8040-02" (VSID: -3) to bcomd at 127.0.0.1] 00000d93.00000c08 243bc868 Thu Sep 26 2019 04:33:30 -06:00 [kern_mgwd:info:2284] 0x81b0a4d00: 0: WARNING: ECVCache::MONITOR: src/tables/ECV_Monitor.cc:publishMain:248 Failed to update bcomd with ECV (will retry): RPC: Program not registered [from mgwd on node "nmg-fas8040-02" (VSID: -3) to bcomd at 127.0.0.1] 00000d93.00000c09 243bc87c Thu Sep 26 2019 04:33:32 -06:00 [kern_mgwd:info:2284] 0x81b0a4d00: 0: WARNING: ECVCache::MONITOR: src/tables/ECV_Monitor.cc:publishMain:248 Failed to update bcomd with ECV (will retry): RPC: Program not registered [from mgwd on node "nmg-fas8040-02" (VSID: -3) to bcomd at 127.0.0.1] 00000d93.00000c0a 243bc8a4 Thu Sep 26 2019 04:33:34 -06:00 [kern_mgwd:info:2284] 0x81b0a4d00: 0: WARNING: ECVCache::MONITOR: src/tables/ECV_Monitor.cc:publishMain:248 Failed to update bcomd with ECV (will retry): RPC: Program not registered [from mgwd on node "nmg-fas8040-02" (VSID: -3) to bcomd at 127.0.0.1] 00000d93.00000c0b 243bc8a4 Thu Sep 26 2019 04:33:35 -06:00 [kern_mgwd:info:2284] 0x81b7bc900: 0: NOTICE: net_ports: update_operational_fields:src/tables/net_ports.cc:497 installing new/updated mac based on driver state: mac=02:a0:98:57:30:67 00000d93.00000c0c 243bc8a4 Thu Sep 26 2019 04:33:35 -06:00 [kern_mgwd:info:2284] 0x81b244200: 0: NOTICE: net_ports: update_operational_fields:src/tables/net_ports.cc:497 installing new/updated mac based on driver state: mac=02:a0:98:57:30:67 00000d93.00000c0f 243bc8a4 Thu Sep 26 2019 04:33:36 -06:00 [kern_mgwd:info:2284] Notice: init_config_br_threadf: config_backup_restore_ops::populate_default_settings succeeded. 00000d93.00000c10 243bc8a4 Thu Sep 26 2019 04:33:36 -06:00 [kern_mgwd:info:2284] 0x81b245100: 0: ERR: RPCServer: mgwd_ready_for_replay: vifmgr is not ready (state: unknown) 00000d93.00000c12 243bc8a4 Thu Sep 26 2019 04:33:36 -06:00 [kern_mgwd:info:2284] 0x81b0a4d00: 0: WARNING: ECVCache::MONITOR: src/tables/ECV_Monitor.cc:publishMain:248 Failed to update bcomd with ECV (will retry): RPC: Program not registered [from mgwd on node "nmg-fas8040-02" (VSID: -3) to bcomd at 127.0.0.1] 00000d93.00000c31 243bc8be Thu Sep 26 2019 04:33:38 -06:00 [kern_mgwd:info:2284] 0x81b0a4d00: 0: WARNING: ECVCache::MONITOR: src/tables/ECV_Monitor.cc:publishMain:248 Failed to update bcomd with ECV (will retry): RPC: Program not registered [from mgwd on node "nmg-fas8040-02" (VSID: -3) to bcomd at 127.0.0.1] 00000d93.00000c56 243bc8be Thu Sep 26 2019 04:33:39 -06:00 [kern_mgwd:info:2284] 0x81b514900: 0: NOTICE: RpcConnectionCache: clearFailure: called for 127.0.0.1:536873492(vifmgr)/1[tcp]{fffffffd|8} 00000d93.00000c60 243bc8bf Thu Sep 26 2019 04:33:39 -06:00 [kern_mgwd:info:2284] 0x81b244c00: 0: NOTICE: Tables:web: reconf_cluster_proxy_async (local): Reconfiguration is already in progress 00000d93.00000cb9 243bc8bf Thu Sep 26 2019 04:33:40 -06:00 [kern_mgwd:info:2284] 0x81b244c00: 0: NOTICE: net_ports: update_operational_fields:src/tables/net_ports.cc:497 installing new/updated mac based on driver state: mac=02:a0:98:57:30:67 00000d93.00000cba 243bc8bf Thu Sep 26 2019 04:33:40 -06:00 [kern_mgwd:info:2284] 0x81b244c00: 0: NOTICE: net_ports: update_operational_fields:src/tables/net_ports.cc:497 installing new/updated mac based on driver state: mac=02:a0:98:57:30:67 00000d93.00000d11 243bc8bf Thu Sep 26 2019 04:33:40 -06:00 [kern_mgwd:info:2284] 0x81b244c00: 0: INFO: Net::: Switchless cluster option is modified to 0 00000d93.00000d12 243bc8bf Thu Sep 26 2019 04:33:40 -06:00 [kern_mgwd:info:2284] 0x81b244c00: 0: INFO: Net::: smdb_error netopts_switchless_cluster_iterator::modify_no_detect(): kernel is configured with switchless disabled 00000d93.00000d13 243bc8bf Thu Sep 26 2019 04:33:40 -06:00 [kern_mgwd:info:2284] 0x81b244c00: 0: INFO: Net::: smdb_error netopts_switchless_cluster_iterator::modify_no_detect(): rdb is updated to switchless disabled 00000d93.00000d14 243bc8d5 Thu Sep 26 2019 04:33:40 -06:00 [kern_mgwd:info:2284] 0x81b244c00: 0: NOTICE: net_ports: update_operational_fields:src/tables/net_ports.cc:497 installing new/updated mac based on driver state: mac=02:a0:98:57:30:67 00000d93.00000d15 243bc8d5 Thu Sep 26 2019 04:33:40 -06:00 [kern_mgwd:info:2284] 0x81b244c00: 0: NOTICE: net_ports: update_operational_fields:src/tables/net_ports.cc:497 installing new/updated mac based on driver state: mac=02:a0:98:57:30:67 00000d93.00000d16 243bc8d5 Thu Sep 26 2019 04:33:40 -06:00 [kern_mgwd:info:2284] 0x81b244c00: 0: NOTICE: net_ports: update_operational_fields:src/tables/net_ports.cc:497 installing new/updated mac based on driver state: mac=02:a0:98:57:30:67 00000d93.00000d17 243bc8d5 Thu Sep 26 2019 04:33:40 -06:00 [kern_mgwd:info:2284] 0x81b244c00: 0: NOTICE: net_ports: update_operational_fields:src/tables/net_ports.cc:497 installing new/updated mac based on driver state: mac=02:a0:98:57:30:67 00000d93.00000d18 243bc8d5 Thu Sep 26 2019 04:33:40 -06:00 [kern_mgwd:info:2284] 0x81dace200: 0: NOTICE: RpcConnectionCache: clearFailure: called for 127.0.0.1:536873504(bcomd)/1[tcp]{fffffffd|8} 00000d93.00000da8 243bc8d6 Thu Sep 26 2019 04:33:41 -06:00 [kern_mgwd:info:2284] 0x81b13fb00: 0: NOTICE: RpcConnectionCache: clearFailure: called for 127.0.0.1:536873490(vldb)/1[tcp]{fffffffd|8} 00000d93.00000da9 243bc8d6 Thu Sep 26 2019 04:33:41 -06:00 [kern_mgwd:info:2284] 0x81b13fb00: 0: NOTICE: RpcConnectionCache: clearFailure: called for 127.0.0.1:536873552(crs)/1[tcp]{fffffffd|8} 00000d93.00000dac 243bc912 Thu Sep 26 2019 04:33:46 -06:00 [kern_mgwd:info:2284] 0x81b7bce00: 0: ERR: RPCServer: mgwd_ready_for_replay: vifmgr is not ready (state: started) 00000d93.00000dc4 243bc912 Thu Sep 26 2019 04:33:46 -06:00 [kern_mgwd:info:2284] 0x81b244200: 0: NOTICE: Tables:web: reconf_cluster_proxy_async (all nodes): Reconfiguration is already in progress 00000d93.00000dca 243bc912 Thu Sep 26 2019 04:33:47 -06:00 [kern_mgwd:info:2284] 0x81d24c600: 86ffff0000000039: WARNING: tables::nblade_iter: Postponed resync since the vifmgr is not running 00000d93.00000dd2 243bc912 Thu Sep 26 2019 04:33:47 -06:00 [kern_mgwd:info:2284] 0x81b24a600: 0: ERR: CAPM_SERVER: register_capability_change:src/capmserver/capm_server.cc:390 Failed to register id:1 for capability DATA_ONTAP.8.4.0. Error:duplicate entry 00000d93.00000e67 243bc949 Thu Sep 26 2019 04:33:49 -06:00 [kern_mgwd:info:2284] 0x81b7bc900: 0: INFO: Net::: Switchless cluster option is modified to 0 00000d93.00000e68 243bc949 Thu Sep 26 2019 04:33:49 -06:00 [kern_mgwd:info:2284] 0x81b518000: 0: INFO: SFO_Ready|Cluster_Apps: wait_for_apps_online (LIF_PLACEMENT_THREAD): vifmgr is online. Number of tries: 18. 00000d93.00000e69 243bc949 Thu Sep 26 2019 04:33:49 -06:00 [kern_mgwd:info:2284] 0x81b518000: 0: INFO: SFO_Ready|Cluster_Apps: wait_for_apps_online (LIF_PLACEMENT_THREAD): vifmgr is in running state. Number of tries: 18. 00000d93.00000e6a 243bc949 Thu Sep 26 2019 04:33:49 -06:00 [kern_mgwd:info:2284] 0x81b51a300: 0: INFO: SFO_Ready|Cluster_Apps: wait_for_apps_online (LIF_PLACEMENT_THREAD): vifmgr is online. Number of tries: 18. 00000d93.00000e6b 243bc949 Thu Sep 26 2019 04:33:49 -06:00 [kern_mgwd:info:2284] 0x81b51a300: 0: INFO: SFO_Ready|Cluster_Apps: wait_for_apps_online (LIF_PLACEMENT_THREAD): vifmgr is in running state. Number of tries: 18. 00000d93.00000e6c 243bc949 Thu Sep 26 2019 04:33:49 -06:00 [kern_mgwd:info:2284] 0x81b7bc900: 0: INFO: Net::: smdb_error netopts_switchless_cluster_iterator::modify_no_detect(): kernel is configured with switchless disabled 00000d93.00000e6d 243bc949 Thu Sep 26 2019 04:33:49 -06:00 [kern_mgwd:info:2284] 0x81b7bc900: 0: INFO: Net::: smdb_error netopts_switchless_cluster_iterator::modify_no_detect(): rdb is updated to switchless disabled 00000d93.00000e71 243bc949 Thu Sep 26 2019 04:33:50 -06:00 [kern_mgwd:info:2284] 0x81dace200: 0: INFO: SFO_Ready|Cluster_Apps: wait_for_apps_online (RECONF_CLUSTER_PROXY_THREAD): vldb is online. Number of tries: 17. 00000d93.00000e72 243bc949 Thu Sep 26 2019 04:33:50 -06:00 [kern_mgwd:info:2284] 0x81dace200: 0: INFO: SFO_Ready|Cluster_Apps: wait_for_apps_online (RECONF_CLUSTER_PROXY_THREAD): vifmgr is online. Number of tries: 17. 00000d93.00000e73 243bc949 Thu Sep 26 2019 04:33:50 -06:00 [kern_mgwd:info:2284] 0x81dace200: 0: INFO: SFO_Ready|Cluster_Apps: wait_for_apps_online (RECONF_CLUSTER_PROXY_THREAD): vifmgr is in running state. Number of tries: 17. 00000d93.00000e74 243bc949 Thu Sep 26 2019 04:33:50 -06:00 [kern_mgwd:info:2284] 0x81dace200: 0: INFO: SFO_Ready|Cluster_Apps: wait_for_apps_online (RECONF_CLUSTER_PROXY_THREAD): bcomd is online. Number of tries: 17. 00000d93.00000e75 243bc949 Thu Sep 26 2019 04:33:50 -06:00 [kern_mgwd:info:2284] 0x81dace200: 0: INFO: SFO_Ready|Cluster_Apps: wait_for_apps_online (RECONF_CLUSTER_PROXY_THREAD): crs is online. Number of tries: 17. 00000d93.00000e76 243bc949 Thu Sep 26 2019 04:33:50 -06:00 [kern_mgwd:info:2284] 0x81dace200: 0: INFO: SFO_Ready|Cluster_Apps: wait_for_apps_online (waiter: RECONF_CLUSTER_PROXY_THREAD): Finished after 16 tries. 00000d93.00000e77 243bc949 Thu Sep 26 2019 04:33:50 -06:00 [kern_mgwd:info:2284] 0x81dace200: 0: NOTICE: Tables:web: SMF DSMDB service ready 00000d93.00000e78 243bc949 Thu Sep 26 2019 04:33:50 -06:00 [kern_mgwd:info:2284] 0x81dace200: 0: NOTICE: Tables:web: Updated the proxy server on node nmg-fas8040-01 00000d93.00000e79 243bc949 Thu Sep 26 2019 04:33:50 -06:00 [kern_mgwd:info:2284] 0x81dace200: 0: NOTICE: Tables:web: Updated the proxy server on node nmg-fas8040-02 00000d93.00000e7a 243bc949 Thu Sep 26 2019 04:33:50 -06:00 [kern_mgwd:info:2284] 0x81dace200: 0: NOTICE: Tables:web: Completed cluster proxy reconfig thread 00000d93.00000e7d 243bc949 Thu Sep 26 2019 04:33:50 -06:00 [kern_mgwd:info:2284] 0x81dad0a00: 0: NOTICE: QOS::rpc: qos_workload_data_object_sync_driver: VLDB online. Resyncing data object configuration 00000d93.00000e7e 243bc949 Thu Sep 26 2019 04:33:50 -06:00 [kern_mgwd:info:2284] 0x81dad0a00: 0: NOTICE: QOS::rpc: qos_workload_data_object_sync_driver: Processing workloads for data object attribute sync 00000d93.00000e7f 243bc949 Thu Sep 26 2019 04:33:50 -06:00 [kern_mgwd:info:2284] 0x81dad0a00: 0: NOTICE: QOS::rpc: qos_workload_data_object_sync_driver: Sent 5 workload dobj updates to kernel 00000d93.00000e80 243bc949 Thu Sep 26 2019 04:33:50 -06:00 [kern_mgwd:info:2284] 0x81dad0a00: 0: NOTICE: QOS::rpc: qos_workload_maintainer: QoS workload data object info sync succesful 00000d93.00000e81 243bc949 Thu Sep 26 2019 04:33:51 -06:00 [kern_mgwd:info:2284] 0x81b7bbf00: 0: INFO: SFO_Ready|Cluster_Apps: wait_for_apps_online (Configure mdnsd): vifmgr is online. Number of tries: 19. 00000d93.00000e82 243bc949 Thu Sep 26 2019 04:33:51 -06:00 [kern_mgwd:info:2284] 0x81b7bbf00: 0: INFO: SFO_Ready|Cluster_Apps: wait_for_apps_online (Configure mdnsd): vifmgr is in running state. Number of tries: 19. 00000d93.00000e83 243bc949 Thu Sep 26 2019 04:33:51 -06:00 [kern_mgwd:info:2284] 0x81b7bbf00: 0: INFO: SFO_Ready|Cluster_Apps: wait_for_apps_online (waiter: Configure mdnsd): Finished after 18 tries. 00000d93.00000e84 243bc949 Thu Sep 26 2019 04:33:51 -06:00 [kern_mgwd:info:2284] 0x81b7bbf00: 0: INFO: MGWD::mdnsd_utils: stop_mdnsd:src/tables/mdnsd_utils.cc:555 mdnsd process stop. ref_cnt=0-1. 00000d93.00000e85 243bc949 Thu Sep 26 2019 04:33:51 -06:00 [kern_mgwd:info:2284] 0x81b7bbf00: 0: INFO: MGWD::mdnsd_utils: stop_mdnsd:src/tables/mdnsd_utils.cc:563 mdnsd process is already stopped. 00000d93.00000e86 243bc949 Thu Sep 26 2019 04:33:51 -06:00 [kern_mgwd:info:2284] 0x81b518500: 0: INFO: SFO_Ready|Cluster_Apps: wait_for_apps_online (RECONF_CLUSTER_PROXY_THREAD): vldb is online. Number of tries: 18. 00000d93.00000e87 243bc949 Thu Sep 26 2019 04:33:51 -06:00 [kern_mgwd:info:2284] 0x81b13fb00: 0: INFO: SFO_Ready|Cluster_Apps: wait_for_apps_online (Startup stability test): vldb is online. Number of tries: 18. 00000d93.00000e88 243bc949 Thu Sep 26 2019 04:33:51 -06:00 [kern_mgwd:info:2284] mhost_boot: MHOST_SF: Stable VLDB online 00000d93.00000e89 243bc949 Thu Sep 26 2019 04:33:51 -06:00 [kern_mgwd:info:2284] 0x81b518500: 0: INFO: SFO_Ready|Cluster_Apps: wait_for_apps_online (RECONF_CLUSTER_PROXY_THREAD): vifmgr is online. Number of tries: 18. 00000d93.00000e8a 243bc949 Thu Sep 26 2019 04:33:51 -06:00 [kern_mgwd:info:2284] 0x81b518500: 0: INFO: SFO_Ready|Cluster_Apps: wait_for_apps_online (RECONF_CLUSTER_PROXY_THREAD): vifmgr is in running state. Number of tries: 18. 00000d93.00000e8b 243bc949 Thu Sep 26 2019 04:33:51 -06:00 [kern_mgwd:info:2284] 0x81b13fb00: 0: INFO: SFO_Ready|Cluster_Apps: wait_for_apps_online (Startup stability test): vifmgr is online. Number of tries: 18. 00000d93.00000e8c 243bc949 Thu Sep 26 2019 04:33:51 -06:00 [kern_mgwd:info:2284] mhost_boot: MHOST_SF: Stable VIFMGR online 00000d93.00000e8d 243bc949 Thu Sep 26 2019 04:33:51 -06:00 [kern_mgwd:info:2284] 0x81b13fb00: 0: INFO: SFO_Ready|Cluster_Apps: wait_for_apps_online (Startup stability test): vifmgr is in running state. Number of tries: 18. 00000d93.00000e8e 243bc949 Thu Sep 26 2019 04:33:51 -06:00 [kern_mgwd:info:2284] mhost_boot: MHOST_SF: Stable VIFMGR running 00000d93.00000e8f 243bc949 Thu Sep 26 2019 04:33:51 -06:00 [kern_mgwd:info:2284] 0x81b518500: 0: INFO: SFO_Ready|Cluster_Apps: wait_for_apps_online (RECONF_CLUSTER_PROXY_THREAD): bcomd is online. Number of tries: 18. 00000d93.00000e90 243bc949 Thu Sep 26 2019 04:33:51 -06:00 [kern_mgwd:info:2284] 0x81b13fb00: 0: INFO: SFO_Ready|Cluster_Apps: wait_for_apps_online (Startup stability test): bcomd is online. Number of tries: 18. 00000d93.00000e91 243bc949 Thu Sep 26 2019 04:33:51 -06:00 [kern_mgwd:info:2284] mhost_boot: MHOST_SF: Stable BCOMD online 00000d93.00000e92 243bc949 Thu Sep 26 2019 04:33:51 -06:00 [kern_mgwd:info:2284] 0x81b518500: 0: INFO: SFO_Ready|Cluster_Apps: wait_for_apps_online (RECONF_CLUSTER_PROXY_THREAD): crs is online. Number of tries: 18. 00000d93.00000e93 243bc949 Thu Sep 26 2019 04:33:51 -06:00 [kern_mgwd:info:2284] 0x81b518500: 0: INFO: SFO_Ready|Cluster_Apps: wait_for_apps_online (waiter: RECONF_CLUSTER_PROXY_THREAD): Finished after 17 tries. 00000d93.00000e94 243bc949 Thu Sep 26 2019 04:33:51 -06:00 [kern_mgwd:info:2284] 0x81b518500: 0: NOTICE: Tables:web: SMF DSMDB service ready 00000d93.00000e95 243bc949 Thu Sep 26 2019 04:33:51 -06:00 [kern_mgwd:info:2284] 0x81b13fb00: 0: INFO: SFO_Ready|Cluster_Apps: wait_for_apps_online (Startup stability test): crs is online. Number of tries: 18. 00000d93.00000e96 243bc949 Thu Sep 26 2019 04:33:51 -06:00 [kern_mgwd:info:2284] mhost_boot: MHOST_SF: Stable CRS online 00000d93.00000e97 243bc949 Thu Sep 26 2019 04:33:51 -06:00 [kern_mgwd:info:2284] 0x81b13fb00: 0: INFO: SFO_Ready|Cluster_Apps: wait_for_apps_online (waiter: Startup stability test): Finished after 17 tries. 00000d93.00000e98 243bc949 Thu Sep 26 2019 04:33:51 -06:00 [kern_mgwd:info:2284] 0x81b13fb00: 0: INFO: SFO_Ready|Cluster_Apps: Startup stability test: All mhost apps have gone online. 00000d93.00000e99 243bc949 Thu Sep 26 2019 04:33:51 -06:00 [kern_mgwd:info:2284] mhost_boot: MHOST_SF: Stability Apps end 00000d93.00000e9a 243bc949 Thu Sep 26 2019 04:33:51 -06:00 [kern_mgwd:info:2284] 0x81b13fb00: 0: INFO: SFO_Ready|Cluster_Apps: Startup stability test: wait until FRS initialized. 00000d93.00000e9b 243bc949 Thu Sep 26 2019 04:33:51 -06:00 [kern_mgwd:info:2284] Startup stability test: waiting for file replication to initialize. 00000d93.00000e9c 243bc949 Thu Sep 26 2019 04:33:51 -06:00 [kern_mgwd:info:2284] 0x81b512b00: 0: INFO: SFO_Ready|Cluster_Apps: wait_for_stable_mhost (waiter: Coredump), maxTime 180 sec; stable after 37 sec. 00000d93.00000e9d 243bc949 Thu Sep 26 2019 04:33:51 -06:00 [kern_mgwd:info:2284] 0x81b517100: 0: INFO: SFO_Ready|Cluster_Apps: wait_for_stable_mhost (waiter: Setup), maxTime 900 sec; stable after 37 sec. 00000d93.00000e9e 243bc949 Thu Sep 26 2019 04:33:51 -06:00 [kern_mgwd:info:2284] mhost_boot: MHOST_SF: CFO stable apps wait OK/end 00000d93.00000e9f 243bc949 Thu Sep 26 2019 04:33:51 -06:00 [kern_mgwd:info:2284] 0x81b517100: 0: INFO: SFO_Ready|inform_dblade: inform_dblade_when_apps_online_pthread_func:69 finished wait_for_stable with 900 waitTime 00000d93.00000ea0 243bc949 Thu Sep 26 2019 04:33:51 -06:00 [kern_mgwd:info:2284] mhost_boot: MHOST_SF: inform dblade mhost stable 00000d93.00000ea1 243bc949 Thu Sep 26 2019 04:33:51 -06:00 [kern_mgwd:info:2284] wait_for_scsi_blade (waiter: Setup), maxTime 863 sec: starting 00000d93.00000ea2 243bc949 Thu Sep 26 2019 04:33:51 -06:00 [kern_mgwd:info:2284] 0x81b517100: 0: INFO: SFO_Ready|Cluster_Apps: wait_for_in_cluster (waiter: Setup), maxTime 863 sec: starting 00000d93.00000ea3 243bc949 Thu Sep 26 2019 04:33:51 -06:00 [kern_mgwd:info:2284] 0x81b517100: 0: INFO: SFO_Ready|Cluster_Apps: wait_for_in_cluster (waiter: Setup), maxTime 863 sec: in cluster after 0 sec. 00000d93.00000ea4 243bc949 Thu Sep 26 2019 04:33:51 -06:00 [kern_mgwd:info:2284] 0x81b517100: 0: INFO: SFO_Ready|Cluster_Apps: wait_for_apps_online (waiter: Setup) apps: bcomd, maxTime 863 sec: starting 00000d93.00000ea5 243bc949 Thu Sep 26 2019 04:33:51 -06:00 [kern_mgwd:info:2284] 0x81b517100: 0: INFO: SFO_Ready|Cluster_Apps: wait_for_apps_online (Setup): bcomd is online. Number of tries: 1. 00000d93.00000ea6 243bc949 Thu Sep 26 2019 04:33:51 -06:00 [kern_mgwd:info:2284] mhost_boot: MHOST_SF: CFO BCOMD online 00000d93.00000ea7 243bc949 Thu Sep 26 2019 04:33:51 -06:00 [kern_mgwd:info:2284] Coredump AutoSave: Queued job coredump-autosave 00000d93.00000ea8 243bc949 Thu Sep 26 2019 04:33:51 -06:00 [kern_mgwd:info:2284] In coredump_module::queue_coredump_job_thread() exit 00000d93.00000ea9 243bc949 Thu Sep 26 2019 04:33:51 -06:00 [kern_mgwd:info:2284] coredumpSaveJob: run 00000d93.00000eaa 243bc949 Thu Sep 26 2019 04:33:51 -06:00 [kern_mgwd:info:2284] coredumpSaveJob: execute 00000d93.00000eab 243bc949 Thu Sep 26 2019 04:33:51 -06:00 [kern_mgwd:info:2284] 0x81b517100: 0: INFO: SFO_Ready|Cluster_Apps: wait_for_apps_online (waiter: Setup): Finished after 0 tries. 00000d93.00000eac 243bc949 Thu Sep 26 2019 04:33:51 -06:00 [kern_mgwd:info:2284] 0x81b518500: 0: NOTICE: Tables:web: Updated the proxy server on node nmg-fas8040-02 00000d93.00000ead 243bc949 Thu Sep 26 2019 04:33:51 -06:00 [kern_mgwd:info:2284] 0x81b518500: 0: NOTICE: Tables:web: Completed cluster proxy reconfig thread 00000d93.00000eae 243bc949 Thu Sep 26 2019 04:33:51 -06:00 [kern_mgwd:info:2284] 0x81b518000: 0: INFO: SFO_Ready|Cluster_Apps: wait_for_apps_online (LIF_PLACEMENT_THREAD): bcomd is online. Number of tries: 19. 00000d93.00000eaf 243bc949 Thu Sep 26 2019 04:33:51 -06:00 [kern_mgwd:info:2284] 0x81b518000: 0: INFO: SFO_Ready|Cluster_Apps: wait_for_apps_online (waiter: LIF_PLACEMENT_THREAD): Finished after 18 tries. 00000d93.00000eb0 243bc949 Thu Sep 26 2019 04:33:51 -06:00 [kern_mgwd:info:2284] trigger_lif_modify_update 2765 LIF PLACEMENT THREAD: LIF MODIFY UPDATE: vifmgr and bcomd are UP and ONLINE 00000d93.00000eb1 243bc949 Thu Sep 26 2019 04:33:51 -06:00 [kern_mgwd:info:2284] 0x81b51a300: 0: INFO: SFO_Ready|Cluster_Apps: wait_for_apps_online (LIF_PLACEMENT_THREAD): bcomd is online. Number of tries: 19. 00000d93.00000eb2 243bc949 Thu Sep 26 2019 04:33:51 -06:00 [kern_mgwd:info:2284] 0x81b51a300: 0: INFO: SFO_Ready|Cluster_Apps: wait_for_apps_online (waiter: LIF_PLACEMENT_THREAD): Finished after 18 tries. 00000d93.00000eb3 243bc949 Thu Sep 26 2019 04:33:51 -06:00 [kern_mgwd:info:2284] handle_lif_placement 2673 HANDLE LIF PLACEMENT: vifmgr and bcomd are UP and ONLINE 00000d93.00000eb4 243bc949 Thu Sep 26 2019 04:33:52 -06:00 [kern_mgwd:info:2284] 0x81b13fb00: 0: WARNING: FileReplication: fr_notification_manager is initializing. Starting the work queue. 00000d93.00000eb5 243bc949 Thu Sep 26 2019 04:33:52 -06:00 [kern_mgwd:info:2284] 0x81b13fb00: 0: WARNING: FileReplication: fr_notification_manager successfully initialized. 00000d93.00000eb6 243bc949 Thu Sep 26 2019 04:33:52 -06:00 [kern_mgwd:info:2284] Startup stability test: file replication initialized successfully. 00000d93.00000eb7 243bc949 Thu Sep 26 2019 04:33:52 -06:00 [kern_mgwd:info:2284] 0x81b13fb00: 0: INFO: SFO_Ready|Cluster_Apps: Startup stability test: FRS initialized. 00000d93.00000eb8 243bc949 Thu Sep 26 2019 04:33:52 -06:00 [kern_mgwd:info:2284] 0x81b13fb00: 0: INFO: SFO_Ready|Cluster_Apps: Startup stability test: wait until XC_MGR initialized. 00000d93.00000eb9 243bc949 Thu Sep 26 2019 04:33:52 -06:00 [kern_mgwd:info:2284] 0x81b13fb00: 0: INFO: SFO_Ready|Cluster_Apps: Startup stability test: XC_MGR initialized. 00000d93.00000eba 243bc949 Thu Sep 26 2019 04:33:52 -06:00 [kern_mgwd:info:2284] 0x81b13fb00: 0: INFO: SFO_Ready|Cluster_Apps: Startup stability test: done. 00000d93.00000ebb 243bc949 Thu Sep 26 2019 04:33:52 -06:00 [kern_mgwd:info:2284] mhost_boot: MHOST_SF: Stability end 00000d93.00000ebc 243bc949 Thu Sep 26 2019 04:33:52 -06:00 [kern_mgwd:info:2284] 0x81d24da00: 8603e900000000c3: WARNING: FileReplication: Completed consistency scan of local pubroot. 00000d93.00000eef 243bc949 Thu Sep 26 2019 04:33:52 -06:00 [kern_mgwd:info:2284] coredumpSaveJob: At least one disk 4c.30.0 of container type 'unknown' homed locally (homeid: 536876827) is currently owned by another node (ownerid: 536877897), and is not a candidate for savecore, and is being skipped. 00000d93.00000ef0 243bc949 Thu Sep 26 2019 04:33:52 -06:00 [kern_mgwd:info:2284] coredumpSaveJob: At least one disk 0b.30.1 of container type 'unknown' homed locally (homeid: 536876827) is currently owned by another node (ownerid: 536877897), and is not a candidate for savecore, and is being skipped. 00000d93.00000ef1 243bc949 Thu Sep 26 2019 04:33:52 -06:00 [kern_mgwd:info:2284] coredumpSaveJob: At least one disk 4c.30.2 of container type 'unknown' homed locally (homeid: 536876827) is currently owned by another node (ownerid: 536877897), and is not a candidate for savecore, and is being skipped. 00000d93.00000ef2 243bc949 Thu Sep 26 2019 04:33:52 -06:00 [kern_mgwd:info:2284] coredumpSaveJob: At least one disk 0b.30.3 of container type 'unknown' homed locally (homeid: 536876827) is currently owned by another node (ownerid: 536877897), and is not a candidate for savecore, and is being skipped. 00000d93.00000ef3 243bc949 Thu Sep 26 2019 04:33:52 -06:00 [kern_mgwd:info:2284] coredumpSaveJob: At least one disk 4c.30.4 of container type 'unknown' homed locally (homeid: 536876827) is currently owned by another node (ownerid: 536877897), and is not a candidate for savecore, and is being skipped. 00000d93.00000ef4 243bc949 Thu Sep 26 2019 04:33:52 -06:00 [kern_mgwd:info:2284] coredumpSaveJob: At least one disk 0b.30.5 of container type 'unknown' homed locally (homeid: 536876827) is currently owned by another node (ownerid: 536877897), and is not a candidate for savecore, and is being skipped. 00000d93.00000ef5 243bc949 Thu Sep 26 2019 04:33:52 -06:00 [kern_mgwd:info:2284] coredumpSaveJob: At least one disk 4a.42.0 of container type 'unknown' homed locally (homeid: 536876827) is currently owned by another node (ownerid: 536877897), and is not a candidate for savecore, and is being skipped. 00000d93.00000ef6 243bc949 Thu Sep 26 2019 04:33:52 -06:00 [kern_mgwd:info:2284] coredumpSaveJob: At least one disk 4a.42.2 of container type 'unknown' homed locally (homeid: 536876827) is currently owned by another node (ownerid: 536877897), and is not a candidate for savecore, and is being skipped. 00000d93.00000ef7 243bc949 Thu Sep 26 2019 04:33:52 -06:00 [kern_mgwd:info:2284] coredumpSaveJob: At least one disk 4a.42.3 of container type 'unknown' homed locally (homeid: 536876827) is currently owned by another node (ownerid: 536877897), and is not a candidate for savecore, and is being skipped. 00000d93.00000ef8 243bc949 Thu Sep 26 2019 04:33:52 -06:00 [kern_mgwd:info:2284] coredumpSaveJob: At least one disk 4a.42.5 of container type 'unknown' homed locally (homeid: 536876827) is currently owned by another node (ownerid: 536877897), and is not a candidate for savecore, and is being skipped. 00000d93.00000ef9 243bc949 Thu Sep 26 2019 04:33:52 -06:00 [kern_mgwd:info:2284] coredumpSaveJob: At least one disk 4a.42.7 of container type 'unknown' homed locally (homeid: 536876827) is currently owned by another node (ownerid: 536877897), and is not a candidate for savecore, and is being skipped. 00000d93.00000efa 243bc949 Thu Sep 26 2019 04:33:52 -06:00 [kern_mgwd:info:2284] coredumpSaveJob: At least one disk 4a.42.8 of container type 'unknown' homed locally (homeid: 536876827) is currently owned by another node (ownerid: 536877897), and is not a candidate for savecore, and is being skipped. 00000d93.00000efb 243bc949 Thu Sep 26 2019 04:33:52 -06:00 [kern_mgwd:info:2284] coredumpSaveJob: At least one disk 4a.42.11 of container type 'unknown' homed locally (homeid: 536876827) is currently owned by another node (ownerid: 536877897), and is not a candidate for savecore, and is being skipped. 00000d93.00000efc 243bc949 Thu Sep 26 2019 04:33:52 -06:00 [kern_mgwd:info:2284] coredumpSaveJob: At least one disk 4a.42.4 of container type 'unknown' homed locally (homeid: 536876827) is currently owned by another node (ownerid: 536877897), and is not a candidate for savecore, and is being skipped. 00000d93.00000efd 243bc949 Thu Sep 26 2019 04:33:52 -06:00 [kern_mgwd:info:2284] coredumpSaveJob: At least one disk 4a.42.9 of container type 'unknown' homed locally (homeid: 536876827) is currently owned by another node (ownerid: 536877897), and is not a candidate for savecore, and is being skipped. 00000d93.00000efe 243bc949 Thu Sep 26 2019 04:33:52 -06:00 [kern_mgwd:info:2284] coredumpSaveJob: At least one disk 4a.42.10 of container type 'unknown' homed locally (homeid: 536876827) is currently owned by another node (ownerid: 536877897), and is not a candidate for savecore, and is being skipped. 00000d93.00000eff 243bc949 Thu Sep 26 2019 04:33:52 -06:00 [kern_mgwd:info:2284] coredumpSaveJob: At least one disk 4a.42.6 of container type 'unknown' homed locally (homeid: 536876827) is currently owned by another node (ownerid: 536877897), and is not a candidate for savecore, and is being skipped. 00000d93.00000f00 243bc949 Thu Sep 26 2019 04:33:52 -06:00 [kern_mgwd:info:2284] coredumpSaveJob: At least one disk 4a.42.13 of container type 'unknown' homed locally (homeid: 536876827) is currently owned by another node (ownerid: 536877897), and is not a candidate for savecore, and is being skipped. 00000d93.00000f01 243bc949 Thu Sep 26 2019 04:33:52 -06:00 [kern_mgwd:info:2284] coredumpSaveJob: At least one disk 4a.42.15 of container type 'unknown' homed locally (homeid: 536876827) is currently owned by another node (ownerid: 536877897), and is not a candidate for savecore, and is being skipped. 00000d93.00000f02 243bc949 Thu Sep 26 2019 04:33:52 -06:00 [kern_mgwd:info:2284] coredumpSaveJob: At least one disk 4a.42.12 of container type 'unknown' homed locally (homeid: 536876827) is currently owned by another node (ownerid: 536877897), and is not a candidate for savecore, and is being skipped. 00000d93.00000f03 243bc949 Thu Sep 26 2019 04:33:52 -06:00 [kern_mgwd:info:2284] coredumpSaveJob: At least one disk 0d.41.1 of container type 'unknown' homed locally (homeid: 536876827) is currently owned by another node (ownerid: 536877897), and is not a candidate for savecore, and is being skipped. 00000d93.00000f04 243bc949 Thu Sep 26 2019 04:33:52 -06:00 [kern_mgwd:info:2284] coredumpSaveJob: At least one disk 4a.41.2 of container type 'unknown' homed locally (homeid: 536876827) is currently owned by another node (ownerid: 536877897), and is not a candidate for savecore, and is being skipped. 00000d93.00000f05 243bc949 Thu Sep 26 2019 04:33:52 -06:00 [kern_mgwd:info:2284] coredumpSaveJob: At least one disk 4a.41.0 of container type 'unknown' homed locally (homeid: 536876827) is currently owned by another node (ownerid: 536877897), and is not a candidate for savecore, and is being skipped. 00000d93.00000f06 243bc949 Thu Sep 26 2019 04:33:52 -06:00 [kern_mgwd:info:2284] coredumpSaveJob: At least one disk 0d.41.3 of container type 'unknown' homed locally (homeid: 536876827) is currently owned by another node (ownerid: 536877897), and is not a candidate for savecore, and is being skipped. 00000d93.00000f07 243bc949 Thu Sep 26 2019 04:33:52 -06:00 [kern_mgwd:info:2284] coredumpSaveJob: At least one disk 4a.41.6 of container type 'unknown' homed locally (homeid: 536876827) is currently owned by another node (ownerid: 536877897), and is not a candidate for savecore, and is being skipped. 00000d93.00000f08 243bc949 Thu Sep 26 2019 04:33:52 -06:00 [kern_mgwd:info:2284] coredumpSaveJob: At least one disk 0d.41.5 of container type 'unknown' homed locally (homeid: 536876827) is currently owned by another node (ownerid: 536877897), and is not a candidate for savecore, and is being skipped. 00000d93.00000f09 243bc949 Thu Sep 26 2019 04:33:52 -06:00 [kern_mgwd:info:2284] coredumpSaveJob: At least one disk 4a.42.14 of container type 'unknown' homed locally (homeid: 536876827) is currently owned by another node (ownerid: 536877897), and is not a candidate for savecore, and is being skipped. 00000d93.00000f0a 243bc949 Thu Sep 26 2019 04:33:52 -06:00 [kern_mgwd:info:2284] coredumpSaveJob: At least one disk 4a.41.8 of container type 'unknown' homed locally (homeid: 536876827) is currently owned by another node (ownerid: 536877897), and is not a candidate for savecore, and is being skipped. 00000d93.00000f0b 243bc949 Thu Sep 26 2019 04:33:52 -06:00 [kern_mgwd:info:2284] coredumpSaveJob: At least one disk 0d.41.7 of container type 'unknown' homed locally (homeid: 536876827) is currently owned by another node (ownerid: 536877897), and is not a candidate for savecore, and is being skipped. 00000d93.00000f0c 243bc949 Thu Sep 26 2019 04:33:52 -06:00 [kern_mgwd:info:2284] coredumpSaveJob: At least one disk 4a.41.4 of container type 'unknown' homed locally (homeid: 536876827) is currently owned by another node (ownerid: 536877897), and is not a candidate for savecore, and is being skipped. 00000d93.00000f0d 243bc949 Thu Sep 26 2019 04:33:52 -06:00 [kern_mgwd:info:2284] coredumpSaveJob: At least one disk 0d.41.9 of container type 'unknown' homed locally (homeid: 536876827) is currently owned by another node (ownerid: 536877897), and is not a candidate for savecore, and is being skipped. 00000d93.00000f0e 243bc949 Thu Sep 26 2019 04:33:52 -06:00 [kern_mgwd:info:2284] coredumpSaveJob: At least one disk 4a.41.12 of container type 'unknown' homed locally (homeid: 536876827) is currently owned by another node (ownerid: 536877897), and is not a candidate for savecore, and is being skipped. 00000d93.00000f0f 243bc949 Thu Sep 26 2019 04:33:52 -06:00 [kern_mgwd:info:2284] coredumpSaveJob: At least one disk 4a.41.14 of container type 'unknown' homed locally (homeid: 536876827) is currently owned by another node (ownerid: 536877897), and is not a candidate for savecore, and is being skipped. 00000d93.00000f10 243bc949 Thu Sep 26 2019 04:33:52 -06:00 [kern_mgwd:info:2284] coredumpSaveJob: At least one disk 0d.41.11 of container type 'unknown' homed locally (homeid: 536876827) is currently owned by another node (ownerid: 536877897), and is not a candidate for savecore, and is being skipped. 00000d93.00000f11 243bc949 Thu Sep 26 2019 04:33:52 -06:00 [kern_mgwd:info:2284] coredumpSaveJob: At least one disk 0d.41.15 of container type 'unknown' homed locally (homeid: 536876827) is currently owned by another node (ownerid: 536877897), and is not a candidate for savecore, and is being skipped. 00000d93.00000f12 243bc949 Thu Sep 26 2019 04:33:52 -06:00 [kern_mgwd:info:2284] coredumpSaveJob: At least one disk 0d.41.21 of container type 'unknown' homed locally (homeid: 536876827) is currently owned by another node (ownerid: 536877897), and is not a candidate for savecore, and is being skipped. 00000d93.00000f13 243bc949 Thu Sep 26 2019 04:33:52 -06:00 [kern_mgwd:info:2284] coredumpSaveJob: At least one disk 4a.41.18 of container type 'unknown' homed locally (homeid: 536876827) is currently owned by another node (ownerid: 536877897), and is not a candidate for savecore, and is being skipped. 00000d93.00000f14 243bc949 Thu Sep 26 2019 04:33:52 -06:00 [kern_mgwd:info:2284] coredumpSaveJob: At least one disk 0d.41.13 of container type 'unknown' homed locally (homeid: 536876827) is currently owned by another node (ownerid: 536877897), and is not a candidate for savecore, and is being skipped. 00000d93.00000f15 243bc949 Thu Sep 26 2019 04:33:52 -06:00 [kern_mgwd:info:2284] coredumpSaveJob: At least one disk 0d.41.23 of container type 'unknown' homed locally (homeid: 536876827) is currently owned by another node (ownerid: 536877897), and is not a candidate for savecore, and is being skipped. 00000d93.00000f16 243bc949 Thu Sep 26 2019 04:33:52 -06:00 [kern_mgwd:info:2284] coredumpSaveJob: At least one disk 4a.41.16 of container type 'unknown' homed locally (homeid: 536876827) is currently owned by another node (ownerid: 536877897), and is not a candidate for savecore, and is being skipped. 00000d93.00000f17 243bc949 Thu Sep 26 2019 04:33:52 -06:00 [kern_mgwd:info:2284] coredumpSaveJob: At least one disk 4a.41.20 of container type 'unknown' homed locally (homeid: 536876827) is currently owned by another node (ownerid: 536877897), and is not a candidate for savecore, and is being skipped. 00000d93.00000f18 243bc949 Thu Sep 26 2019 04:33:52 -06:00 [kern_mgwd:info:2284] coredumpSaveJob: At least one disk 4a.41.22 of container type 'unknown' homed locally (homeid: 536876827) is currently owned by another node (ownerid: 536877897), and is not a candidate for savecore, and is being skipped. 00000d93.00000f19 243bc94a Thu Sep 26 2019 04:33:52 -06:00 [kern_mgwd:info:2284] coredumpSaveJob: At least one disk 0d.40.0 of container type 'unknown' homed locally (homeid: 536876827) is currently owned by another node (ownerid: 536877897), and is not a candidate for savecore, and is being skipped. 00000d93.00000f1a 243bc94a Thu Sep 26 2019 04:33:52 -06:00 [kern_mgwd:info:2284] coredumpSaveJob: At least one disk 0d.40.2 of container type 'unknown' homed locally (homeid: 536876827) is currently owned by another node (ownerid: 536877897), and is not a candidate for savecore, and is being skipped. 00000d93.00000f1b 243bc94a Thu Sep 26 2019 04:33:52 -06:00 [kern_mgwd:info:2284] coredumpSaveJob: At least one disk 0d.40.1 of container type 'unknown' homed locally (homeid: 536876827) is currently owned by another node (ownerid: 536877897), and is not a candidate for savecore, and is being skipped. 00000d93.00000f1c 243bc94a Thu Sep 26 2019 04:33:52 -06:00 [kern_mgwd:info:2284] coredumpSaveJob: At least one disk 0d.40.4 of container type 'unknown' homed locally (homeid: 536876827) is currently owned by another node (ownerid: 536877897), and is not a candidate for savecore, and is being skipped. 00000d93.00000f1d 243bc94a Thu Sep 26 2019 04:33:52 -06:00 [kern_mgwd:info:2284] coredumpSaveJob: At least one disk 0d.40.6 of container type 'unknown' homed locally (homeid: 536876827) is currently owned by another node (ownerid: 536877897), and is not a candidate for savecore, and is being skipped. 00000d93.00000f1e 243bc94a Thu Sep 26 2019 04:33:52 -06:00 [kern_mgwd:info:2284] coredumpSaveJob: At least one disk 0d.40.9 of container type 'unknown' homed locally (homeid: 536876827) is currently owned by another node (ownerid: 536877897), and is not a candidate for savecore, and is being skipped. 00000d93.00000f1f 243bc94a Thu Sep 26 2019 04:33:52 -06:00 [kern_mgwd:info:2284] coredumpSaveJob: At least one disk 0d.40.10 of container type 'unknown' homed locally (homeid: 536876827) is currently owned by another node (ownerid: 536877897), and is not a candidate for savecore, and is being skipped. 00000d93.00000f20 243bc94a Thu Sep 26 2019 04:33:52 -06:00 [kern_mgwd:info:2284] coredumpSaveJob: At least one disk 0d.40.5 of container type 'unknown' homed locally (homeid: 536876827) is currently owned by another node (ownerid: 536877897), and is not a candidate for savecore, and is being skipped. 00000d93.00000f21 243bc94a Thu Sep 26 2019 04:33:52 -06:00 [kern_mgwd:info:2284] coredumpSaveJob: At least one disk 0d.40.8 of container type 'unknown' homed locally (homeid: 536876827) is currently owned by another node (ownerid: 536877897), and is not a candidate for savecore, and is being skipped. 00000d93.00000f22 243bc94a Thu Sep 26 2019 04:33:52 -06:00 [kern_mgwd:info:2284] coredumpSaveJob: At least one disk 0d.40.11 of container type 'unknown' homed locally (homeid: 536876827) is currently owned by another node (ownerid: 536877897), and is not a candidate for savecore, and is being skipped. 00000d93.00000f23 243bc94a Thu Sep 26 2019 04:33:52 -06:00 [kern_mgwd:info:2284] coredumpSaveJob: At least one disk 0d.40.13 of container type 'unknown' homed locally (homeid: 536876827) is currently owned by another node (ownerid: 536877897), and is not a candidate for savecore, and is being skipped. 00000d93.00000f24 243bc94a Thu Sep 26 2019 04:33:52 -06:00 [kern_mgwd:info:2284] coredumpSaveJob: At least one disk 0d.40.7 of container type 'unknown' homed locally (homeid: 536876827) is currently owned by another node (ownerid: 536877897), and is not a candidate for savecore, and is being skipped. 00000d93.00000f25 243bc94a Thu Sep 26 2019 04:33:52 -06:00 [kern_mgwd:info:2284] coredumpSaveJob: At least one disk 0d.40.14 of container type 'unknown' homed locally (homeid: 536876827) is currently owned by another node (ownerid: 536877897), and is not a candidate for savecore, and is being skipped. 00000d93.00000f26 243bc94a Thu Sep 26 2019 04:33:52 -06:00 [kern_mgwd:info:2284] coredumpSaveJob: At least one disk 0d.40.12 of container type 'unknown' homed locally (homeid: 536876827) is currently owned by another node (ownerid: 536877897), and is not a candidate for savecore, and is being skipped. 00000d93.00000f27 243bc94a Thu Sep 26 2019 04:33:52 -06:00 [kern_mgwd:info:2284] coredumpSaveJob: At least one disk 0d.40.16 of container type 'unknown' homed locally (homeid: 536876827) is currently owned by another node (ownerid: 536877897), and is not a candidate for savecore, and is being skipped. 00000d93.00000f28 243bc94a Thu Sep 26 2019 04:33:52 -06:00 [kern_mgwd:info:2284] coredumpSaveJob: At least one disk 0d.40.15 of container type 'unknown' homed locally (homeid: 536876827) is currently owned by another node (ownerid: 536877897), and is not a candidate for savecore, and is being skipped. 00000d93.00000f29 243bc94a Thu Sep 26 2019 04:33:52 -06:00 [kern_mgwd:info:2284] coredumpSaveJob: At least one disk 0d.40.18 of container type 'unknown' homed locally (homeid: 536876827) is currently owned by another node (ownerid: 536877897), and is not a candidate for savecore, and is being skipped. 00000d93.00000f2a 243bc94a Thu Sep 26 2019 04:33:52 -06:00 [kern_mgwd:info:2284] coredumpSaveJob: At least one disk 0d.41.17 of container type 'unknown' homed locally (homeid: 536876827) is currently owned by another node (ownerid: 536877897), and is not a candidate for savecore, and is being skipped. 00000d93.00000f2b 243bc94a Thu Sep 26 2019 04:33:52 -06:00 [kern_mgwd:info:2284] coredumpSaveJob: At least one disk 0d.40.20 of container type 'unknown' homed locally (homeid: 536876827) is currently owned by another node (ownerid: 536877897), and is not a candidate for savecore, and is being skipped. 00000d93.00000f2c 243bc94a Thu Sep 26 2019 04:33:52 -06:00 [kern_mgwd:info:2284] coredumpSaveJob: At least one disk 4a.41.10 of container type 'unknown' homed locally (homeid: 536876827) is currently owned by another node (ownerid: 536877897), and is not a candidate for savecore, and is being skipped. 00000d93.00000f2d 243bc94a Thu Sep 26 2019 04:33:52 -06:00 [kern_mgwd:info:2284] coredumpSaveJob: At least one disk 0d.40.22 of container type 'unknown' homed locally (homeid: 536876827) is currently owned by another node (ownerid: 536877897), and is not a candidate for savecore, and is being skipped. 00000d93.00000f2e 243bc94a Thu Sep 26 2019 04:33:52 -06:00 [kern_mgwd:info:2284] coredumpSaveJob: At least one disk 0d.40.17 of container type 'unknown' homed locally (homeid: 536876827) is currently owned by another node (ownerid: 536877897), and is not a candidate for savecore, and is being skipped. 00000d93.00000f2f 243bc94a Thu Sep 26 2019 04:33:52 -06:00 [kern_mgwd:info:2284] coredumpSaveJob: At least one disk 0d.40.23 of container type 'unknown' homed locally (homeid: 536876827) is currently owned by another node (ownerid: 536877897), and is not a candidate for savecore, and is being skipped. 00000d93.00000f30 243bc94a Thu Sep 26 2019 04:33:52 -06:00 [kern_mgwd:info:2284] coredumpSaveJob: At least one disk 0d.40.19 of container type 'unknown' homed locally (homeid: 536876827) is currently owned by another node (ownerid: 536877897), and is not a candidate for savecore, and is being skipped. 00000d93.00000f31 243bc94a Thu Sep 26 2019 04:33:52 -06:00 [kern_mgwd:info:2284] coredumpSaveJob: At least one disk 0d.40.21 of container type 'unknown' homed locally (homeid: 536876827) is currently owned by another node (ownerid: 536877897), and is not a candidate for savecore, and is being skipped. 00000d93.00000f32 243bc94a Thu Sep 26 2019 04:33:52 -06:00 [kern_mgwd:info:2284] coredumpSaveJob: At least one disk 0d.41.19 of container type 'unknown' homed locally (homeid: 536876827) is currently owned by another node (ownerid: 536877897), and is not a candidate for savecore, and is being skipped. 00000d93.00000f33 243bc94a Thu Sep 26 2019 04:33:52 -06:00 [kern_mgwd:info:2284] coredumpSaveJob: At least one disk 0d.40.3 of container type 'unknown' homed locally (homeid: 536876827) is currently owned by another node (ownerid: 536877897), and is not a candidate for savecore, and is being skipped. 00000d93.00000f34 243bc94a Thu Sep 26 2019 04:33:52 -06:00 [kern_mgwd:info:2284] coredumpSaveJob: At least one disk 4b.20.3 of container type 'unknown' homed locally (homeid: 536876827) is currently owned by another node (ownerid: 536877897), and is not a candidate for savecore, and is being skipped. 00000d93.00000f35 243bc94a Thu Sep 26 2019 04:33:52 -06:00 [kern_mgwd:info:2284] 0x81d248000: 0: NOTICE: RpcConnectionCache: clearFailure: called for 127.0.0.1:536873528(upgrademgr)/1[tcp]{fffffffd|8} 00000d93.00000f37 243bc94a Thu Sep 26 2019 04:33:52 -06:00 [kern_mgwd:info:2284] 0x81d248000: 0: NOTICE: rdb_sitelist: slpush: successfully pushed site information to _push_sitelist_id[20] nodelist_status non_authoritative 00000d93.00000f38 243bc94a Thu Sep 26 2019 04:33:52 -06:00 [kern_mgwd:info:2284] 0x81d248000: 0: NOTICE: RpcConnectionCache: clearFailure: called for 127.0.0.1:536873556(vserverdr)/1[tcp]{fffffffd|8} 00000d93.00000f3a 243bc94a Thu Sep 26 2019 04:33:52 -06:00 [kern_mgwd:info:2284] 0x81d248000: 0: NOTICE: rdb_sitelist: slpush: successfully pushed site information to _push_sitelist_id[34] nodelist_status non_authoritative 00000d93.00000f3b 243bc94a Thu Sep 26 2019 04:33:52 -06:00 [kern_mgwd:info:2284] 0x81d248000: 0: NOTICE: RpcConnectionCache: clearFailure: called for 127.0.0.1:536873564(No name for prog:536873564)/1[tcp]{fffffffd|8} 00000d93.00000f3d 243bc94a Thu Sep 26 2019 04:33:52 -06:00 [kern_mgwd:info:2284] 0x81d248000: 0: NOTICE: rdb_sitelist: slpush: successfully pushed site information to _push_sitelist_id[38] nodelist_status non_authoritative 00000d93.00000f3f 243bc94a Thu Sep 26 2019 04:33:53 -06:00 [kern_mgwd:info:2284] Process coreseg_segment_start request: 0 00000d93.00000f40 243bc94a Thu Sep 26 2019 04:33:53 -06:00 [kern_mgwd:info:2284] coredumpSaveJob: complete 00000d93.00000f41 243bc96d Thu Sep 26 2019 04:33:55 -06:00 [kern_mgwd:info:2284] 0x81b512600: 0: ERR: security_mgwd::tables::certificate::certificate: write_hash_files:src/tables/certificate/certificate.cc:1985 SSL: Failed to create symlink for file '/mroot/etc/vserver_4294967295/certificates/ssl/serverCa/99+4A538C28+EntrustRootCertificationAuthority-G2+455f1b52.0.pem' with '/mroot/etc/vserver_4294967295/certificates/ssl/serverCa/455f1b52.0'. Reason: 17 00000d93.00000f42 243bc96d Thu Sep 26 2019 04:33:56 -06:00 [kern_mgwd:info:2284] 0x81b7bce00: 0: ERR: RPCServer: mgwd_ready_for_replay: mgwd and vifmgr ready 00000d93.00000f63 243bca00 Thu Sep 26 2019 04:34:10 -06:00 [kern_mgwd:info:2284] 0x81b24a600: 8303e90000000004: NOTICE: MGWD::VSERVER_INFO: src/tables/vserver_export/vserver_info.cc:assign_tgt_dev_id_imp:332 enter: vs: d5e08d96-c771-11e7-948b-00a09857ee8f 00000d93.00000f64 243bca00 Thu Sep 26 2019 04:34:10 -06:00 [kern_mgwd:info:2284] 0x81b24a600: 8303e90000000004: NOTICE: MGWD::VSERVER_INFO: src/tables/vserver_export/vserver_info.cc:assign_tgt_dev_id_imp:363 returning: 0/0 - success 00000d93.00000f6a 243bca00 Thu Sep 26 2019 04:34:11 -06:00 [kern_mgwd:info:2284] wait_for_scsi_blade (Setup): scsi_blade is initialized. Number of tries: 11. 00000d93.00000f6b 243bca00 Thu Sep 26 2019 04:34:11 -06:00 [kern_mgwd:info:2284] wait_for_scsi_blade (waiter: Setup), maxTime 863 sec; initialized after 20 sec. 00000d93.00000f6c 243bca00 Thu Sep 26 2019 04:34:11 -06:00 [kern_mgwd:info:2284] mhost_boot: MHOST_SF: inform dblade scsi done 00000d93.00000f6d 243bca00 Thu Sep 26 2019 04:34:11 -06:00 [kern_mgwd:info:2284] wait_for_clam_quorum (waiter: Setup), maxTime 806 sec: starting 00000d93.00000f6e 243bca00 Thu Sep 26 2019 04:34:11 -06:00 [kern_mgwd:info:2284] wait_for_clam_quorum (Setup): CLAM is in quorum. Number of tries: 1. 00000d93.00000f6f 243bca00 Thu Sep 26 2019 04:34:11 -06:00 [kern_mgwd:info:2284] wait_for_clam_quorum (waiter: Setup), maxTime 806 sec; initialized after 0 sec. 00000d93.00000f70 243bca00 Thu Sep 26 2019 04:34:11 -06:00 [kern_mgwd:info:2284] mhost_boot: MHOST_SF: inform dblade clam done 00000d93.00000f71 243bca00 Thu Sep 26 2019 04:34:11 -06:00 [kern_mgwd:info:2284] 0x81b517100: 0: INFO: SFO_Ready|inform_dblade: inform_dblade_when_apps_online_pthread_func:114 done waiting 00000d93.00000f72 243bca00 Thu Sep 26 2019 04:34:11 -06:00 [kern_mgwd:info:2284] mhost_boot: MHOST_SF: async upgrade check 00000d93.00000f73 243bca00 Thu Sep 26 2019 04:34:11 -06:00 [kern_mgwd:info:2284] 0x81b517100: 0: INFO: SFO_Ready|inform_dblade: inform_dblade_when_apps_online_pthread_func:120 done async_upgrade_check 00000d93.00000f74 243bca00 Thu Sep 26 2019 04:34:11 -06:00 [kern_mgwd:info:2284] mhost_boot: MHOST_SF: set system stable start 00000d93.00000f75 243bca00 Thu Sep 26 2019 04:34:11 -06:00 [kern_mgwd:info:2284] mhost_boot: MHOST_SF: set system stable end 00000d93.00000f76 243bca00 Thu Sep 26 2019 04:34:11 -06:00 [kern_mgwd:info:2284] ksmf set_system_stable_after_startup_iterator succeeded. 00000d93.00000f77 243bca00 Thu Sep 26 2019 04:34:11 -06:00 [kern_mgwd:info:2284] 0x81b517100: 0: INFO: SFO_Ready|inform_dblade: inform_dblade_when_apps_online_pthread_func:142 done setting system stable 00000d93.00000f78 243bca00 Thu Sep 26 2019 04:34:11 -06:00 [kern_mgwd:info:2284] Notified nwd in the kernel 00000d93.00000f79 243bca00 Thu Sep 26 2019 04:34:11 -06:00 [kern_mgwd:info:2284] 0x81b517100: 0: INFO: SFO_Ready|inform_dblade: inform_dblade_when_apps_online_pthread_func:147 NWD started 00000d93.00000f7a 243bca00 Thu Sep 26 2019 04:34:11 -06:00 [kern_mgwd:info:2284] During startup: got cluster_uuid: 635f71dd-b79d-11e7-9b07-00a09857ee8f (is-nil: false) from local cluster_id table. 00000d93.00000f7b 243bca00 Thu Sep 26 2019 04:34:11 -06:00 [kern_mgwd:info:2284] During startup: Verified that the cluster uuid matches the local mailbox cluster uuid: 635f71dd-b79d-11e7-9b07-00a09857ee8f. 00000d93.00000f7c 243bca00 Thu Sep 26 2019 04:34:11 -06:00 [kern_mgwd:info:2284] 0x81b517100: 0: INFO: SFO_Ready|inform_dblade: inform_dblade_when_apps_online_pthread_func:158 returning 00000d93.00000f7d 243bca00 Thu Sep 26 2019 04:34:11 -06:00 [kern_mgwd:info:2284] mhost_boot: MHOST_SF: inform dblade end 00000d93.00000f81 243bca1e Thu Sep 26 2019 04:34:13 -06:00 [kern_mgwd:info:2284] check_defaults(): RDB Unit is online. Trying to create old and new defaults. 00000d93.00000f82 243bca1e Thu Sep 26 2019 04:34:13 -06:00 [kern_mgwd:info:2284] check_defaults(): Registering old table callbacks with RDB 00000d93.00000f83 243bca1e Thu Sep 26 2019 04:34:13 -06:00 [kern_mgwd:info:2284] check_defaults(): Successfully unregistered EMS config callbacks 00000d93.00000f84 243bca1e Thu Sep 26 2019 04:34:13 -06:00 [kern_mgwd:info:2284] check_defaults(): Registering new table callbacks with RDB 00000d93.00000f85 243bca1e Thu Sep 26 2019 04:34:13 -06:00 [kern_mgwd:info:2284] 0x81d249400: 0: INFO: EMS::router_tables: ems_notifyd_destination_synchronizer::update_notifyd_table: re-sync'ing notifyd table. 00000d93.00000f86 243bca1e Thu Sep 26 2019 04:34:13 -06:00 [kern_mgwd:info:2284] 0x81d249400: 0: INFO: EMS::router_tables: ems_notifyd_destination_synchronizer::update_notifyd_table: Successfully updated table. 00000d93.00000f87 243bca1e Thu Sep 26 2019 04:34:13 -06:00 [kern_mgwd:info:2284] 0x81d249400: 0: INFO: EMS::router_tables: ems_notifyd_route_synchronizer::update_notifyd_table: re-sync'ing notifyd tables. 00000d93.00000f88 243bca1e Thu Sep 26 2019 04:34:13 -06:00 [kern_mgwd:info:2284] 0x81d249400: 0: INFO: EMS::router_tables: ems_notifyd_route_synchronizer::update_notifyd_table: Successfully updated notifyd table. 00000d93.00000f89 243bca1e Thu Sep 26 2019 04:34:13 -06:00 [kern_mgwd:info:2284] 0x81d249400: 0: INFO: EMS::MgwdNdTableSync: mgwd_nd_table_synchronizer::update_notifyd_table: re-sync'ing all tables in notifyd... 00000d93.00000f9f 243bca1e Thu Sep 26 2019 04:34:13 -06:00 [kern_mgwd:info:2284] 0x81d249400: 0: INFO: EMS::MgwdNdTableSync: mgwd_nd_table_synchronizer::clearNdTables: all tables in notifyd successfully cleared. 00000d93.00000fa0 243bca1e Thu Sep 26 2019 04:34:13 -06:00 [kern_mgwd:info:2284] 0x81d249400: 0: INFO: EMS::DEST_CALLBACK: ems_nd_event_destination_synchronizer::update_notifyd_table: re-sync'ing notifyd table. 00000d93.00000fa1 243bca1e Thu Sep 26 2019 04:34:13 -06:00 [kern_mgwd:info:2284] 0x81d249400: 0: INFO: EMS::DEST_CALLBACK: ems_nd_event_destination_synchronizer::update_notifyd_table: Updating snmp-traphost 00000d93.00000fa6 243bca1e Thu Sep 26 2019 04:34:13 -06:00 [kern_mgwd:info:2284] 0x81d249400: 0: INFO: EMS::DEST_CALLBACK: ems_nd_event_destination_synchronizer::update_notifyd_table: Successfully updated notifyd table. 00000d93.00000fa7 243bca1e Thu Sep 26 2019 04:34:13 -06:00 [kern_mgwd:info:2284] 0x81d249400: 0: INFO: EMS::FILTER_CALLBACK: ems_nd_event_filter_synchronizer::update_notifyd_table: re-sync'ing notifyd tables. 00000d93.00000fab 243bca1e Thu Sep 26 2019 04:34:13 -06:00 [kern_mgwd:info:2284] 0x81d249400: 0: INFO: EMS::FILTER_CALLBACK: ems_nd_event_filter_synchronizer::update_notifyd_table: Successfully updated notifyd table. 00000d93.00000fac 243bca1e Thu Sep 26 2019 04:34:13 -06:00 [kern_mgwd:info:2284] 0x81d249400: 0: INFO: EMS::NOTIFICATION_CALLBACK: ems_nd_event_notification_synchronizer::update_notifyd_table: re-sync'ing notifyd table. 00000d93.00000fb2 243bca1e Thu Sep 26 2019 04:34:13 -06:00 [kern_mgwd:info:2284] 0x81d249400: 0: INFO: EMS::NOTIFICATION_CALLBACK: ems_nd_event_notification_synchronizer::update_notifyd_table: Successfully updated notifyd table. 00000d93.00000fb3 243bca1e Thu Sep 26 2019 04:34:13 -06:00 [kern_mgwd:info:2284] 0x81d249400: 0: INFO: EMS::router_tables: ems_notifyd_log_config_synchronizer::update_notifyd_table: re-sync'ing notifyd table. 00000d93.00000fb6 243bca1e Thu Sep 26 2019 04:34:13 -06:00 [kern_mgwd:info:2284] 0x81d249400: 0: INFO: EMS::router_tables: ems_notifyd_log_config_synchronizer::update_notifyd_table: Successfully updated notifyd table. 00000d93.00000fb7 243bca1e Thu Sep 26 2019 04:34:13 -06:00 [kern_mgwd:info:2284] 0x81d249400: 0: INFO: EMS::MgwdNdTableSync: mgwd_nd_table_synchronizer::performResync: all tables in notifyd re-sync'ed successfully. 00000d93.00000fb8 243bca1e Thu Sep 26 2019 04:34:13 -06:00 [kern_mgwd:info:2284] 0x81d249400: 0: INFO: EMS::MgwdNdTableSync: mgwd_nd_table_synchronizer::update_notifyd_table: all tables in notifyd already up-to-date. Ignoring online re-sync. 00000d93.00000fb9 243bca1e Thu Sep 26 2019 04:34:14 -06:00 [kern_mgwd:info:2284] Starting a backup of the var filesystem. 00000d93.00000fcd 243bcb9f Thu Sep 26 2019 04:34:51 -06:00 [kern_mgwd:info:2284] 0x81b518000: 0: ERR: Net_Vif::: trigger_lif_modify_update: Node name is nmg-fas8040-02 00000d93.00000fe2 243bcc8f Thu Sep 26 2019 04:35:16 -06:00 [kern_mgwd:info:2284] 0x81b51c100: 0: ERR: job_manager::job_mgr_job_ops: action:src/job_mgr_job_ops.cc:115 job: adcf59b4-b79d-11e7-9b07-00a09857ee8f: pause returned 23 00000d93.00000fe3 243bcc8f Thu Sep 26 2019 04:35:16 -06:00 [kern_mgwd:info:2284] Notice: _initADBJobs: Ensure Auto Balance Aggregate daemons are active: succeeded. 00000d93.00000fe7 243bccbd Thu Sep 26 2019 04:35:21 -06:00 [kern_mgwd:info:2284] 0x81b51a300: 0: INFO: Net::DR: src/tables/net.cc:9116 get_imp:VIF entry not found in failed_lif_placement table 00000d93.00000fe8 243bccbd Thu Sep 26 2019 04:35:21 -06:00 [kern_mgwd:info:2284] 0x81b51a300: 0: INFO: Net::DR: src/tables/net.cc:9130 next_imp:VIF entry not found in failed_lif_placement table 00000d93.00000fe9 243bccbd Thu Sep 26 2019 04:35:21 -06:00 [kern_mgwd:info:2284] No entries to be deleted from failed_lif_placement table 00000d93.00000ff1 243bcce2 Thu Sep 26 2019 04:35:25 -06:00 [kern_mgwd:info:2284] 0x81b24a600: 0: ERR: RPCServer: mgwd_xml_start_1_svc: Called for asup_pathname(/mroot/etc/log/autosupport/201909260435.0.files) 00000d93.00000ff2 243bcce2 Thu Sep 26 2019 04:35:25 -06:00 [kern_mgwd:info:2284] 0x81b24a600: 0: ERR: RPCServer: mgwd_xml_start_1_svc: ASUP Started SUCCESS 00000d93.00001009 243bce0a Thu Sep 26 2019 04:35:54 -06:00 [kern_mgwd:info:2284] 0x81b245100: 0: ERR: RPCServer: mgwd_xml_start_1_svc: Called for asup_pathname(/mroot/etc/log/autosupport/201909260435.1.files) 00000d93.0000100a 243bce0a Thu Sep 26 2019 04:35:54 -06:00 [kern_mgwd:info:2284] 0x81b245100: 0: ERR: RPCServer: mgwd_xml_start_1_svc: current_asup_xml exists. State: (2) 00000d93.0000100b 243bce0a Thu Sep 26 2019 04:35:54 -06:00 [kern_mgwd:info:2284] 0x81b245100: 0: ERR: RPCServer: mgwd_xml_start_1_svc: ASUP Started SUCCESS 00000d93.0000103e 243bce4a Thu Sep 26 2019 04:35:56 -06:00 [kern_mgwd:info:2284] 0x81b51cb00: 0: ERR: perf::perf_asup_info: _validateCounterMapCache: asup store enabled - caching only mhost objects 00000d93.0000103f 243bce4a Thu Sep 26 2019 04:35:56 -06:00 [kern_mgwd:info:2284] Error: get_preset_detail: failure occurred while retrieving performance preset detail: entry doesn't exist 00000d93.00001040 243bce4a Thu Sep 26 2019 04:35:56 -06:00 [kern_mgwd:info:2284] 0x81b51cb00: 0: ERR: perf::cm_preset_detail_ui: get_preset_detail: get failed for obj ic_viif_debug_stats 00000d93.00001041 243bce4a Thu Sep 26 2019 04:35:56 -06:00 [kern_mgwd:info:2284] Error: get_preset_detail: failure occurred while retrieving performance preset detail: no process bitmap 00000d93.00001042 243bce4a Thu Sep 26 2019 04:35:56 -06:00 [kern_mgwd:info:2284] 0x81b51cb00: 0: ERR: perf::cm_preset_detail_ui: get_preset_detail: have process bitmap failed for obj ic_viif_debug_stats 00000d93.00001043 243bce4a Thu Sep 26 2019 04:35:56 -06:00 [kern_mgwd:info:2284] Error: get_preset_detail: failure occurred while retrieving performance preset detail: entry doesn't exist 00000d93.00001044 243bce4a Thu Sep 26 2019 04:35:56 -06:00 [kern_mgwd:info:2284] 0x81b51cb00: 0: ERR: perf::cm_preset_detail_ui: get_preset_detail: get failed for obj nfsv3_client:constituent 00000d93.00001045 243bce4a Thu Sep 26 2019 04:35:56 -06:00 [kern_mgwd:info:2284] Error: get_preset_detail: failure occurred while retrieving performance preset detail: no process bitmap 00000d93.00001046 243bce4a Thu Sep 26 2019 04:35:56 -06:00 [kern_mgwd:info:2284] 0x81b51cb00: 0: ERR: perf::cm_preset_detail_ui: get_preset_detail: have process bitmap failed for obj nfsv3_client:constituent 00000d93.00001047 243bce4a Thu Sep 26 2019 04:35:56 -06:00 [kern_mgwd:info:2284] Error: get_preset_detail: failure occurred while retrieving performance preset detail: entry doesn't exist 00000d93.00001048 243bce4a Thu Sep 26 2019 04:35:56 -06:00 [kern_mgwd:info:2284] 0x81b51cb00: 0: ERR: perf::cm_preset_detail_ui: get_preset_detail: get failed for obj nfsv4_1_client:constituent 00000d93.00001049 243bce4a Thu Sep 26 2019 04:35:56 -06:00 [kern_mgwd:info:2284] Error: get_preset_detail: failure occurred while retrieving performance preset detail: no process bitmap 00000d93.0000104a 243bce4a Thu Sep 26 2019 04:35:56 -06:00 [kern_mgwd:info:2284] 0x81b51cb00: 0: ERR: perf::cm_preset_detail_ui: get_preset_detail: have process bitmap failed for obj nfsv4_1_client:constituent 00000d93.0000104b 243bce4a Thu Sep 26 2019 04:35:56 -06:00 [kern_mgwd:info:2284] Error: get_preset_detail: failure occurred while retrieving performance preset detail: entry doesn't exist 00000d93.0000104c 243bce4a Thu Sep 26 2019 04:35:56 -06:00 [kern_mgwd:info:2284] 0x81b51cb00: 0: ERR: perf::cm_preset_detail_ui: get_preset_detail: get failed for obj nfsv4_client:constituent 00000d93.0000104d 243bce4a Thu Sep 26 2019 04:35:56 -06:00 [kern_mgwd:info:2284] Error: get_preset_detail: failure occurred while retrieving performance preset detail: no process bitmap 00000d93.0000104e 243bce4a Thu Sep 26 2019 04:35:56 -06:00 [kern_mgwd:info:2284] 0x81b51cb00: 0: ERR: perf::cm_preset_detail_ui: get_preset_detail: have process bitmap failed for obj nfsv4_client:constituent 00000d93.0000104f 243bce4a Thu Sep 26 2019 04:35:56 -06:00 [kern_mgwd:info:2284] Error: get_preset_detail: failure occurred while retrieving performance preset detail: entry doesn't exist 00000d93.00001050 243bce4a Thu Sep 26 2019 04:35:56 -06:00 [kern_mgwd:info:2284] 0x81b51cb00: 0: ERR: perf::cm_preset_detail_ui: get_preset_detail: get failed for obj repl_granular_sync_cg 00000d93.00001051 243bce4a Thu Sep 26 2019 04:35:56 -06:00 [kern_mgwd:info:2284] Error: get_preset_detail: failure occurred while retrieving performance preset detail: no process bitmap 00000d93.00001052 243bce4a Thu Sep 26 2019 04:35:56 -06:00 [kern_mgwd:info:2284] 0x81b51cb00: 0: ERR: perf::cm_preset_detail_ui: get_preset_detail: have process bitmap failed for obj repl_granular_sync_cg 00000d93.00001053 243bce4a Thu Sep 26 2019 04:35:56 -06:00 [kern_mgwd:info:2284] Error: get_preset_detail: failure occurred while retrieving performance preset detail: entry doesn't exist 00000d93.00001054 243bce4a Thu Sep 26 2019 04:35:56 -06:00 [kern_mgwd:info:2284] 0x81b51cb00: 0: ERR: perf::cm_preset_detail_ui: get_preset_detail: get failed for obj repl_granular_sync_dst 00000d93.00001055 243bce4a Thu Sep 26 2019 04:35:56 -06:00 [kern_mgwd:info:2284] Error: get_preset_detail: failure occurred while retrieving performance preset detail: no process bitmap 00000d93.00001056 243bce4a Thu Sep 26 2019 04:35:56 -06:00 [kern_mgwd:info:2284] 0x81b51cb00: 0: ERR: perf::cm_preset_detail_ui: get_preset_detail: have process bitmap failed for obj repl_granular_sync_dst 00000d93.00001057 243bce4a Thu Sep 26 2019 04:35:56 -06:00 [kern_mgwd:info:2284] Error: get_preset_detail: failure occurred while retrieving performance preset detail: entry doesn't exist 00000d93.00001058 243bce4a Thu Sep 26 2019 04:35:56 -06:00 [kern_mgwd:info:2284] 0x81b51cb00: 0: ERR: perf::cm_preset_detail_ui: get_preset_detail: get failed for obj smc_group 00000d93.00001059 243bce4a Thu Sep 26 2019 04:35:56 -06:00 [kern_mgwd:info:2284] Error: get_preset_detail: failure occurred while retrieving performance preset detail: no process bitmap 00000d93.0000105a 243bce4a Thu Sep 26 2019 04:35:56 -06:00 [kern_mgwd:info:2284] 0x81b51cb00: 0: ERR: perf::cm_preset_detail_ui: get_preset_detail: have process bitmap failed for obj smc_group 00000d93.0000105b 243bce4a Thu Sep 26 2019 04:35:56 -06:00 [kern_mgwd:info:2284] 0x81b51cb00: 0: ERR: perf::perf_asup_info: _validateCounterMapCache: asup store enabled - caching only mhost objects 00000d93.0000105d 243bce4a Thu Sep 26 2019 04:35:56 -06:00 [kern_mgwd:info:2284] 0x81b51cb00: 0: ERR: perf::perf_asup_info: _validateCounterMapCache: asup store enabled - caching only mhost objects 00000d93.0000105e 243bce4a Thu Sep 26 2019 04:35:56 -06:00 [kern_mgwd:info:2284] 0x81b51cb00: 0: ERR: perf::perf_asup_info: _validateCounterMapCache: asup store enabled - caching only mhost objects 00000d93.0000105f 243bce4a Thu Sep 26 2019 04:35:56 -06:00 [kern_mgwd:info:2284] 0x81b51cb00: 0: ERR: perf::perf_asup_info: _validateCounterMapCache: asup store enabled - caching only mhost objects 00000d93.00001061 243bce4a Thu Sep 26 2019 04:35:56 -06:00 [kern_mgwd:info:2284] 0x81b51cb00: 0: ERR: perf::perf_asup_info: _validateCounterMapCache: asup store enabled - caching only mhost objects 00000d93.00001063 243bce4a Thu Sep 26 2019 04:35:56 -06:00 [kern_mgwd:info:2284] 0x81b51cb00: 0: ERR: perf::perf_asup_info: _validateCounterMapCache: asup store enabled - caching only mhost objects 00000d93.00001065 243bce4a Thu Sep 26 2019 04:35:56 -06:00 [kern_mgwd:info:2284] 0x81b51cb00: 0: ERR: perf::perf_asup_info: _validateCounterMapCache: asup store enabled - caching only mhost objects 00000d93.00001067 243bce4a Thu Sep 26 2019 04:35:56 -06:00 [kern_mgwd:info:2284] 0x81b51cb00: 0: ERR: perf::perf_asup_info: _validateCounterMapCache: asup store enabled - caching only mhost objects 00000d93.00001068 243bce4a Thu Sep 26 2019 04:35:56 -06:00 [kern_mgwd:info:2284] 0x81b51cb00: 0: ERR: perf::perf_asup_info: _validateCounterMapCache: asup store enabled - caching only mhost objects 00000d93.00001069 243bce4a Thu Sep 26 2019 04:35:56 -06:00 [kern_mgwd:info:2284] 0x81b51cb00: 0: ERR: perf::perf_asup_info: _validateCounterMapCache: asup store enabled - caching only mhost objects 00000d93.0000106b 243bce4a Thu Sep 26 2019 04:35:56 -06:00 [kern_mgwd:info:2284] 0x81b51cb00: 0: ERR: metrocluster::fcvi: get failed on kern_fcvi_info table for port 0, error = entry doesn't exist {in get_imp() at src/tables/sysconfig_ic_xml.cc:105} 00000d93.0000106c 243bce4a Thu Sep 26 2019 04:35:56 -06:00 [kern_mgwd:info:2284] 0x81b51cb00: 0: ERR: metrocluster::fcvi: get failed on kern_fcvi_info table for port 0, error = entry doesn't exist {in next_imp() at src/tables/sysconfig_ic_xml.cc:137} 00000d93.0000106d 243bce4a Thu Sep 26 2019 04:35:56 -06:00 [kern_mgwd:info:2284] Error: Loop detected in next() for table kern_ethernet_controller_info. Next on "0" returned "0". 00000d93.0000106e 243bce4a Thu Sep 26 2019 04:35:56 -06:00 [kern_mgwd:info:2284] Warning: Unable to list entries on node nmg-fas8040-02. System Disk details are not available on this platform. 00000d93.00001071 243bce4a Thu Sep 26 2019 04:35:57 -06:00 [kern_mgwd:info:2284] 0x81b51cb00: 0: ERR: AggrEfficiency: src/tables/aggr_efficiency.cc:smdb_error aggr_efficiency_iterator::_getAggrDetails(const smdb_type_aggr_name &, smdb_type_Uuid &, smdb_type_filerid &):122 Failed to get aggregate information from VLDB for aggregate , Reason: entry doesn't exist 00000d93.00001075 243bce4a Thu Sep 26 2019 04:35:57 -06:00 [kern_mgwd:info:2284] 0x81b51cb00: 0: NOTICE: TABLES::disk: Added 168 disk recs to the shared reference cache (size 84) 00000d93.0000107c 243bce4a Thu Sep 26 2019 04:35:59 -06:00 [kern_mgwd:info:2284] 0x81b51cb00: 0: ERR: SAN::FCP::INITIATOR: src/tables/san/fcp_initiator.cc:get_imp:134 returning: 82/30 - Invalid WWN "": Valid WWN format is XX:XX:XX:XX:XX:XX:XX:XX, where X is a hexadecimal digit. Example: "01:02:03:04:0a:0b:0c:0d". 00000d93.0000107d 243bce4a Thu Sep 26 2019 04:35:59 -06:00 [kern_mgwd:info:2284] 0x81b51cb00: 0: ERR: SAN::MGWD::UTILS: src/tables/san/bcomUtils.cc:getVserverUuid:1013 Failed getting UUID for vserver : entry doesn't exist 00000d93.0000107e 243bce4a Thu Sep 26 2019 04:35:59 -06:00 [kern_mgwd:info:2284] 0x81b51cb00: 0: ERR: SAN::ISCSI::INTERFACE: src/tables/san/iscsi_interface.cc:get_imp:422 returning: 40/24 - Failed to lookup identifier for Vserver . entry doesn't exist. 00000d93.00001080 243bce4a Thu Sep 26 2019 04:35:59 -06:00 [kern_mgwd:info:2284] get_next_imp(): owner_get_names() returned: Owner lookup failed. Name Service Switch (NSS) configuration not found for Vserver "Main_SVM". 00000d93.00001081 243bce4a Thu Sep 26 2019 04:35:59 -06:00 [kern_mgwd:info:2284] get_next_imp(): owner_get_names() returned: Owner lookup failed. Name Service Switch (NSS) configuration not found for Vserver "Main_SVM". 00000d93.00001082 243bce4a Thu Sep 26 2019 04:35:59 -06:00 [kern_mgwd:info:2284] get_next_imp(): owner_get_names() returned: Owner lookup failed. Name Service Switch (NSS) configuration not found for Vserver "Main_SVM". 00000d93.00001083 243bce4a Thu Sep 26 2019 04:35:59 -06:00 [kern_mgwd:info:2284] get_next_imp(): owner_get_names() returned: Owner lookup failed. Name Service Switch (NSS) configuration not found for Vserver "Main_SVM". 00000d93.00001084 243bce4a Thu Sep 26 2019 04:35:59 -06:00 [kern_mgwd:info:2284] get_next_imp(): owner_get_names() returned: Owner lookup failed. Name Service Switch (NSS) configuration not found for Vserver "Main_SVM". 00000d93.00001085 243bce4a Thu Sep 26 2019 04:35:59 -06:00 [kern_mgwd:info:2284] 0x81b51cb00: 0: WARNING: VOPL::DbladeZapiServerConnection: fillVolAttrs 3305 failed to get creation-time 00000d93.00001086 243bce4a Thu Sep 26 2019 04:35:59 -06:00 [kern_mgwd:info:2284] 0x81b51cb00: 0: WARNING: VOPL::DbladeZapiServerConnection: fillVolAttrs 3716 failed fo get snaplock type 00000d93.00001087 243bce4a Thu Sep 26 2019 04:35:59 -06:00 [kern_mgwd:info:2284] 0x81b51cb00: 0: INFO: SFO_Ready|Cluster_Apps: wait_for_apps_online (waiter: VReport) apps: vldb, maxTime 5 sec: starting 00000d93.00001088 243bce4a Thu Sep 26 2019 04:35:59 -06:00 [kern_mgwd:info:2284] 0x81b51cb00: 0: INFO: SFO_Ready|Cluster_Apps: wait_for_apps_online (VReport): vldb is online. Number of tries: 1. 00000d93.00001089 243bce4a Thu Sep 26 2019 04:35:59 -06:00 [kern_mgwd:info:2284] 0x81b51cb00: 0: INFO: SFO_Ready|Cluster_Apps: wait_for_apps_online (waiter: VReport): Finished after 0 tries. 00000d93.0000108a 243bce4a Thu Sep 26 2019 04:35:59 -06:00 [kern_mgwd:info:2284] 0x81b51cb00: 0: INFO: SFO_Ready|Cluster_Apps: wait_for_apps_online (waiter: VReport) apps: vldb, maxTime 5 sec: starting 00000d93.0000108b 243bce4a Thu Sep 26 2019 04:35:59 -06:00 [kern_mgwd:info:2284] 0x81b51cb00: 0: INFO: SFO_Ready|Cluster_Apps: wait_for_apps_online (VReport): vldb is online. Number of tries: 1. 00000d93.0000108c 243bce4a Thu Sep 26 2019 04:35:59 -06:00 [kern_mgwd:info:2284] 0x81b51cb00: 0: INFO: SFO_Ready|Cluster_Apps: wait_for_apps_online (waiter: VReport): Finished after 0 tries. 00000d93.0000108d 243bce4a Thu Sep 26 2019 04:35:59 -06:00 [kern_mgwd:info:2284] 0x81b51cb00: 0: WARNING: MGWD::volume_cache: populateCache SetVolumeCache 546 d_volume_list_info_iter_next with nextTag: cookie=0,desired_attrs=name,owning-vserver-uuid,is-home,extent-size,dsid,msid,state,vol-access,language-code,containing-aggregate,mirror-types,containing-aggregate-uuid,data-version,uuid,clone-parent,is-cluster-volume,fc-cache-policy-idx,transition,iv-constituent-role,creation-time,snaplock,flexgroup-index,flexgroup-msid,flexgroup-uuid,ondisk-quiesced,encrypt,key-id,encryption-blob,key-generation,buftree-id,btuuid response from node: 169.254.5.219 is: 00000d93.0000108e 243bce4a Thu Sep 26 2019 04:35:59 -06:00 [kern_mgwd:info:2284] 0x81b51cb00: 0: WARNING: MGWD::volume_cache: populateCache SetVolumeCache 546 d_volume_list_info_iter_next with nextTag: cookie=f26ab847-ca47-11e7-8fa8-00a098573067,desired_attrs=name,owning-vserver-uuid,is-home,extent-size,dsid,msid,state,vol-access,language-code,containing-aggregate,mirror-types,containing-aggregate-uuid,data-version,uuid,clone-parent,is-cluster-volume,fc-cache-policy-idx,transition,iv-constituent-role,creation-time,snaplock,flexgroup-index,flexgroup-msid,flexgroup-uuid,ondisk-quiesced,encrypt,key-id,encryption-blob,key-generation,buftree-id,btuuid response from node: 169.254.5.219 is: 00000d93.0000108f 243bce4a Thu Sep 26 2019 04:35:59 -06:00 [kern_mgwd:info:2284] 0x81b51cb00: 0: WARNING: MGWD::volume_cache: populateCache SetVolumeCache 546 d_volume_list_info_iter_next with nextTag: cookie=0,desired_attrs=name,owning-vserver-uuid,is-home,extent-size,dsid,msid,state,vol-access,language-code,containing-aggregate,mirror-types,containing-aggregate-uuid,data-version,uuid,clone-parent,is-cluster-volume,fc-cache-policy-idx,transition,iv-constituent-role,creation-time,snaplock,flexgroup-index,flexgroup-msid,flexgroup-uuid,ondisk-quiesced,encrypt,key-id,encryption-blob,key-generation,buftree-id,btuuid response from node: 169.254.23.82 is: 00000d93.00001090 243bce4a Thu Sep 26 2019 04:35:59 -06:00 [kern_mgwd:info:2284] 0x81b51cb00: 0: WARNING: MGWD::volume_cache: populateCache SetVolumeCache 546 d_volume_list_info_iter_next with nextTag: cookie=c5105621-51ff-44fe-9abe-69b0d9bee4da,desired_attrs=name,owning-vserver-uuid,is-home,extent-size,dsid,msid,state,vol-access,language-code,containing-aggregate,mirror-types,containing-aggregate-uuid,data-version,uuid,clone-parent,is-cluster-volume,fc-cache-policy-idx,transition,iv-constituent-role,creation-time,snaplock,flexgroup-index,flexgroup-msid,flexgroup-uuid,ondisk-quiesced,encrypt,key-id,encryption-blob,key-generation,buftree-id,btuuid response from node: 169.254.23.82 is: 00000d93.00001091 243bce4a Thu Sep 26 2019 04:35:59 -06:00 [kern_mgwd:info:2284] 0x81b51cb00: 0: ERR: tables::vreport: Object Name not found in vReport cache. 00000d93.00001092 243bce4a Thu Sep 26 2019 04:35:59 -06:00 [kern_mgwd:info:2284] 0x81b51cb00: 0: ERR: tables::vreport: isMCCPostHealRoots: Cannot get metrocluster state. Error: entry doesn't exist 00000d93.00001093 243bce4a Thu Sep 26 2019 04:35:59 -06:00 [kern_mgwd:info:2284] 0x81b51cb00: 0: ERR: MGWD::VsMSID: src/vreport/vreport_junction.h 125: toStr(): vsName and vsid are both empty. 00000d93.00001094 243bce4a Thu Sep 26 2019 04:35:59 -06:00 [kern_mgwd:info:2284] 0x81b51cb00: 0: ERR: MGWD::VsMSID: src/vreport/vreport_junction.h 134: toStr(): volName and volMsid are both empty. The object is 00000d93.00001095 243bce4a Thu Sep 26 2019 04:35:59 -06:00 [kern_mgwd:info:2284] 0x81b51cb00: 0: ERR: MGWD::VsMSID: src/vreport/vreport_junction.h 125: toStr(): vsName and vsid are both empty. 00000d93.00001096 243bce4a Thu Sep 26 2019 04:35:59 -06:00 [kern_mgwd:info:2284] 0x81b51cb00: 0: ERR: MGWD::VsMSID: src/vreport/vreport_junction.h 134: toStr(): volName and volMsid are both empty. The object is 00000d93.00001097 243bce4a Thu Sep 26 2019 04:35:59 -06:00 [kern_mgwd:info:2284] Info: WAFL and VLDB volume/aggregate records are consistent. 00000d93.000010ac 243bce4a Thu Sep 26 2019 04:36:01 -06:00 [kern_mgwd:info:2284] 0x81b51cb00: 0: ERR: perf::perf_asup_info: _validateCounterMapCache: asup store enabled - caching only mhost objects 00000d93.000010ae 243bce4b Thu Sep 26 2019 04:36:01 -06:00 [kern_mgwd:info:2284] 0x81b51cb00: 0: ERR: perf::perf_asup_info: _validateCounterMapCache: asup store enabled - caching only mhost objects 00000d93.000010af 243bce4b Thu Sep 26 2019 04:36:01 -06:00 [kern_mgwd:info:2284] 0x81b51cb00: 0: NOTICE: RpcConnectionCache: markFailure: markFailure: New Quarantine for 127.0.0.1:536873568(No name for prog:536873568)/1[tcp]{fffffffd|8} with timeout(5.000s) Reason:: RPC: Program not registered 00000d93.000010da 243bd348 Thu Sep 26 2019 04:38:08 -06:00 [kern_mgwd:info:2284] Info: HM asup thread exiting normally 00000d93.000010ec 243bdf7b Thu Sep 26 2019 04:43:21 -06:00 [kern_mgwd:info:2284] 0x81d251600: 86ffff0000000035: WARNING: job_manager::job_store: run:src/job_store_support.cc:568 Job Mgr: rdb_transaction history: Timeout/0 Timeout/300 OK/600 00000d93.00001126 243bf120 Thu Sep 26 2019 04:50:52 -06:00 [kern_mgwd:info:2284] 0x81b244c00: 0: ERR: RPCServer: mgwd_xml_start_1_svc: Called for asup_pathname(/mroot/etc/log/autosupport/201909260450.0.files) 00000d93.00001127 243bf120 Thu Sep 26 2019 04:50:52 -06:00 [kern_mgwd:info:2284] 0x81b244c00: 0: ERR: RPCServer: mgwd_xml_start_1_svc: current_asup_xml exists. State: (2) 00000d93.00001128 243bf120 Thu Sep 26 2019 04:50:52 -06:00 [kern_mgwd:info:2284] 0x81b244c00: 0: ERR: RPCServer: mgwd_xml_start_1_svc: ASUP Started SUCCESS 00000d93.0000119a 243c02a3 Thu Sep 26 2019 04:58:21 -06:00 [kern_mgwd:info:2284] 0x81d251600: 86ffff0000000035: WARNING: job_manager::job_store: run:src/job_store_support.cc:568 Job Mgr: rdb_transaction history: Timeout/0 Timeout/300 Timeout/600 00000d93.000011e7 243c0e64 Thu Sep 26 2019 05:03:22 -06:00 [kern_mgwd:info:2284] 0x81d24d500: 86ffff0000000050: WARNING: MGWD:NetworkDiagJob::: [src/tables/network_diag_job.cc:109] 1 inconsistency found by network debug checker; see vifmgr.log for details 00000d93.000012d7 243c0e65 Thu Sep 26 2019 05:03:22 -06:00 [kern_mgwd:info:2284] 0x81d24d500: 86ffff0000000050: WARNING: job_manager::scheduler_other: get_next:src/scheduler.cc:440 onetime_scheduler::get_next: reschedule not allowed. line: 440 00000d93.000012d8 243c0e65 Thu Sep 26 2019 05:03:22 -06:00 [kern_mgwd:info:2284] 0x81d24d500: 86ffff0000000050: WARNING: job_manager::job_record: get_next_run:src/job_record.cc:647 get_next_run(reschedule), returning 0, schedulenext is 0 for job: 8856 - 0db07695-e049-11e9-a00d-00a098573067 00000d93.0000131c 243c29d8 Thu Sep 26 2019 05:15:05 -06:00 [kern_mgwd:info:2284] 0x81d251600: 86ffff0000000035: WARNING: job_manager::job_store: run:src/job_store_support.cc:568 Job Mgr: rdb_transaction history: Timeout/0 OK/300 Timeout/403 OK/703 Timeout/704 00000d93.0000135a 243c4166 Thu Sep 26 2019 05:25:08 -06:00 [kern_mgwd:info:2284] 0x81d251600: 86ffff0000000035: WARNING: job_manager::job_store: run:src/job_store_support.cc:568 Job Mgr: rdb_transaction history: Timeout/0 Timeout/300 OK/600 00000d93.000013d0 243c549e Thu Sep 26 2019 05:33:20 -06:00 [kern_mgwd:info:2284] 0x81d24bc00: 8603e90000000160: WARNING: job_manager::job_sched: modify_schedule:src/job_sched.cc:800 job_manager::job_sched::modify_schedule: Calling set_scheduler on the record: Current schedule = 0x81b84a980; New schedule = 0x81c4b5520 00000d93.000013d4 243c562e Thu Sep 26 2019 05:34:00 -06:00 [kern_mgwd:info:2284] 0x81d24e900: 8603e900000000c4: WARNING: job_manager::scheduler_other: get_next:src/scheduler.cc:440 onetime_scheduler::get_next: reschedule not allowed. line: 440 00000d93.000013d5 243c562e Thu Sep 26 2019 05:34:00 -06:00 [kern_mgwd:info:2284] 0x81d24e900: 8603e900000000c4: WARNING: job_manager::job_record: get_next_run:src/job_record.cc:647 get_next_run(reschedule), returning 0, schedulenext is 0 for job: 21833 - 26d996b5-e049-11e9-a00d-00a098573067 00000d93.000013d6 243c5642 Thu Sep 26 2019 05:34:02 -06:00 [kern_mgwd:info:2284] 0x81b512600: 0: ERR: security_mgwd::tables::certificate::certificate: write_hash_files:src/tables/certificate/certificate.cc:1985 SSL: Failed to create symlink for file '/mroot/etc/vserver_4294967295/certificates/ssl/serverCa/99+4A538C28+EntrustRootCertificationAuthority-G2+455f1b52.0.pem' with '/mroot/etc/vserver_4294967295/certificates/ssl/serverCa/455f1b52.0'. Reason: 17 00000d93.0000140b 243c5fed Thu Sep 26 2019 05:38:09 -06:00 [kern_mgwd:info:2284] Starting a backup of the var filesystem. 00000d93.0000140e 243c6058 Thu Sep 26 2019 05:38:20 -06:00 [kern_mgwd:info:2284] 0x81d251600: 86ffff0000000035: WARNING: job_manager::job_store: run:src/job_store_support.cc:568 Job Mgr: rdb_transaction history: Timeout/0 OK/300 Timeout/492 00000d93.00001480 243c87b6 Thu Sep 26 2019 05:55:08 -06:00 [kern_mgwd:info:2284] 0x81d251600: 86ffff0000000035: WARNING: job_manager::job_store: run:src/job_store_support.cc:568 Job Mgr: rdb_transaction history: Timeout/0 OK/300 Timeout/408 Timeout/708 00000d93.00001515 243caade Thu Sep 26 2019 06:10:08 -06:00 [kern_mgwd:info:2284] 0x81d251600: 86ffff0000000035: WARNING: job_manager::job_store: run:src/job_store_support.cc:568 Job Mgr: rdb_transaction history: Timeout/0 OK/300 Timeout/600 00000d93.000015ea 243ccdb6 Thu Sep 26 2019 06:25:00 -06:00 [kern_mgwd:info:2284] 0x81d24e900: 86ffff000000002c: WARNING: job_manager::job_store: claim_job_tx:src/job_rec_int.cc:581 claim_job: Attempted to claim a job that was already owned: 489d5bff-b80c-11e7-964b-00a098573067 owner: f28b11a3-b769-11e7-9b07-00a09857ee8f 00000d93.000015eb 243ccdb6 Thu Sep 26 2019 06:25:00 -06:00 [kern_mgwd:info:2284] 0x81d24e900: 86ffff000000002c: WARNING: job_manager::job_sched: _check_claimable:src/job_sched.cc:2225 UIID:19 UUID:489d5bff-b80c-11e7-964b-00a098573067 name:Network Recovery Job - 5min rc:3 _check_claimable: failed to claim jobs: 12 00000d93.000015ec 243cce06 Thu Sep 26 2019 06:25:08 -06:00 [kern_mgwd:info:2284] 0x81d251600: 86ffff0000000035: WARNING: job_manager::job_store: run:src/job_store_support.cc:568 Job Mgr: rdb_transaction history: Timeout/0 OK/300 Timeout/301 OK/601 00000d93.0000162c 243ce2cd Thu Sep 26 2019 06:34:00 -06:00 [kern_mgwd:info:2284] 0x81d24df00: 8603e90000000164: WARNING: job_manager::scheduler_other: get_next:src/scheduler.cc:440 onetime_scheduler::get_next: reschedule not allowed. line: 440 00000d93.0000162d 243ce2cd Thu Sep 26 2019 06:34:00 -06:00 [kern_mgwd:info:2284] 0x81d24df00: 8603e90000000164: WARNING: job_manager::job_record: get_next_run:src/job_record.cc:647 get_next_run(reschedule), returning 0, schedulenext is 0 for job: 21834 - 8c16cd37-e051-11e9-a00d-00a098573067 00000d93.0000162e 243ce2de Thu Sep 26 2019 06:34:01 -06:00 [kern_mgwd:info:2284] 0x81b512600: 0: ERR: security_mgwd::tables::certificate::certificate: write_hash_files:src/tables/certificate/certificate.cc:1985 SSL: Failed to create symlink for file '/mroot/etc/vserver_4294967295/certificates/ssl/serverCa/99+4A538C28+EntrustRootCertificationAuthority-G2+455f1b52.0.pem' with '/mroot/etc/vserver_4294967295/certificates/ssl/serverCa/455f1b52.0'. Reason: 17 00000d93.00001669 243cf12e Thu Sep 26 2019 06:40:08 -06:00 [kern_mgwd:info:2284] 0x81d251600: 86ffff0000000035: WARNING: job_manager::job_store: run:src/job_store_support.cc:568 Job Mgr: rdb_transaction history: Timeout/0 OK/300 Timeout/600 00000d93.00001704 243d1456 Thu Sep 26 2019 06:55:08 -06:00 [kern_mgwd:info:2284] 0x81d251600: 86ffff0000000035: WARNING: job_manager::job_store: run:src/job_store_support.cc:568 Job Mgr: rdb_transaction history: Timeout/0 Timeout/300 Timeout/600 00000d93.0000179c 243d377e Thu Sep 26 2019 07:10:08 -06:00 [kern_mgwd:info:2284] 0x81d251600: 86ffff0000000035: WARNING: job_manager::job_store: run:src/job_store_support.cc:568 Job Mgr: rdb_transaction history: Timeout/0 Timeout/300 Timeout/600 00000d93.00001815 243d5a92 Thu Sep 26 2019 07:25:06 -06:00 [kern_mgwd:info:2284] 0x81d251600: 86ffff0000000035: WARNING: job_manager::job_store: run:src/job_store_support.cc:568 Job Mgr: rdb_transaction history: Timeout/0 Timeout/300 OK/600 00000d93.00001852 243d6df2 Thu Sep 26 2019 07:33:22 -06:00 [kern_mgwd:info:2284] 0x81d24cb00: 8603e90000000273: WARNING: job_manager::job_sched: modify_schedule:src/job_sched.cc:800 job_manager::job_sched::modify_schedule: Calling set_scheduler on the record: Current schedule = 0x81b91d040; New schedule = 0x81b429e80 00000d93.00001856 243d6f6d Thu Sep 26 2019 07:34:00 -06:00 [kern_mgwd:info:2284] 0x81d24d500: 8603e900000001f3: WARNING: job_manager::scheduler_other: get_next:src/scheduler.cc:440 onetime_scheduler::get_next: reschedule not allowed. line: 440 00000d93.00001857 243d6f6d Thu Sep 26 2019 07:34:00 -06:00 [kern_mgwd:info:2284] 0x81d24d500: 8603e900000001f3: WARNING: job_manager::job_record: get_next_run:src/job_record.cc:647 get_next_run(reschedule), returning 0, schedulenext is 0 for job: 21835 - ed48f0c0-e059-11e9-a00d-00a098573067 00000d93.00001858 243d6f80 Thu Sep 26 2019 07:34:01 -06:00 [kern_mgwd:info:2284] 0x81b512600: 0: ERR: security_mgwd::tables::certificate::certificate: write_hash_files:src/tables/certificate/certificate.cc:1985 SSL: Failed to create symlink for file '/mroot/etc/vserver_4294967295/certificates/ssl/serverCa/99+4A538C28+EntrustRootCertificationAuthority-G2+455f1b52.0.pem' with '/mroot/etc/vserver_4294967295/certificates/ssl/serverCa/455f1b52.0'. Reason: 17 00000d93.0000186e 243d79aa Thu Sep 26 2019 07:38:22 -06:00 [kern_mgwd:info:2284] 0x81d251600: 86ffff0000000035: WARNING: job_manager::job_store: run:src/job_store_support.cc:568 Job Mgr: rdb_transaction history: Timeout/0 OK/300 Timeout/496 00000d93.000018d2 243d9cd2 Thu Sep 26 2019 07:53:22 -06:00 [kern_mgwd:info:2284] 0x81d251600: 86ffff0000000035: WARNING: job_manager::job_store: run:src/job_store_support.cc:568 Job Mgr: rdb_transaction history: Timeout/0 Timeout/300 Timeout/600 00000d93.0000194d 243dbffa Thu Sep 26 2019 08:08:22 -06:00 [kern_mgwd:info:2284] 0x81d251600: 86ffff0000000035: WARNING: job_manager::job_store: run:src/job_store_support.cc:568 Job Mgr: rdb_transaction history: Timeout/0 Timeout/300 Timeout/600 00000d93.000019b6 243de323 Thu Sep 26 2019 08:23:22 -06:00 [kern_mgwd:info:2284] 0x81d251600: 86ffff0000000035: WARNING: job_manager::job_store: run:src/job_store_support.cc:568 Job Mgr: rdb_transaction history: Timeout/0 Timeout/300 Timeout/600 00000d93.00001a05 243dfaa1 Thu Sep 26 2019 08:33:23 -06:00 [kern_mgwd:info:2284] 0x81d24cb00: 8603e900000002f8: WARNING: job_manager::job_sched: modify_schedule:src/job_sched.cc:800 job_manager::job_sched::modify_schedule: Calling set_scheduler on the record: Current schedule = 0x81b84a800; New schedule = 0x81b417a20 00000d93.00001a06 243dfaa1 Thu Sep 26 2019 08:33:23 -06:00 [kern_mgwd:info:2284] 0x81d251600: 86ffff0000000035: WARNING: job_manager::job_store: run:src/job_store_support.cc:568 Job Mgr: rdb_transaction history: OK/0 Timeout/195 OK/495 00000d93.00001a0a 243dfc0d Thu Sep 26 2019 08:34:00 -06:00 [kern_mgwd:info:2284] 0x81d24d500: 8603e90000000277: WARNING: job_manager::scheduler_other: get_next:src/scheduler.cc:440 onetime_scheduler::get_next: reschedule not allowed. line: 440 00000d93.00001a0b 243dfc0d Thu Sep 26 2019 08:34:00 -06:00 [kern_mgwd:info:2284] 0x81d24d500: 8603e90000000277: WARNING: job_manager::job_record: get_next_run:src/job_record.cc:647 get_next_run(reschedule), returning 0, schedulenext is 0 for job: 21836 - 4f263d52-e062-11e9-a00d-00a098573067 00000d93.00001a0c 243dfc1f Thu Sep 26 2019 08:34:01 -06:00 [kern_mgwd:info:2284] 0x81b512600: 0: ERR: security_mgwd::tables::certificate::certificate: write_hash_files:src/tables/certificate/certificate.cc:1985 SSL: Failed to create symlink for file '/mroot/etc/vserver_4294967295/certificates/ssl/serverCa/99+4A538C28+EntrustRootCertificationAuthority-G2+455f1b52.0.pem' with '/mroot/etc/vserver_4294967295/certificates/ssl/serverCa/455f1b52.0'. Reason: 17 00000d93.00001a7d 243e21de Thu Sep 26 2019 08:50:08 -06:00 [kern_mgwd:info:2284] 0x81d251600: 86ffff0000000035: WARNING: job_manager::job_store: run:src/job_store_support.cc:568 Job Mgr: rdb_transaction history: Timeout/0 Timeout/300 OK/600 Timeout/705 00000d93.00001ad3 243e450a Thu Sep 26 2019 09:05:08 -06:00 [kern_mgwd:info:2284] 0x81d251600: 86ffff0000000035: WARNING: job_manager::job_store: run:src/job_store_support.cc:568 Job Mgr: rdb_transaction history: Timeout/0 Timeout/300 OK/600 00000d93.00001b6e 243e682d Thu Sep 26 2019 09:20:08 -06:00 [kern_mgwd:info:2284] 0x81d251600: 86ffff0000000035: WARNING: job_manager::job_store: run:src/job_store_support.cc:568 Job Mgr: rdb_transaction history: OK/0 Timeout/300 OK/600 00000d93.00001be5 243e88b1 Thu Sep 26 2019 09:34:00 -06:00 [kern_mgwd:info:2284] 0x81d24da00: 8603e900000002fe: WARNING: job_manager::scheduler_other: get_next:src/scheduler.cc:440 onetime_scheduler::get_next: reschedule not allowed. line: 440 00000d93.00001be6 243e88b1 Thu Sep 26 2019 09:34:00 -06:00 [kern_mgwd:info:2284] 0x81d24da00: 8603e900000002fe: WARNING: job_manager::job_record: get_next_run:src/job_record.cc:647 get_next_run(reschedule), returning 0, schedulenext is 0 for job: 21837 - b0d6f813-e06a-11e9-a00d-00a098573067 00000d93.00001be8 243e88be Thu Sep 26 2019 09:34:01 -06:00 [kern_mgwd:info:2284] 0x81b512600: 0: ERR: security_mgwd::tables::certificate::certificate: write_hash_files:src/tables/certificate/certificate.cc:1985 SSL: Failed to create symlink for file '/mroot/etc/vserver_4294967295/certificates/ssl/serverCa/99+4A538C28+EntrustRootCertificationAuthority-G2+455f1b52.0.pem' with '/mroot/etc/vserver_4294967295/certificates/ssl/serverCa/455f1b52.0'. Reason: 17 00000d93.00001bed 243e8b55 Thu Sep 26 2019 09:35:08 -06:00 [kern_mgwd:info:2284] 0x81d251600: 86ffff0000000035: WARNING: job_manager::job_store: run:src/job_store_support.cc:568 Job Mgr: rdb_transaction history: Timeout/0 Timeout/300 Timeout/600 00000d93.00001c4e 243eae7e Thu Sep 26 2019 09:50:08 -06:00 [kern_mgwd:info:2284] 0x81d251600: 86ffff0000000035: WARNING: job_manager::job_store: run:src/job_store_support.cc:568 Job Mgr: rdb_transaction history: Timeout/0 Timeout/300 Timeout/600 00000d93.00001ca8 243ed1a6 Thu Sep 26 2019 10:05:08 -06:00 [kern_mgwd:info:2284] 0x81d251600: 86ffff0000000035: WARNING: job_manager::job_store: run:src/job_store_support.cc:568 Job Mgr: rdb_transaction history: Timeout/0 Timeout/300 Timeout/600 00000d93.00001d1e 243eefb7 Thu Sep 26 2019 10:17:57 -06:00 [kern_mgwd:info:2284] 0x81d24b200: 8603e90000000403: WARNING: job_manager::job_sched: process_modify:src/job_sched.cc:487 UIID:8857 UUID:c9f4e95f-e078-11e9-93b8-00a09857ee8f name:Local backup job rc:1 Doing nothing in modify callback for job, since it's pending cleanup 00000d93.00001d1f 243eefcd Thu Sep 26 2019 10:18:00 -06:00 [kern_mgwd:info:2284] Info: smf-net: failed due to no more data or timeout. timeout=0 sec, fd=749, numFdReady=0, bytesRcvd=0, FD_ISSET=false, buflength=512 00000d93.00001d20 243eefdb Thu Sep 26 2019 10:18:01 -06:00 [kern_mgwd:info:2284] 0x81dacd800: 0: WARNING: FileReplication: The file replication service successfully distributed package nmg-fas8040-02.8hour.2019-09-26.10_15_00.7z:0 to 169.254.23.82. 00000d93.00001d21 243ef0dc Thu Sep 26 2019 10:18:27 -06:00 [kern_mgwd:info:2284] 0x81d24bc00: 8603e9000000040a: WARNING: FileReplication: Starting subscription (download) request for package nmg-fas8040.8hour.2019-09-26.10_15_00.7z:latest with old API extract target '/mroot/etc/backups/.tmp/nmg-fas8040.8hour.2019-09-26.10_15_00.7z'. 00000d93.00001d22 243ef117 Thu Sep 26 2019 10:18:33 -06:00 [kern_mgwd:info:2284] 0x81d251600: 86ffff0000000035: WARNING: job_manager::job_store: run:src/job_store_support.cc:568 Job Mgr: rdb_transaction history: OK/0 OK/35 Timeout/205 Timeout/505 00000d93.00001d23 243ef117 Thu Sep 26 2019 10:18:33 -06:00 [kern_mgwd:info:2284] 0x81d24b200: 8603e9000000040c: WARNING: job_manager::job_sched: process_modify:src/job_sched.cc:487 UIID:8859 UUID:4562d3a7-e079-11e9-93b8-00a09857ee8f name:Receive backup job rc:1 Doing nothing in modify callback for job, since it's pending cleanup 00000d93.00001d63 243f0d37 Thu Sep 26 2019 10:30:33 -06:00 [kern_mgwd:info:2284] 0x81d251600: 86ffff0000000035: WARNING: job_manager::job_store: run:src/job_store_support.cc:568 Job Mgr: rdb_transaction history: Timeout/0 Timeout/300 OK/600 OK/625 OK/636 00000d93.00001d8d 243f154c Thu Sep 26 2019 10:34:00 -06:00 [kern_mgwd:info:2284] 0x81d24d500: 8603e90000000389: WARNING: job_manager::scheduler_other: get_next:src/scheduler.cc:440 onetime_scheduler::get_next: reschedule not allowed. line: 440 00000d93.00001d8e 243f154c Thu Sep 26 2019 10:34:00 -06:00 [kern_mgwd:info:2284] 0x81d24d500: 8603e90000000389: WARNING: job_manager::job_record: get_next_run:src/job_record.cc:647 get_next_run(reschedule), returning 0, schedulenext is 0 for job: 21838 - 129b76d3-e073-11e9-a00d-00a098573067 00000d93.00001d8f 243f155f Thu Sep 26 2019 10:34:01 -06:00 [kern_mgwd:info:2284] 0x81b512600: 0: ERR: security_mgwd::tables::certificate::certificate: write_hash_files:src/tables/certificate/certificate.cc:1985 SSL: Failed to create symlink for file '/mroot/etc/vserver_4294967295/certificates/ssl/serverCa/99+4A538C28+EntrustRootCertificationAuthority-G2+455f1b52.0.pem' with '/mroot/etc/vserver_4294967295/certificates/ssl/serverCa/455f1b52.0'. Reason: 17 00000d93.00001e17 243f305f Thu Sep 26 2019 10:45:33 -06:00 [kern_mgwd:info:2284] 0x81d251600: 86ffff0000000035: WARNING: job_manager::job_store: run:src/job_store_support.cc:568 Job Mgr: rdb_transaction history: Timeout/0 Timeout/300 Timeout/600 00000d93.00001e7a 243f528d Thu Sep 26 2019 11:00:08 -06:00 [kern_mgwd:info:2284] 0x81d251600: 86ffff0000000035: WARNING: job_manager::job_store: run:src/job_store_support.cc:568 Job Mgr: rdb_transaction history: OK/0 Timeout/275 Timeout/575 00000d93.00001ed2 243f75b5 Thu Sep 26 2019 11:15:08 -06:00 [kern_mgwd:info:2284] 0x81d251600: 86ffff0000000035: WARNING: job_manager::job_store: run:src/job_store_support.cc:568 Job Mgr: rdb_transaction history: OK/0 Timeout/300 Timeout/600 00000d93.00001f4b 243f98dd Thu Sep 26 2019 11:30:08 -06:00 [kern_mgwd:info:2284] 0x81d251600: 86ffff0000000035: WARNING: job_manager::job_store: run:src/job_store_support.cc:568 Job Mgr: rdb_transaction history: OK/0 OK/300 Timeout/302 OK/602 00000d93.00001f5b 243fa0e9 Thu Sep 26 2019 11:33:33 -06:00 [kern_mgwd:info:2284] 0x81d24e400: 8603e900000004cb: WARNING: job_manager::job_sched: modify_schedule:src/job_sched.cc:800 job_manager::job_sched::modify_schedule: Calling set_scheduler on the record: Current schedule = 0x81b84ae60; New schedule = 0x81b4e2fc0 00000d93.00001f65 243fa1ec Thu Sep 26 2019 11:34:00 -06:00 [kern_mgwd:info:2284] 0x81d24e900: 8603e90000000440: WARNING: job_manager::scheduler_other: get_next:src/scheduler.cc:440 onetime_scheduler::get_next: reschedule not allowed. line: 440 00000d93.00001f66 243fa1ec Thu Sep 26 2019 11:34:00 -06:00 [kern_mgwd:info:2284] 0x81d24e900: 8603e90000000440: WARNING: job_manager::job_record: get_next_run:src/job_record.cc:647 get_next_run(reschedule), returning 0, schedulenext is 0 for job: 21840 - 7459007d-e07b-11e9-a00d-00a098573067 00000d93.00001f67 243fa201 Thu Sep 26 2019 11:34:01 -06:00 [kern_mgwd:info:2284] 0x81b512600: 0: ERR: security_mgwd::tables::certificate::certificate: write_hash_files:src/tables/certificate/certificate.cc:1985 SSL: Failed to create symlink for file '/mroot/etc/vserver_4294967295/certificates/ssl/serverCa/99+4A538C28+EntrustRootCertificationAuthority-G2+455f1b52.0.pem' with '/mroot/etc/vserver_4294967295/certificates/ssl/serverCa/455f1b52.0'. Reason: 17 00000d93.00001fc8 243fbbe7 Thu Sep 26 2019 11:45:05 -06:00 [kern_mgwd:info:2284] 0x81d251600: 86ffff0000000035: WARNING: job_manager::job_store: run:src/job_store_support.cc:568 Job Mgr: rdb_transaction history: Timeout/0 Timeout/300 OK/600 OK/692 00000d93.00002008 243fd357 Thu Sep 26 2019 11:55:05 -06:00 [kern_mgwd:info:2284] 0x81d251600: 86ffff0000000035: WARNING: job_manager::job_store: run:src/job_store_support.cc:568 Job Mgr: rdb_transaction history: Timeout/0 OK/300 OK/597 00000d93.00002048 243feac7 Thu Sep 26 2019 12:05:05 -06:00 [kern_mgwd:info:2284] 0x81d251600: 86ffff0000000035: WARNING: job_manager::job_store: run:src/job_store_support.cc:568 Job Mgr: rdb_transaction history: OK/0 Timeout/297 OK/597 00000d93.000020bf 24400df5 Thu Sep 26 2019 12:20:05 -06:00 [kern_mgwd:info:2284] 0x81d251600: 86ffff0000000035: WARNING: job_manager::job_store: run:src/job_store_support.cc:568 Job Mgr: rdb_transaction history: Timeout/0 OK/300 Timeout/600 00000d93.000020fe 2440255e Thu Sep 26 2019 12:30:05 -06:00 [kern_mgwd:info:2284] 0x81d251600: 86ffff0000000035: WARNING: job_manager::job_store: run:src/job_store_support.cc:568 Job Mgr: rdb_transaction history: OK/0 Timeout/299 OK/599 00000d93.00002106 24402e9e Thu Sep 26 2019 12:34:00 -06:00 [kern_mgwd:info:2284] 0x81d24e400: 8603e900000004cf: WARNING: job_manager::scheduler_other: get_next:src/scheduler.cc:440 onetime_scheduler::get_next: reschedule not allowed. line: 440 00000d93.00002107 24402e9e Thu Sep 26 2019 12:34:00 -06:00 [kern_mgwd:info:2284] 0x81d24e400: 8603e900000004cf: WARNING: job_manager::job_record: get_next_run:src/job_record.cc:647 get_next_run(reschedule), returning 0, schedulenext is 0 for job: 21841 - d61152f0-e083-11e9-a00d-00a098573067 00000d93.00002109 24402e9e Thu Sep 26 2019 12:34:01 -06:00 [kern_mgwd:info:2284] 0x81b512600: 0: ERR: security_mgwd::tables::certificate::certificate: write_hash_files:src/tables/certificate/certificate.cc:1985 SSL: Failed to create symlink for file '/mroot/etc/vserver_4294967295/certificates/ssl/serverCa/99+4A538C28+EntrustRootCertificationAuthority-G2+455f1b52.0.pem' with '/mroot/etc/vserver_4294967295/certificates/ssl/serverCa/455f1b52.0'. Reason: 17 00000d93.00002169 24404887 Thu Sep 26 2019 12:45:05 -06:00 [kern_mgwd:info:2284] 0x81d251600: 86ffff0000000035: WARNING: job_manager::job_store: run:src/job_store_support.cc:568 Job Mgr: rdb_transaction history: OK/0 OK/300 OK/600 00000d93.000021f3 24406baf Thu Sep 26 2019 13:00:05 -06:00 [kern_mgwd:info:2284] 0x81d251600: 86ffff0000000035: WARNING: job_manager::job_store: run:src/job_store_support.cc:568 Job Mgr: rdb_transaction history: Timeout/0 Timeout/300 Timeout/600 00000d93.0000226e 24408ed7 Thu Sep 26 2019 13:15:05 -06:00 [kern_mgwd:info:2284] 0x81d251600: 86ffff0000000035: WARNING: job_manager::job_store: run:src/job_store_support.cc:568 Job Mgr: rdb_transaction history: Timeout/0 Timeout/300 Timeout/600 00000d93.000022e6 2440b1fe Thu Sep 26 2019 13:30:05 -06:00 [kern_mgwd:info:2284] 0x81d251600: 86ffff0000000035: WARNING: job_manager::job_store: run:src/job_store_support.cc:568 Job Mgr: rdb_transaction history: Timeout/0 Timeout/300 OK/600 00000d93.000022ea 2440ba33 Thu Sep 26 2019 13:33:35 -06:00 [kern_mgwd:info:2284] 0x81d24bc00: 8603e900000005e0: WARNING: job_manager::job_sched: modify_schedule:src/job_sched.cc:800 job_manager::job_sched::modify_schedule: Calling set_scheduler on the record: Current schedule = 0x81b91d780; New schedule = 0x81c158740 00000d93.000022ed 2440bb2c Thu Sep 26 2019 13:34:00 -06:00 [kern_mgwd:info:2284] 0x81d24cb00: 8603e90000000560: WARNING: job_manager::scheduler_other: get_next:src/scheduler.cc:440 onetime_scheduler::get_next: reschedule not allowed. line: 440 00000d93.000022ee 2440bb2c Thu Sep 26 2019 13:34:00 -06:00 [kern_mgwd:info:2284] 0x81d24cb00: 8603e90000000560: WARNING: job_manager::job_record: get_next_run:src/job_record.cc:647 get_next_run(reschedule), returning 0, schedulenext is 0 for job: 21842 - 38231c9a-e08c-11e9-a00d-00a098573067 00000d93.000022ef 2440bb3d Thu Sep 26 2019 13:34:01 -06:00 [kern_mgwd:info:2284] 0x81b512600: 0: ERR: security_mgwd::tables::certificate::certificate: write_hash_files:src/tables/certificate/certificate.cc:1985 SSL: Failed to create symlink for file '/mroot/etc/vserver_4294967295/certificates/ssl/serverCa/99+4A538C28+EntrustRootCertificationAuthority-G2+455f1b52.0.pem' with '/mroot/etc/vserver_4294967295/certificates/ssl/serverCa/455f1b52.0'. Reason: 17 00000d93.00002343 2440d544 Thu Sep 26 2019 13:45:08 -06:00 [kern_mgwd:info:2284] 0x81d251600: 86ffff0000000035: WARNING: job_manager::job_store: run:src/job_store_support.cc:568 Job Mgr: rdb_transaction history: Timeout/0 Timeout/300 OK/600 OK/693 00000d93.000023a5 2440f86c Thu Sep 26 2019 14:00:08 -06:00 [kern_mgwd:info:2284] 0x81d251600: 86ffff0000000035: WARNING: job_manager::job_store: run:src/job_store_support.cc:568 Job Mgr: rdb_transaction history: Timeout/0 Timeout/300 OK/600 00000d93.00002420 24411b76 Thu Sep 26 2019 14:15:05 -06:00 [kern_mgwd:info:2284] 0x81d251600: 86ffff0000000035: WARNING: job_manager::job_store: run:src/job_store_support.cc:568 Job Mgr: rdb_transaction history: OK/0 OK/300 OK/600 00000d93.00002489 24413e9e Thu Sep 26 2019 14:30:05 -06:00 [kern_mgwd:info:2284] 0x81d251600: 86ffff0000000035: WARNING: job_manager::job_store: run:src/job_store_support.cc:568 Job Mgr: rdb_transaction history: Timeout/0 OK/300 OK/600 00000d93.0000249e 244146dc Thu Sep 26 2019 14:33:36 -06:00 [kern_mgwd:info:2284] 0x81d24df00: 8603e9000000066d: WARNING: job_manager::job_sched: modify_schedule:src/job_sched.cc:800 job_manager::job_sched::modify_schedule: Calling set_scheduler on the record: Current schedule = 0x81b84a1a0; New schedule = 0x81b417a20 00000d93.0000249f 244147cb Thu Sep 26 2019 14:34:00 -06:00 [kern_mgwd:info:2284] 0x81d24d500: 8603e900000005e4: WARNING: job_manager::scheduler_other: get_next:src/scheduler.cc:440 onetime_scheduler::get_next: reschedule not allowed. line: 440 00000d93.000024a0 244147cb Thu Sep 26 2019 14:34:00 -06:00 [kern_mgwd:info:2284] 0x81d24d500: 8603e900000005e4: WARNING: job_manager::job_record: get_next_run:src/job_record.cc:647 get_next_run(reschedule), returning 0, schedulenext is 0 for job: 21843 - 999f48b3-e094-11e9-a00d-00a098573067 00000d93.000024a1 244147dd Thu Sep 26 2019 14:34:01 -06:00 [kern_mgwd:info:2284] 0x81b512600: 0: ERR: security_mgwd::tables::certificate::certificate: write_hash_files:src/tables/certificate/certificate.cc:1985 SSL: Failed to create symlink for file '/mroot/etc/vserver_4294967295/certificates/ssl/serverCa/99+4A538C28+EntrustRootCertificationAuthority-G2+455f1b52.0.pem' with '/mroot/etc/vserver_4294967295/certificates/ssl/serverCa/455f1b52.0'. Reason: 17 00000d93.000024e3 244161c6 Thu Sep 26 2019 14:45:05 -06:00 [kern_mgwd:info:2284] 0x81d251600: 86ffff0000000035: WARNING: job_manager::job_store: run:src/job_store_support.cc:568 Job Mgr: rdb_transaction history: Timeout/0 OK/300 OK/600 OK/689 00000d93.00002558 244184ee Thu Sep 26 2019 15:00:05 -06:00 [kern_mgwd:info:2284] 0x81d251600: 86ffff0000000035: WARNING: job_manager::job_store: run:src/job_store_support.cc:568 Job Mgr: rdb_transaction history: Timeout/0 Timeout/300 Timeout/600 00000d93.00002590 244191a9 Thu Sep 26 2019 15:05:29 -06:00 [kern_mgwd:info:2284] Starting a backup of the var filesystem. 00000d93.000025be 24419c7c Thu Sep 26 2019 15:10:08 -06:00 [kern_mgwd:info:2284] 0x81d251600: 86ffff0000000035: WARNING: job_manager::job_store: run:src/job_store_support.cc:568 Job Mgr: rdb_transaction history: OK/0 Timeout/300 OK/600 00000d93.00002637 2441bf86 Thu Sep 26 2019 15:25:05 -06:00 [kern_mgwd:info:2284] 0x81d251600: 86ffff0000000035: WARNING: job_manager::job_store: run:src/job_store_support.cc:568 Job Mgr: rdb_transaction history: Timeout/0 OK/300 OK/597 00000d93.00002677 2441d386 Thu Sep 26 2019 15:33:37 -06:00 [kern_mgwd:info:2284] 0x81d24c600: 8603e90000000703: WARNING: job_manager::job_sched: modify_schedule:src/job_sched.cc:800 job_manager::job_sched::modify_schedule: Calling set_scheduler on the record: Current schedule = 0x81b91d100; New schedule = 0x81b4ed0a0 00000d93.00002678 2441d386 Thu Sep 26 2019 15:33:37 -06:00 [kern_mgwd:info:2284] 0x81d251600: 86ffff0000000035: WARNING: job_manager::job_store: run:src/job_store_support.cc:568 Job Mgr: rdb_transaction history: OK/0 OK/209 Timeout/209 OK/509 00000d93.00002679 2441d46b Thu Sep 26 2019 15:34:00 -06:00 [kern_mgwd:info:2284] 0x81d24e400: 8603e90000000671: WARNING: job_manager::scheduler_other: get_next:src/scheduler.cc:440 onetime_scheduler::get_next: reschedule not allowed. line: 440 00000d93.0000267a 2441d46b Thu Sep 26 2019 15:34:00 -06:00 [kern_mgwd:info:2284] 0x81d24e400: 8603e90000000671: WARNING: job_manager::job_record: get_next_run:src/job_record.cc:647 get_next_run(reschedule), returning 0, schedulenext is 0 for job: 21844 - fb8cdb7b-e09c-11e9-a00d-00a098573067 00000d93.0000267b 2441d47d Thu Sep 26 2019 15:34:01 -06:00 [kern_mgwd:info:2284] 0x81b512600: 0: ERR: security_mgwd::tables::certificate::certificate: write_hash_files:src/tables/certificate/certificate.cc:1985 SSL: Failed to create symlink for file '/mroot/etc/vserver_4294967295/certificates/ssl/serverCa/99+4A538C28+EntrustRootCertificationAuthority-G2+455f1b52.0.pem' with '/mroot/etc/vserver_4294967295/certificates/ssl/serverCa/455f1b52.0'. Reason: 17 00000d93.000026e3 2441f6ae Thu Sep 26 2019 15:48:37 -06:00 [kern_mgwd:info:2284] 0x81d251600: 86ffff0000000035: WARNING: job_manager::job_store: run:src/job_store_support.cc:568 Job Mgr: rdb_transaction history: Timeout/0 Timeout/300 Timeout/600 00000d93.00002739 244219da Thu Sep 26 2019 16:03:37 -06:00 [kern_mgwd:info:2284] 0x81d251600: 86ffff0000000035: WARNING: job_manager::job_store: run:src/job_store_support.cc:568 Job Mgr: rdb_transaction history: Timeout/0 Timeout/300 Timeout/600 00000d93.00002771 244229ed Thu Sep 26 2019 16:10:29 -06:00 [kern_mgwd:info:2284] Starting a backup of the var filesystem. 00000d93.000027af 2442408c Thu Sep 26 2019 16:20:08 -06:00 [kern_mgwd:info:2284] 0x81d251600: 86ffff0000000035: WARNING: job_manager::job_store: run:src/job_store_support.cc:568 Job Mgr: rdb_transaction history: Timeout/0 Timeout/300 OK/600 Timeout/691 00000d93.0000282b 24426030 Thu Sep 26 2019 16:33:38 -06:00 [kern_mgwd:info:2284] 0x81d24c100: 8603e90000000787: WARNING: job_manager::job_sched: modify_schedule:src/job_sched.cc:800 job_manager::job_sched::modify_schedule: Calling set_scheduler on the record: Current schedule = 0x81b91d4a0; New schedule = 0x81b2191e0 00000d93.0000282c 24426030 Thu Sep 26 2019 16:33:38 -06:00 [kern_mgwd:info:2284] 0x81d251600: 86ffff0000000035: WARNING: job_manager::job_store: run:src/job_store_support.cc:568 Job Mgr: rdb_transaction history: OK/0 Timeout/211 OK/511 00000d93.0000282d 2442610b Thu Sep 26 2019 16:34:00 -06:00 [kern_mgwd:info:2284] 0x81d24cb00: 8603e90000000707: WARNING: job_manager::scheduler_other: get_next:src/scheduler.cc:440 onetime_scheduler::get_next: reschedule not allowed. line: 440 00000d93.0000282e 2442610b Thu Sep 26 2019 16:34:00 -06:00 [kern_mgwd:info:2284] 0x81d24cb00: 8603e90000000707: WARNING: job_manager::job_record: get_next_run:src/job_record.cc:647 get_next_run(reschedule), returning 0, schedulenext is 0 for job: 21845 - 5d104d43-e0a5-11e9-a00d-00a098573067 00000d93.0000282f 2442611d Thu Sep 26 2019 16:34:01 -06:00 [kern_mgwd:info:2284] 0x81b512600: 0: ERR: security_mgwd::tables::certificate::certificate: write_hash_files:src/tables/certificate/certificate.cc:1985 SSL: Failed to create symlink for file '/mroot/etc/vserver_4294967295/certificates/ssl/serverCa/99+4A538C28+EntrustRootCertificationAuthority-G2+455f1b52.0.pem' with '/mroot/etc/vserver_4294967295/certificates/ssl/serverCa/455f1b52.0'. Reason: 17 00000d93.0000289d 244286dc Thu Sep 26 2019 16:50:08 -06:00 [kern_mgwd:info:2284] 0x81d251600: 86ffff0000000035: WARNING: job_manager::job_store: run:src/job_store_support.cc:568 Job Mgr: rdb_transaction history: Timeout/0 Timeout/300 OK/600 Timeout/690 00000d93.000028f3 2442aa08 Thu Sep 26 2019 17:05:08 -06:00 [kern_mgwd:info:2284] 0x81d251600: 86ffff0000000035: WARNING: job_manager::job_store: run:src/job_store_support.cc:568 Job Mgr: rdb_transaction history: Timeout/0 Timeout/300 OK/600 00000d93.0000292e 2442bdf7 Thu Sep 26 2019 17:13:37 -06:00 [kern_mgwd:info:2284] Info: sp_passwd_update: sp user update: passwd_rdb callback, rdb_op = 0 00000d93.0000292f 2442bdf7 Thu Sep 26 2019 17:13:37 -06:00 [kern_mgwd:info:2284] Info: sp_passwd_update: sp user update: passwd_rdb callback, rdb_op = 0 00000d93.00002930 2442bdf7 Thu Sep 26 2019 17:13:38 -06:00 [kern_mgwd:info:2284] Starting a backup of the var filesystem. 00000d93.00002934 2442be5c Thu Sep 26 2019 17:13:48 -06:00 [kern_mgwd:info:2284] Info: sp_passwd_update: sp user update: passwd_rdb callback, rdb_op = 0 00000d93.00002966 2442cd2c Thu Sep 26 2019 17:20:08 -06:00 [kern_mgwd:info:2284] 0x81d251600: 86ffff0000000035: WARNING: job_manager::job_store: run:src/job_store_support.cc:568 Job Mgr: rdb_transaction history: Timeout/0 Timeout/300 Timeout/600 00000d93.000029da 2442edb0 Thu Sep 26 2019 17:34:00 -06:00 [kern_mgwd:info:2284] 0x81d24df00: 8603e9000000078d: WARNING: job_manager::scheduler_other: get_next:src/scheduler.cc:440 onetime_scheduler::get_next: reschedule not allowed. line: 440 00000d93.000029db 2442edb0 Thu Sep 26 2019 17:34:00 -06:00 [kern_mgwd:info:2284] 0x81d24df00: 8603e9000000078d: WARNING: job_manager::job_record: get_next_run:src/job_record.cc:647 get_next_run(reschedule), returning 0, schedulenext is 0 for job: 21846 - bee9af40-e0ad-11e9-a00d-00a098573067 00000d93.000029dd 2442edbc Thu Sep 26 2019 17:34:01 -06:00 [kern_mgwd:info:2284] 0x81b512600: 0: ERR: security_mgwd::tables::certificate::certificate: write_hash_files:src/tables/certificate/certificate.cc:1985 SSL: Failed to create symlink for file '/mroot/etc/vserver_4294967295/certificates/ssl/serverCa/99+4A538C28+EntrustRootCertificationAuthority-G2+455f1b52.0.pem' with '/mroot/etc/vserver_4294967295/certificates/ssl/serverCa/455f1b52.0'. Reason: 17 00000d93.000029e4 2442f03f Thu Sep 26 2019 17:35:06 -06:00 [kern_mgwd:info:2284] 0x81d251600: 86ffff0000000035: WARNING: job_manager::job_store: run:src/job_store_support.cc:568 Job Mgr: rdb_transaction history: Timeout/0 Timeout/300 OK/600 00000d93.00002a41 24431367 Thu Sep 26 2019 17:50:06 -06:00 [kern_mgwd:info:2284] 0x81d251600: 86ffff0000000035: WARNING: job_manager::job_store: run:src/job_store_support.cc:568 Job Mgr: rdb_transaction history: Timeout/0 Timeout/300 Timeout/600 00000d93.00002ba9 244336a3 Thu Sep 26 2019 18:05:08 -06:00 [kern_mgwd:info:2284] 0x81d251600: 86ffff0000000035: WARNING: job_manager::job_store: run:src/job_store_support.cc:568 Job Mgr: rdb_transaction history: Timeout/0 OK/300 Timeout/302 Timeout/602 00000d93.00002be7 24434b0b Thu Sep 26 2019 18:13:50 -06:00 [kern_mgwd:info:2284] Starting a backup of the var filesystem. 00000d93.00002c1d 244354a2 Thu Sep 26 2019 18:17:54 -06:00 [kern_mgwd:info:2284] 0x81d24b200: 8603e90000000892: WARNING: job_manager::job_sched: process_modify:src/job_sched.cc:487 UIID:8860 UUID:d8183955-e0bb-11e9-93b8-00a09857ee8f name:Local backup job rc:1 Doing nothing in modify callback for job, since it's pending cleanup 00000d93.00002c20 244354cb Thu Sep 26 2019 18:18:00 -06:00 [kern_mgwd:info:2284] Info: smf-net: failed due to no more data or timeout. timeout=0 sec, fd=42, numFdReady=0, bytesRcvd=0, FD_ISSET=false, buflength=512 00000d93.00002c21 244354d9 Thu Sep 26 2019 18:18:01 -06:00 [kern_mgwd:info:2284] 0x81dacd800: 0: WARNING: FileReplication: The file replication service successfully distributed package nmg-fas8040-02.8hour.2019-09-26.18_15_00.7z:0 to 169.254.23.82. 00000d93.00002c24 244355dc Thu Sep 26 2019 18:18:27 -06:00 [kern_mgwd:info:2284] 0x81d24d500: 8603e90000000899: WARNING: FileReplication: Starting subscription (download) request for package nmg-fas8040.8hour.2019-09-26.18_15_00.7z:latest with old API extract target '/mroot/etc/backups/.tmp/nmg-fas8040.8hour.2019-09-26.18_15_00.7z'. 00000d93.00002c25 24435617 Thu Sep 26 2019 18:18:33 -06:00 [kern_mgwd:info:2284] 0x81d251600: 86ffff0000000035: WARNING: job_manager::job_store: run:src/job_store_support.cc:568 Job Mgr: rdb_transaction history: OK/0 OK/38 Timeout/205 Timeout/505 00000d93.00002c26 24435617 Thu Sep 26 2019 18:18:33 -06:00 [kern_mgwd:info:2284] 0x81d24b200: 8603e9000000089d: WARNING: job_manager::job_sched: process_modify:src/job_sched.cc:487 UIID:8862 UUID:5398f184-e0bc-11e9-93b8-00a09857ee8f name:Receive backup job rc:1 Doing nothing in modify callback for job, since it's pending cleanup 00000d93.00002c66 24437239 Thu Sep 26 2019 18:30:33 -06:00 [kern_mgwd:info:2284] 0x81d251600: 86ffff0000000035: WARNING: job_manager::job_store: run:src/job_store_support.cc:568 Job Mgr: rdb_transaction history: Timeout/0 Timeout/300 OK/600 OK/639 00000d93.00002cbe 24437a4a Thu Sep 26 2019 18:34:00 -06:00 [kern_mgwd:info:2284] 0x81d24cb00: 8603e90000000815: WARNING: job_manager::scheduler_other: get_next:src/scheduler.cc:440 onetime_scheduler::get_next: reschedule not allowed. line: 440 00000d93.00002cbf 24437a4a Thu Sep 26 2019 18:34:00 -06:00 [kern_mgwd:info:2284] 0x81d24cb00: 8603e90000000815: WARNING: job_manager::job_record: get_next_run:src/job_record.cc:647 get_next_run(reschedule), returning 0, schedulenext is 0 for job: 21847 - 20badb03-e0b6-11e9-a00d-00a098573067 00000d93.00002cc0 24437a60 Thu Sep 26 2019 18:34:02 -06:00 [kern_mgwd:info:2284] 0x81b512600: 0: ERR: security_mgwd::tables::certificate::certificate: write_hash_files:src/tables/certificate/certificate.cc:1985 SSL: Failed to create symlink for file '/mroot/etc/vserver_4294967295/certificates/ssl/serverCa/99+4A538C28+EntrustRootCertificationAuthority-G2+455f1b52.0.pem' with '/mroot/etc/vserver_4294967295/certificates/ssl/serverCa/455f1b52.0'. Reason: 17 00000d93.00002d15 24439463 Thu Sep 26 2019 18:45:08 -06:00 [kern_mgwd:info:2284] 0x81d251600: 86ffff0000000035: WARNING: job_manager::job_store: run:src/job_store_support.cc:568 Job Mgr: rdb_transaction history: OK/0 Timeout/275 Timeout/575 00000d93.00002d75 2443b78b Thu Sep 26 2019 19:00:08 -06:00 [kern_mgwd:info:2284] 0x81d251600: 86ffff0000000035: WARNING: job_manager::job_store: run:src/job_store_support.cc:568 Job Mgr: rdb_transaction history: OK/0 Timeout/300 Timeout/600 00000d93.00002dff 2443dab3 Thu Sep 26 2019 19:15:08 -06:00 [kern_mgwd:info:2284] 0x81d251600: 86ffff0000000035: WARNING: job_manager::job_store: run:src/job_store_support.cc:568 Job Mgr: rdb_transaction history: OK/0 Timeout/300 Timeout/600 00000d93.00002e77 2443fddb Thu Sep 26 2019 19:30:08 -06:00 [kern_mgwd:info:2284] 0x81d251600: 86ffff0000000035: WARNING: job_manager::job_store: run:src/job_store_support.cc:568 Job Mgr: rdb_transaction history: OK/0 Timeout/300 Timeout/600 00000d93.00002e8a 24440672 Thu Sep 26 2019 19:33:48 -06:00 [kern_mgwd:info:2284] 0x81d24bc00: 86ffff0000000023: WARNING: job_manager::job_store: claim_job_tx:src/job_rec_int.cc:581 claim_job: Attempted to claim a job that was already owned: 179bd90b-b79e-11e7-9b07-00a09857ee8f owner: f28b11a3-b769-11e7-9b07-00a09857ee8f 00000d93.00002e8b 24440672 Thu Sep 26 2019 19:33:48 -06:00 [kern_mgwd:info:2284] 0x81d24bc00: 86ffff0000000023: WARNING: job_manager::job_sched: _check_claimable:src/job_sched.cc:2225 UIID:14 UUID:179bd90b-b79e-11e7-9b07-00a09857ee8f name:Vol Reaper rc:3 _check_claimable: failed to claim jobs: 12 00000d93.00002e8f 244406ea Thu Sep 26 2019 19:34:00 -06:00 [kern_mgwd:info:2284] 0x81d24c600: 8603e900000008ca: WARNING: job_manager::scheduler_other: get_next:src/scheduler.cc:440 onetime_scheduler::get_next: reschedule not allowed. line: 440 00000d93.00002e90 244406ea Thu Sep 26 2019 19:34:00 -06:00 [kern_mgwd:info:2284] 0x81d24c600: 8603e900000008ca: WARNING: job_manager::job_record: get_next_run:src/job_record.cc:647 get_next_run(reschedule), returning 0, schedulenext is 0 for job: 21849 - 830ee9ff-e0be-11e9-a00d-00a098573067 00000d93.00002e91 24440703 Thu Sep 26 2019 19:34:01 -06:00 [kern_mgwd:info:2284] 0x81b512600: 0: ERR: security_mgwd::tables::certificate::certificate: write_hash_files:src/tables/certificate/certificate.cc:1985 SSL: Failed to create symlink for file '/mroot/etc/vserver_4294967295/certificates/ssl/serverCa/99+4A538C28+EntrustRootCertificationAuthority-G2+455f1b52.0.pem' with '/mroot/etc/vserver_4294967295/certificates/ssl/serverCa/455f1b52.0'. Reason: 17 00000d93.00002ebf 244413d3 Thu Sep 26 2019 19:39:30 -06:00 [kern_mgwd:info:2284] Starting a backup of the var filesystem. 00000d93.00002ec0 244415c1 Thu Sep 26 2019 19:40:19 -06:00 [kern_mgwd:info:2284] A [src/rdb/HAM.cc 1233 (0x81b51a300)]: checkHASFOConfigVerbose: haConfigured: True, result HA_AGENT_CONFIGURED. 00000d93.00002ef8 24442103 Thu Sep 26 2019 19:45:08 -06:00 [kern_mgwd:info:2284] 0x81d251600: 86ffff0000000035: WARNING: job_manager::job_store: run:src/job_store_support.cc:568 Job Mgr: rdb_transaction history: Timeout/0 Timeout/300 Timeout/600 00000d93.00002eff 24442c6a Thu Sep 26 2019 19:50:00 -06:00 [kern_mgwd:info:2284] 0x81d24da00: 86ffff0000000029: WARNING: job_manager::job_store: claim_job_tx:src/job_rec_int.cc:581 claim_job: Attempted to claim a job that was already owned: 489d5bff-b80c-11e7-964b-00a098573067 owner: f28b11a3-b769-11e7-9b07-00a09857ee8f 00000d93.00002f00 24442c6a Thu Sep 26 2019 19:50:00 -06:00 [kern_mgwd:info:2284] 0x81d24da00: 86ffff0000000029: WARNING: job_manager::job_sched: _check_claimable:src/job_sched.cc:2225 UIID:19 UUID:489d5bff-b80c-11e7-964b-00a098573067 name:Network Recovery Job - 5min rc:3 _check_claimable: failed to claim jobs: 12 00000d93.00002f32 244434ad Thu Sep 26 2019 19:53:31 -06:00 [kern_mgwd:info:2284] 0x81dacd300: 0: ERR: perf::statistics: fail open on /mroot/etc/log/stats/samples/. 00000d93.00002f50 2444442b Thu Sep 26 2019 20:00:08 -06:00 [kern_mgwd:info:2284] 0x81d251600: 86ffff0000000035: WARNING: job_manager::job_store: run:src/job_store_support.cc:568 Job Mgr: rdb_transaction history: Timeout/0 Timeout/300 Timeout/600 00000d93.00002fbd 24446753 Thu Sep 26 2019 20:15:08 -06:00 [kern_mgwd:info:2284] 0x81d251600: 86ffff0000000035: WARNING: job_manager::job_store: run:src/job_store_support.cc:568 Job Mgr: rdb_transaction history: Timeout/0 Timeout/300 Timeout/600 00000d93.00003036 24448a5c Thu Sep 26 2019 20:30:05 -06:00 [kern_mgwd:info:2284] 0x81d251600: 86ffff0000000035: WARNING: job_manager::job_store: run:src/job_store_support.cc:568 Job Mgr: rdb_transaction history: Timeout/0 OK/300 Timeout/597 00000d93.0000303c 2444931c Thu Sep 26 2019 20:33:49 -06:00 [kern_mgwd:info:2284] 0x81d24c100: 8603e90000000a06: WARNING: job_manager::job_sched: modify_schedule:src/job_sched.cc:800 job_manager::job_sched::modify_schedule: Calling set_scheduler on the record: Current schedule = 0x81b91d9a0; New schedule = 0x81c158180 00000d93.0000303d 2444938a Thu Sep 26 2019 20:34:00 -06:00 [kern_mgwd:info:2284] 0x81d24da00: 8603e90000000959: WARNING: job_manager::scheduler_other: get_next:src/scheduler.cc:440 onetime_scheduler::get_next: reschedule not allowed. line: 440 00000d93.0000303e 2444938a Thu Sep 26 2019 20:34:00 -06:00 [kern_mgwd:info:2284] 0x81d24da00: 8603e90000000959: WARNING: job_manager::job_record: get_next_run:src/job_record.cc:647 get_next_run(reschedule), returning 0, schedulenext is 0 for job: 21850 - e41d9819-e0c6-11e9-a00d-00a098573067 00000d93.0000303f 2444939d Thu Sep 26 2019 20:34:01 -06:00 [kern_mgwd:info:2284] 0x81b512600: 0: ERR: security_mgwd::tables::certificate::certificate: write_hash_files:src/tables/certificate/certificate.cc:1985 SSL: Failed to create symlink for file '/mroot/etc/vserver_4294967295/certificates/ssl/serverCa/99+4A538C28+EntrustRootCertificationAuthority-G2+455f1b52.0.pem' with '/mroot/etc/vserver_4294967295/certificates/ssl/serverCa/455f1b52.0'. Reason: 17 00000d93.0000307d 24449ed5 Thu Sep 26 2019 20:38:49 -06:00 [kern_mgwd:info:2284] 0x81d251600: 86ffff0000000035: WARNING: job_manager::job_store: run:src/job_store_support.cc:568 Job Mgr: rdb_transaction history: Timeout/0 OK/300 OK/521