Active IQ and AutoSupport Discussions

cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Active IQ and AutoSupport Discussions

Ask a Question

why ASUP process hung in transmitting status?

maqiuming
‎2017-06-21 11:34 PM
5,675 Views

Platform : FAS3250

version: 8.1.2 cluster mode

company:IGT

 

symptom: ASUP can not received by EMail .

 

ASUP can't received form 6/15 , seq 3207  Autosupport log continue under transmitting status , does it process hung  ? can we kill it via systemshell?

 

there no cancel transmit command in 8.1.2 C-mode

 

1. ASUP enabled

Node             StateFrom     To       Mail Hosts

--------------------- --------- ------------- ------------- ----------

PEKNAP-01        enablepeknap01@igt.com
                                         support@netapp.com, qiuming.ma@igt.com
                                                       10.210.41.203
PEKNAP-02        enablepeknap02@igt.com
                                         qiuming.ma@igt.com, support@netapp.com
                                                       10.210.41.203

2 entries were displayed.

 

2.ping SMTP mail server is alive

PEKNAP::system node autosupport*> net ping -node PEKNAP-01 -destination 10.210.41.203

  (network ping)

10.210.41.203 is alive

 

3.detail setting in ASUP CHECK log, notifyd log -->

 

4.

PEKNAP::system node autosupport*> history show

                                                                                              

             Seq                                    Attempt  Last

Node         Num   Destination Status               Count    Update

------------ ----- ----------- -------------------- -------- -----------------

PEKNAP-01    3218

                   http        queued               -        6/15/2017 11:48:21

                   noteto      queued               -        6/15/2017 11:48:21

             3217

                   smtp        queued               -        6/15/2017 09:01:02

                   http        queued               -        6/15/2017 09:01:02

                   noteto      queued               -        6/15/2017 09:01:02

             3216

                   smtp        queued               -        6/15/2017 08:59:50

                   http        queued               -        6/15/2017 08:59:50

                   noteto      queued               -        6/15/2017 08:59:50

             3215

                   smtp        re-queued            -        6/15/2017 07:54:43

                   http        re-queued            -        6/15/2017 07:54:43

                   noteto      re-queued            -        6/15/2017 07:54:43

             3214

                   smtp        re-queued            -        6/15/2017 07:54:18

                   http        re-queued            -        6/15/2017 07:54:18

                   noteto      re-queued            -        6/15/2017 07:54:18

             3213

                   smtp        collection-failed    -        6/15/2017 07:53:29

                   http        collection-failed    -        6/15/2017 07:53:29

                   noteto      collection-failed    -        6/15/2017 07:53:29

             3212

                   smtp        collection-failed    -        6/15/2017 07:53:02

                   http        collection-failed    -        6/15/2017 07:53:02

                   noteto      collection-failed    -        6/15/2017 07:53:02

             3211

                   smtp        collection-failed    -        6/15/2017 07:52:34

                   http        collection-failed    -        6/15/2017 07:52:34

                   noteto      collection-failed    -        6/15/2017 07:52:34

             3210

                   smtp        collection-failed    -        6/15/2017 07:52:15

                   http        collection-failed    -        6/15/2017 07:52:15

                   noteto      collection-failed    -        6/15/2017 07:52:15

             3209

                   smtp        collection-failed    -        6/15/2017 07:52:00

                   http        collection-failed    -        6/15/2017 07:52:00

                   noteto      collection-failed    -        6/15/2017 07:52:00

             3208

                   smtp        re-queued            -        6/15/2017 07:46:42

                   http        re-queued            -        6/15/2017 07:46:42

                   noteto      re-queued            -        6/15/2017 07:46:42

             3207

                   smtp        transmitting         -        6/15/2017 08:58:44

                   http        re-queued            -        6/15/2017 07:45:16

                   noteto      ignore               1        6/15/2017 07:53:10

                   retransmit  queued               -        6/22/2017 08:06:04

             3206

                   smtp        sent-successful      1        6/15/2017 07:53:10

                   http        ignore               1        6/15/2017 07:53:10

                   noteto      sent-successful      1        6/15/2017 07:52:58

             3205

                   smtp        sent-successful      1        6/15/2017 07:52:56

                   http        ignore               1        6/15/2017 07:52:56

                   noteto      sent-successful      1        6/15/2017 07:52:53

             3204

                   smtp        collection-failed    -        6/15/2017 07:34:46

                   http        collection-failed    -        6/15/2017 07:34:46

                   noteto      collection-failed    -        6/15/2017 07:34:46

             3203

                   smtp        sent-successful      1        6/15/2017 07:47:29

                   http        ignore               1        6/15/2017 07:47:29

                   noteto      ignore               1        6/15/2017 07:47:25

             3202

                   smtp        sent-successful      1        6/15/2017 07:47:25

                   http        ignore               1        6/15/2017 07:47:25

                   noteto      sent-successful      1        6/15/2017 07:47:19

             3201

                   smtp        sent-successful      1        6/15/2017 07:47:17

                   http        ignore               1        6/15/2017 07:32:26

                   noteto      ignore               1        6/15/2017 07:47:15

             3200

                   smtp        sent-successful      1        6/15/2017 07:47:15

                   http        ignore               1        6/15/2017 07:32:26

                   noteto      ignore               1        6/15/2017 07:47:11

             3199

                   smtp        sent-successful      1        6/15/2017 07:47:11

                   http        ignore               1        6/15/2017 07:32:26

                   noteto      ignore               1        6/15/2017 07:47:08

             3198

                   smtp        sent-successful      1        6/15/2017 07:47:08

                   http        ignore               1        6/15/2017 07:32:26

 

PEKNAP::system node autosupport*> history show -seq-num 3206 -instance

 

                            Node: PEKNAP-01

     AutoSupport Sequence Number: 3206

Destination for this AutoSupport: smtp

                   Trigger Event: callhome.sfo.takeover

             Time of Last Update: 6/15/2017 07:53:10

              Status of Delivery: sent-successful

               Delivery Attempts: 1

             AutoSupport Subject: CONTROLLER TAKEOVER COMPLETE AUTOMATIC

                    Delivery URI: mailto:autosupport@netapp.com,certified@eccom.com.cn,qiuming.ma@igt.com,support@netapp.com

                      Last Error: -

 

                            Node: PEKNAP-01

     AutoSupport Sequence Number: 3206

Destination for this AutoSupport: http

                   Trigger Event: callhome.sfo.takeover

             Time of Last Update: 6/15/2017 07:53:10

              Status of Delivery: ignore

               Delivery Attempts: 1

             AutoSupport Subject: CONTROLLER TAKEOVER COMPLETE AUTOMATIC

                    Delivery URI: -

                      Last Error: -

 

                            Node: PEKNAP-01

     AutoSupport Sequence Number: 3206

Destination for this AutoSupport: noteto

                   Trigger Event: callhome.sfo.takeover

             Time of Last Update: 6/15/2017 07:52:58

              Status of Delivery: sent-successful

               Delivery Attempts: 1

             AutoSupport Subject: CONTROLLER TAKEOVER COMPLETE AUTOMATIC

                    Delivery URI: mailto:jason.tang@igt.com,jianhui.du@igt.com,rui.wang@igt.com

                      Last Error: -

3 entries were displayed.

 

PEKNAP::system node autosupport*> history show -seq-num 3206 -instance  -instance  7 -instance  -instance

 

                            Node: PEKNAP-01

     AutoSupport Sequence Number: 3207

Destination for this AutoSupport: smtp

                   Trigger Event: callhome.reboot.takeover

             Time of Last Update: 6/15/2017 08:58:44

              Status of Delivery: transmitting

               Delivery Attempts: -

             AutoSupport Subject: PARTNER REBOOT (CONTROLLER TAKEOVER)

                    Delivery URI: -

                      Last Error: -

 

                            Node: PEKNAP-01

     AutoSupport Sequence Number: 3207

Destination for this AutoSupport: http

                   Trigger Event: callhome.reboot.takeover

             Time of Last Update: 6/15/2017 07:45:16

              Status of Delivery: re-queued

               Delivery Attempts: -

             AutoSupport Subject: PARTNER REBOOT (CONTROLLER TAKEOVER)

                    Delivery URI: -

                      Last Error: -

 

                            Node: PEKNAP-01

     AutoSupport Sequence Number: 3207

Destination for this AutoSupport: noteto

                   Trigger Event: callhome.reboot.takeover

             Time of Last Update: 6/15/2017 07:53:10

              Status of Delivery: ignore

               Delivery Attempts: 1

             AutoSupport Subject: PARTNER REBOOT (CONTROLLER TAKEOVER)

                    Delivery URI: -

                      Last Error: -

 

                            Node: PEKNAP-01

     AutoSupport Sequence Number: 3207

Destination for this AutoSupport: retransmit

                   Trigger Event: callhome.reboot.takeover

             Time of Last Update: 6/22/2017 08:06:04

              Status of Delivery: queued

               Delivery Attempts: -

             AutoSupport Subject: PARTNER REBOOT (CONTROLLER TAKEOVER)

                    Delivery URI: https://support.netapp.com/asupprod/post/1.0/postAsup

                      Last Error: -

4 entries were displayed.

0 Kudos
Tags (1)
3 REPLIES 3

maqiuming
‎2017-06-21 11:43 PM
5,668 Views
@maqiuming wrote:

Platform : FAS3250

version: 8.1.2 cluster mode

company:IGT

 

symptom: ASUP can not received by EMail .

 

ASUP can't received form 6/15 , seq 3207  Autosupport log continue under transmitting status , does it process hung  ? can we kill it via systemshell?

 

there no cancel transmit command in 8.1.2 C-mode

 

1. ASUP enabled

Node             StateFrom     To       Mail Hosts

--------------------- --------- ------------- ------------- ----------

PEKNAP-01        enablepeknap01@igt.com
                                         support@netapp.com, qiuming.ma@igt.com
                                                       10.210.41.203
PEKNAP-02        enablepeknap02@igt.com
                                         qiuming.ma@igt.com, support@netapp.com
                                                       10.210.41.203

2 entries were displayed.

 

3.detail setting in ASUP CHECK log, notifyd log -->

 

2.ping SMTP mail server is alive

PEKNAP::system node autosupport*> net ping -node PEKNAP-01 -destination 10.210.41.203

  (network ping)

10.210.41.203 is alive

 

4.

PEKNAP::system node autosupport*> history show

                                                                                              

             Seq                                    Attempt  Last

Node         Num   Destination Status               Count    Update

------------ ----- ----------- -------------------- -------- -----------------

PEKNAP-01    3218

                   http        queued               -        6/15/2017 11:48:21

                   noteto      queued               -        6/15/2017 11:48:21

             3217

                   smtp        queued               -        6/15/2017 09:01:02

                   http        queued               -        6/15/2017 09:01:02

                   noteto      queued               -        6/15/2017 09:01:02

             3216

                   smtp        queued               -        6/15/2017 08:59:50

                   http        queued               -        6/15/2017 08:59:50

                   noteto      queued               -        6/15/2017 08:59:50

             3215

                   smtp        re-queued            -        6/15/2017 07:54:43

                   http        re-queued            -        6/15/2017 07:54:43

                   noteto      re-queued            -        6/15/2017 07:54:43

             3214

                   smtp        re-queued            -        6/15/2017 07:54:18

                   http        re-queued            -        6/15/2017 07:54:18

                   noteto      re-queued            -        6/15/2017 07:54:18

             3213

                   smtp        collection-failed    -        6/15/2017 07:53:29

                   http        collection-failed    -        6/15/2017 07:53:29

                   noteto      collection-failed    -        6/15/2017 07:53:29

             3212

                   smtp        collection-failed    -        6/15/2017 07:53:02

                   http        collection-failed    -        6/15/2017 07:53:02

                   noteto      collection-failed    -        6/15/2017 07:53:02

             3211

                   smtp        collection-failed    -        6/15/2017 07:52:34

                   http        collection-failed    -        6/15/2017 07:52:34

                   noteto      collection-failed    -        6/15/2017 07:52:34

             3210

                   smtp        collection-failed    -        6/15/2017 07:52:15

                   http        collection-failed    -        6/15/2017 07:52:15

                   noteto      collection-failed    -        6/15/2017 07:52:15

             3209

                   smtp        collection-failed    -        6/15/2017 07:52:00

                   http        collection-failed    -        6/15/2017 07:52:00

                   noteto      collection-failed    -        6/15/2017 07:52:00

             3208

                   smtp        re-queued            -        6/15/2017 07:46:42

                   http        re-queued            -        6/15/2017 07:46:42

                   noteto      re-queued            -        6/15/2017 07:46:42

             3207

                   smtp        transmitting         -        6/15/2017 08:58:44

                   http        re-queued            -        6/15/2017 07:45:16

                   noteto      ignore               1        6/15/2017 07:53:10

                   retransmit  queued               -        6/22/2017 08:06:04

             3206

                   smtp        sent-successful      1        6/15/2017 07:53:10

                   http        ignore               1        6/15/2017 07:53:10

                   noteto      sent-successful      1        6/15/2017 07:52:58

             3205

                   smtp        sent-successful      1        6/15/2017 07:52:56

                   http        ignore               1        6/15/2017 07:52:56

                   noteto      sent-successful      1        6/15/2017 07:52:53

             3204

                   smtp        collection-failed    -        6/15/2017 07:34:46

                   http        collection-failed    -        6/15/2017 07:34:46

                   noteto      collection-failed    -        6/15/2017 07:34:46

             3203

                   smtp        sent-successful      1        6/15/2017 07:47:29

                   http        ignore               1        6/15/2017 07:47:29

                   noteto      ignore               1        6/15/2017 07:47:25

             3202

                   smtp        sent-successful      1        6/15/2017 07:47:25

                   http        ignore               1        6/15/2017 07:47:25

                   noteto      sent-successful      1        6/15/2017 07:47:19

             3201

                   smtp        sent-successful      1        6/15/2017 07:47:17

                   http        ignore               1        6/15/2017 07:32:26

                   noteto      ignore               1        6/15/2017 07:47:15

             3200

                   smtp        sent-successful      1        6/15/2017 07:47:15

                   http        ignore               1        6/15/2017 07:32:26

                   noteto      ignore               1        6/15/2017 07:47:11

             3199

                   smtp        sent-successful      1        6/15/2017 07:47:11

                   http        ignore               1        6/15/2017 07:32:26

                   noteto      ignore               1        6/15/2017 07:47:08

             3198

                   smtp        sent-successful      1        6/15/2017 07:47:08

                   http        ignore               1        6/15/2017 07:32:26

 

PEKNAP::system node autosupport*> history show -seq-num 3206 -instance

 

                            Node: PEKNAP-01

     AutoSupport Sequence Number: 3206

Destination for this AutoSupport: smtp

                   Trigger Event: callhome.sfo.takeover

             Time of Last Update: 6/15/2017 07:53:10

              Status of Delivery: sent-successful

               Delivery Attempts: 1

             AutoSupport Subject: CONTROLLER TAKEOVER COMPLETE AUTOMATIC

                    Delivery URI: mailto:autosupport@netapp.com,certified@eccom.com.cn,qiuming.ma@igt.com,support@netapp.com

                      Last Error: -

 

                            Node: PEKNAP-01

     AutoSupport Sequence Number: 3206

Destination for this AutoSupport: http

                   Trigger Event: callhome.sfo.takeover

             Time of Last Update: 6/15/2017 07:53:10

              Status of Delivery: ignore

               Delivery Attempts: 1

             AutoSupport Subject: CONTROLLER TAKEOVER COMPLETE AUTOMATIC

                    Delivery URI: -

                      Last Error: -

 

                            Node: PEKNAP-01

     AutoSupport Sequence Number: 3206

Destination for this AutoSupport: noteto

                   Trigger Event: callhome.sfo.takeover

             Time of Last Update: 6/15/2017 07:52:58

              Status of Delivery: sent-successful

               Delivery Attempts: 1

             AutoSupport Subject: CONTROLLER TAKEOVER COMPLETE AUTOMATIC

                    Delivery URI: mailto:jason.tang@igt.com,jianhui.du@igt.com,rui.wang@igt.com

                      Last Error: -

3 entries were displayed.

 

PEKNAP::system node autosupport*> history show -seq-num 3206 -instance  -instance  7 -instance  -instance

 

                            Node: PEKNAP-01

     AutoSupport Sequence Number: 3207

Destination for this AutoSupport: smtp

                   Trigger Event: callhome.reboot.takeover

             Time of Last Update: 6/15/2017 08:58:44

              Status of Delivery: transmitting

               Delivery Attempts: -

             AutoSupport Subject: PARTNER REBOOT (CONTROLLER TAKEOVER)

                    Delivery URI: -

                      Last Error: -

 

                            Node: PEKNAP-01

     AutoSupport Sequence Number: 3207

Destination for this AutoSupport: http

                   Trigger Event: callhome.reboot.takeover

             Time of Last Update: 6/15/2017 07:45:16

              Status of Delivery: re-queued

               Delivery Attempts: -

             AutoSupport Subject: PARTNER REBOOT (CONTROLLER TAKEOVER)

                    Delivery URI: -

                      Last Error: -

 

                            Node: PEKNAP-01

     AutoSupport Sequence Number: 3207

Destination for this AutoSupport: noteto

                   Trigger Event: callhome.reboot.takeover

             Time of Last Update: 6/15/2017 07:53:10

              Status of Delivery: ignore

               Delivery Attempts: 1

             AutoSupport Subject: PARTNER REBOOT (CONTROLLER TAKEOVER)

                    Delivery URI: -

                      Last Error: -

 

                            Node: PEKNAP-01

     AutoSupport Sequence Number: 3207

Destination for this AutoSupport: retransmit

                   Trigger Event: callhome.reboot.takeover

             Time of Last Update: 6/22/2017 08:06:04

              Status of Delivery: queued

               Delivery Attempts: -

             AutoSupport Subject: PARTNER REBOOT (CONTROLLER TAKEOVER)

                    Delivery URI: https://support.netapp.com/asupprod/post/1.0/postAsup

                      Last Error: -

4 entries were displayed.


can not upload log , place here directly , may be large , will cut out the import time.

 

question is , everythings for config looks normal form the CLI , but the ASUP still can not send out , does some command can kill the transmitting session and restart form the least one ? 

 

00000016.00000b31 000013d3 Thu Jun 15 2017 07:46:44 +00:00 [kern_notifyd:info:737] Successfully delivered smtp asup with sequence: 3196
00000016.00000b32 000013d3 Thu Jun 15 2017 07:46:44 +00:00 [kern_notifyd:info:737] Beginning to deliver asup with sequence number 3197, subject HA Group Notification from PEKNAP-01 (MANAGEMENT_LOG) INFO
00000016.00000b33 000013d3 Thu Jun 15 2017 07:46:44 +00:00 [kern_notifyd:info:737] Successfully delivered noteto asup with sequence: 3197
00000016.00000b34 000013d3 Thu Jun 15 2017 07:46:44 +00:00 [kern_notifyd:info:737] 0x8072d9e00: ERR: Autosupport::Delivery: created compressor with out_filename = /mroot/etc/log/autosupport/201706130030.0.smtp_tmp
00000016.00000b3d 00001492 Thu Jun 15 2017 07:47:03 +00:00 [kern_notifyd:info:737] Address Info (0x807b1f2c0) [ai_flags: 1025, ai_family: 2, ai_socktype: 1, ai_protocol: 6, ai_addrlen: 16, ai_addr: 0x807b1f2f0, ai_canonname: (null), ai_next: 0x0]
00000016.00000b3e 000014bb Thu Jun 15 2017 07:47:04 +00:00 [kern_notifyd:info:737] Subject: HA Group Notification from PEKNAP-01 (MANAGEMENT_LOG) INFO
00000016.00000b3f 000014bb Thu Jun 15 2017 07:47:05 +00:00 [kern_notifyd:info:737] Successfully delivered smtp asup with sequence: 3197
00000016.00000b40 000014bb Thu Jun 15 2017 07:47:05 +00:00 [kern_notifyd:info:737] Beginning to deliver asup with sequence number 3198, subject HA Group Notification from PEKNAP-01 (PERFORMANCE DATA) INFO
00000016.00000b41 000014bb Thu Jun 15 2017 07:47:05 +00:00 [kern_notifyd:info:737] Successfully delivered noteto asup with sequence: 3198
00000016.00000b42 000014bb Thu Jun 15 2017 07:47:05 +00:00 [kern_notifyd:info:737] 0x8072d9e00: ERR: Autosupport::Delivery: created compressor with out_filename = /mroot/etc/log/autosupport/201706140000.0.smtp_tmp
00000016.00000b43 000014bb Thu Jun 15 2017 07:47:06 +00:00 [kern_notifyd:info:737] Address Info (0x807b1f2c0) [ai_flags: 1025, ai_family: 2, ai_socktype: 1, ai_protocol: 6, ai_addrlen: 16, ai_addr: 0x807b1f2f0, ai_canonname: (null), ai_next: 0x0]
00000016.00000b46 000014bb Thu Jun 15 2017 07:47:07 +00:00 [kern_notifyd:info:737] Subject: HA Group Notification from PEKNAP-01 (PERFORMANCE DATA) INFO
00000016.00000b47 000014eb Thu Jun 15 2017 07:47:08 +00:00 [kern_notifyd:info:737] Successfully delivered smtp asup with sequence: 3198
00000016.00000b48 000014eb Thu Jun 15 2017 07:47:08 +00:00 [kern_notifyd:info:737] Beginning to deliver asup with sequence number 3199, subject HA Group Notification from PEKNAP-01 (MANAGEMENT_LOG) INFO
00000016.00000b49 000014eb Thu Jun 15 2017 07:47:08 +00:00 [kern_notifyd:info:737] Successfully delivered noteto asup with sequence: 3199
00000016.00000b4a 000014eb Thu Jun 15 2017 07:47:08 +00:00 [kern_notifyd:info:737] 0x8072d9e00: ERR: Autosupport::Delivery: created compressor with out_filename = /mroot/etc/log/autosupport/201706140020.0.smtp_tmp
00000016.00000b4b 000014eb Thu Jun 15 2017 07:47:09 +00:00 [kern_notifyd:info:737] Address Info (0x807b1f2c0) [ai_flags: 1025, ai_family: 2, ai_socktype: 1, ai_protocol: 6, ai_addrlen: 16, ai_addr: 0x807b1f2f0, ai_canonname: (null), ai_next: 0x0]
00000016.00000b50 000014eb Thu Jun 15 2017 07:47:10 +00:00 [kern_notifyd:info:737] Subject: HA Group Notification from PEKNAP-01 (MANAGEMENT_LOG) INFO
00000016.00000b51 000014eb Thu Jun 15 2017 07:47:11 +00:00 [kern_notifyd:info:737] Successfully delivered smtp asup with sequence: 3199
00000016.00000b52 000014eb Thu Jun 15 2017 07:47:11 +00:00 [kern_notifyd:info:737] Beginning to deliver asup with sequence number 3200, subject HA Group Notification from PEKNAP-01 (PERFORMANCE DATA) INFO
00000016.00000b53 000014eb Thu Jun 15 2017 07:47:11 +00:00 [kern_notifyd:info:737] Successfully delivered noteto asup with sequence: 3200
00000016.00000b54 000014eb Thu Jun 15 2017 07:47:11 +00:00 [kern_notifyd:info:737] 0x8072d9e00: ERR: Autosupport::Delivery: created compressor with out_filename = /mroot/etc/log/autosupport/201706150000.0.smtp_tmp
00000016.00000b55 000014eb Thu Jun 15 2017 07:47:12 +00:00 [kern_notifyd:info:737] Address Info (0x807b1fb00) [ai_flags: 1025, ai_family: 2, ai_socktype: 1, ai_protocol: 6, ai_addrlen: 16, ai_addr: 0x807b1fb30, ai_canonname: (null), ai_next: 0x0]
00000016.00000b56 000014f8 Thu Jun 15 2017 07:47:13 +00:00 [kern_notifyd:info:737] Subject: HA Group Notification from PEKNAP-01 (PERFORMANCE DATA) INFO
00000016.00000b57 0000150e Thu Jun 15 2017 07:47:15 +00:00 [kern_notifyd:info:737] Successfully delivered smtp asup with sequence: 3200
00000016.00000b58 0000150e Thu Jun 15 2017 07:47:15 +00:00 [kern_notifyd:info:737] Beginning to deliver asup with sequence number 3201, subject HA Group Notification from PEKNAP-01 (MANAGEMENT_LOG) INFO
00000016.00000b59 0000150e Thu Jun 15 2017 07:47:15 +00:00 [kern_notifyd:info:737] Successfully delivered noteto asup with sequence: 3201
00000016.00000b5a 0000150e Thu Jun 15 2017 07:47:15 +00:00 [kern_notifyd:info:737] 0x8072d9e00: ERR: Autosupport::Delivery: created compressor with out_filename = /mroot/etc/log/autosupport/201706150010.0.smtp_tmp
00000016.00000b5b 0000150e Thu Jun 15 2017 07:47:15 +00:00 [kern_notifyd:info:737] Address Info (0x807b1fb00) [ai_flags: 1025, ai_family: 2, ai_socktype: 1, ai_protocol: 6, ai_addrlen: 16, ai_addr: 0x807b1fb30, ai_canonname: (null), ai_next: 0x0]
00000016.00000b5c 00001521 Thu Jun 15 2017 07:47:16 +00:00 [kern_notifyd:info:737] Subject: HA Group Notification from PEKNAP-01 (MANAGEMENT_LOG) INFO
00000016.00000b5d 00001521 Thu Jun 15 2017 07:47:17 +00:00 [kern_notifyd:info:737] Successfully delivered smtp asup with sequence: 3201
00000016.00000b5e 00001521 Thu Jun 15 2017 07:47:17 +00:00 [kern_notifyd:info:737] Beginning to deliver asup with sequence number 3202, subject HA Group Notification from PEKNAP-01 (REBOOT (power on)) INFO
00000016.00000b5f 00001521 Thu Jun 15 2017 07:47:17 +00:00 [kern_notifyd:info:737] Address Info (0x807b1f0c0) [ai_flags: 1025, ai_family: 2, ai_socktype: 1, ai_protocol: 6, ai_addrlen: 16, ai_addr: 0x807b1f0f0, ai_canonname: (null), ai_next: 0x0]
00000016.00000b60 00001533 Thu Jun 15 2017 07:47:19 +00:00 [kern_notifyd:info:737] Successfully delivered noteto asup with sequence: 3202
00000016.00000b61 00001533 Thu Jun 15 2017 07:47:19 +00:00 [kern_notifyd:info:737] 0x8072d9e00: ERR: Autosupport::Delivery: created compressor with out_filename = /mroot/etc/log/autosupport/201706150729.0.smtp_tmp
00000016.00000b62 0000155e Thu Jun 15 2017 07:47:23 +00:00 [kern_notifyd:info:737] Address Info (0x807b1fb00) [ai_flags: 1025, ai_family: 2, ai_socktype: 1, ai_protocol: 6, ai_addrlen: 16, ai_addr: 0x807b1fb30, ai_canonname: (null), ai_next: 0x0]
00000016.00000b63 0000156b Thu Jun 15 2017 07:47:24 +00:00 [kern_notifyd:info:737] Subject: HA Group Notification from PEKNAP-01 (REBOOT (power on)) INFO
00000016.00000b64 00001588 Thu Jun 15 2017 07:47:25 +00:00 [kern_notifyd:info:737] Successfully delivered smtp asup with sequence: 3202
00000016.00000b65 00001588 Thu Jun 15 2017 07:47:25 +00:00 [kern_notifyd:info:737] 0x8072d9e00: ERR: Autosupport::Delivery: created compressor with out_filename = /mroot/etc/log/autosupport/201706150729.0.http_tmp
00000016.00000b66 00001588 Thu Jun 15 2017 07:47:25 +00:00 [kern_notifyd:info:737] Successfully delivered http asup with sequence: 3202
00000016.00000b67 00001588 Thu Jun 15 2017 07:47:25 +00:00 [kern_notifyd:info:737] Beginning to deliver asup with sequence number 3203, subject HA Group Notification from PEKNAP-01 (PERFORMANCE SNAPSHOT) INFO
00000016.00000b68 00001588 Thu Jun 15 2017 07:47:25 +00:00 [kern_notifyd:info:737] Successfully delivered noteto asup with sequence: 3203
00000016.00000b69 00001588 Thu Jun 15 2017 07:47:25 +00:00 [kern_notifyd:info:737] 0x8072d9e00: ERR: Autosupport::Delivery: created compressor with out_filename = /mroot/etc/log/autosupport/201706150730.0.smtp_tmp
00000016.00000b6a 00001588 Thu Jun 15 2017 07:47:26 +00:00 [kern_notifyd:info:737] Address Info (0x807b1fb00) [ai_flags: 1025, ai_family: 2, ai_socktype: 1, ai_protocol: 6, ai_addrlen: 16, ai_addr: 0x807b1fb30, ai_canonname: (null), ai_next: 0x0]
00000016.00000b6d 00001588 Thu Jun 15 2017 07:47:27 +00:00 [kern_notifyd:info:737] Subject: HA Group Notification from PEKNAP-01 (PERFORMANCE SNAPSHOT) INFO
00000016.00000b6e 00001598 Thu Jun 15 2017 07:47:29 +00:00 [kern_notifyd:info:737] Successfully delivered smtp asup with sequence: 3203
00000016.00000b6f 00001598 Thu Jun 15 2017 07:47:29 +00:00 [kern_notifyd:info:737] 0x8072d9e00: ERR: Autosupport::Delivery: created compressor with out_filename = /mroot/etc/log/autosupport/201706150730.0.http_tmp
00000016.00000b70 00001598 Thu Jun 15 2017 07:47:29 +00:00 [kern_notifyd:info:737] Successfully delivered http asup with sequence: 3203
00000016.00000bf9 00001fe5 Thu Jun 15 2017 07:51:52 +00:00 [kern_notifyd:info:737] start Health Dashboard job for event cf.fsm.autoGivebackStarted (929904) worked t=815
00000016.00000bfb 00001fe5 Thu Jun 15 2017 07:51:52 +00:00 [kern_notifyd:info:737] start Health Dashboard job for event cf.fm.givebackStarted (929905) worked t=815
00000016.00000bfc 00001fe5 Thu Jun 15 2017 07:51:52 +00:00 [kern_notifyd:info:737] start SNMP job for event cf.fm.givebackStarted (929905) worked t=815
00000016.00000c02 0000200b Thu Jun 15 2017 07:51:55 +00:00 [kern_notifyd:info:737] start Health Dashboard job for event cf.fm.givebackComplete (929919) worked t=818
00000016.00000c03 0000200b Thu Jun 15 2017 07:51:55 +00:00 [kern_notifyd:info:737] start SNMP job for event cf.fm.givebackComplete (929919) worked t=818
00000016.00000c05 0000200b Thu Jun 15 2017 07:51:55 +00:00 [kern_notifyd:info:737] start SNMP job for event cf.fm.givebackDuration (929920) worked t=818
00000016.00000c1e 00002032 Thu Jun 15 2017 07:52:00 +00:00 [kern_notifyd:info:737] ems receiver for callhome: callhome.spares.low
00000016.00000c20 00002032 Thu Jun 15 2017 07:52:00 +00:00 [kern_notifyd:info:737] read callhome event: callhome.spares.low
00000016.00000c21 00002032 Thu Jun 15 2017 07:52:00 +00:00 [kern_notifyd:info:737] first level screening for callhome event: callhome.spares.low (name callhome.spares.low, src kernel)
00000016.00000c22 00002032 Thu Jun 15 2017 07:52:00 +00:00 [kern_notifyd:info:737] ems_asup::handle for event callhome.spares.low (929946) t=822
00000016.00000c23 00002032 Thu Jun 15 2017 07:52:00 +00:00 [kern_notifyd:info:737] ems_asup::handle job for event callhome.spares.low (929946) worked t=822
00000016.00000c24 00002032 Thu Jun 15 2017 07:52:00 +00:00 [kern_notifyd:info:737] asup_job::run started t=822
00000016.00000c25 00002032 Thu Jun 15 2017 07:52:00 +00:00 [kern_notifyd:info:737] asup_job::run is triggered by event callhome.spares.low (929946) t=822
00000016.00000c26 00002032 Thu Jun 15 2017 07:52:00 +00:00 [kern_notifyd:info:737] asup_job::run calling smtp_file for 'SPARES_LOW'
00000016.00000c27 00002032 Thu Jun 15 2017 07:52:00 +00:00 [kern_notifyd:info:737] Creating autosupport with sequence number 3209, event callhome.spares.low
00000016.00000c28 00002032 Thu Jun 15 2017 07:52:00 +00:00 [kern_notifyd:info:737] Beginning collection for autosupport with seq num 3209
00000016.00000c29 00002032 Thu Jun 15 2017 07:52:00 +00:00 [kern_notifyd:info:737] (category: 737:3209:collect) (emittime: 6/15/2017 07:51:59) (message: asup d-asup-start directory_path='/etc/log/autosupport/201706150752.0.files'  (complete))
00000016.00000c2a 00002032 Thu Jun 15 2017 07:52:00 +00:00 [kern_notifyd:info:737] (category: 737:3209:collect) (emittime: 6/15/2017 07:51:59) (message: asup ZAPI d-asup-start failed. Status: 1, Reason: AutoSupport Giveback in progress, Error: 13928)
00000016.00000c2b 00002032 Thu Jun 15 2017 07:52:00 +00:00 [kern_notifyd:info:737] asup_job::run smtp_file failed, requeue
00000016.00000c34 000020b7 Thu Jun 15 2017 07:52:14 +00:00 [kern_notifyd:info:737] asup_job::run started t=836
00000016.00000c35 000020b7 Thu Jun 15 2017 07:52:14 +00:00 [kern_notifyd:info:737] asup_job::run is triggered by event callhome.spares.low (929946) t=836
00000016.00000c36 000020b7 Thu Jun 15 2017 07:52:14 +00:00 [kern_notifyd:info:737] asup_job::run calling smtp_file for 'SPARES_LOW'
00000016.00000c37 000020b7 Thu Jun 15 2017 07:52:14 +00:00 [kern_notifyd:info:737] Creating autosupport with sequence number 3210, event callhome.spares.low
00000016.00000c38 000020b7 Thu Jun 15 2017 07:52:14 +00:00 [kern_notifyd:info:737] Beginning collection for autosupport with seq num 3210
00000016.00000c39 000020c1 Thu Jun 15 2017 07:52:15 +00:00 [kern_notifyd:info:737] (category: 737:3210:collect) (emittime: 6/15/2017 07:52:13) (message: asup lstat '/mroot/etc/log/autosupport/201706150752.0.files' exists)
00000016.00000c3a 000020c1 Thu Jun 15 2017 07:52:15 +00:00 [kern_notifyd:info:737] (category: 737:3210:collect) (emittime: 6/15/2017 07:52:14) (message: asup d-asup-start directory_path='/etc/log/autosupport/201706150752.1.files'  (complete))
00000016.00000c3b 000020c1 Thu Jun 15 2017 07:52:15 +00:00 [kern_notifyd:info:737] asup_job::run smtp_file failed, requeue
00000016.00000c3c 000020c1 Thu Jun 15 2017 07:52:15 +00:00 [kern_notifyd:info:737] (category: 737:3210:collect) (emittime: 6/15/2017 07:52:14) (message: asup ZAPI d-asup-start failed. Status: 1, Reason: AutoSupport Giveback in progress, Error: 13928)
00000016.00000c54 0000216b Thu Jun 15 2017 07:52:32 +00:00 [kern_notifyd:info:737] asup_job::run started t=854
00000016.00000c55 0000216b Thu Jun 15 2017 07:52:32 +00:00 [kern_notifyd:info:737] asup_job::run is triggered by event callhome.spares.low (929946) t=854
00000016.00000c56 0000216b Thu Jun 15 2017 07:52:32 +00:00 [kern_notifyd:info:737] asup_job::run calling smtp_file for 'SPARES_LOW'
00000016.00000c57 0000216b Thu Jun 15 2017 07:52:32 +00:00 [kern_notifyd:info:737] Creating autosupport with sequence number 3211, event callhome.spares.low
00000016.00000c58 0000216b Thu Jun 15 2017 07:52:32 +00:00 [kern_notifyd:info:737] Beginning collection for autosupport with seq num 3211
00000016.00000c5f 0000218b Thu Jun 15 2017 07:52:34 +00:00 [kern_notifyd:info:737] (category: 737:3211:collect) (emittime: 6/15/2017 07:52:31) (message: asup lstat '/mroot/etc/log/autosupport/201706150752.0.files' exists)
00000016.00000c60 0000218b Thu Jun 15 2017 07:52:34 +00:00 [kern_notifyd:info:737] (category: 737:3211:collect) (emittime: 6/15/2017 07:52:32) (message: asup lstat '/mroot/etc/log/autosupport/201706150752.1.files' exists)
00000016.00000c61 0000218b Thu Jun 15 2017 07:52:34 +00:00 [kern_notifyd:info:737] (category: 737:3211:collect) (emittime: 6/15/2017 07:52:33) (message: asup d-asup-start directory_path='/etc/log/autosupport/201706150752.2.files'  (complete))
00000016.00000c62 0000218b Thu Jun 15 2017 07:52:34 +00:00 [kern_notifyd:info:737] (category: 737:3211:collect) (emittime: 6/15/2017 07:52:33) (message: asup ZAPI d-asup-start failed. Status: 1, Reason: AutoSupport Giveback in progress, Error: 13928)
00000016.00000c63 0000218b Thu Jun 15 2017 07:52:34 +00:00 [kern_notifyd:info:737] asup_job::run smtp_file failed, requeue
00000016.00000c7a 00002229 Thu Jun 15 2017 07:52:51 +00:00 [kern_notifyd:info:737] Beginning to deliver asup with sequence number 3205, subject HA Group Notification from PEKNAP-01 (REBOOT (power on)) INFO
00000016.00000c7b 00002229 Thu Jun 15 2017 07:52:51 +00:00 [kern_notifyd:info:737] Address Info (0x807b7a700) [ai_flags: 1025, ai_family: 2, ai_socktype: 1, ai_protocol: 6, ai_addrlen: 16, ai_addr: 0x807b7a730, ai_canonname: (null), ai_next: 0x0]
00000016.00000c7c 0000223c Thu Jun 15 2017 07:52:53 +00:00 [kern_notifyd:info:737] Successfully delivered noteto asup with sequence: 3205
00000016.00000c7d 0000223c Thu Jun 15 2017 07:52:53 +00:00 [kern_notifyd:info:737] 0x8072d9e00: ERR: Autosupport::Delivery: created compressor with out_filename = /mroot/etc/log/autosupport/201706150741.0.smtp_tmp
00000016.00000c7e 00002249 Thu Jun 15 2017 07:52:54 +00:00 [kern_notifyd:info:737] Address Info (0x807b7a700) [ai_flags: 1025, ai_family: 2, ai_socktype: 1, ai_protocol: 6, ai_addrlen: 16, ai_addr: 0x807b7a730, ai_canonname: (null), ai_next: 0x0]
00000016.00000c7f 00002256 Thu Jun 15 2017 07:52:55 +00:00 [kern_notifyd:info:737] Subject: HA Group Notification from PEKNAP-01 (REBOOT (power on)) INFO
00000016.00000c80 00002260 Thu Jun 15 2017 07:52:56 +00:00 [kern_notifyd:info:737] Successfully delivered smtp asup with sequence: 3205
00000016.00000c81 00002260 Thu Jun 15 2017 07:52:56 +00:00 [kern_notifyd:info:737] 0x8072d9e00: ERR: Autosupport::Delivery: created compressor with out_filename = /mroot/etc/log/autosupport/201706150741.0.http_tmp
00000016.00000c82 00002260 Thu Jun 15 2017 07:52:56 +00:00 [kern_notifyd:info:737] Successfully delivered http asup with sequence: 3205
00000016.00000c83 00002260 Thu Jun 15 2017 07:52:56 +00:00 [kern_notifyd:info:737] Beginning to deliver asup with sequence number 3206, subject HA Group Notification from PEKNAP-01 (CONTROLLER TAKEOVER COMPLETE AUTOMATIC) ERROR
00000016.00000c84 00002260 Thu Jun 15 2017 07:52:56 +00:00 [kern_notifyd:info:737] Address Info (0x807b7a6c0) [ai_flags: 1025, ai_family: 2, ai_socktype: 1, ai_protocol: 6, ai_addrlen: 16, ai_addr: 0x807b7a6f0, ai_canonname: (null), ai_next: 0x0]
00000016.00000c85 00002287 Thu Jun 15 2017 07:52:58 +00:00 [kern_notifyd:info:737] Successfully delivered noteto asup with sequence: 3206
00000016.00000c86 00002287 Thu Jun 15 2017 07:52:58 +00:00 [kern_notifyd:info:737] 0x8072d9e00: ERR: Autosupport::Delivery: created compressor with out_filename = /mroot/etc/log/autosupport/201706150742.0.smtp_tmp
00000016.00000cb5 00002287 Thu Jun 15 2017 07:53:00 +00:00 [kern_notifyd:info:737] start SNMP job for event monitor.globalStatus.nonCritical (929968) worked t=882
00000016.00000cb6 00002287 Thu Jun 15 2017 07:53:00 +00:00 [kern_notifyd:info:737] Address Info (0x807b7a700) [ai_flags: 1025, ai_family: 2, ai_socktype: 1, ai_protocol: 6, ai_addrlen: 16, ai_addr: 0x807b7a730, ai_canonname: (null), ai_next: 0x0]
00000016.00000cb9 00002297 Thu Jun 15 2017 07:53:02 +00:00 [kern_notifyd:info:737] Subject: HA Group Notification from PEKNAP-01 (CONTROLLER TAKEOVER COMPLETE AUTOMATIC) ERROR
00000016.00000cba 00002297 Thu Jun 15 2017 07:53:02 +00:00 [kern_notifyd:info:737] asup_job::run started t=884
00000016.00000cbb 00002297 Thu Jun 15 2017 07:53:02 +00:00 [kern_notifyd:info:737] asup_job::run is triggered by event callhome.spares.low (929946) t=884
00000016.00000cbc 00002297 Thu Jun 15 2017 07:53:02 +00:00 [kern_notifyd:info:737] asup_job::run calling smtp_file for 'SPARES_LOW'
00000016.00000cbd 00002297 Thu Jun 15 2017 07:53:02 +00:00 [kern_notifyd:info:737] Creating autosupport with sequence number 3212, event callhome.spares.low
00000016.00000cbe 00002297 Thu Jun 15 2017 07:53:02 +00:00 [kern_notifyd:info:737] Beginning collection for autosupport with seq num 3212
00000016.00000cbf 00002298 Thu Jun 15 2017 07:53:02 +00:00 [kern_notifyd:info:737] (category: 737:3212:collect) (emittime: 6/15/2017 07:53:01) (message: asup d-asup-start directory_path='/etc/log/autosupport/201706150753.0.files'  (complete))
00000016.00000cc0 00002298 Thu Jun 15 2017 07:53:02 +00:00 [kern_notifyd:info:737] asup_job::run smtp_file failed, requeue
00000016.00000cc1 00002298 Thu Jun 15 2017 07:53:02 +00:00 [kern_notifyd:info:737] (category: 737:3212:collect) (emittime: 6/15/2017 07:53:01) (message: asup ZAPI d-asup-start failed. Status: 1, Reason: AutoSupport Giveback in progress, Error: 13928)
00000016.00000cc6 000022cd Thu Jun 15 2017 07:53:05 +00:00 [kern_notifyd:info:737] start Health Dashboard job for event raid.vol.state.changed (929986) worked t=887
00000016.00000cc7 000022cd Thu Jun 15 2017 07:53:05 +00:00 [kern_notifyd:info:737] start SNMP job for event raid.vol.state.changed (929986) worked t=887
00000016.00000d14 000022d9 Thu Jun 15 2017 07:53:07 +00:00 [kern_notifyd:info:737] ems receiver for callhome: callhome.sfo.giveback
00000016.00000d15 000022d9 Thu Jun 15 2017 07:53:07 +00:00 [kern_notifyd:info:737] read callhome event: callhome.sfo.giveback
00000016.00000d17 000022d9 Thu Jun 15 2017 07:53:07 +00:00 [kern_notifyd:info:737] first level screening for callhome event: callhome.sfo.giveback (name callhome.sfo.giveback, src kernel)
00000016.00000d18 000022d9 Thu Jun 15 2017 07:53:07 +00:00 [kern_notifyd:info:737] ems_asup::handle for event callhome.sfo.giveback (930058) t=889
00000016.00000d19 000022d9 Thu Jun 15 2017 07:53:07 +00:00 [kern_notifyd:info:737] ems_asup::handle job for event callhome.sfo.giveback (930058) worked t=889
00000016.00000d1a 000022ee Thu Jun 15 2017 07:53:10 +00:00 [kern_notifyd:info:737] Successfully delivered smtp asup with sequence: 3206
00000016.00000d1b 000022ee Thu Jun 15 2017 07:53:10 +00:00 [kern_notifyd:info:737] 0x8072d9e00: ERR: Autosupport::Delivery: created compressor with out_filename = /mroot/etc/log/autosupport/201706150742.0.http_tmp
00000016.00000d1c 000022ee Thu Jun 15 2017 07:53:10 +00:00 [kern_notifyd:info:737] Successfully delivered http asup with sequence: 3206
00000016.00000d1d 000022ee Thu Jun 15 2017 07:53:10 +00:00 [kern_notifyd:info:737] Beginning to deliver asup with sequence number 3207, subject HA Group Notification from PEKNAP-01 (PARTNER REBOOT (CONTROLLER TAKEOVER)) WARNING
00000016.00000d1e 000022ee Thu Jun 15 2017 07:53:10 +00:00 [kern_notifyd:info:737] Successfully delivered noteto asup with sequence: 3207
00000016.00000d1f 000022ee Thu Jun 15 2017 07:53:10 +00:00 [kern_notifyd:info:737] 0x8072d9e00: ERR: Autosupport::Delivery: created compressor with out_filename = /mroot/etc/log/autosupport/201706150744.0.smtp_tmp
00000016.00000d20 00002316 Thu Jun 15 2017 07:53:14 +00:00 [kern_notifyd:info:737] Address Info (0x807b194c0) [ai_flags: 1025, ai_family: 2, ai_socktype: 1, ai_protocol: 6, ai_addrlen: 16, ai_addr: 0x807b194f0, ai_canonname: (null), ai_next: 0x0]
00000016.00000d21 00002323 Thu Jun 15 2017 07:53:16 +00:00 [kern_notifyd:info:737] Subject: HA Group Notification from PEKNAP-01 (PARTNER REBOOT (CONTROLLER TAKEOVER)) WARNING
00000016.00000d22 0000239c Thu Jun 15 2017 07:53:28 +00:00 [kern_notifyd:info:737] asup_job::run started t=910
00000016.00000d23 0000239c Thu Jun 15 2017 07:53:28 +00:00 [kern_notifyd:info:737] asup_job::run is triggered by event callhome.spares.low (929946) t=910
00000016.00000d24 0000239c Thu Jun 15 2017 07:53:28 +00:00 [kern_notifyd:info:737] asup_job::run calling smtp_file for 'SPARES_LOW'
00000016.00000d25 0000239c Thu Jun 15 2017 07:53:28 +00:00 [kern_notifyd:info:737] Creating autosupport with sequence number 3213, event callhome.spares.low
00000016.00000d26 0000239c Thu Jun 15 2017 07:53:28 +00:00 [kern_notifyd:info:737] Beginning collection for autosupport with seq num 3213
00000016.00000d27 000023a6 Thu Jun 15 2017 07:53:29 +00:00 [kern_notifyd:info:737] (category: 737:3213:collect) (emittime: 6/15/2017 07:53:27) (message: asup lstat '/mroot/etc/log/autosupport/201706150753.0.files' exists)
00000016.00000d28 000023a6 Thu Jun 15 2017 07:53:29 +00:00 [kern_notifyd:info:737] (category: 737:3213:collect) (emittime: 6/15/2017 07:53:28) (message: asup d-asup-start directory_path='/etc/log/autosupport/201706150753.1.files'  (complete))
00000016.00000d29 000023a6 Thu Jun 15 2017 07:53:29 +00:00 [kern_notifyd:info:737] asup_job::run smtp_file failed, requeue
00000016.00000d2a 000023a6 Thu Jun 15 2017 07:53:29 +00:00 [kern_notifyd:info:737] (category: 737:3213:collect) (emittime: 6/15/2017 07:53:28) (message: asup ZAPI d-asup-start failed. Status: 1, Reason: AutoSupport Giveback in progress, Error: 13928)
00000016.00000d2b 00002496 Thu Jun 15 2017 07:53:53 +00:00 [kern_notifyd:info:737] asup_job::run started t=935
00000016.00000d2c 00002496 Thu Jun 15 2017 07:53:53 +00:00 [kern_notifyd:info:737] asup_job::run is triggered by event callhome.spares.low (929946) t=935
00000016.00000d2d 00002496 Thu Jun 15 2017 07:53:53 +00:00 [kern_notifyd:info:737] asup_job::run calling smtp_file for 'SPARES_LOW'
00000016.00000d2e 00002496 Thu Jun 15 2017 07:53:53 +00:00 [kern_notifyd:info:737] Creating autosupport with sequence number 3214, event callhome.spares.low
00000016.00000d2f 00002496 Thu Jun 15 2017 07:53:53 +00:00 [kern_notifyd:info:737] Beginning collection for autosupport with seq num 3214
00000016.00000d32 000024b2 Thu Jun 15 2017 07:53:55 +00:00 [kern_notifyd:info:737] 0x8072d9f90: ERR: Autosupport::Collection: NotifyD collector: [AutoSupport_Custom_Content]    Target: custom_fx Subsys: log_files Type: basic Minimal: 0 Mode Flags: 15 Outfile: MESSAGES.gz Cmd: Custom function Query: messages   --  manifest_status = content-empty , error = This file has not changed since previously sent AutoSupport seq( 3206 )
00000016.00000d3b 000024b2 Thu Jun 15 2017 07:53:55 +00:00 [kern_notifyd:info:737] get name 'autosupport.content' privs 'anon,admin,advanced,diag,test'
00000016.00000d3c 000024b2 Thu Jun 15 2017 07:53:55 +00:00 [kern_notifyd:info:737] get name 'autosupport.doit' privs 'anon,admin,advanced,diag,test'
00000016.00000d3d 000024b2 Thu Jun 15 2017 07:53:55 +00:00 [kern_notifyd:info:737] do it get
00000016.00000d3e 000024b2 Thu Jun 15 2017 07:53:55 +00:00 [kern_notifyd:info:737] get name 'autosupport.enable' privs 'anon,admin,advanced,diag,test'
00000016.00000d3f 000024b2 Thu Jun 15 2017 07:53:55 +00:00 [kern_notifyd:info:737] get name 'autosupport.from' privs 'anon,admin,advanced,diag,test'
00000016.00000d40 000024b2 Thu Jun 15 2017 07:53:55 +00:00 [kern_notifyd:info:737] get name 'autosupport.local_collection' privs 'anon,admin,advanced,diag,test'
00000016.00000d41 000024b2 Thu Jun 15 2017 07:53:55 +00:00 [kern_notifyd:info:737] get name 'autosupport.mailhost' privs 'anon,admin,advanced,diag,test'
00000016.00000d42 000024b2 Thu Jun 15 2017 07:53:55 +00:00 [kern_notifyd:info:737] get name 'autosupport.max_http_size' privs 'anon,admin,advanced,diag,test'
00000016.00000d43 000024b2 Thu Jun 15 2017 07:53:55 +00:00 [kern_notifyd:info:737] get name 'autosupport.max_smtp_size' privs 'anon,admin,advanced,diag,test'
00000016.00000d44 000024b2 Thu Jun 15 2017 07:53:55 +00:00 [kern_notifyd:info:737] get name 'autosupport.minimal.subject.id' privs 'anon,admin,advanced,diag,test'
00000016.00000d45 000024b2 Thu Jun 15 2017 07:53:55 +00:00 [kern_notifyd:info:737] get name 'autosupport.nht_data.enable' privs 'anon,admin,advanced,diag,test'
00000016.00000d46 000024b2 Thu Jun 15 2017 07:53:55 +00:00 [kern_notifyd:info:737] get name 'autosupport.noteto' privs 'anon,admin,advanced,diag,test'
00000016.00000d47 000024b2 Thu Jun 15 2017 07:53:55 +00:00 [kern_notifyd:info:737] get name 'autosupport.partner.to' privs 'anon,admin,advanced,diag,test'
00000016.00000d48 000024b2 Thu Jun 15 2017 07:53:55 +00:00 [kern_notifyd:info:737] get name 'autosupport.payload_format' privs 'anon,admin,advanced,diag,test'
00000016.00000d49 000024b2 Thu Jun 15 2017 07:53:55 +00:00 [kern_notifyd:info:737] get name 'autosupport.performance_data.doit' privs 'anon,admin,advanced,diag,test'
00000016.00000d4a 000024b2 Thu Jun 15 2017 07:53:56 +00:00 [kern_notifyd:info:737] get name 'autosupport.performance_data.enable' privs 'anon,admin,advanced,diag,test'
00000016.00000d4b 000024b2 Thu Jun 15 2017 07:53:56 +00:00 [kern_notifyd:info:737] get name 'autosupport.periodic.tx_window' privs 'anon,admin,advanced,diag,test'
00000016.00000d4c 000024b2 Thu Jun 15 2017 07:53:56 +00:00 [kern_notifyd:info:737] get name 'autosupport.retry.count' privs 'anon,admin,advanced,diag,test'
00000016.00000d4d 000024b2 Thu Jun 15 2017 07:53:56 +00:00 [kern_notifyd:info:737] get name 'autosupport.retry.interval' privs 'anon,admin,advanced,diag,test'
00000016.00000d4e 000024b2 Thu Jun 15 2017 07:53:56 +00:00 [kern_notifyd:info:737] get name 'autosupport.support.enable' privs 'anon,admin,advanced,diag,test'
00000016.00000d4f 000024b2 Thu Jun 15 2017 07:53:56 +00:00 [kern_notifyd:info:737] get name 'autosupport.support.proxy' privs 'anon,admin,advanced,diag,test'
00000016.00000d50 000024b2 Thu Jun 15 2017 07:53:56 +00:00 [kern_notifyd:info:737] get name 'autosupport.support.put_url' privs 'anon,admin,advanced,diag,test'
00000016.00000d51 000024db Thu Jun 15 2017 07:53:59 +00:00 [kern_notifyd:info:737] get name 'autosupport.support.to' privs 'anon,admin,advanced,diag,test'
00000016.00000d52 000024db Thu Jun 15 2017 07:53:59 +00:00 [kern_notifyd:info:737] get name 'autosupport.support.transport' privs 'anon,admin,advanced,diag,test'
00000016.00000d53 000024db Thu Jun 15 2017 07:53:59 +00:00 [kern_notifyd:info:737] get name 'autosupport.support.url' privs 'anon,admin,advanced,diag,test'
00000016.00000d54 000024db Thu Jun 15 2017 07:53:59 +00:00 [kern_notifyd:info:737] get name 'autosupport.throttle' privs 'anon,admin,advanced,diag,test'
00000016.00000d55 000024db Thu Jun 15 2017 07:53:59 +00:00 [kern_notifyd:info:737] get name 'autosupport.to' privs 'anon,admin,advanced,diag,test'
00000016.00000d56 000024db Thu Jun 15 2017 07:53:59 +00:00 [kern_notifyd:info:737] get name 'autosupport.validate_digital_certificate' privs 'anon,admin,advanced,diag,test'
00000016.00000d57 000024db Thu Jun 15 2017 07:53:59 +00:00 [kern_notifyd:info:737] 0x8072d9f90: ERR: Autosupport::Collection: NotifyD collector: [AutoSupport_Custom_Content]    Target: custom_fx Subsys: log_files Type: troubleshooting Minimal: 0 Mode Flags: 15 Outfile: backup.gz Cmd: Custom function Query: backup   --  manifest_status = content-empty , error = This file has not changed since previously sent AutoSupport seq( 3205 )
00000016.00000d58 000024db Thu Jun 15 2017 07:53:59 +00:00 [kern_notifyd:info:737] 0x8072d9f90: ERR: Autosupport::Collection: NotifyD collector: [AutoSupport_Custom_Content]    Target: custom_fx Subsys: log_files Type: troubleshooting Minimal: 0 Mode Flags: 15 Outfile: sis-log.gz Cmd: Custom function Query: sis   --  manifest_status = content-empty , error = This file has not changed since previously sent AutoSupport seq( 3205 )
00000016.00000d59 000024db Thu Jun 15 2017 07:53:59 +00:00 [kern_notifyd:info:737] 0x8072d9f90: ERR: Autosupport::Collection: NotifyD collector: [AutoSupport_Custom_Content]    Target: custom_fx Subsys: log_files Type: troubleshooting Minimal: 0 Mode Flags: 15 Outfile: acplog.gz Cmd: Custom function Query: acplog   --  manifest_status = content-empty , error = This file has not changed since previously sent AutoSupport seq( 2708 )
00000016.00000d5a 000024db Thu Jun 15 2017 07:53:59 +00:00 [kern_notifyd:info:737] 0x8072d9f90: ERR: Autosupport::Collection: NotifyD collector: [AutoSupport_Custom_Content]    Target: custom_fx Subsys: log_files Type: troubleshooting Minimal: 0 Mode Flags: 15 Outfile: shelf-log-sas.gz Cmd: Custom function Query: shelflog_sas   --  manifest_status = content-empty , error = This file has not changed since previously sent AutoSupport seq( 0 )
00000016.00000d5b 000024db Thu Jun 15 2017 07:53:59 +00:00 [kern_notifyd:info:737] 0x8072d9f90: ERR: Autosupport::Collection: NotifyD collector: [AutoSupport_Custom_Content]    Target: custom_fx Subsys: log_files Type: troubleshooting Minimal: 0 Mode Flags: 15 Outfile: shelf-log-esh.gz Cmd: Custom function Query: shelflog_esh   --  manifest_status = content-empty , error = This file has not changed since previously sent AutoSupport seq( 0 )
00000016.00000d5c 000024db Thu Jun 15 2017 07:53:59 +00:00 [kern_notifyd:info:737] 0x8072d9f90: ERR: Autosupport::Collection: NotifyD collector: [AutoSupport_Custom_Content]    Target: custom_fx Subsys: log_files Type: troubleshooting Minimal: 0 Mode Flags: 15 Outfile: shelf-log-ata.gz Cmd: Custom function Query: shelflog_ata   --  manifest_status = content-empty , error = This file has not changed since previously sent AutoSupport seq( 0 )
00000016.00000d5d 000024db Thu Jun 15 2017 07:53:59 +00:00 [kern_notifyd:info:737] 0x8072d9f90: ERR: Autosupport::Collection: NotifyD collector: [AutoSupport_Custom_Content]    Target: custom_fx Subsys: log_files Type: troubleshooting Minimal: 0 Mode Flags: 15 Outfile: ndmpd.gz Cmd: Custom function Query: ndmpd   --  manifest_status = content-empty , error = This file has not changed since previously sent AutoSupport seq( 3205 )
00000016.00000d5e 000024db Thu Jun 15 2017 07:53:59 +00:00 [kern_notifyd:info:737] 0x8072d9f90: ERR: Autosupport::Collection: NotifyD collector: [AutoSupport_Custom_Content]    Target: custom_fx Subsys: log_files Type: troubleshooting Minimal: 0 Mode Flags: 15 Outfile: secd.gz Cmd: Custom function Query: secd   --  manifest_status = content-empty , error = This file has not changed since previously sent AutoSupport seq( 3206 )
00000016.00000d5f 000024db Thu Jun 15 2017 07:53:59 +00:00 [kern_notifyd:info:737] 0x8072d9f90: ERR: Autosupport::Collection: NotifyD collector: [AutoSupport_Custom_Content]    Target: custom_fx Subsys: log_files Type: troubleshooting Minimal: 0 Mode Flags: 15 Outfile: spmd.gz Cmd: Custom function Query: spmd   --  manifest_status = content-empty , error = This file has not changed since previously sent AutoSupport seq( 3205 )
00000016.00000d60 000024db Thu Jun 15 2017 07:53:59 +00:00 [kern_notifyd:info:737] 0x8072d9f90: ERR: Autosupport::Collection: NotifyD collector: [AutoSupport_Custom_Content]    Target: custom_fx Subsys: log_files Type: troubleshooting Minimal: 0 Mode Flags: 2 Outfile: apache_access.gz Cmd: Custom function Query: apache_access   --  manifest_status = content-empty , error = This file has not changed since previously sent AutoSupport seq( 3202 )
00000016.00000d61 000024db Thu Jun 15 2017 07:53:59 +00:00 [kern_notifyd:info:737] 0x8072d9f90: ERR: Autosupport::Collection: NotifyD collector: [AutoSupport_Custom_Content]    Target: custom_fx Subsys: log_files Type: troubleshooting Minimal: 0 Mode Flags: 2 Outfile: apache_error.gz Cmd: Custom function Query: apache_error   --  manifest_status = content-empty , error = This file has not changed since previously sent AutoSupport seq( 3205 )
00000016.00000d62 000024db Thu Jun 15 2017 07:53:59 +00:00 [kern_notifyd:info:737] 0x8072d9f90: ERR: Autosupport::Collection: NotifyD collector: [AutoSupport_File_Content]    Target: file Subsys: platform Type: troubleshooting Minimal: 0 Mode Flags: 15 Outfile: ssram.txt Cmd: /mroot/etc/log/ssram/ssram.log Query: 0   --  manifest_status = file-not-found , error = Cannot find AutoSupport source file /mroot/etc/log/ssram/ssram.log to collect for AutoSupport
00000016.00000d63 000024db Thu Jun 15 2017 07:53:59 +00:00 [kern_notifyd:info:737] 0x8072d9f90: ERR: Autosupport::Collection: NotifyD collector: [AutoSupport_File_Content]    Target: file Subsys: platform Type: troubleshooting Minimal: 0 Mode Flags: 15 Outfile: splog-subset-wdr.txt Cmd: /mroot/etc/log/splog_subset_wdr.txt Query: 0   --  manifest_status = file-not-found , error = Cannot find AutoSupport source file /mroot/etc/log/splog_subset_wdr.txt to collect for AutoSupport
00000016.00000d65 000024db Thu Jun 15 2017 07:54:00 +00:00 [kern_notifyd:info:737] start SNMP job for event monitor.globalStatus.ok (930059) worked t=942
00000016.00000d69 000025aa Thu Jun 15 2017 07:54:18 +00:00 [kern_notifyd:info:737] Successfully completed collection for autosupport with sequence number 3214
00000016.00000d6a 000025aa Thu Jun 15 2017 07:54:18 +00:00 [kern_notifyd:info:737] asup_job::run got '' for ASUP file
00000016.00000d6b 000025aa Thu Jun 15 2017 07:54:18 +00:00 [kern_notifyd:info:737] asup_job::run started t=961
00000016.00000d6c 000025aa Thu Jun 15 2017 07:54:18 +00:00 [kern_notifyd:info:737] asup_job::run is triggered by event callhome.sfo.giveback (930058) t=961
00000016.00000d6d 000025aa Thu Jun 15 2017 07:54:18 +00:00 [kern_notifyd:info:737] asup_job::run calling smtp_file for 'CONTROLLER GIVEBACK COMPLETE'
00000016.00000d6e 000025aa Thu Jun 15 2017 07:54:18 +00:00 [kern_notifyd:info:737] Creating autosupport with sequence number 3215, event callhome.sfo.giveback
00000016.00000d6f 000025aa Thu Jun 15 2017 07:54:18 +00:00 [kern_notifyd:info:737] Beginning collection for autosupport with seq num 3215
00000016.00000d72 000025aa Thu Jun 15 2017 07:54:18 +00:00 [kern_notifyd:info:737] 0x8072d9f90: ERR: Autosupport::Collection: NotifyD collector: [AutoSupport_Custom_Content]    Target: custom_fx Subsys: log_files Type: basic Minimal: 0 Mode Flags: 15 Outfile: MESSAGES.gz Cmd: Custom function Query: messages   --  manifest_status = content-empty , error = This file has not changed since previously sent AutoSupport seq( 3206 )
00000016.00000d7b 000025aa Thu Jun 15 2017 07:54:19 +00:00 [kern_notifyd:info:737] get name 'autosupport.content' privs 'anon,admin,advanced,diag,test'
00000016.00000d7c 000025aa Thu Jun 15 2017 07:54:19 +00:00 [kern_notifyd:info:737] get name 'autosupport.doit' privs 'anon,admin,advanced,diag,test'
00000016.00000d7d 000025aa Thu Jun 15 2017 07:54:19 +00:00 [kern_notifyd:info:737] do it get
00000016.00000d7e 000025aa Thu Jun 15 2017 07:54:19 +00:00 [kern_notifyd:info:737] get name 'autosupport.enable' privs 'anon,admin,advanced,diag,test'
00000016.00000d7f 000025aa Thu Jun 15 2017 07:54:19 +00:00 [kern_notifyd:info:737] get name 'autosupport.from' privs 'anon,admin,advanced,diag,test'
00000016.00000d80 000025aa Thu Jun 15 2017 07:54:19 +00:00 [kern_notifyd:info:737] get name 'autosupport.local_collection' privs 'anon,admin,advanced,diag,test'
00000016.00000d81 000025aa Thu Jun 15 2017 07:54:19 +00:00 [kern_notifyd:info:737] get name 'autosupport.mailhost' privs 'anon,admin,advanced,diag,test'
00000016.00000d82 000025aa Thu Jun 15 2017 07:54:19 +00:00 [kern_notifyd:info:737] get name 'autosupport.max_http_size' privs 'anon,admin,advanced,diag,test'
00000016.00000d83 000025aa Thu Jun 15 2017 07:54:19 +00:00 [kern_notifyd:info:737] get name 'autosupport.max_smtp_size' privs 'anon,admin,advanced,diag,test'
00000016.00000d84 000025aa Thu Jun 15 2017 07:54:19 +00:00 [kern_notifyd:info:737] get name 'autosupport.minimal.subject.id' privs 'anon,admin,advanced,diag,test'
00000016.00000d85 000025aa Thu Jun 15 2017 07:54:19 +00:00 [kern_notifyd:info:737] get name 'autosupport.nht_data.enable' privs 'anon,admin,advanced,diag,test'
00000016.00000d86 000025aa Thu Jun 15 2017 07:54:19 +00:00 [kern_notifyd:info:737] get name 'autosupport.noteto' privs 'anon,admin,advanced,diag,test'
00000016.00000d87 000025aa Thu Jun 15 2017 07:54:19 +00:00 [kern_notifyd:info:737] get name 'autosupport.partner.to' privs 'anon,admin,advanced,diag,test'
00000016.00000d88 000025aa Thu Jun 15 2017 07:54:19 +00:00 [kern_notifyd:info:737] get name 'autosupport.payload_format' privs 'anon,admin,advanced,diag,test'
00000016.00000d89 000025aa Thu Jun 15 2017 07:54:19 +00:00 [kern_notifyd:info:737] get name 'autosupport.performance_data.doit' privs 'anon,admin,advanced,diag,test'
00000016.00000d8a 000025aa Thu Jun 15 2017 07:54:19 +00:00 [kern_notifyd:info:737] get name 'autosupport.performance_data.enable' privs 'anon,admin,advanced,diag,test'
00000016.00000d8b 000025aa Thu Jun 15 2017 07:54:19 +00:00 [kern_notifyd:info:737] get name 'autosupport.periodic.tx_window' privs 'anon,admin,advanced,diag,test'
00000016.00000d8c 000025aa Thu Jun 15 2017 07:54:19 +00:00 [kern_notifyd:info:737] get name 'autosupport.retry.count' privs 'anon,admin,advanced,diag,test'
00000016.00000d8d 000025aa Thu Jun 15 2017 07:54:19 +00:00 [kern_notifyd:info:737] get name 'autosupport.retry.interval' privs 'anon,admin,advanced,diag,test'
00000016.00000d8e 000025aa Thu Jun 15 2017 07:54:19 +00:00 [kern_notifyd:info:737] get name 'autosupport.support.enable' privs 'anon,admin,advanced,diag,test'
00000016.00000d8f 000025aa Thu Jun 15 2017 07:54:19 +00:00 [kern_notifyd:info:737] get name 'autosupport.support.proxy' privs 'anon,admin,advanced,diag,test'
00000016.00000d90 000025aa Thu Jun 15 2017 07:54:19 +00:00 [kern_notifyd:info:737] get name 'autosupport.support.put_url' privs 'anon,admin,advanced,diag,test'
00000016.00000d91 000025aa Thu Jun 15 2017 07:54:19 +00:00 [kern_notifyd:info:737] get name 'autosupport.support.to' privs 'anon,admin,advanced,diag,test'
00000016.00000d92 000025aa Thu Jun 15 2017 07:54:19 +00:00 [kern_notifyd:info:737] get name 'autosupport.support.transport' privs 'anon,admin,advanced,diag,test'
00000016.00000d93 000025aa Thu Jun 15 2017 07:54:19 +00:00 [kern_notifyd:info:737] get name 'autosupport.support.url' privs 'anon,admin,advanced,diag,test'
00000016.00000d94 000025aa Thu Jun 15 2017 07:54:19 +00:00 [kern_notifyd:info:737] get name 'autosupport.throttle' privs 'anon,admin,advanced,diag,test'
00000016.00000d95 000025aa Thu Jun 15 2017 07:54:19 +00:00 [kern_notifyd:info:737] get name 'autosupport.to' privs 'anon,admin,advanced,diag,test'
00000016.00000d96 000025aa Thu Jun 15 2017 07:54:19 +00:00 [kern_notifyd:info:737] get name 'autosupport.validate_digital_certificate' privs 'anon,admin,advanced,diag,test'
00000016.00000da2 000025c8 Thu Jun 15 2017 07:54:23 +00:00 [kern_notifyd:info:737] 0x8072d9f90: ERR: Autosupport::Collection: asup[737:0] process_run for /usr/sbin/nperf: process exit status 255
00000016.00000dae 0000260a Thu Jun 15 2017 07:54:29 +00:00 [kern_notifyd:info:737] 0x8072d9f90: ERR: Autosupport::Collection: NotifyD collector: [AutoSupport_File_Content]    Target: file Subsys: wafl Type: basic Minimal: 0 Mode Flags: 15 Outfile: memstats.txt Cmd: /membufstats.log Query: 0   --  manifest_status = file-not-found , error = Cannot find AutoSupport source file /membufstats.log to collect for AutoSupport
00000016.00000daf 0000260a Thu Jun 15 2017 07:54:29 +00:00 [kern_notifyd:info:737] 0x8072d9f90: ERR: Autosupport::Collection: NotifyD collector: [AutoSupport_File_Content]    Target: file Subsys: platform Type: troubleshooting Minimal: 0 Mode Flags: 15 Outfile: ssram.txt Cmd: /mroot/etc/log/ssram/ssram.log Query: 0   --  manifest_status = file-not-found , error = Cannot find AutoSupport source file /mroot/etc/log/ssram/ssram.log to collect for AutoSupport
00000016.00000db0 0000260a Thu Jun 15 2017 07:54:29 +00:00 [kern_notifyd:info:737] 0x8072d9f90: ERR: Autosupport::Collection: NotifyD collector: [AutoSupport_File_Content]    Target: file Subsys: platform Type: troubleshooting Minimal: 0 Mode Flags: 15 Outfile: splog-subset-wdr.txt Cmd: /mroot/etc/log/splog_subset_wdr.txt Query: 0   --  manifest_status = file-not-found , error = Cannot find AutoSupport source file /mroot/etc/log/splog_subset_wdr.txt to collect for AutoSupport
00000016.00000dd3 0000265b Thu Jun 15 2017 07:54:37 +00:00 [kern_notifyd:info:737] Unable to modify manifest row due to subsystem size limit: asup_seqnum = 3215 , manifest_item = 219 , subsystem = san , size_collected = 15675721 , size_remaining = 2019628
00000016.00000dd4 0000265b Thu Jun 15 2017 07:54:37 +00:00 [kern_notifyd:info:737] 0x8072d9f90: ERR: Autosupport::Collection: NotifyD collector: [AutoSupport_Custom_Content]    Target: custom_fx Subsys: log_files Type: troubleshooting Minimal: 0 Mode Flags: 15 Outfile: backup.gz Cmd: Custom function Query: backup   --  manifest_status = content-empty , error = This file has not changed since previously sent AutoSupport seq( 3205 )
00000016.00000dd5 0000265b Thu Jun 15 2017 07:54:37 +00:00 [kern_notifyd:info:737] 0x8072d9f90: ERR: Autosupport::Collection: NotifyD collector: [AutoSupport_Custom_Content]    Target: custom_fx Subsys: log_files Type: troubleshooting Minimal: 0 Mode Flags: 15 Outfile: sis-log.gz Cmd: Custom function Query: sis   --  manifest_status = content-empty , error = This file has not changed since previously sent AutoSupport seq( 3205 )
00000016.00000dd6 0000265b Thu Jun 15 2017 07:54:37 +00:00 [kern_notifyd:info:737] 0x8072d9f90: ERR: Autosupport::Collection: NotifyD collector: [AutoSupport_Custom_Content]    Target: custom_fx Subsys: log_files Type: troubleshooting Minimal: 0 Mode Flags: 15 Outfile: acplog.gz Cmd: Custom function Query: acplog   --  manifest_status = content-empty , error = This file has not changed since previously sent AutoSupport seq( 2708 )
00000016.00000dd7 0000265b Thu Jun 15 2017 07:54:37 +00:00 [kern_notifyd:info:737] 0x8072d9f90: ERR: Autosupport::Collection: NotifyD collector: [AutoSupport_Custom_Content]    Target: custom_fx Subsys: log_files Type: troubleshooting Minimal: 0 Mode Flags: 15 Outfile: shelf-log-sas.gz Cmd: Custom function Query: shelflog_sas   --  manifest_status = content-empty , error = This file has not changed since previously sent AutoSupport seq( 0 )
00000016.00000dd8 0000265b Thu Jun 15 2017 07:54:37 +00:00 [kern_notifyd:info:737] 0x8072d9f90: ERR: Autosupport::Collection: NotifyD collector: [AutoSupport_Custom_Content]    Target: custom_fx Subsys: log_files Type: troubleshooting Minimal: 0 Mode Flags: 15 Outfile: shelf-log-esh.gz Cmd: Custom function Query: shelflog_esh   --  manifest_status = content-empty , error = This file has not changed since previously sent AutoSupport seq( 0 )
00000016.00000dd9 0000265b Thu Jun 15 2017 07:54:37 +00:00 [kern_notifyd:info:737] 0x8072d9f90: ERR: Autosupport::Collection: NotifyD collector: [AutoSupport_Custom_Content]    Target: custom_fx Subsys: log_files Type: troubleshooting Minimal: 0 Mode Flags: 15 Outfile: shelf-log-ata.gz Cmd: Custom function Query: shelflog_ata   --  manifest_status = content-empty , error = This file has not changed since previously sent AutoSupport seq( 0 )
00000016.00000dda 0000265b Thu Jun 15 2017 07:54:37 +00:00 [kern_notifyd:info:737] 0x8072d9f90: ERR: Autosupport::Collection: NotifyD collector: [AutoSupport_Custom_Content]    Target: custom_fx Subsys: log_files Type: troubleshooting Minimal: 0 Mode Flags: 15 Outfile: shelf-log-iom.gz Cmd: Custom function Query: shelflog_iom   --  manifest_status = content-empty , error = This file has not changed since previously sent AutoSupport seq( 3214 )
00000016.00000ddb 0000265b Thu Jun 15 2017 07:54:37 +00:00 [kern_notifyd:info:737] 0x8072d9f90: ERR: Autosupport::Collection: NotifyD collector: [AutoSupport_Custom_Content]    Target: custom_fx Subsys: log_files Type: troubleshooting Minimal: 0 Mode Flags: 2 Outfile: sktlogd.gz Cmd: Custom function Query: sktlogd   --  manifest_status = content-empty , error = This file has not changed since previously sent AutoSupport seq( 3214 )
00000016.00000ddc 0000265b Thu Jun 15 2017 07:54:37 +00:00 [kern_notifyd:info:737] 0x8072d9f90: ERR: Autosupport::Collection: NotifyD collector: [AutoSupport_Custom_Content]    Target: custom_fx Subsys: log_files Type: troubleshooting Minimal: 0 Mode Flags: 15 Outfile: ndmpd.gz Cmd: Custom function Query: ndmpd   --  manifest_status = content-empty , error = This file has not changed since previously sent AutoSupport seq( 3205 )
00000016.00000ddd 0000265b Thu Jun 15 2017 07:54:37 +00:00 [kern_notifyd:info:737] 0x8072d9f90: ERR: Autosupport::Collection: NotifyD collector: [AutoSupport_Custom_Content]    Target: custom_fx Subsys: log_files Type: troubleshooting Minimal: 0 Mode Flags: 15 Outfile: secd.gz Cmd: Custom function Query: secd   --  manifest_status = content-empty , error = This file has not changed since previously sent AutoSupport seq( 3206 )
00000016.00000dde 0000265b Thu Jun 15 2017 07:54:37 +00:00 [kern_notifyd:info:737] 0x8072d9f90: ERR: Autosupport::Collection: NotifyD collector: [AutoSupport_Custom_Content]    Target: custom_fx Subsys: log_files Type: troubleshooting Minimal: 0 Mode Flags: 15 Outfile: spmd.gz Cmd: Custom function Query: spmd   --  manifest_status = content-empty , error = This file has not changed since previously sent AutoSupport seq( 3205 )
00000016.00000ddf 0000265b Thu Jun 15 2017 07:54:37 +00:00 [kern_notifyd:info:737] 0x8072d9f90: ERR: Autosupport::Collection: NotifyD collector: [AutoSupport_Custom_Content]    Target: custom_fx Subsys: log_files Type: troubleshooting Minimal: 0 Mode Flags: 2 Outfile: vifmgr.gz Cmd: Custom function Query: vifmgr   --  manifest_status = content-empty , error = This file has not changed since previously sent AutoSupport seq( 3214 )
00000016.00000de0 0000265b Thu Jun 15 2017 07:54:37 +00:00 [kern_notifyd:info:737] 0x8072d9f90: ERR: Autosupport::Collection: NotifyD collector: [AutoSupport_Custom_Content]    Target: custom_fx Subsys: log_files Type: troubleshooting Minimal: 0 Mode Flags: 2 Outfile: vldb.gz Cmd: Custom function Query: vldb   --  manifest_status = content-empty , error = This file has not changed since previously sent AutoSupport seq( 3214 )
00000016.00000de1 0000265b Thu Jun 15 2017 07:54:37 +00:00 [kern_notifyd:info:737] 0x8072d9f90: ERR: Autosupport::Collection: NotifyD collector: [AutoSupport_Custom_Content]    Target: custom_fx Subsys: log_files Type: troubleshooting Minimal: 0 Mode Flags: 2 Outfile: bcomd.gz Cmd: Custom function Query: bcomd   --  manifest_status = content-empty , error = This file has not changed since previously sent AutoSupport seq( 3214 )
00000016.00000de2 0000265b Thu Jun 15 2017 07:54:37 +00:00 [kern_notifyd:info:737] 0x8072d9f90: ERR: Autosupport::Collection: NotifyD collector: [AutoSupport_Custom_Content]    Target: custom_fx Subsys: log_files Type: troubleshooting Minimal: 0 Mode Flags: 2 Outfile: apache_access.gz Cmd: Custom function Query: apache_access   --  manifest_status = content-empty , error = This file has not changed since previously sent AutoSupport seq( 3202 )
00000016.00000de3 0000265b Thu Jun 15 2017 07:54:37 +00:00 [kern_notifyd:info:737] 0x8072d9f90: ERR: Autosupport::Collection: NotifyD collector: [AutoSupport_Custom_Content]    Target: custom_fx Subsys: log_files Type: troubleshooting Minimal: 0 Mode Flags: 2 Outfile: apache_error.gz Cmd: Custom function Query: apache_error   --  manifest_status = content-empty , error = This file has not changed since previously sent AutoSupport seq( 3205 )
00000016.00000dea 00002691 Thu Jun 15 2017 07:54:43 +00:00 [kern_notifyd:info:737] Successfully completed collection for autosupport with sequence number 3215
00000016.00000deb 00002691 Thu Jun 15 2017 07:54:43 +00:00 [kern_notifyd:info:737] asup_job::run got '' for ASUP file
00000016.00000e1d 00005ddc Thu Jun 15 2017 08:18:19 +00:00 [kern_notifyd:info:737] start Health Dashboard job for event vifmgr.portup (930121) worked t=2401
00000016.00000e20 00005e04 Thu Jun 15 2017 08:18:23 +00:00 [kern_notifyd:info:737] start Health Dashboard job for event vifmgr.portup (930122) worked t=2405
00000016.00000e23 00005e04 Thu Jun 15 2017 08:18:23 +00:00 [kern_notifyd:info:737] start Health Dashboard job for event vifmgr.portup (930123) worked t=2405
00000017.00000003 00000551 Thu Jun 15 2017 08:57:09 +00:00 [user_notifyd:notice] SteadyUptime: CPU frequency: 2333427663  initialized at 134 (uptime at startup).
00000017.00000004 00000551 Thu Jun 15 2017 08:57:10 +00:00 [kern_notifyd:info:737] .------------------------------------------------------------------------------.
00000017.00000005 00000551 Thu Jun 15 2017 08:57:10 +00:00 [kern_notifyd:info:737]                    Loading notifyd at: Thu Jun 15 08:57:10 2017
00000017.00000006 00000551 Thu Jun 15 2017 08:57:10 +00:00 [kern_notifyd:info:737] .------------------------------------------------------------------------------.
00000017.00000007 00000551 Thu Jun 15 2017 08:57:10 +00:00 [kern_notifyd:info:737] Is_7_Mode: kenv for 'bootarg.init.boot_clustered' returned 'true', using 10-mode
00000017.00000008 00000551 Thu Jun 15 2017 08:57:10 +00:00 [kern_notifyd:info:737] isUpgrade(): OPTOUT Can't read bootarg.upgrade.
00000017.00000009 00000551 Thu Jun 15 2017 08:57:10 +00:00 [kern_notifyd:info:737] using program number: 536871426 for RPC server.
00000017.0000000a 00000551 Thu Jun 15 2017 08:57:10 +00:00 [kern_notifyd:info:737] using program number: 536873502 for dsmdb agent.
00000017.0000000b 00000551 Thu Jun 15 2017 08:57:10 +00:00 [kern_notifyd:info:737] using program number: 536873503 for dsmdb registry.
00000017.00000013 0000055c Thu Jun 15 2017 08:57:11 +00:00 [kern_notifyd:info:737] Initializing ZapiWrapper Library
00000017.00000014 0000055c Thu Jun 15 2017 08:57:11 +00:00 [kern_notifyd:info:737] Initializing ZapiConnectionCache
00000017.0000001b 00000598 Thu Jun 15 2017 08:57:17 +00:00 [kern_notifyd:info:737] na_zsm_socket_open(127.0.20.1): shttpc_connect() returns -1: err=60
00000017.0000001c 00000598 Thu Jun 15 2017 08:57:17 +00:00 [kern_notifyd:info:737] 0x8009020b0: ERR: ZapiConnection: registry-set: connection failed; reason='ZSM - Can't connect to host (60) (13055)' address=127.0.0.1:1,0:3:0:2:951:10:5
00000017.0000001d 00000598 Thu Jun 15 2017 08:57:17 +00:00 [kern_notifyd:info:737] 0x8009020b0: ERR: ZapiConnection: registry-set: failed to connect to '127.0.0.1:1,0:3:0:2:951:10:5' duration 6
00000017.0000001e 00000598 Thu Jun 15 2017 08:57:17 +00:00 [kern_notifyd:info:737] 0x8009020b0: ERR: AutoSupport::AsupConfig: Unable to set optout time in the registry, 0
00000017.0000001f 00000598 Thu Jun 15 2017 08:57:17 +00:00 [kern_notifyd:info:737] 0x8009020b0: ERR: AutoSupport::AsupConfig: OPTOUT:modify failed diable notify_imp
00000017.00000022 00000604 Thu Jun 15 2017 08:57:27 +00:00 [kern_notifyd:info:737] na_zsm_socket_open(127.0.20.1): shttpc_connect() returns -1: err=60
00000017.00000023 00000604 Thu Jun 15 2017 08:57:27 +00:00 [kern_notifyd:info:737] 0x80726b950: ERR: ZapiConnection: no proc: connection failed; reason='ZSM - Can't connect to host (60) (13055)' address=127.0.0.1:1,0:3:0:2:951:10:5
00000017.00000038 00000673 Thu Jun 15 2017 08:57:38 +00:00 [kern_notifyd:info:737] na_zsm_socket_open(127.0.20.1): shttpc_connect() returns -1: err=60
00000017.00000039 00000673 Thu Jun 15 2017 08:57:38 +00:00 [kern_notifyd:info:737] 0x80726b950: ERR: ZapiConnection: no proc: connection failed; reason='ZSM - Can't connect to host (60) (13055)' address=127.0.0.1:1,0:3:0:2:951:10:5
00000017.0000003a 00000673 Thu Jun 15 2017 08:57:39 +00:00 [kern_notifyd:info:737] EMS: first id=1497517022/1 for 'kern.cli.cmd'
00000017.0000003b 00000673 Thu Jun 15 2017 08:57:39 +00:00 [kern_notifyd:info:737] EMS kernel ORDER 1497517022/1 for 'kern.cli.cmd' (last 0/0)
00000017.00000041 000006d7 Thu Jun 15 2017 08:57:49 +00:00 [kern_notifyd:info:737] na_zsm_socket_open(127.0.20.1): shttpc_connect() returns -1: err=60
00000017.00000042 000006d7 Thu Jun 15 2017 08:57:49 +00:00 [kern_notifyd:info:737] 0x80726b950: ERR: ZapiConnection: no proc: connection failed; reason='ZSM - Can't connect to host (60) (13055)' address=127.0.0.1:1,0:3:0:2:951:10:5
00000017.00000043 00000747 Thu Jun 15 2017 08:58:00 +00:00 [kern_notifyd:info:737] na_zsm_socket_open(127.0.20.1): shttpc_connect() returns -1: err=60
00000017.00000044 00000747 Thu Jun 15 2017 08:58:00 +00:00 [kern_notifyd:info:737] 0x80726b950: ERR: ZapiConnection: no proc: connection failed; reason='ZSM - Can't connect to host (60) (13055)' address=127.0.0.1:1,0:3:0:2:951:10:5
00000017.00000064 000008a6 Thu Jun 15 2017 08:58:35 +00:00 [kern_notifyd:info:737] ems receiver for callhome: callhome.reboot.poweron
00000017.0000006d 000008cc Thu Jun 15 2017 08:58:37 +00:00 [kern_notifyd:info:737] EMS: pre-mroot first id=1497517022/1
00000017.0000006e 000008cc Thu Jun 15 2017 08:58:37 +00:00 [kern_notifyd:info:737] EMS: pre-mroot last id=1497517022/256
00000017.00000074 000008cc Thu Jun 15 2017 08:58:37 +00:00 [kern_notifyd:info:737] EMS manager 1.1

0 Kudos

loganb
NetApp
‎2017-06-22 01:31 PM
In response to maqiuming
5,620 Views

please open a support ticket for further assistance.

 

 

looks like the notifyd process is wedged.

 

 

0 Kudos

andris
NetApp
‎2017-06-22 08:04 AM
5,637 Views
8.1.2 is very old ONTAP software. Upgrade to the latest recommended release for your FAS model.
0 Kudos
All Community Forums
Public