Active IQ and AutoSupport Discussions

why ASUP process hung in transmitting status?

maqiuming
5,198 Views

Platform : FAS3250

version: 8.1.2 cluster mode

company:IGT

 

symptom: ASUP can not received by EMail .

 

ASUP can't received form 6/15 , seq 3207  Autosupport log continue under transmitting status , does it process hung  ? can we kill it via systemshell?

 

there no cancel transmit command in 8.1.2 C-mode

 

1. ASUP enabled

Node             StateFrom     To       Mail Hosts

--------------------- --------- ------------- ------------- ----------

PEKNAP-01        enablepeknap01@igt.com
                                         support@netapp.com, qiuming.ma@igt.com
                                                       10.210.41.203
PEKNAP-02        enablepeknap02@igt.com
                                         qiuming.ma@igt.com, support@netapp.com
                                                       10.210.41.203

2 entries were displayed.

 

2.ping SMTP mail server is alive

PEKNAP::system node autosupport*> net ping -node PEKNAP-01 -destination 10.210.41.203

  (network ping)

10.210.41.203 is alive

 

3.detail setting in ASUP CHECK log, notifyd log -->

 

4.

PEKNAP::system node autosupport*> history show

                                                                                              

             Seq                                    Attempt  Last

Node         Num   Destination Status               Count    Update

------------ ----- ----------- -------------------- -------- -----------------

PEKNAP-01    3218

                   http        queued               -        6/15/2017 11:48:21

                   noteto      queued               -        6/15/2017 11:48:21

             3217

                   smtp        queued               -        6/15/2017 09:01:02

                   http        queued               -        6/15/2017 09:01:02

                   noteto      queued               -        6/15/2017 09:01:02

             3216

                   smtp        queued               -        6/15/2017 08:59:50

                   http        queued               -        6/15/2017 08:59:50

                   noteto      queued               -        6/15/2017 08:59:50

             3215

                   smtp        re-queued            -        6/15/2017 07:54:43

                   http        re-queued            -        6/15/2017 07:54:43

                   noteto      re-queued            -        6/15/2017 07:54:43

             3214

                   smtp        re-queued            -        6/15/2017 07:54:18

                   http        re-queued            -        6/15/2017 07:54:18

                   noteto      re-queued            -        6/15/2017 07:54:18

             3213

                   smtp        collection-failed    -        6/15/2017 07:53:29

                   http        collection-failed    -        6/15/2017 07:53:29

                   noteto      collection-failed    -        6/15/2017 07:53:29

             3212

                   smtp        collection-failed    -        6/15/2017 07:53:02

                   http        collection-failed    -        6/15/2017 07:53:02

                   noteto      collection-failed    -        6/15/2017 07:53:02

             3211

                   smtp        collection-failed    -        6/15/2017 07:52:34

                   http        collection-failed    -        6/15/2017 07:52:34

                   noteto      collection-failed    -        6/15/2017 07:52:34

             3210

                   smtp        collection-failed    -        6/15/2017 07:52:15

                   http        collection-failed    -        6/15/2017 07:52:15

                   noteto      collection-failed    -        6/15/2017 07:52:15

             3209

                   smtp        collection-failed    -        6/15/2017 07:52:00

                   http        collection-failed    -        6/15/2017 07:52:00

                   noteto      collection-failed    -        6/15/2017 07:52:00

             3208

                   smtp        re-queued            -        6/15/2017 07:46:42

                   http        re-queued            -        6/15/2017 07:46:42

                   noteto      re-queued            -        6/15/2017 07:46:42

             3207

                   smtp        transmitting         -        6/15/2017 08:58:44

                   http        re-queued            -        6/15/2017 07:45:16

                   noteto      ignore               1        6/15/2017 07:53:10

                   retransmit  queued               -        6/22/2017 08:06:04

             3206

                   smtp        sent-successful      1        6/15/2017 07:53:10

                   http        ignore               1        6/15/2017 07:53:10

                   noteto      sent-successful      1        6/15/2017 07:52:58

             3205

                   smtp        sent-successful      1        6/15/2017 07:52:56

                   http        ignore               1        6/15/2017 07:52:56

                   noteto      sent-successful      1        6/15/2017 07:52:53

             3204

                   smtp        collection-failed    -        6/15/2017 07:34:46

                   http        collection-failed    -        6/15/2017 07:34:46

                   noteto      collection-failed    -        6/15/2017 07:34:46

             3203

                   smtp        sent-successful      1        6/15/2017 07:47:29

                   http        ignore               1        6/15/2017 07:47:29

                   noteto      ignore               1        6/15/2017 07:47:25

             3202

                   smtp        sent-successful      1        6/15/2017 07:47:25

                   http        ignore               1        6/15/2017 07:47:25

                   noteto      sent-successful      1        6/15/2017 07:47:19

             3201

                   smtp        sent-successful      1        6/15/2017 07:47:17

                   http        ignore               1        6/15/2017 07:32:26

                   noteto      ignore               1        6/15/2017 07:47:15

             3200

                   smtp        sent-successful      1        6/15/2017 07:47:15

                   http        ignore               1        6/15/2017 07:32:26

                   noteto      ignore               1        6/15/2017 07:47:11

             3199

                   smtp        sent-successful      1        6/15/2017 07:47:11

                   http        ignore               1        6/15/2017 07:32:26

                   noteto      ignore               1        6/15/2017 07:47:08

             3198

                   smtp        sent-successful      1        6/15/2017 07:47:08

                   http        ignore               1        6/15/2017 07:32:26

 

PEKNAP::system node autosupport*> history show -seq-num 3206 -instance

 

                            Node: PEKNAP-01

     AutoSupport Sequence Number: 3206

Destination for this AutoSupport: smtp

                   Trigger Event: callhome.sfo.takeover

             Time of Last Update: 6/15/2017 07:53:10

              Status of Delivery: sent-successful

               Delivery Attempts: 1

             AutoSupport Subject: CONTROLLER TAKEOVER COMPLETE AUTOMATIC

                    Delivery URI: mailto:autosupport@netapp.com,certified@eccom.com.cn,qiuming.ma@igt.com,support@netapp.com

                      Last Error: -

 

                            Node: PEKNAP-01

     AutoSupport Sequence Number: 3206

Destination for this AutoSupport: http

                   Trigger Event: callhome.sfo.takeover

             Time of Last Update: 6/15/2017 07:53:10

              Status of Delivery: ignore

               Delivery Attempts: 1

             AutoSupport Subject: CONTROLLER TAKEOVER COMPLETE AUTOMATIC

                    Delivery URI: -

                      Last Error: -

 

                            Node: PEKNAP-01

     AutoSupport Sequence Number: 3206

Destination for this AutoSupport: noteto

                   Trigger Event: callhome.sfo.takeover

             Time of Last Update: 6/15/2017 07:52:58

              Status of Delivery: sent-successful

               Delivery Attempts: 1

             AutoSupport Subject: CONTROLLER TAKEOVER COMPLETE AUTOMATIC

                    Delivery URI: mailto:jason.tang@igt.com,jianhui.du@igt.com,rui.wang@igt.com

                      Last Error: -

3 entries were displayed.

 

PEKNAP::system node autosupport*> history show -seq-num 3206 -instance  -instance  7 -instance  -instance

 

                            Node: PEKNAP-01

     AutoSupport Sequence Number: 3207

Destination for this AutoSupport: smtp

                   Trigger Event: callhome.reboot.takeover

             Time of Last Update: 6/15/2017 08:58:44

              Status of Delivery: transmitting

               Delivery Attempts: -

             AutoSupport Subject: PARTNER REBOOT (CONTROLLER TAKEOVER)

                    Delivery URI: -

                      Last Error: -

 

                            Node: PEKNAP-01

     AutoSupport Sequence Number: 3207

Destination for this AutoSupport: http

                   Trigger Event: callhome.reboot.takeover

             Time of Last Update: 6/15/2017 07:45:16

              Status of Delivery: re-queued

               Delivery Attempts: -

             AutoSupport Subject: PARTNER REBOOT (CONTROLLER TAKEOVER)

                    Delivery URI: -

                      Last Error: -

 

                            Node: PEKNAP-01

     AutoSupport Sequence Number: 3207

Destination for this AutoSupport: noteto

                   Trigger Event: callhome.reboot.takeover

             Time of Last Update: 6/15/2017 07:53:10

              Status of Delivery: ignore

               Delivery Attempts: 1

             AutoSupport Subject: PARTNER REBOOT (CONTROLLER TAKEOVER)

                    Delivery URI: -

                      Last Error: -

 

                            Node: PEKNAP-01

     AutoSupport Sequence Number: 3207

Destination for this AutoSupport: retransmit

                   Trigger Event: callhome.reboot.takeover

             Time of Last Update: 6/22/2017 08:06:04

              Status of Delivery: queued

               Delivery Attempts: -

             AutoSupport Subject: PARTNER REBOOT (CONTROLLER TAKEOVER)

                    Delivery URI: https://support.netapp.com/asupprod/post/1.0/postAsup

                      Last Error: -

4 entries were displayed.

3 REPLIES 3

maqiuming
5,191 Views

@maqiuming wrote:

Platform : FAS3250

version: 8.1.2 cluster mode

company:IGT

 

symptom: ASUP can not received by EMail .

 

ASUP can't received form 6/15 , seq 3207  Autosupport log continue under transmitting status , does it process hung  ? can we kill it via systemshell?

 

there no cancel transmit command in 8.1.2 C-mode

 

1. ASUP enabled

Node             StateFrom     To       Mail Hosts

--------------------- --------- ------------- ------------- ----------

PEKNAP-01        enablepeknap01@igt.com
                                         support@netapp.com, qiuming.ma@igt.com
                                                       10.210.41.203
PEKNAP-02        enablepeknap02@igt.com
                                         qiuming.ma@igt.com, support@netapp.com
                                                       10.210.41.203

2 entries were displayed.

 

3.detail setting in ASUP CHECK log, notifyd log -->

 

2.ping SMTP mail server is alive

PEKNAP::system node autosupport*> net ping -node PEKNAP-01 -destination 10.210.41.203

  (network ping)

10.210.41.203 is alive

 

4.

PEKNAP::system node autosupport*> history show

                                                                                              

             Seq                                    Attempt  Last

Node         Num   Destination Status               Count    Update

------------ ----- ----------- -------------------- -------- -----------------

PEKNAP-01    3218

                   http        queued               -        6/15/2017 11:48:21

                   noteto      queued               -        6/15/2017 11:48:21

             3217

                   smtp        queued               -        6/15/2017 09:01:02

                   http        queued               -        6/15/2017 09:01:02

                   noteto      queued               -        6/15/2017 09:01:02

             3216

                   smtp        queued               -        6/15/2017 08:59:50

                   http        queued               -        6/15/2017 08:59:50

                   noteto      queued               -        6/15/2017 08:59:50

             3215

                   smtp        re-queued            -        6/15/2017 07:54:43

                   http        re-queued            -        6/15/2017 07:54:43

                   noteto      re-queued            -        6/15/2017 07:54:43

             3214

                   smtp        re-queued            -        6/15/2017 07:54:18

                   http        re-queued            -        6/15/2017 07:54:18

                   noteto      re-queued            -        6/15/2017 07:54:18

             3213

                   smtp        collection-failed    -        6/15/2017 07:53:29

                   http        collection-failed    -        6/15/2017 07:53:29

                   noteto      collection-failed    -        6/15/2017 07:53:29

             3212

                   smtp        collection-failed    -        6/15/2017 07:53:02

                   http        collection-failed    -        6/15/2017 07:53:02

                   noteto      collection-failed    -        6/15/2017 07:53:02

             3211

                   smtp        collection-failed    -        6/15/2017 07:52:34

                   http        collection-failed    -        6/15/2017 07:52:34

                   noteto      collection-failed    -        6/15/2017 07:52:34

             3210

                   smtp        collection-failed    -        6/15/2017 07:52:15

                   http        collection-failed    -        6/15/2017 07:52:15

                   noteto      collection-failed    -        6/15/2017 07:52:15

             3209

                   smtp        collection-failed    -        6/15/2017 07:52:00

                   http        collection-failed    -        6/15/2017 07:52:00

                   noteto      collection-failed    -        6/15/2017 07:52:00

             3208

                   smtp        re-queued            -        6/15/2017 07:46:42

                   http        re-queued            -        6/15/2017 07:46:42

                   noteto      re-queued            -        6/15/2017 07:46:42

             3207

                   smtp        transmitting         -        6/15/2017 08:58:44

                   http        re-queued            -        6/15/2017 07:45:16

                   noteto      ignore               1        6/15/2017 07:53:10

                   retransmit  queued               -        6/22/2017 08:06:04

             3206

                   smtp        sent-successful      1        6/15/2017 07:53:10

                   http        ignore               1        6/15/2017 07:53:10

                   noteto      sent-successful      1        6/15/2017 07:52:58

             3205

                   smtp        sent-successful      1        6/15/2017 07:52:56

                   http        ignore               1        6/15/2017 07:52:56

                   noteto      sent-successful      1        6/15/2017 07:52:53

             3204

                   smtp        collection-failed    -        6/15/2017 07:34:46

                   http        collection-failed    -        6/15/2017 07:34:46

                   noteto      collection-failed    -        6/15/2017 07:34:46

             3203

                   smtp        sent-successful      1        6/15/2017 07:47:29

                   http        ignore               1        6/15/2017 07:47:29

                   noteto      ignore               1        6/15/2017 07:47:25

             3202

                   smtp        sent-successful      1        6/15/2017 07:47:25

                   http        ignore               1        6/15/2017 07:47:25

                   noteto      sent-successful      1        6/15/2017 07:47:19

             3201

                   smtp        sent-successful      1        6/15/2017 07:47:17

                   http        ignore               1        6/15/2017 07:32:26

                   noteto      ignore               1        6/15/2017 07:47:15

             3200

                   smtp        sent-successful      1        6/15/2017 07:47:15

                   http        ignore               1        6/15/2017 07:32:26

                   noteto      ignore               1        6/15/2017 07:47:11

             3199

                   smtp        sent-successful      1        6/15/2017 07:47:11

                   http        ignore               1        6/15/2017 07:32:26

                   noteto      ignore               1        6/15/2017 07:47:08

             3198

                   smtp        sent-successful      1        6/15/2017 07:47:08

                   http        ignore               1        6/15/2017 07:32:26

 

PEKNAP::system node autosupport*> history show -seq-num 3206 -instance

 

                            Node: PEKNAP-01

     AutoSupport Sequence Number: 3206

Destination for this AutoSupport: smtp

                   Trigger Event: callhome.sfo.takeover

             Time of Last Update: 6/15/2017 07:53:10

              Status of Delivery: sent-successful

               Delivery Attempts: 1

             AutoSupport Subject: CONTROLLER TAKEOVER COMPLETE AUTOMATIC

                    Delivery URI: mailto:autosupport@netapp.com,certified@eccom.com.cn,qiuming.ma@igt.com,support@netapp.com

                      Last Error: -

 

                            Node: PEKNAP-01

     AutoSupport Sequence Number: 3206

Destination for this AutoSupport: http

                   Trigger Event: callhome.sfo.takeover

             Time of Last Update: 6/15/2017 07:53:10

              Status of Delivery: ignore

               Delivery Attempts: 1

             AutoSupport Subject: CONTROLLER TAKEOVER COMPLETE AUTOMATIC

                    Delivery URI: -

                      Last Error: -

 

                            Node: PEKNAP-01

     AutoSupport Sequence Number: 3206

Destination for this AutoSupport: noteto

                   Trigger Event: callhome.sfo.takeover

             Time of Last Update: 6/15/2017 07:52:58

              Status of Delivery: sent-successful

               Delivery Attempts: 1

             AutoSupport Subject: CONTROLLER TAKEOVER COMPLETE AUTOMATIC

                    Delivery URI: mailto:jason.tang@igt.com,jianhui.du@igt.com,rui.wang@igt.com

                      Last Error: -

3 entries were displayed.

 

PEKNAP::system node autosupport*> history show -seq-num 3206 -instance  -instance  7 -instance  -instance

 

                            Node: PEKNAP-01

     AutoSupport Sequence Number: 3207

Destination for this AutoSupport: smtp

                   Trigger Event: callhome.reboot.takeover

             Time of Last Update: 6/15/2017 08:58:44

              Status of Delivery: transmitting

               Delivery Attempts: -

             AutoSupport Subject: PARTNER REBOOT (CONTROLLER TAKEOVER)

                    Delivery URI: -

                      Last Error: -

 

                            Node: PEKNAP-01

     AutoSupport Sequence Number: 3207

Destination for this AutoSupport: http

                   Trigger Event: callhome.reboot.takeover

             Time of Last Update: 6/15/2017 07:45:16

              Status of Delivery: re-queued

               Delivery Attempts: -

             AutoSupport Subject: PARTNER REBOOT (CONTROLLER TAKEOVER)

                    Delivery URI: -

                      Last Error: -

 

                            Node: PEKNAP-01

     AutoSupport Sequence Number: 3207

Destination for this AutoSupport: noteto

                   Trigger Event: callhome.reboot.takeover

             Time of Last Update: 6/15/2017 07:53:10

              Status of Delivery: ignore

               Delivery Attempts: 1

             AutoSupport Subject: PARTNER REBOOT (CONTROLLER TAKEOVER)

                    Delivery URI: -

                      Last Error: -

 

                            Node: PEKNAP-01

     AutoSupport Sequence Number: 3207

Destination for this AutoSupport: retransmit

                   Trigger Event: callhome.reboot.takeover

             Time of Last Update: 6/22/2017 08:06:04

              Status of Delivery: queued

               Delivery Attempts: -

             AutoSupport Subject: PARTNER REBOOT (CONTROLLER TAKEOVER)

                    Delivery URI: https://support.netapp.com/asupprod/post/1.0/postAsup

                      Last Error: -

4 entries were displayed.



can not upload log , place here directly , may be large , will cut out the import time.

 

question is , everythings for config looks normal form the CLI , but the ASUP still can not send out , does some command can kill the transmitting session and restart form the least one ? 

 

00000016.00000b31 000013d3 Thu Jun 15 2017 07:46:44 +00:00 [kern_notifyd:info:737] Successfully delivered smtp asup with sequence: 3196
00000016.00000b32 000013d3 Thu Jun 15 2017 07:46:44 +00:00 [kern_notifyd:info:737] Beginning to deliver asup with sequence number 3197, subject HA Group Notification from PEKNAP-01 (MANAGEMENT_LOG) INFO
00000016.00000b33 000013d3 Thu Jun 15 2017 07:46:44 +00:00 [kern_notifyd:info:737] Successfully delivered noteto asup with sequence: 3197
00000016.00000b34 000013d3 Thu Jun 15 2017 07:46:44 +00:00 [kern_notifyd:info:737] 0x8072d9e00: ERR: Autosupport::Delivery: created compressor with out_filename = /mroot/etc/log/autosupport/201706130030.0.smtp_tmp
00000016.00000b3d 00001492 Thu Jun 15 2017 07:47:03 +00:00 [kern_notifyd:info:737] Address Info (0x807b1f2c0) [ai_flags: 1025, ai_family: 2, ai_socktype: 1, ai_protocol: 6, ai_addrlen: 16, ai_addr: 0x807b1f2f0, ai_canonname: (null), ai_next: 0x0]
00000016.00000b3e 000014bb Thu Jun 15 2017 07:47:04 +00:00 [kern_notifyd:info:737] Subject: HA Group Notification from PEKNAP-01 (MANAGEMENT_LOG) INFO
00000016.00000b3f 000014bb Thu Jun 15 2017 07:47:05 +00:00 [kern_notifyd:info:737] Successfully delivered smtp asup with sequence: 3197
00000016.00000b40 000014bb Thu Jun 15 2017 07:47:05 +00:00 [kern_notifyd:info:737] Beginning to deliver asup with sequence number 3198, subject HA Group Notification from PEKNAP-01 (PERFORMANCE DATA) INFO
00000016.00000b41 000014bb Thu Jun 15 2017 07:47:05 +00:00 [kern_notifyd:info:737] Successfully delivered noteto asup with sequence: 3198
00000016.00000b42 000014bb Thu Jun 15 2017 07:47:05 +00:00 [kern_notifyd:info:737] 0x8072d9e00: ERR: Autosupport::Delivery: created compressor with out_filename = /mroot/etc/log/autosupport/201706140000.0.smtp_tmp
00000016.00000b43 000014bb Thu Jun 15 2017 07:47:06 +00:00 [kern_notifyd:info:737] Address Info (0x807b1f2c0) [ai_flags: 1025, ai_family: 2, ai_socktype: 1, ai_protocol: 6, ai_addrlen: 16, ai_addr: 0x807b1f2f0, ai_canonname: (null), ai_next: 0x0]
00000016.00000b46 000014bb Thu Jun 15 2017 07:47:07 +00:00 [kern_notifyd:info:737] Subject: HA Group Notification from PEKNAP-01 (PERFORMANCE DATA) INFO
00000016.00000b47 000014eb Thu Jun 15 2017 07:47:08 +00:00 [kern_notifyd:info:737] Successfully delivered smtp asup with sequence: 3198
00000016.00000b48 000014eb Thu Jun 15 2017 07:47:08 +00:00 [kern_notifyd:info:737] Beginning to deliver asup with sequence number 3199, subject HA Group Notification from PEKNAP-01 (MANAGEMENT_LOG) INFO
00000016.00000b49 000014eb Thu Jun 15 2017 07:47:08 +00:00 [kern_notifyd:info:737] Successfully delivered noteto asup with sequence: 3199
00000016.00000b4a 000014eb Thu Jun 15 2017 07:47:08 +00:00 [kern_notifyd:info:737] 0x8072d9e00: ERR: Autosupport::Delivery: created compressor with out_filename = /mroot/etc/log/autosupport/201706140020.0.smtp_tmp
00000016.00000b4b 000014eb Thu Jun 15 2017 07:47:09 +00:00 [kern_notifyd:info:737] Address Info (0x807b1f2c0) [ai_flags: 1025, ai_family: 2, ai_socktype: 1, ai_protocol: 6, ai_addrlen: 16, ai_addr: 0x807b1f2f0, ai_canonname: (null), ai_next: 0x0]
00000016.00000b50 000014eb Thu Jun 15 2017 07:47:10 +00:00 [kern_notifyd:info:737] Subject: HA Group Notification from PEKNAP-01 (MANAGEMENT_LOG) INFO
00000016.00000b51 000014eb Thu Jun 15 2017 07:47:11 +00:00 [kern_notifyd:info:737] Successfully delivered smtp asup with sequence: 3199
00000016.00000b52 000014eb Thu Jun 15 2017 07:47:11 +00:00 [kern_notifyd:info:737] Beginning to deliver asup with sequence number 3200, subject HA Group Notification from PEKNAP-01 (PERFORMANCE DATA) INFO
00000016.00000b53 000014eb Thu Jun 15 2017 07:47:11 +00:00 [kern_notifyd:info:737] Successfully delivered noteto asup with sequence: 3200
00000016.00000b54 000014eb Thu Jun 15 2017 07:47:11 +00:00 [kern_notifyd:info:737] 0x8072d9e00: ERR: Autosupport::Delivery: created compressor with out_filename = /mroot/etc/log/autosupport/201706150000.0.smtp_tmp
00000016.00000b55 000014eb Thu Jun 15 2017 07:47:12 +00:00 [kern_notifyd:info:737] Address Info (0x807b1fb00) [ai_flags: 1025, ai_family: 2, ai_socktype: 1, ai_protocol: 6, ai_addrlen: 16, ai_addr: 0x807b1fb30, ai_canonname: (null), ai_next: 0x0]
00000016.00000b56 000014f8 Thu Jun 15 2017 07:47:13 +00:00 [kern_notifyd:info:737] Subject: HA Group Notification from PEKNAP-01 (PERFORMANCE DATA) INFO
00000016.00000b57 0000150e Thu Jun 15 2017 07:47:15 +00:00 [kern_notifyd:info:737] Successfully delivered smtp asup with sequence: 3200
00000016.00000b58 0000150e Thu Jun 15 2017 07:47:15 +00:00 [kern_notifyd:info:737] Beginning to deliver asup with sequence number 3201, subject HA Group Notification from PEKNAP-01 (MANAGEMENT_LOG) INFO
00000016.00000b59 0000150e Thu Jun 15 2017 07:47:15 +00:00 [kern_notifyd:info:737] Successfully delivered noteto asup with sequence: 3201
00000016.00000b5a 0000150e Thu Jun 15 2017 07:47:15 +00:00 [kern_notifyd:info:737] 0x8072d9e00: ERR: Autosupport::Delivery: created compressor with out_filename = /mroot/etc/log/autosupport/201706150010.0.smtp_tmp
00000016.00000b5b 0000150e Thu Jun 15 2017 07:47:15 +00:00 [kern_notifyd:info:737] Address Info (0x807b1fb00) [ai_flags: 1025, ai_family: 2, ai_socktype: 1, ai_protocol: 6, ai_addrlen: 16, ai_addr: 0x807b1fb30, ai_canonname: (null), ai_next: 0x0]
00000016.00000b5c 00001521 Thu Jun 15 2017 07:47:16 +00:00 [kern_notifyd:info:737] Subject: HA Group Notification from PEKNAP-01 (MANAGEMENT_LOG) INFO
00000016.00000b5d 00001521 Thu Jun 15 2017 07:47:17 +00:00 [kern_notifyd:info:737] Successfully delivered smtp asup with sequence: 3201
00000016.00000b5e 00001521 Thu Jun 15 2017 07:47:17 +00:00 [kern_notifyd:info:737] Beginning to deliver asup with sequence number 3202, subject HA Group Notification from PEKNAP-01 (REBOOT (power on)) INFO
00000016.00000b5f 00001521 Thu Jun 15 2017 07:47:17 +00:00 [kern_notifyd:info:737] Address Info (0x807b1f0c0) [ai_flags: 1025, ai_family: 2, ai_socktype: 1, ai_protocol: 6, ai_addrlen: 16, ai_addr: 0x807b1f0f0, ai_canonname: (null), ai_next: 0x0]
00000016.00000b60 00001533 Thu Jun 15 2017 07:47:19 +00:00 [kern_notifyd:info:737] Successfully delivered noteto asup with sequence: 3202
00000016.00000b61 00001533 Thu Jun 15 2017 07:47:19 +00:00 [kern_notifyd:info:737] 0x8072d9e00: ERR: Autosupport::Delivery: created compressor with out_filename = /mroot/etc/log/autosupport/201706150729.0.smtp_tmp
00000016.00000b62 0000155e Thu Jun 15 2017 07:47:23 +00:00 [kern_notifyd:info:737] Address Info (0x807b1fb00) [ai_flags: 1025, ai_family: 2, ai_socktype: 1, ai_protocol: 6, ai_addrlen: 16, ai_addr: 0x807b1fb30, ai_canonname: (null), ai_next: 0x0]
00000016.00000b63 0000156b Thu Jun 15 2017 07:47:24 +00:00 [kern_notifyd:info:737] Subject: HA Group Notification from PEKNAP-01 (REBOOT (power on)) INFO
00000016.00000b64 00001588 Thu Jun 15 2017 07:47:25 +00:00 [kern_notifyd:info:737] Successfully delivered smtp asup with sequence: 3202
00000016.00000b65 00001588 Thu Jun 15 2017 07:47:25 +00:00 [kern_notifyd:info:737] 0x8072d9e00: ERR: Autosupport::Delivery: created compressor with out_filename = /mroot/etc/log/autosupport/201706150729.0.http_tmp
00000016.00000b66 00001588 Thu Jun 15 2017 07:47:25 +00:00 [kern_notifyd:info:737] Successfully delivered http asup with sequence: 3202
00000016.00000b67 00001588 Thu Jun 15 2017 07:47:25 +00:00 [kern_notifyd:info:737] Beginning to deliver asup with sequence number 3203, subject HA Group Notification from PEKNAP-01 (PERFORMANCE SNAPSHOT) INFO
00000016.00000b68 00001588 Thu Jun 15 2017 07:47:25 +00:00 [kern_notifyd:info:737] Successfully delivered noteto asup with sequence: 3203
00000016.00000b69 00001588 Thu Jun 15 2017 07:47:25 +00:00 [kern_notifyd:info:737] 0x8072d9e00: ERR: Autosupport::Delivery: created compressor with out_filename = /mroot/etc/log/autosupport/201706150730.0.smtp_tmp
00000016.00000b6a 00001588 Thu Jun 15 2017 07:47:26 +00:00 [kern_notifyd:info:737] Address Info (0x807b1fb00) [ai_flags: 1025, ai_family: 2, ai_socktype: 1, ai_protocol: 6, ai_addrlen: 16, ai_addr: 0x807b1fb30, ai_canonname: (null), ai_next: 0x0]
00000016.00000b6d 00001588 Thu Jun 15 2017 07:47:27 +00:00 [kern_notifyd:info:737] Subject: HA Group Notification from PEKNAP-01 (PERFORMANCE SNAPSHOT) INFO
00000016.00000b6e 00001598 Thu Jun 15 2017 07:47:29 +00:00 [kern_notifyd:info:737] Successfully delivered smtp asup with sequence: 3203
00000016.00000b6f 00001598 Thu Jun 15 2017 07:47:29 +00:00 [kern_notifyd:info:737] 0x8072d9e00: ERR: Autosupport::Delivery: created compressor with out_filename = /mroot/etc/log/autosupport/201706150730.0.http_tmp
00000016.00000b70 00001598 Thu Jun 15 2017 07:47:29 +00:00 [kern_notifyd:info:737] Successfully delivered http asup with sequence: 3203
00000016.00000bf9 00001fe5 Thu Jun 15 2017 07:51:52 +00:00 [kern_notifyd:info:737] start Health Dashboard job for event cf.fsm.autoGivebackStarted (929904) worked t=815
00000016.00000bfb 00001fe5 Thu Jun 15 2017 07:51:52 +00:00 [kern_notifyd:info:737] start Health Dashboard job for event cf.fm.givebackStarted (929905) worked t=815
00000016.00000bfc 00001fe5 Thu Jun 15 2017 07:51:52 +00:00 [kern_notifyd:info:737] start SNMP job for event cf.fm.givebackStarted (929905) worked t=815
00000016.00000c02 0000200b Thu Jun 15 2017 07:51:55 +00:00 [kern_notifyd:info:737] start Health Dashboard job for event cf.fm.givebackComplete (929919) worked t=818
00000016.00000c03 0000200b Thu Jun 15 2017 07:51:55 +00:00 [kern_notifyd:info:737] start SNMP job for event cf.fm.givebackComplete (929919) worked t=818
00000016.00000c05 0000200b Thu Jun 15 2017 07:51:55 +00:00 [kern_notifyd:info:737] start SNMP job for event cf.fm.givebackDuration (929920) worked t=818
00000016.00000c1e 00002032 Thu Jun 15 2017 07:52:00 +00:00 [kern_notifyd:info:737] ems receiver for callhome: callhome.spares.low
00000016.00000c20 00002032 Thu Jun 15 2017 07:52:00 +00:00 [kern_notifyd:info:737] read callhome event: callhome.spares.low
00000016.00000c21 00002032 Thu Jun 15 2017 07:52:00 +00:00 [kern_notifyd:info:737] first level screening for callhome event: callhome.spares.low (name callhome.spares.low, src kernel)
00000016.00000c22 00002032 Thu Jun 15 2017 07:52:00 +00:00 [kern_notifyd:info:737] ems_asup::handle for event callhome.spares.low (929946) t=822
00000016.00000c23 00002032 Thu Jun 15 2017 07:52:00 +00:00 [kern_notifyd:info:737] ems_asup::handle job for event callhome.spares.low (929946) worked t=822
00000016.00000c24 00002032 Thu Jun 15 2017 07:52:00 +00:00 [kern_notifyd:info:737] asup_job::run started t=822
00000016.00000c25 00002032 Thu Jun 15 2017 07:52:00 +00:00 [kern_notifyd:info:737] asup_job::run is triggered by event callhome.spares.low (929946) t=822
00000016.00000c26 00002032 Thu Jun 15 2017 07:52:00 +00:00 [kern_notifyd:info:737] asup_job::run calling smtp_file for 'SPARES_LOW'
00000016.00000c27 00002032 Thu Jun 15 2017 07:52:00 +00:00 [kern_notifyd:info:737] Creating autosupport with sequence number 3209, event callhome.spares.low
00000016.00000c28 00002032 Thu Jun 15 2017 07:52:00 +00:00 [kern_notifyd:info:737] Beginning collection for autosupport with seq num 3209
00000016.00000c29 00002032 Thu Jun 15 2017 07:52:00 +00:00 [kern_notifyd:info:737] (category: 737:3209:collect) (emittime: 6/15/2017 07:51:59) (message: asup d-asup-start directory_path='/etc/log/autosupport/201706150752.0.files'  (complete))
00000016.00000c2a 00002032 Thu Jun 15 2017 07:52:00 +00:00 [kern_notifyd:info:737] (category: 737:3209:collect) (emittime: 6/15/2017 07:51:59) (message: asup ZAPI d-asup-start failed. Status: 1, Reason: AutoSupport Giveback in progress, Error: 13928)
00000016.00000c2b 00002032 Thu Jun 15 2017 07:52:00 +00:00 [kern_notifyd:info:737] asup_job::run smtp_file failed, requeue
00000016.00000c34 000020b7 Thu Jun 15 2017 07:52:14 +00:00 [kern_notifyd:info:737] asup_job::run started t=836
00000016.00000c35 000020b7 Thu Jun 15 2017 07:52:14 +00:00 [kern_notifyd:info:737] asup_job::run is triggered by event callhome.spares.low (929946) t=836
00000016.00000c36 000020b7 Thu Jun 15 2017 07:52:14 +00:00 [kern_notifyd:info:737] asup_job::run calling smtp_file for 'SPARES_LOW'
00000016.00000c37 000020b7 Thu Jun 15 2017 07:52:14 +00:00 [kern_notifyd:info:737] Creating autosupport with sequence number 3210, event callhome.spares.low
00000016.00000c38 000020b7 Thu Jun 15 2017 07:52:14 +00:00 [kern_notifyd:info:737] Beginning collection for autosupport with seq num 3210
00000016.00000c39 000020c1 Thu Jun 15 2017 07:52:15 +00:00 [kern_notifyd:info:737] (category: 737:3210:collect) (emittime: 6/15/2017 07:52:13) (message: asup lstat '/mroot/etc/log/autosupport/201706150752.0.files' exists)
00000016.00000c3a 000020c1 Thu Jun 15 2017 07:52:15 +00:00 [kern_notifyd:info:737] (category: 737:3210:collect) (emittime: 6/15/2017 07:52:14) (message: asup d-asup-start directory_path='/etc/log/autosupport/201706150752.1.files'  (complete))
00000016.00000c3b 000020c1 Thu Jun 15 2017 07:52:15 +00:00 [kern_notifyd:info:737] asup_job::run smtp_file failed, requeue
00000016.00000c3c 000020c1 Thu Jun 15 2017 07:52:15 +00:00 [kern_notifyd:info:737] (category: 737:3210:collect) (emittime: 6/15/2017 07:52:14) (message: asup ZAPI d-asup-start failed. Status: 1, Reason: AutoSupport Giveback in progress, Error: 13928)
00000016.00000c54 0000216b Thu Jun 15 2017 07:52:32 +00:00 [kern_notifyd:info:737] asup_job::run started t=854
00000016.00000c55 0000216b Thu Jun 15 2017 07:52:32 +00:00 [kern_notifyd:info:737] asup_job::run is triggered by event callhome.spares.low (929946) t=854
00000016.00000c56 0000216b Thu Jun 15 2017 07:52:32 +00:00 [kern_notifyd:info:737] asup_job::run calling smtp_file for 'SPARES_LOW'
00000016.00000c57 0000216b Thu Jun 15 2017 07:52:32 +00:00 [kern_notifyd:info:737] Creating autosupport with sequence number 3211, event callhome.spares.low
00000016.00000c58 0000216b Thu Jun 15 2017 07:52:32 +00:00 [kern_notifyd:info:737] Beginning collection for autosupport with seq num 3211
00000016.00000c5f 0000218b Thu Jun 15 2017 07:52:34 +00:00 [kern_notifyd:info:737] (category: 737:3211:collect) (emittime: 6/15/2017 07:52:31) (message: asup lstat '/mroot/etc/log/autosupport/201706150752.0.files' exists)
00000016.00000c60 0000218b Thu Jun 15 2017 07:52:34 +00:00 [kern_notifyd:info:737] (category: 737:3211:collect) (emittime: 6/15/2017 07:52:32) (message: asup lstat '/mroot/etc/log/autosupport/201706150752.1.files' exists)
00000016.00000c61 0000218b Thu Jun 15 2017 07:52:34 +00:00 [kern_notifyd:info:737] (category: 737:3211:collect) (emittime: 6/15/2017 07:52:33) (message: asup d-asup-start directory_path='/etc/log/autosupport/201706150752.2.files'  (complete))
00000016.00000c62 0000218b Thu Jun 15 2017 07:52:34 +00:00 [kern_notifyd:info:737] (category: 737:3211:collect) (emittime: 6/15/2017 07:52:33) (message: asup ZAPI d-asup-start failed. Status: 1, Reason: AutoSupport Giveback in progress, Error: 13928)
00000016.00000c63 0000218b Thu Jun 15 2017 07:52:34 +00:00 [kern_notifyd:info:737] asup_job::run smtp_file failed, requeue
00000016.00000c7a 00002229 Thu Jun 15 2017 07:52:51 +00:00 [kern_notifyd:info:737] Beginning to deliver asup with sequence number 3205, subject HA Group Notification from PEKNAP-01 (REBOOT (power on)) INFO
00000016.00000c7b 00002229 Thu Jun 15 2017 07:52:51 +00:00 [kern_notifyd:info:737] Address Info (0x807b7a700) [ai_flags: 1025, ai_family: 2, ai_socktype: 1, ai_protocol: 6, ai_addrlen: 16, ai_addr: 0x807b7a730, ai_canonname: (null), ai_next: 0x0]
00000016.00000c7c 0000223c Thu Jun 15 2017 07:52:53 +00:00 [kern_notifyd:info:737] Successfully delivered noteto asup with sequence: 3205
00000016.00000c7d 0000223c Thu Jun 15 2017 07:52:53 +00:00 [kern_notifyd:info:737] 0x8072d9e00: ERR: Autosupport::Delivery: created compressor with out_filename = /mroot/etc/log/autosupport/201706150741.0.smtp_tmp
00000016.00000c7e 00002249 Thu Jun 15 2017 07:52:54 +00:00 [kern_notifyd:info:737] Address Info (0x807b7a700) [ai_flags: 1025, ai_family: 2, ai_socktype: 1, ai_protocol: 6, ai_addrlen: 16, ai_addr: 0x807b7a730, ai_canonname: (null), ai_next: 0x0]
00000016.00000c7f 00002256 Thu Jun 15 2017 07:52:55 +00:00 [kern_notifyd:info:737] Subject: HA Group Notification from PEKNAP-01 (REBOOT (power on)) INFO
00000016.00000c80 00002260 Thu Jun 15 2017 07:52:56 +00:00 [kern_notifyd:info:737] Successfully delivered smtp asup with sequence: 3205
00000016.00000c81 00002260 Thu Jun 15 2017 07:52:56 +00:00 [kern_notifyd:info:737] 0x8072d9e00: ERR: Autosupport::Delivery: created compressor with out_filename = /mroot/etc/log/autosupport/201706150741.0.http_tmp
00000016.00000c82 00002260 Thu Jun 15 2017 07:52:56 +00:00 [kern_notifyd:info:737] Successfully delivered http asup with sequence: 3205
00000016.00000c83 00002260 Thu Jun 15 2017 07:52:56 +00:00 [kern_notifyd:info:737] Beginning to deliver asup with sequence number 3206, subject HA Group Notification from PEKNAP-01 (CONTROLLER TAKEOVER COMPLETE AUTOMATIC) ERROR
00000016.00000c84 00002260 Thu Jun 15 2017 07:52:56 +00:00 [kern_notifyd:info:737] Address Info (0x807b7a6c0) [ai_flags: 1025, ai_family: 2, ai_socktype: 1, ai_protocol: 6, ai_addrlen: 16, ai_addr: 0x807b7a6f0, ai_canonname: (null), ai_next: 0x0]
00000016.00000c85 00002287 Thu Jun 15 2017 07:52:58 +00:00 [kern_notifyd:info:737] Successfully delivered noteto asup with sequence: 3206
00000016.00000c86 00002287 Thu Jun 15 2017 07:52:58 +00:00 [kern_notifyd:info:737] 0x8072d9e00: ERR: Autosupport::Delivery: created compressor with out_filename = /mroot/etc/log/autosupport/201706150742.0.smtp_tmp
00000016.00000cb5 00002287 Thu Jun 15 2017 07:53:00 +00:00 [kern_notifyd:info:737] start SNMP job for event monitor.globalStatus.nonCritical (929968) worked t=882
00000016.00000cb6 00002287 Thu Jun 15 2017 07:53:00 +00:00 [kern_notifyd:info:737] Address Info (0x807b7a700) [ai_flags: 1025, ai_family: 2, ai_socktype: 1, ai_protocol: 6, ai_addrlen: 16, ai_addr: 0x807b7a730, ai_canonname: (null), ai_next: 0x0]
00000016.00000cb9 00002297 Thu Jun 15 2017 07:53:02 +00:00 [kern_notifyd:info:737] Subject: HA Group Notification from PEKNAP-01 (CONTROLLER TAKEOVER COMPLETE AUTOMATIC) ERROR
00000016.00000cba 00002297 Thu Jun 15 2017 07:53:02 +00:00 [kern_notifyd:info:737] asup_job::run started t=884
00000016.00000cbb 00002297 Thu Jun 15 2017 07:53:02 +00:00 [kern_notifyd:info:737] asup_job::run is triggered by event callhome.spares.low (929946) t=884
00000016.00000cbc 00002297 Thu Jun 15 2017 07:53:02 +00:00 [kern_notifyd:info:737] asup_job::run calling smtp_file for 'SPARES_LOW'
00000016.00000cbd 00002297 Thu Jun 15 2017 07:53:02 +00:00 [kern_notifyd:info:737] Creating autosupport with sequence number 3212, event callhome.spares.low
00000016.00000cbe 00002297 Thu Jun 15 2017 07:53:02 +00:00 [kern_notifyd:info:737] Beginning collection for autosupport with seq num 3212
00000016.00000cbf 00002298 Thu Jun 15 2017 07:53:02 +00:00 [kern_notifyd:info:737] (category: 737:3212:collect) (emittime: 6/15/2017 07:53:01) (message: asup d-asup-start directory_path='/etc/log/autosupport/201706150753.0.files'  (complete))
00000016.00000cc0 00002298 Thu Jun 15 2017 07:53:02 +00:00 [kern_notifyd:info:737] asup_job::run smtp_file failed, requeue
00000016.00000cc1 00002298 Thu Jun 15 2017 07:53:02 +00:00 [kern_notifyd:info:737] (category: 737:3212:collect) (emittime: 6/15/2017 07:53:01) (message: asup ZAPI d-asup-start failed. Status: 1, Reason: AutoSupport Giveback in progress, Error: 13928)
00000016.00000cc6 000022cd Thu Jun 15 2017 07:53:05 +00:00 [kern_notifyd:info:737] start Health Dashboard job for event raid.vol.state.changed (929986) worked t=887
00000016.00000cc7 000022cd Thu Jun 15 2017 07:53:05 +00:00 [kern_notifyd:info:737] start SNMP job for event raid.vol.state.changed (929986) worked t=887
00000016.00000d14 000022d9 Thu Jun 15 2017 07:53:07 +00:00 [kern_notifyd:info:737] ems receiver for callhome: callhome.sfo.giveback
00000016.00000d15 000022d9 Thu Jun 15 2017 07:53:07 +00:00 [kern_notifyd:info:737] read callhome event: callhome.sfo.giveback
00000016.00000d17 000022d9 Thu Jun 15 2017 07:53:07 +00:00 [kern_notifyd:info:737] first level screening for callhome event: callhome.sfo.giveback (name callhome.sfo.giveback, src kernel)
00000016.00000d18 000022d9 Thu Jun 15 2017 07:53:07 +00:00 [kern_notifyd:info:737] ems_asup::handle for event callhome.sfo.giveback (930058) t=889
00000016.00000d19 000022d9 Thu Jun 15 2017 07:53:07 +00:00 [kern_notifyd:info:737] ems_asup::handle job for event callhome.sfo.giveback (930058) worked t=889
00000016.00000d1a 000022ee Thu Jun 15 2017 07:53:10 +00:00 [kern_notifyd:info:737] Successfully delivered smtp asup with sequence: 3206
00000016.00000d1b 000022ee Thu Jun 15 2017 07:53:10 +00:00 [kern_notifyd:info:737] 0x8072d9e00: ERR: Autosupport::Delivery: created compressor with out_filename = /mroot/etc/log/autosupport/201706150742.0.http_tmp
00000016.00000d1c 000022ee Thu Jun 15 2017 07:53:10 +00:00 [kern_notifyd:info:737] Successfully delivered http asup with sequence: 3206
00000016.00000d1d 000022ee Thu Jun 15 2017 07:53:10 +00:00 [kern_notifyd:info:737] Beginning to deliver asup with sequence number 3207, subject HA Group Notification from PEKNAP-01 (PARTNER REBOOT (CONTROLLER TAKEOVER)) WARNING
00000016.00000d1e 000022ee Thu Jun 15 2017 07:53:10 +00:00 [kern_notifyd:info:737] Successfully delivered noteto asup with sequence: 3207
00000016.00000d1f 000022ee Thu Jun 15 2017 07:53:10 +00:00 [kern_notifyd:info:737] 0x8072d9e00: ERR: Autosupport::Delivery: created compressor with out_filename = /mroot/etc/log/autosupport/201706150744.0.smtp_tmp
00000016.00000d20 00002316 Thu Jun 15 2017 07:53:14 +00:00 [kern_notifyd:info:737] Address Info (0x807b194c0) [ai_flags: 1025, ai_family: 2, ai_socktype: 1, ai_protocol: 6, ai_addrlen: 16, ai_addr: 0x807b194f0, ai_canonname: (null), ai_next: 0x0]
00000016.00000d21 00002323 Thu Jun 15 2017 07:53:16 +00:00 [kern_notifyd:info:737] Subject: HA Group Notification from PEKNAP-01 (PARTNER REBOOT (CONTROLLER TAKEOVER)) WARNING
00000016.00000d22 0000239c Thu Jun 15 2017 07:53:28 +00:00 [kern_notifyd:info:737] asup_job::run started t=910
00000016.00000d23 0000239c Thu Jun 15 2017 07:53:28 +00:00 [kern_notifyd:info:737] asup_job::run is triggered by event callhome.spares.low (929946) t=910
00000016.00000d24 0000239c Thu Jun 15 2017 07:53:28 +00:00 [kern_notifyd:info:737] asup_job::run calling smtp_file for 'SPARES_LOW'
00000016.00000d25 0000239c Thu Jun 15 2017 07:53:28 +00:00 [kern_notifyd:info:737] Creating autosupport with sequence number 3213, event callhome.spares.low
00000016.00000d26 0000239c Thu Jun 15 2017 07:53:28 +00:00 [kern_notifyd:info:737] Beginning collection for autosupport with seq num 3213
00000016.00000d27 000023a6 Thu Jun 15 2017 07:53:29 +00:00 [kern_notifyd:info:737] (category: 737:3213:collect) (emittime: 6/15/2017 07:53:27) (message: asup lstat '/mroot/etc/log/autosupport/201706150753.0.files' exists)
00000016.00000d28 000023a6 Thu Jun 15 2017 07:53:29 +00:00 [kern_notifyd:info:737] (category: 737:3213:collect) (emittime: 6/15/2017 07:53:28) (message: asup d-asup-start directory_path='/etc/log/autosupport/201706150753.1.files'  (complete))
00000016.00000d29 000023a6 Thu Jun 15 2017 07:53:29 +00:00 [kern_notifyd:info:737] asup_job::run smtp_file failed, requeue
00000016.00000d2a 000023a6 Thu Jun 15 2017 07:53:29 +00:00 [kern_notifyd:info:737] (category: 737:3213:collect) (emittime: 6/15/2017 07:53:28) (message: asup ZAPI d-asup-start failed. Status: 1, Reason: AutoSupport Giveback in progress, Error: 13928)
00000016.00000d2b 00002496 Thu Jun 15 2017 07:53:53 +00:00 [kern_notifyd:info:737] asup_job::run started t=935
00000016.00000d2c 00002496 Thu Jun 15 2017 07:53:53 +00:00 [kern_notifyd:info:737] asup_job::run is triggered by event callhome.spares.low (929946) t=935
00000016.00000d2d 00002496 Thu Jun 15 2017 07:53:53 +00:00 [kern_notifyd:info:737] asup_job::run calling smtp_file for 'SPARES_LOW'
00000016.00000d2e 00002496 Thu Jun 15 2017 07:53:53 +00:00 [kern_notifyd:info:737] Creating autosupport with sequence number 3214, event callhome.spares.low
00000016.00000d2f 00002496 Thu Jun 15 2017 07:53:53 +00:00 [kern_notifyd:info:737] Beginning collection for autosupport with seq num 3214
00000016.00000d32 000024b2 Thu Jun 15 2017 07:53:55 +00:00 [kern_notifyd:info:737] 0x8072d9f90: ERR: Autosupport::Collection: NotifyD collector: [AutoSupport_Custom_Content]    Target: custom_fx Subsys: log_files Type: basic Minimal: 0 Mode Flags: 15 Outfile: MESSAGES.gz Cmd: Custom function Query: messages   --  manifest_status = content-empty , error = This file has not changed since previously sent AutoSupport seq( 3206 )
00000016.00000d3b 000024b2 Thu Jun 15 2017 07:53:55 +00:00 [kern_notifyd:info:737] get name 'autosupport.content' privs 'anon,admin,advanced,diag,test'
00000016.00000d3c 000024b2 Thu Jun 15 2017 07:53:55 +00:00 [kern_notifyd:info:737] get name 'autosupport.doit' privs 'anon,admin,advanced,diag,test'
00000016.00000d3d 000024b2 Thu Jun 15 2017 07:53:55 +00:00 [kern_notifyd:info:737] do it get
00000016.00000d3e 000024b2 Thu Jun 15 2017 07:53:55 +00:00 [kern_notifyd:info:737] get name 'autosupport.enable' privs 'anon,admin,advanced,diag,test'
00000016.00000d3f 000024b2 Thu Jun 15 2017 07:53:55 +00:00 [kern_notifyd:info:737] get name 'autosupport.from' privs 'anon,admin,advanced,diag,test'
00000016.00000d40 000024b2 Thu Jun 15 2017 07:53:55 +00:00 [kern_notifyd:info:737] get name 'autosupport.local_collection' privs 'anon,admin,advanced,diag,test'
00000016.00000d41 000024b2 Thu Jun 15 2017 07:53:55 +00:00 [kern_notifyd:info:737] get name 'autosupport.mailhost' privs 'anon,admin,advanced,diag,test'
00000016.00000d42 000024b2 Thu Jun 15 2017 07:53:55 +00:00 [kern_notifyd:info:737] get name 'autosupport.max_http_size' privs 'anon,admin,advanced,diag,test'
00000016.00000d43 000024b2 Thu Jun 15 2017 07:53:55 +00:00 [kern_notifyd:info:737] get name 'autosupport.max_smtp_size' privs 'anon,admin,advanced,diag,test'
00000016.00000d44 000024b2 Thu Jun 15 2017 07:53:55 +00:00 [kern_notifyd:info:737] get name 'autosupport.minimal.subject.id' privs 'anon,admin,advanced,diag,test'
00000016.00000d45 000024b2 Thu Jun 15 2017 07:53:55 +00:00 [kern_notifyd:info:737] get name 'autosupport.nht_data.enable' privs 'anon,admin,advanced,diag,test'
00000016.00000d46 000024b2 Thu Jun 15 2017 07:53:55 +00:00 [kern_notifyd:info:737] get name 'autosupport.noteto' privs 'anon,admin,advanced,diag,test'
00000016.00000d47 000024b2 Thu Jun 15 2017 07:53:55 +00:00 [kern_notifyd:info:737] get name 'autosupport.partner.to' privs 'anon,admin,advanced,diag,test'
00000016.00000d48 000024b2 Thu Jun 15 2017 07:53:55 +00:00 [kern_notifyd:info:737] get name 'autosupport.payload_format' privs 'anon,admin,advanced,diag,test'
00000016.00000d49 000024b2 Thu Jun 15 2017 07:53:55 +00:00 [kern_notifyd:info:737] get name 'autosupport.performance_data.doit' privs 'anon,admin,advanced,diag,test'
00000016.00000d4a 000024b2 Thu Jun 15 2017 07:53:56 +00:00 [kern_notifyd:info:737] get name 'autosupport.performance_data.enable' privs 'anon,admin,advanced,diag,test'
00000016.00000d4b 000024b2 Thu Jun 15 2017 07:53:56 +00:00 [kern_notifyd:info:737] get name 'autosupport.periodic.tx_window' privs 'anon,admin,advanced,diag,test'
00000016.00000d4c 000024b2 Thu Jun 15 2017 07:53:56 +00:00 [kern_notifyd:info:737] get name 'autosupport.retry.count' privs 'anon,admin,advanced,diag,test'
00000016.00000d4d 000024b2 Thu Jun 15 2017 07:53:56 +00:00 [kern_notifyd:info:737] get name 'autosupport.retry.interval' privs 'anon,admin,advanced,diag,test'
00000016.00000d4e 000024b2 Thu Jun 15 2017 07:53:56 +00:00 [kern_notifyd:info:737] get name 'autosupport.support.enable' privs 'anon,admin,advanced,diag,test'
00000016.00000d4f 000024b2 Thu Jun 15 2017 07:53:56 +00:00 [kern_notifyd:info:737] get name 'autosupport.support.proxy' privs 'anon,admin,advanced,diag,test'
00000016.00000d50 000024b2 Thu Jun 15 2017 07:53:56 +00:00 [kern_notifyd:info:737] get name 'autosupport.support.put_url' privs 'anon,admin,advanced,diag,test'
00000016.00000d51 000024db Thu Jun 15 2017 07:53:59 +00:00 [kern_notifyd:info:737] get name 'autosupport.support.to' privs 'anon,admin,advanced,diag,test'
00000016.00000d52 000024db Thu Jun 15 2017 07:53:59 +00:00 [kern_notifyd:info:737] get name 'autosupport.support.transport' privs 'anon,admin,advanced,diag,test'
00000016.00000d53 000024db Thu Jun 15 2017 07:53:59 +00:00 [kern_notifyd:info:737] get name 'autosupport.support.url' privs 'anon,admin,advanced,diag,test'
00000016.00000d54 000024db Thu Jun 15 2017 07:53:59 +00:00 [kern_notifyd:info:737] get name 'autosupport.throttle' privs 'anon,admin,advanced,diag,test'
00000016.00000d55 000024db Thu Jun 15 2017 07:53:59 +00:00 [kern_notifyd:info:737] get name 'autosupport.to' privs 'anon,admin,advanced,diag,test'
00000016.00000d56 000024db Thu Jun 15 2017 07:53:59 +00:00 [kern_notifyd:info:737] get name 'autosupport.validate_digital_certificate' privs 'anon,admin,advanced,diag,test'
00000016.00000d57 000024db Thu Jun 15 2017 07:53:59 +00:00 [kern_notifyd:info:737] 0x8072d9f90: ERR: Autosupport::Collection: NotifyD collector: [AutoSupport_Custom_Content]    Target: custom_fx Subsys: log_files Type: troubleshooting Minimal: 0 Mode Flags: 15 Outfile: backup.gz Cmd: Custom function Query: backup   --  manifest_status = content-empty , error = This file has not changed since previously sent AutoSupport seq( 3205 )
00000016.00000d58 000024db Thu Jun 15 2017 07:53:59 +00:00 [kern_notifyd:info:737] 0x8072d9f90: ERR: Autosupport::Collection: NotifyD collector: [AutoSupport_Custom_Content]    Target: custom_fx Subsys: log_files Type: troubleshooting Minimal: 0 Mode Flags: 15 Outfile: sis-log.gz Cmd: Custom function Query: sis   --  manifest_status = content-empty , error = This file has not changed since previously sent AutoSupport seq( 3205 )
00000016.00000d59 000024db Thu Jun 15 2017 07:53:59 +00:00 [kern_notifyd:info:737] 0x8072d9f90: ERR: Autosupport::Collection: NotifyD collector: [AutoSupport_Custom_Content]    Target: custom_fx Subsys: log_files Type: troubleshooting Minimal: 0 Mode Flags: 15 Outfile: acplog.gz Cmd: Custom function Query: acplog   --  manifest_status = content-empty , error = This file has not changed since previously sent AutoSupport seq( 2708 )
00000016.00000d5a 000024db Thu Jun 15 2017 07:53:59 +00:00 [kern_notifyd:info:737] 0x8072d9f90: ERR: Autosupport::Collection: NotifyD collector: [AutoSupport_Custom_Content]    Target: custom_fx Subsys: log_files Type: troubleshooting Minimal: 0 Mode Flags: 15 Outfile: shelf-log-sas.gz Cmd: Custom function Query: shelflog_sas   --  manifest_status = content-empty , error = This file has not changed since previously sent AutoSupport seq( 0 )
00000016.00000d5b 000024db Thu Jun 15 2017 07:53:59 +00:00 [kern_notifyd:info:737] 0x8072d9f90: ERR: Autosupport::Collection: NotifyD collector: [AutoSupport_Custom_Content]    Target: custom_fx Subsys: log_files Type: troubleshooting Minimal: 0 Mode Flags: 15 Outfile: shelf-log-esh.gz Cmd: Custom function Query: shelflog_esh   --  manifest_status = content-empty , error = This file has not changed since previously sent AutoSupport seq( 0 )
00000016.00000d5c 000024db Thu Jun 15 2017 07:53:59 +00:00 [kern_notifyd:info:737] 0x8072d9f90: ERR: Autosupport::Collection: NotifyD collector: [AutoSupport_Custom_Content]    Target: custom_fx Subsys: log_files Type: troubleshooting Minimal: 0 Mode Flags: 15 Outfile: shelf-log-ata.gz Cmd: Custom function Query: shelflog_ata   --  manifest_status = content-empty , error = This file has not changed since previously sent AutoSupport seq( 0 )
00000016.00000d5d 000024db Thu Jun 15 2017 07:53:59 +00:00 [kern_notifyd:info:737] 0x8072d9f90: ERR: Autosupport::Collection: NotifyD collector: [AutoSupport_Custom_Content]    Target: custom_fx Subsys: log_files Type: troubleshooting Minimal: 0 Mode Flags: 15 Outfile: ndmpd.gz Cmd: Custom function Query: ndmpd   --  manifest_status = content-empty , error = This file has not changed since previously sent AutoSupport seq( 3205 )
00000016.00000d5e 000024db Thu Jun 15 2017 07:53:59 +00:00 [kern_notifyd:info:737] 0x8072d9f90: ERR: Autosupport::Collection: NotifyD collector: [AutoSupport_Custom_Content]    Target: custom_fx Subsys: log_files Type: troubleshooting Minimal: 0 Mode Flags: 15 Outfile: secd.gz Cmd: Custom function Query: secd   --  manifest_status = content-empty , error = This file has not changed since previously sent AutoSupport seq( 3206 )
00000016.00000d5f 000024db Thu Jun 15 2017 07:53:59 +00:00 [kern_notifyd:info:737] 0x8072d9f90: ERR: Autosupport::Collection: NotifyD collector: [AutoSupport_Custom_Content]    Target: custom_fx Subsys: log_files Type: troubleshooting Minimal: 0 Mode Flags: 15 Outfile: spmd.gz Cmd: Custom function Query: spmd   --  manifest_status = content-empty , error = This file has not changed since previously sent AutoSupport seq( 3205 )
00000016.00000d60 000024db Thu Jun 15 2017 07:53:59 +00:00 [kern_notifyd:info:737] 0x8072d9f90: ERR: Autosupport::Collection: NotifyD collector: [AutoSupport_Custom_Content]    Target: custom_fx Subsys: log_files Type: troubleshooting Minimal: 0 Mode Flags: 2 Outfile: apache_access.gz Cmd: Custom function Query: apache_access   --  manifest_status = content-empty , error = This file has not changed since previously sent AutoSupport seq( 3202 )
00000016.00000d61 000024db Thu Jun 15 2017 07:53:59 +00:00 [kern_notifyd:info:737] 0x8072d9f90: ERR: Autosupport::Collection: NotifyD collector: [AutoSupport_Custom_Content]    Target: custom_fx Subsys: log_files Type: troubleshooting Minimal: 0 Mode Flags: 2 Outfile: apache_error.gz Cmd: Custom function Query: apache_error   --  manifest_status = content-empty , error = This file has not changed since previously sent AutoSupport seq( 3205 )
00000016.00000d62 000024db Thu Jun 15 2017 07:53:59 +00:00 [kern_notifyd:info:737] 0x8072d9f90: ERR: Autosupport::Collection: NotifyD collector: [AutoSupport_File_Content]    Target: file Subsys: platform Type: troubleshooting Minimal: 0 Mode Flags: 15 Outfile: ssram.txt Cmd: /mroot/etc/log/ssram/ssram.log Query: 0   --  manifest_status = file-not-found , error = Cannot find AutoSupport source file /mroot/etc/log/ssram/ssram.log to collect for AutoSupport
00000016.00000d63 000024db Thu Jun 15 2017 07:53:59 +00:00 [kern_notifyd:info:737] 0x8072d9f90: ERR: Autosupport::Collection: NotifyD collector: [AutoSupport_File_Content]    Target: file Subsys: platform Type: troubleshooting Minimal: 0 Mode Flags: 15 Outfile: splog-subset-wdr.txt Cmd: /mroot/etc/log/splog_subset_wdr.txt Query: 0   --  manifest_status = file-not-found , error = Cannot find AutoSupport source file /mroot/etc/log/splog_subset_wdr.txt to collect for AutoSupport
00000016.00000d65 000024db Thu Jun 15 2017 07:54:00 +00:00 [kern_notifyd:info:737] start SNMP job for event monitor.globalStatus.ok (930059) worked t=942
00000016.00000d69 000025aa Thu Jun 15 2017 07:54:18 +00:00 [kern_notifyd:info:737] Successfully completed collection for autosupport with sequence number 3214
00000016.00000d6a 000025aa Thu Jun 15 2017 07:54:18 +00:00 [kern_notifyd:info:737] asup_job::run got '' for ASUP file
00000016.00000d6b 000025aa Thu Jun 15 2017 07:54:18 +00:00 [kern_notifyd:info:737] asup_job::run started t=961
00000016.00000d6c 000025aa Thu Jun 15 2017 07:54:18 +00:00 [kern_notifyd:info:737] asup_job::run is triggered by event callhome.sfo.giveback (930058) t=961
00000016.00000d6d 000025aa Thu Jun 15 2017 07:54:18 +00:00 [kern_notifyd:info:737] asup_job::run calling smtp_file for 'CONTROLLER GIVEBACK COMPLETE'
00000016.00000d6e 000025aa Thu Jun 15 2017 07:54:18 +00:00 [kern_notifyd:info:737] Creating autosupport with sequence number 3215, event callhome.sfo.giveback
00000016.00000d6f 000025aa Thu Jun 15 2017 07:54:18 +00:00 [kern_notifyd:info:737] Beginning collection for autosupport with seq num 3215
00000016.00000d72 000025aa Thu Jun 15 2017 07:54:18 +00:00 [kern_notifyd:info:737] 0x8072d9f90: ERR: Autosupport::Collection: NotifyD collector: [AutoSupport_Custom_Content]    Target: custom_fx Subsys: log_files Type: basic Minimal: 0 Mode Flags: 15 Outfile: MESSAGES.gz Cmd: Custom function Query: messages   --  manifest_status = content-empty , error = This file has not changed since previously sent AutoSupport seq( 3206 )
00000016.00000d7b 000025aa Thu Jun 15 2017 07:54:19 +00:00 [kern_notifyd:info:737] get name 'autosupport.content' privs 'anon,admin,advanced,diag,test'
00000016.00000d7c 000025aa Thu Jun 15 2017 07:54:19 +00:00 [kern_notifyd:info:737] get name 'autosupport.doit' privs 'anon,admin,advanced,diag,test'
00000016.00000d7d 000025aa Thu Jun 15 2017 07:54:19 +00:00 [kern_notifyd:info:737] do it get
00000016.00000d7e 000025aa Thu Jun 15 2017 07:54:19 +00:00 [kern_notifyd:info:737] get name 'autosupport.enable' privs 'anon,admin,advanced,diag,test'
00000016.00000d7f 000025aa Thu Jun 15 2017 07:54:19 +00:00 [kern_notifyd:info:737] get name 'autosupport.from' privs 'anon,admin,advanced,diag,test'
00000016.00000d80 000025aa Thu Jun 15 2017 07:54:19 +00:00 [kern_notifyd:info:737] get name 'autosupport.local_collection' privs 'anon,admin,advanced,diag,test'
00000016.00000d81 000025aa Thu Jun 15 2017 07:54:19 +00:00 [kern_notifyd:info:737] get name 'autosupport.mailhost' privs 'anon,admin,advanced,diag,test'
00000016.00000d82 000025aa Thu Jun 15 2017 07:54:19 +00:00 [kern_notifyd:info:737] get name 'autosupport.max_http_size' privs 'anon,admin,advanced,diag,test'
00000016.00000d83 000025aa Thu Jun 15 2017 07:54:19 +00:00 [kern_notifyd:info:737] get name 'autosupport.max_smtp_size' privs 'anon,admin,advanced,diag,test'
00000016.00000d84 000025aa Thu Jun 15 2017 07:54:19 +00:00 [kern_notifyd:info:737] get name 'autosupport.minimal.subject.id' privs 'anon,admin,advanced,diag,test'
00000016.00000d85 000025aa Thu Jun 15 2017 07:54:19 +00:00 [kern_notifyd:info:737] get name 'autosupport.nht_data.enable' privs 'anon,admin,advanced,diag,test'
00000016.00000d86 000025aa Thu Jun 15 2017 07:54:19 +00:00 [kern_notifyd:info:737] get name 'autosupport.noteto' privs 'anon,admin,advanced,diag,test'
00000016.00000d87 000025aa Thu Jun 15 2017 07:54:19 +00:00 [kern_notifyd:info:737] get name 'autosupport.partner.to' privs 'anon,admin,advanced,diag,test'
00000016.00000d88 000025aa Thu Jun 15 2017 07:54:19 +00:00 [kern_notifyd:info:737] get name 'autosupport.payload_format' privs 'anon,admin,advanced,diag,test'
00000016.00000d89 000025aa Thu Jun 15 2017 07:54:19 +00:00 [kern_notifyd:info:737] get name 'autosupport.performance_data.doit' privs 'anon,admin,advanced,diag,test'
00000016.00000d8a 000025aa Thu Jun 15 2017 07:54:19 +00:00 [kern_notifyd:info:737] get name 'autosupport.performance_data.enable' privs 'anon,admin,advanced,diag,test'
00000016.00000d8b 000025aa Thu Jun 15 2017 07:54:19 +00:00 [kern_notifyd:info:737] get name 'autosupport.periodic.tx_window' privs 'anon,admin,advanced,diag,test'
00000016.00000d8c 000025aa Thu Jun 15 2017 07:54:19 +00:00 [kern_notifyd:info:737] get name 'autosupport.retry.count' privs 'anon,admin,advanced,diag,test'
00000016.00000d8d 000025aa Thu Jun 15 2017 07:54:19 +00:00 [kern_notifyd:info:737] get name 'autosupport.retry.interval' privs 'anon,admin,advanced,diag,test'
00000016.00000d8e 000025aa Thu Jun 15 2017 07:54:19 +00:00 [kern_notifyd:info:737] get name 'autosupport.support.enable' privs 'anon,admin,advanced,diag,test'
00000016.00000d8f 000025aa Thu Jun 15 2017 07:54:19 +00:00 [kern_notifyd:info:737] get name 'autosupport.support.proxy' privs 'anon,admin,advanced,diag,test'
00000016.00000d90 000025aa Thu Jun 15 2017 07:54:19 +00:00 [kern_notifyd:info:737] get name 'autosupport.support.put_url' privs 'anon,admin,advanced,diag,test'
00000016.00000d91 000025aa Thu Jun 15 2017 07:54:19 +00:00 [kern_notifyd:info:737] get name 'autosupport.support.to' privs 'anon,admin,advanced,diag,test'
00000016.00000d92 000025aa Thu Jun 15 2017 07:54:19 +00:00 [kern_notifyd:info:737] get name 'autosupport.support.transport' privs 'anon,admin,advanced,diag,test'
00000016.00000d93 000025aa Thu Jun 15 2017 07:54:19 +00:00 [kern_notifyd:info:737] get name 'autosupport.support.url' privs 'anon,admin,advanced,diag,test'
00000016.00000d94 000025aa Thu Jun 15 2017 07:54:19 +00:00 [kern_notifyd:info:737] get name 'autosupport.throttle' privs 'anon,admin,advanced,diag,test'
00000016.00000d95 000025aa Thu Jun 15 2017 07:54:19 +00:00 [kern_notifyd:info:737] get name 'autosupport.to' privs 'anon,admin,advanced,diag,test'
00000016.00000d96 000025aa Thu Jun 15 2017 07:54:19 +00:00 [kern_notifyd:info:737] get name 'autosupport.validate_digital_certificate' privs 'anon,admin,advanced,diag,test'
00000016.00000da2 000025c8 Thu Jun 15 2017 07:54:23 +00:00 [kern_notifyd:info:737] 0x8072d9f90: ERR: Autosupport::Collection: asup[737:0] process_run for /usr/sbin/nperf: process exit status 255
00000016.00000dae 0000260a Thu Jun 15 2017 07:54:29 +00:00 [kern_notifyd:info:737] 0x8072d9f90: ERR: Autosupport::Collection: NotifyD collector: [AutoSupport_File_Content]    Target: file Subsys: wafl Type: basic Minimal: 0 Mode Flags: 15 Outfile: memstats.txt Cmd: /membufstats.log Query: 0   --  manifest_status = file-not-found , error = Cannot find AutoSupport source file /membufstats.log to collect for AutoSupport
00000016.00000daf 0000260a Thu Jun 15 2017 07:54:29 +00:00 [kern_notifyd:info:737] 0x8072d9f90: ERR: Autosupport::Collection: NotifyD collector: [AutoSupport_File_Content]    Target: file Subsys: platform Type: troubleshooting Minimal: 0 Mode Flags: 15 Outfile: ssram.txt Cmd: /mroot/etc/log/ssram/ssram.log Query: 0   --  manifest_status = file-not-found , error = Cannot find AutoSupport source file /mroot/etc/log/ssram/ssram.log to collect for AutoSupport
00000016.00000db0 0000260a Thu Jun 15 2017 07:54:29 +00:00 [kern_notifyd:info:737] 0x8072d9f90: ERR: Autosupport::Collection: NotifyD collector: [AutoSupport_File_Content]    Target: file Subsys: platform Type: troubleshooting Minimal: 0 Mode Flags: 15 Outfile: splog-subset-wdr.txt Cmd: /mroot/etc/log/splog_subset_wdr.txt Query: 0   --  manifest_status = file-not-found , error = Cannot find AutoSupport source file /mroot/etc/log/splog_subset_wdr.txt to collect for AutoSupport
00000016.00000dd3 0000265b Thu Jun 15 2017 07:54:37 +00:00 [kern_notifyd:info:737] Unable to modify manifest row due to subsystem size limit: asup_seqnum = 3215 , manifest_item = 219 , subsystem = san , size_collected = 15675721 , size_remaining = 2019628
00000016.00000dd4 0000265b Thu Jun 15 2017 07:54:37 +00:00 [kern_notifyd:info:737] 0x8072d9f90: ERR: Autosupport::Collection: NotifyD collector: [AutoSupport_Custom_Content]    Target: custom_fx Subsys: log_files Type: troubleshooting Minimal: 0 Mode Flags: 15 Outfile: backup.gz Cmd: Custom function Query: backup   --  manifest_status = content-empty , error = This file has not changed since previously sent AutoSupport seq( 3205 )
00000016.00000dd5 0000265b Thu Jun 15 2017 07:54:37 +00:00 [kern_notifyd:info:737] 0x8072d9f90: ERR: Autosupport::Collection: NotifyD collector: [AutoSupport_Custom_Content]    Target: custom_fx Subsys: log_files Type: troubleshooting Minimal: 0 Mode Flags: 15 Outfile: sis-log.gz Cmd: Custom function Query: sis   --  manifest_status = content-empty , error = This file has not changed since previously sent AutoSupport seq( 3205 )
00000016.00000dd6 0000265b Thu Jun 15 2017 07:54:37 +00:00 [kern_notifyd:info:737] 0x8072d9f90: ERR: Autosupport::Collection: NotifyD collector: [AutoSupport_Custom_Content]    Target: custom_fx Subsys: log_files Type: troubleshooting Minimal: 0 Mode Flags: 15 Outfile: acplog.gz Cmd: Custom function Query: acplog   --  manifest_status = content-empty , error = This file has not changed since previously sent AutoSupport seq( 2708 )
00000016.00000dd7 0000265b Thu Jun 15 2017 07:54:37 +00:00 [kern_notifyd:info:737] 0x8072d9f90: ERR: Autosupport::Collection: NotifyD collector: [AutoSupport_Custom_Content]    Target: custom_fx Subsys: log_files Type: troubleshooting Minimal: 0 Mode Flags: 15 Outfile: shelf-log-sas.gz Cmd: Custom function Query: shelflog_sas   --  manifest_status = content-empty , error = This file has not changed since previously sent AutoSupport seq( 0 )
00000016.00000dd8 0000265b Thu Jun 15 2017 07:54:37 +00:00 [kern_notifyd:info:737] 0x8072d9f90: ERR: Autosupport::Collection: NotifyD collector: [AutoSupport_Custom_Content]    Target: custom_fx Subsys: log_files Type: troubleshooting Minimal: 0 Mode Flags: 15 Outfile: shelf-log-esh.gz Cmd: Custom function Query: shelflog_esh   --  manifest_status = content-empty , error = This file has not changed since previously sent AutoSupport seq( 0 )
00000016.00000dd9 0000265b Thu Jun 15 2017 07:54:37 +00:00 [kern_notifyd:info:737] 0x8072d9f90: ERR: Autosupport::Collection: NotifyD collector: [AutoSupport_Custom_Content]    Target: custom_fx Subsys: log_files Type: troubleshooting Minimal: 0 Mode Flags: 15 Outfile: shelf-log-ata.gz Cmd: Custom function Query: shelflog_ata   --  manifest_status = content-empty , error = This file has not changed since previously sent AutoSupport seq( 0 )
00000016.00000dda 0000265b Thu Jun 15 2017 07:54:37 +00:00 [kern_notifyd:info:737] 0x8072d9f90: ERR: Autosupport::Collection: NotifyD collector: [AutoSupport_Custom_Content]    Target: custom_fx Subsys: log_files Type: troubleshooting Minimal: 0 Mode Flags: 15 Outfile: shelf-log-iom.gz Cmd: Custom function Query: shelflog_iom   --  manifest_status = content-empty , error = This file has not changed since previously sent AutoSupport seq( 3214 )
00000016.00000ddb 0000265b Thu Jun 15 2017 07:54:37 +00:00 [kern_notifyd:info:737] 0x8072d9f90: ERR: Autosupport::Collection: NotifyD collector: [AutoSupport_Custom_Content]    Target: custom_fx Subsys: log_files Type: troubleshooting Minimal: 0 Mode Flags: 2 Outfile: sktlogd.gz Cmd: Custom function Query: sktlogd   --  manifest_status = content-empty , error = This file has not changed since previously sent AutoSupport seq( 3214 )
00000016.00000ddc 0000265b Thu Jun 15 2017 07:54:37 +00:00 [kern_notifyd:info:737] 0x8072d9f90: ERR: Autosupport::Collection: NotifyD collector: [AutoSupport_Custom_Content]    Target: custom_fx Subsys: log_files Type: troubleshooting Minimal: 0 Mode Flags: 15 Outfile: ndmpd.gz Cmd: Custom function Query: ndmpd   --  manifest_status = content-empty , error = This file has not changed since previously sent AutoSupport seq( 3205 )
00000016.00000ddd 0000265b Thu Jun 15 2017 07:54:37 +00:00 [kern_notifyd:info:737] 0x8072d9f90: ERR: Autosupport::Collection: NotifyD collector: [AutoSupport_Custom_Content]    Target: custom_fx Subsys: log_files Type: troubleshooting Minimal: 0 Mode Flags: 15 Outfile: secd.gz Cmd: Custom function Query: secd   --  manifest_status = content-empty , error = This file has not changed since previously sent AutoSupport seq( 3206 )
00000016.00000dde 0000265b Thu Jun 15 2017 07:54:37 +00:00 [kern_notifyd:info:737] 0x8072d9f90: ERR: Autosupport::Collection: NotifyD collector: [AutoSupport_Custom_Content]    Target: custom_fx Subsys: log_files Type: troubleshooting Minimal: 0 Mode Flags: 15 Outfile: spmd.gz Cmd: Custom function Query: spmd   --  manifest_status = content-empty , error = This file has not changed since previously sent AutoSupport seq( 3205 )
00000016.00000ddf 0000265b Thu Jun 15 2017 07:54:37 +00:00 [kern_notifyd:info:737] 0x8072d9f90: ERR: Autosupport::Collection: NotifyD collector: [AutoSupport_Custom_Content]    Target: custom_fx Subsys: log_files Type: troubleshooting Minimal: 0 Mode Flags: 2 Outfile: vifmgr.gz Cmd: Custom function Query: vifmgr   --  manifest_status = content-empty , error = This file has not changed since previously sent AutoSupport seq( 3214 )
00000016.00000de0 0000265b Thu Jun 15 2017 07:54:37 +00:00 [kern_notifyd:info:737] 0x8072d9f90: ERR: Autosupport::Collection: NotifyD collector: [AutoSupport_Custom_Content]    Target: custom_fx Subsys: log_files Type: troubleshooting Minimal: 0 Mode Flags: 2 Outfile: vldb.gz Cmd: Custom function Query: vldb   --  manifest_status = content-empty , error = This file has not changed since previously sent AutoSupport seq( 3214 )
00000016.00000de1 0000265b Thu Jun 15 2017 07:54:37 +00:00 [kern_notifyd:info:737] 0x8072d9f90: ERR: Autosupport::Collection: NotifyD collector: [AutoSupport_Custom_Content]    Target: custom_fx Subsys: log_files Type: troubleshooting Minimal: 0 Mode Flags: 2 Outfile: bcomd.gz Cmd: Custom function Query: bcomd   --  manifest_status = content-empty , error = This file has not changed since previously sent AutoSupport seq( 3214 )
00000016.00000de2 0000265b Thu Jun 15 2017 07:54:37 +00:00 [kern_notifyd:info:737] 0x8072d9f90: ERR: Autosupport::Collection: NotifyD collector: [AutoSupport_Custom_Content]    Target: custom_fx Subsys: log_files Type: troubleshooting Minimal: 0 Mode Flags: 2 Outfile: apache_access.gz Cmd: Custom function Query: apache_access   --  manifest_status = content-empty , error = This file has not changed since previously sent AutoSupport seq( 3202 )
00000016.00000de3 0000265b Thu Jun 15 2017 07:54:37 +00:00 [kern_notifyd:info:737] 0x8072d9f90: ERR: Autosupport::Collection: NotifyD collector: [AutoSupport_Custom_Content]    Target: custom_fx Subsys: log_files Type: troubleshooting Minimal: 0 Mode Flags: 2 Outfile: apache_error.gz Cmd: Custom function Query: apache_error   --  manifest_status = content-empty , error = This file has not changed since previously sent AutoSupport seq( 3205 )
00000016.00000dea 00002691 Thu Jun 15 2017 07:54:43 +00:00 [kern_notifyd:info:737] Successfully completed collection for autosupport with sequence number 3215
00000016.00000deb 00002691 Thu Jun 15 2017 07:54:43 +00:00 [kern_notifyd:info:737] asup_job::run got '' for ASUP file
00000016.00000e1d 00005ddc Thu Jun 15 2017 08:18:19 +00:00 [kern_notifyd:info:737] start Health Dashboard job for event vifmgr.portup (930121) worked t=2401
00000016.00000e20 00005e04 Thu Jun 15 2017 08:18:23 +00:00 [kern_notifyd:info:737] start Health Dashboard job for event vifmgr.portup (930122) worked t=2405
00000016.00000e23 00005e04 Thu Jun 15 2017 08:18:23 +00:00 [kern_notifyd:info:737] start Health Dashboard job for event vifmgr.portup (930123) worked t=2405
00000017.00000003 00000551 Thu Jun 15 2017 08:57:09 +00:00 [user_notifyd:notice] SteadyUptime: CPU frequency: 2333427663  initialized at 134 (uptime at startup).
00000017.00000004 00000551 Thu Jun 15 2017 08:57:10 +00:00 [kern_notifyd:info:737] .------------------------------------------------------------------------------.
00000017.00000005 00000551 Thu Jun 15 2017 08:57:10 +00:00 [kern_notifyd:info:737]                    Loading notifyd at: Thu Jun 15 08:57:10 2017
00000017.00000006 00000551 Thu Jun 15 2017 08:57:10 +00:00 [kern_notifyd:info:737] .------------------------------------------------------------------------------.
00000017.00000007 00000551 Thu Jun 15 2017 08:57:10 +00:00 [kern_notifyd:info:737] Is_7_Mode: kenv for 'bootarg.init.boot_clustered' returned 'true', using 10-mode
00000017.00000008 00000551 Thu Jun 15 2017 08:57:10 +00:00 [kern_notifyd:info:737] isUpgrade(): OPTOUT Can't read bootarg.upgrade.
00000017.00000009 00000551 Thu Jun 15 2017 08:57:10 +00:00 [kern_notifyd:info:737] using program number: 536871426 for RPC server.
00000017.0000000a 00000551 Thu Jun 15 2017 08:57:10 +00:00 [kern_notifyd:info:737] using program number: 536873502 for dsmdb agent.
00000017.0000000b 00000551 Thu Jun 15 2017 08:57:10 +00:00 [kern_notifyd:info:737] using program number: 536873503 for dsmdb registry.
00000017.00000013 0000055c Thu Jun 15 2017 08:57:11 +00:00 [kern_notifyd:info:737] Initializing ZapiWrapper Library
00000017.00000014 0000055c Thu Jun 15 2017 08:57:11 +00:00 [kern_notifyd:info:737] Initializing ZapiConnectionCache
00000017.0000001b 00000598 Thu Jun 15 2017 08:57:17 +00:00 [kern_notifyd:info:737] na_zsm_socket_open(127.0.20.1): shttpc_connect() returns -1: err=60
00000017.0000001c 00000598 Thu Jun 15 2017 08:57:17 +00:00 [kern_notifyd:info:737] 0x8009020b0: ERR: ZapiConnection: registry-set: connection failed; reason='ZSM - Can't connect to host (60) (13055)' address=127.0.0.1:1,0:3:0:2:951:10:5
00000017.0000001d 00000598 Thu Jun 15 2017 08:57:17 +00:00 [kern_notifyd:info:737] 0x8009020b0: ERR: ZapiConnection: registry-set: failed to connect to '127.0.0.1:1,0:3:0:2:951:10:5' duration 6
00000017.0000001e 00000598 Thu Jun 15 2017 08:57:17 +00:00 [kern_notifyd:info:737] 0x8009020b0: ERR: AutoSupport::AsupConfig: Unable to set optout time in the registry, 0
00000017.0000001f 00000598 Thu Jun 15 2017 08:57:17 +00:00 [kern_notifyd:info:737] 0x8009020b0: ERR: AutoSupport::AsupConfig: OPTOUT:modify failed diable notify_imp
00000017.00000022 00000604 Thu Jun 15 2017 08:57:27 +00:00 [kern_notifyd:info:737] na_zsm_socket_open(127.0.20.1): shttpc_connect() returns -1: err=60
00000017.00000023 00000604 Thu Jun 15 2017 08:57:27 +00:00 [kern_notifyd:info:737] 0x80726b950: ERR: ZapiConnection: no proc: connection failed; reason='ZSM - Can't connect to host (60) (13055)' address=127.0.0.1:1,0:3:0:2:951:10:5
00000017.00000038 00000673 Thu Jun 15 2017 08:57:38 +00:00 [kern_notifyd:info:737] na_zsm_socket_open(127.0.20.1): shttpc_connect() returns -1: err=60
00000017.00000039 00000673 Thu Jun 15 2017 08:57:38 +00:00 [kern_notifyd:info:737] 0x80726b950: ERR: ZapiConnection: no proc: connection failed; reason='ZSM - Can't connect to host (60) (13055)' address=127.0.0.1:1,0:3:0:2:951:10:5
00000017.0000003a 00000673 Thu Jun 15 2017 08:57:39 +00:00 [kern_notifyd:info:737] EMS: first id=1497517022/1 for 'kern.cli.cmd'
00000017.0000003b 00000673 Thu Jun 15 2017 08:57:39 +00:00 [kern_notifyd:info:737] EMS kernel ORDER 1497517022/1 for 'kern.cli.cmd' (last 0/0)
00000017.00000041 000006d7 Thu Jun 15 2017 08:57:49 +00:00 [kern_notifyd:info:737] na_zsm_socket_open(127.0.20.1): shttpc_connect() returns -1: err=60
00000017.00000042 000006d7 Thu Jun 15 2017 08:57:49 +00:00 [kern_notifyd:info:737] 0x80726b950: ERR: ZapiConnection: no proc: connection failed; reason='ZSM - Can't connect to host (60) (13055)' address=127.0.0.1:1,0:3:0:2:951:10:5
00000017.00000043 00000747 Thu Jun 15 2017 08:58:00 +00:00 [kern_notifyd:info:737] na_zsm_socket_open(127.0.20.1): shttpc_connect() returns -1: err=60
00000017.00000044 00000747 Thu Jun 15 2017 08:58:00 +00:00 [kern_notifyd:info:737] 0x80726b950: ERR: ZapiConnection: no proc: connection failed; reason='ZSM - Can't connect to host (60) (13055)' address=127.0.0.1:1,0:3:0:2:951:10:5
00000017.00000064 000008a6 Thu Jun 15 2017 08:58:35 +00:00 [kern_notifyd:info:737] ems receiver for callhome: callhome.reboot.poweron
00000017.0000006d 000008cc Thu Jun 15 2017 08:58:37 +00:00 [kern_notifyd:info:737] EMS: pre-mroot first id=1497517022/1
00000017.0000006e 000008cc Thu Jun 15 2017 08:58:37 +00:00 [kern_notifyd:info:737] EMS: pre-mroot last id=1497517022/256
00000017.00000074 000008cc Thu Jun 15 2017 08:58:37 +00:00 [kern_notifyd:info:737] EMS manager 1.1

loganb
5,143 Views

please open a support ticket for further assistance.

 

 

looks like the notifyd process is wedged.

 

 

andris
5,160 Views
8.1.2 is very old ONTAP software. Upgrade to the latest recommended release for your FAS model.
Public