Sun May 14 00:00:00 BNT [filer02: kern.log.rotate:notice]: System filer02 (ID 0142227438) is running NetApp Release 7.3.2
Sun May 14 00:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 18 hours.
Sun May 14 00:00:30 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of nightly.0 snapshot (Snapshot operation failed: No space left on device.)
Sun May 14 00:00:39 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Sun May 14 00:05:04 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Sun May 14 00:09:30 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Sun May 14 00:13:25 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Sun May 14 00:17:54 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Sun May 14 00:37:40 BNT last message repeated 5 times
Sun May 14 00:41:22 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Sun May 14 00:57:43 BNT last message repeated 4 times
Sun May 14 00:57:43 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201705101200.0) for host (0) was not posted to NetApp. The system will drop the message
Sun May 14 01:00:00 BNT [filer02: kern.uptime.filer:info]:   1:00am up  7:30 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 84679 iSCSI ops
Sun May 14 01:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 17 hours.
Sun May 14 01:00:01 BNT [filer02: raid.rg.scrub.start:notice]: /aggr0/plex0/rg0: starting scrub
Sun May 14 02:00:00 BNT [filer02: kern.uptime.filer:info]:   2:00am up  8:30 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 84680 iSCSI ops
Sun May 14 02:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 16 hours.
Sun May 14 03:00:00 BNT [filer02: kern.uptime.filer:info]:   3:00am up  9:30 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 84680 iSCSI ops
Sun May 14 03:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 15 hours.
Sun May 14 03:00:15 BNT [filer02: raid.rg.scrub.done:notice]: /aggr0/plex0/rg0: scrub completed in 2:00:13.54
Sun May 14 03:00:15 BNT [filer02: raid.rg.scrub.summary.pi:notice]: Scrub found 0 parity inconsistencies in /aggr0/plex0/rg0.
Sun May 14 03:00:15 BNT [filer02: raid.rg.scrub.summary.cksum:notice]: Scrub found 0 checksum errors in /aggr0/plex0/rg0.
Sun May 14 03:00:15 BNT [filer02: raid.rg.scrub.summary.media:notice]: Scrub found 0 media errors in /aggr0/plex0/rg0.
Sun May 14 04:00:00 BNT [filer02: kern.uptime.filer:info]:   4:00am up 10:30 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 84680 iSCSI ops
Sun May 14 04:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 14 hours.
Sun May 14 05:00:00 BNT [filer02: kern.uptime.filer:info]:   5:00am up 11:30 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 84680 iSCSI ops
Sun May 14 05:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 13 hours.
Sun May 14 06:00:00 BNT [filer02: kern.uptime.filer:info]:   6:00am up 12:30 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 84680 iSCSI ops
Sun May 14 06:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 12 hours.
Sun May 14 07:00:00 BNT [filer02: kern.uptime.filer:info]:   7:00am up 13:30 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 84680 iSCSI ops
Sun May 14 07:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 11 hours.
Sun May 14 08:00:00 BNT [filer02: kern.uptime.filer:info]:   8:00am up 14:30 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 84680 iSCSI ops
Sun May 14 08:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 10 hours.
Sun May 14 08:00:16 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Sun May 14 08:00:28 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Sun May 14 08:04:24 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Sun May 14 08:08:23 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Sun May 14 08:12:44 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Sun May 14 08:16:56 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Sun May 14 08:35:51 BNT last message repeated 5 times
Sun May 14 08:40:06 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Sun May 14 08:56:26 BNT last message repeated 4 times
Sun May 14 08:56:26 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201705101600.0) for host (0) was not posted to NetApp. The system will drop the message
Sun May 14 09:00:00 BNT [filer02: kern.uptime.filer:info]:   9:00am up 15:30 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 84680 iSCSI ops
Sun May 14 09:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 9 hours.
Sun May 14 10:00:00 BNT [filer02: kern.uptime.filer:info]:  10:00am up 16:30 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 84680 iSCSI ops
Sun May 14 10:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 8 hours.
Sun May 14 11:00:00 BNT [filer02: kern.uptime.filer:info]:  11:00am up 17:30 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 84680 iSCSI ops
Sun May 14 11:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 7 hours.
Sun May 14 12:00:00 BNT [filer02: kern.uptime.filer:info]:  12:00pm up 18:30 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 84684 iSCSI ops
Sun May 14 12:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 6 hours.
Sun May 14 12:00:15 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Sun May 14 12:00:24 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Sun May 14 12:04:06 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Sun May 14 12:08:10 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Sun May 14 12:12:31 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Sun May 14 12:16:40 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Sun May 14 12:36:05 BNT last message repeated 5 times
Sun May 14 12:40:31 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Sun May 14 12:57:06 BNT last message repeated 4 times
Sun May 14 12:57:06 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201705102000.0) for host (0) was not posted to NetApp. The system will drop the message
Sun May 14 13:00:00 BNT [filer02: kern.uptime.filer:info]:   1:00pm up 19:30 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 84684 iSCSI ops
Sun May 14 13:00:01 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 5 hours.
Sun May 14 14:00:00 BNT [filer02: kern.uptime.filer:info]:   2:00pm up 20:31 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 84684 iSCSI ops
Sun May 14 14:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 4 hours.
Sun May 14 15:00:00 BNT [filer02: kern.uptime.filer:info]:   3:00pm up 21:31 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 84684 iSCSI ops
Sun May 14 15:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 3 hours.
Sun May 14 16:00:00 BNT [filer02: kern.uptime.filer:info]:   4:00pm up 22:31 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 84684 iSCSI ops
Sun May 14 16:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 2 hours.
Sun May 14 16:00:00 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Sun May 14 16:00:10 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Sun May 14 16:04:05 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Sun May 14 16:07:58 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Sun May 14 16:11:50 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Sun May 14 16:16:16 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Sun May 14 16:35:50 BNT last message repeated 5 times
Sun May 14 16:39:59 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Sun May 14 16:56:56 BNT last message repeated 4 times
Sun May 14 16:56:56 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201705110000.0) for host (0) was not posted to NetApp. The system will drop the message
Sun May 14 17:00:00 BNT [filer02: kern.uptime.filer:info]:   5:00pm up 23:31 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 84684 iSCSI ops
Sun May 14 17:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 1 hour.
Sun May 14 17:00:00 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] is still present in the system and should be removed.
Sun May 14 17:00:00 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] is still present in the system and should be removed.
Sun May 14 17:00:00 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] is still present in the system and should be removed.
Sun May 14 18:00:00 BNT [filer02: kern.uptime.filer:info]:   6:00pm up  1 day, 31 mins, 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 84684 iSCSI ops
Sun May 14 18:00:00 BNT [filer02: monitor.brokenDisk.notice:notice]: When two disks are broken in raid_dp volume,  the system shuts down automatically every 24  hours to encourage you to replace the disk.  If you reboot the system it will run for another  24 hours before shutting down. (The 24 hour timeout  may be increased by altering the "raid.timeout"  value using the "options" command.)
Sun May 14 18:00:00 BNT [filer02: monitor.shutdown.brokenDisk:EMERGENCY]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system now.
Sun May 14 18:00:03 BNT [filer02: cf.fsm.takeoverOfPartnerDisabled:notice]: Cluster monitor: takeover of filer01 disabled (local halt in progress)
Sun May 14 18:00:06 BNT [filer02: kern.shutdown:notice]: System shut down because : "BROKEN DISK".
Sun May 14 23:46:30 GMT [localhost: rc:notice]: The system was down for 49576 seconds
Mon May 15 07:46:05 BNT [filer02: cf.nm.nicTransitionUp:info]: replayed event: Interconnect link 0 is UP
Mon May 15 07:46:12 BNT [filer02: fci.initialization.failed:error]: replayed event: Initialization failed on Fibre Channel adapter 0b.
Mon May 15 07:46:14 BNT [filer02: nvram.battery.turned.on:info]: replayed event: The NVRAM battery is turned ON. It is turned OFF during system shutdown.
Mon May 15 07:46:17 BNT [filer02: diskown.isEnabled:info]: replayed event: software ownership has been enabled for this system
Mon May 15 07:46:20 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.2 is a local HA mailbox disk.
Mon May 15 07:46:20 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.0 is a local HA mailbox disk.
Mon May 15 07:46:20 BNT [filer02: fmmb.instStat.change:info]: replayed event: normal mailbox instance on local side.
Mon May 15 07:46:21 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.7 is a partner HA mailbox disk.
Mon May 15 07:46:21 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.3 is a partner HA mailbox disk.
Mon May 15 07:46:21 BNT [filer02: fmmb.instStat.change:info]: replayed event: normal mailbox instance on partner side.
Mon May 15 07:46:21 BNT [filer02: cf.fm.partner:info]: replayed event: Cluster monitor: partner 'filer01'
Mon May 15 07:46:21 BNT [filer02: cf.fm.timeMasterStatus:info]: replayed event: Acting as cluster time slave
Mon May 15 07:46:21 BNT [filer02: raid.assim.disk.brokenPreAssim:error]: replayed event: Broken Disk 0a.40 Shelf ? Bay ? [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] detected prior to assimilation. It should be removed.
Mon May 15 07:46:21 BNT [filer02: raid.assim.disk.brokenPreAssim:error]: replayed event: Broken Disk 0a.35 Shelf ? Bay ? [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] detected prior to assimilation. It should be removed.
Mon May 15 07:46:21 BNT [filer02: raid.assim.disk.brokenPreAssim:error]: replayed event: Broken Disk 0a.33 Shelf ? Bay ? [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] detected prior to assimilation. It should be removed.
Mon May 15 07:46:22 BNT [filer02: coredump.spare.none:info]: replayed event: No sparecore disk was found.
Mon May 15 07:46:23 BNT [filer02: raid.cksum.replay.summary:info]: replayed event: Replayed 0 checksum blocks.
Mon May 15 07:46:23 BNT [filer02: raid.stripe.replay.summary:info]: replayed event: Replayed 0 stripes.
Mon May 15 07:46:24 BNT [filer02: cf.fm.launch:info]: replayed event: Launching cluster monitor
Mon May 15 07:46:24 BNT [filer02: cf.fm.partner:info]: replayed event: Cluster monitor: partner 'filer01'
Mon May 15 07:46:25 BNT [filer02: cf.fsm.takeoverOfPartnerDisabled:notice]: replayed event: Cluster monitor: takeover of filer01 disabled (partner booting)
Mon May 15 07:46:29 BNT [filer02: fcp.service.adapter:warning]: replayed event: FCP started but no adapter is present!
Mon May 15 07:46:30 BNT [filer02: kern.syslog.msg:notice]: replayed event: The system was down for 49576 seconds
Mon May 15 07:46:31 BNT [filer02: disk.failmsg:error]: Disk 0a.40 (N00V0W8L): message received.
Mon May 15 07:46:31 BNT [filer02: disk.failmsg:error]: Disk 0a.35 (N00WLAUL): message received.
Mon May 15 07:46:31 BNT [filer02: disk.failmsg:error]: Disk 0a.33 (N00W75PL): message received.
Mon May 15 07:46:31 BNT [filer02: raid.disk.unload.done:info]: Unload of Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] has completed successfully
Mon May 15 07:46:31 BNT [filer02: raid.disk.unload.done:info]: Unload of Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] has completed successfully
Mon May 15 07:46:31 BNT [filer02: raid.disk.unload.done:info]: Unload of Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] has completed successfully
Mon May 15 07:46:31 BNT [filer02: raid.fdr.failed.ok:info]: Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] successfully deleted from spare pool
Mon May 15 07:46:31 BNT [filer02: raid.fdr.failed.ok:info]: Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] successfully deleted from spare pool
Mon May 15 07:46:31 BNT [filer02: raid.fdr.failed.ok:info]: Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] successfully deleted from spare pool
Mon May 15 07:46:31 BNT [filer02: dfu.firmwareUpToDate:info]: Firmware is up-to-date on all disk drives
Mon May 15 07:46:32 BNT [filer02: cf.fsm.takeoverOfPartnerEnabled:notice]: Cluster monitor: takeover of filer01 enabled
Mon May 15 07:46:32 BNT [filer02: cf.fsm.takeoverByPartnerEnabled:notice]: Cluster monitor: takeover of filer02 by filer01 enabled
Mon May 15 07:46:33 BNT [filer02: perf.archive.start:info]: Performance archiver started. Sampling 22 objects and 198 counters.
Mon May 15 07:46:35 BNT [filer02: netif.linkUp:info]: Ethernet e0b: Link up.
Mon May 15 07:46:35 BNT [filer02: netif.linkUp:info]: Ethernet e0a: Link up.
Mon May 15 07:46:36 BNT [filer02: pvif.switchLink:warning]: vif1: switching to e0a
Mon May 15 07:46:37 BNT [filer02: iscsi.service.startup:info]: iSCSI service startup
Mon May 15 07:46:38 BNT [filer02: cmds.sysconf.logErr:error]: sysconfig: Unless directed by NetApp Global Services the cf.takeover.on_panic option should be set to On. .
Mon May 15 07:46:38 BNT [filer02: raid.rg.recons.missing:notice]: RAID group /aggr1/plex0/rg0 is missing 2 disk(s).
Mon May 15 07:46:38 BNT [filer02: raid.rg.recons.cantStart:warning]: The reconstruction cannot start in RAID group /aggr1/plex0/rg0: No 520 bps disk of required size available in spare pool, targeting any  spare pool
Mon May 15 07:46:38 BNT [filer02: mgr.boot.disk_done:info]: NetApp Release 7.3.2 boot complete. Last disk update written at Sun May 14 18:00:08 BNT 2017
Mon May 15 07:46:38 BNT [filer02: monitor.raid.brokenDisk:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken.
Mon May 15 07:46:38 BNT [filer02: monitor.brokenDisk.notice:notice]: When two disks are broken in raid_dp volume,  the system shuts down automatically every 24  hours to encourage you to replace the disk.  If you reboot the system it will run for another  24 hours before shutting down. (The 24 hour timeout  may be increased by altering the "raid.timeout"  value using the "options" command.)
Mon May 15 07:46:38 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] is still present in the system and should be removed.
Mon May 15 07:46:38 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] is still present in the system and should be removed.
Mon May 15 07:46:38 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] is still present in the system and should be removed.
Mon May 15 07:46:38 BNT [filer02: shelf.config.spha:info]: System is using single path HA attached storage only.
Mon May 15 07:46:38 BNT [filer02: mgr.boot.reason_ok:notice]: System rebooted.
Mon May 15 07:46:39 BNT [filer02: iscsi.notice:notice]: ISCSI: New session from initiator iqn.1991-05.com.microsoft:pcb-722-ad1.pcbl722dom.local at IP addr 10.72.4.1
Mon May 15 07:46:52 BNT [filer02: diskown.errorReadingOwnership:warning]: error 3 (disk failed) while reading ownership on disk 0a.24 (S/N N00SSSAL)
Mon May 15 07:47:02 BNT [filer02: raid.rg.spares.low:warning]: /aggr1/plex0/rg0
Mon May 15 07:47:03 BNT [filer02: monitor.globalStatus.nonCritical:warning]: Disk on adapter 0a, shelf 2, bay 8, failed. Disk on adapter 0a, shelf 2, bay 3, failed. Disk on adapter 0a, shelf 2, bay 1, failed.  There are not enough spare disks. /vol/vol1 is full (using or reserving 100% of space and 0% of inodes, using 92% of reserve).
Mon May 15 07:47:03 BNT [filer02: nbt.nbns.registrationComplete:info]: NBT: All CIFS name registrations have completed for the local server.
Mon May 15 07:47:07 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Mon May 15 07:47:46 BNT last message repeated 5 times
Mon May 15 07:51:45 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Mon May 15 07:52:30 BNT [filer02: cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 0a.26 (DB493CEF:613B738F:C6775410:52ABD6E0:00000000:00000000:00000000:00000000:00000000:00000000) has recently changed or the node (filer01) is missing the disk.
Mon May 15 07:52:30 BNT [filer02: cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 0a.33 (016F3AB5:B475F4EC:10505E83:02202755:00000000:00000000:00000000:00000000:00000000:00000000) has recently changed or the node (filer02) is missing the disk.
Mon May 15 07:52:30 BNT [filer02: cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 0a.35 (9115483F:9DD43BC4:27172773:99F7D8B4:00000000:00000000:00000000:00000000:00000000:00000000) has recently changed or the node (filer02) is missing the disk.
Mon May 15 07:52:30 BNT [filer02: cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 0a.40 (072AEB65:C662A9CF:E58B9917:9BFA787B:00000000:00000000:00000000:00000000:00000000:00000000) has recently changed or the node (filer02) is missing the disk.
Mon May 15 07:52:30 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Mon May 15 07:52:36 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Mon May 15 07:57:18 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Mon May 15 08:00:00 BNT [filer02: kern.uptime.filer:info]:   8:00am up 13 mins, 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 2695 iSCSI ops
Mon May 15 08:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 24 hours.
Mon May 15 08:00:01 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Mon May 15 08:00:09 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Mon May 15 08:04:38 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Mon May 15 08:08:27 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Mon May 15 08:12:22 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Mon May 15 08:16:34 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Mon May 15 08:16:34 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201705110742.0) for host (0) was not posted to NetApp. The system will drop the message
Mon May 15 08:53:43 BNT [filer02: cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 0a.26 (DB493CEF:613B738F:C6775410:52ABD6E0:00000000:00000000:00000000:00000000:00000000:00000000) has recently changed or the node (filer01) is missing the disk.
Mon May 15 08:53:43 BNT [filer02: cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 0a.33 (016F3AB5:B475F4EC:10505E83:02202755:00000000:00000000:00000000:00000000:00000000:00000000) has recently changed or the node (filer02) is missing the disk.
Mon May 15 08:53:43 BNT [filer02: cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 0a.35 (9115483F:9DD43BC4:27172773:99F7D8B4:00000000:00000000:00000000:00000000:00000000:00000000) has recently changed or the node (filer02) is missing the disk.
Mon May 15 08:53:43 BNT [filer02: cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 0a.40 (072AEB65:C662A9CF:E58B9917:9BFA787B:00000000:00000000:00000000:00000000:00000000:00000000) has recently changed or the node (filer02) is missing the disk.
Mon May 15 09:00:00 BNT [filer02: kern.uptime.filer:info]:   9:00am up  1:13 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 2695 iSCSI ops
Mon May 15 09:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 23 hours.
Mon May 15 10:00:00 BNT [filer02: kern.uptime.filer:info]:  10:00am up  2:13 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 4879 iSCSI ops
Mon May 15 10:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 22 hours.
Mon May 15 11:00:00 BNT [filer02: kern.uptime.filer:info]:  11:00am up  3:13 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 9655 iSCSI ops
Mon May 15 11:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 21 hours.
Mon May 15 12:00:00 BNT [filer02: kern.uptime.filer:info]:  12:00pm up  4:13 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 11097 iSCSI ops
Mon May 15 12:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 20 hours.
Mon May 15 12:00:45 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Mon May 15 12:00:56 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SHUTDOWN PENDING (degraded mode))
Mon May 15 12:05:20 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SHUTDOWN PENDING (degraded mode))
Mon May 15 12:08:50 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SHUTDOWN PENDING (degraded mode))
Mon May 15 12:12:30 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SHUTDOWN PENDING (degraded mode))
Mon May 15 12:16:10 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SHUTDOWN PENDING (degraded mode))
Mon May 15 12:35:25 BNT last message repeated 5 times
Mon May 15 12:39:06 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SHUTDOWN PENDING (degraded mode))
Mon May 15 12:54:57 BNT last message repeated 4 times
Mon May 15 12:54:57 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201705110742.1) for host (0) was not posted to NetApp. The system will drop the message
Mon May 15 13:00:00 BNT [filer02: kern.uptime.filer:info]:   1:00pm up  5:13 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 11329 iSCSI ops
Mon May 15 13:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 19 hours.
Mon May 15 14:00:00 BNT [filer02: kern.uptime.filer:info]:   2:00pm up  6:13 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 48218 iSCSI ops
Mon May 15 14:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 18 hours.
Mon May 15 15:00:00 BNT [filer02: kern.uptime.filer:info]:   3:00pm up  7:13 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 50861 iSCSI ops
Mon May 15 15:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 17 hours.
Mon May 15 16:00:00 BNT [filer02: kern.uptime.filer:info]:   4:00pm up  8:13 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 52166 iSCSI ops
Mon May 15 16:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 16 hours.
Mon May 15 16:00:46 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Mon May 15 16:01:01 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (REBOOT (power on))
Mon May 15 16:04:49 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (REBOOT (power on))
Mon May 15 16:08:54 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (REBOOT (power on))
Mon May 15 16:12:40 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (REBOOT (power on))
Mon May 15 16:17:01 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (REBOOT (power on))
Mon May 15 16:33:27 BNT last message repeated 4 times
Mon May 15 16:37:55 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (REBOOT (power on))
Mon May 15 16:57:40 BNT last message repeated 5 times
Mon May 15 16:57:40 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201705110743.0) for host (0) was not posted to NetApp. The system will drop the message
Mon May 15 17:00:00 BNT [filer02: kern.uptime.filer:info]:   5:00pm up  9:13 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 54853 iSCSI ops
Mon May 15 17:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 15 hours.
Mon May 15 18:00:00 BNT [filer02: kern.uptime.filer:info]:   6:00pm up 10:13 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 167277 iSCSI ops
Mon May 15 18:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 14 hours.
Mon May 15 19:00:00 BNT [filer02: kern.uptime.filer:info]:   7:00pm up 11:13 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 207721 iSCSI ops
Mon May 15 19:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 13 hours.
Mon May 15 20:00:00 BNT [filer02: kern.uptime.filer:info]:   8:00pm up 12:13 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 326111 iSCSI ops
Mon May 15 20:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 12 hours.
Mon May 15 20:00:31 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Mon May 15 20:00:44 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SPARES_LOW)
Mon May 15 20:04:21 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SPARES_LOW)
Mon May 15 20:08:49 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SPARES_LOW)
Mon May 15 20:12:54 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SPARES_LOW)
Mon May 15 20:17:08 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SPARES_LOW)
Mon May 15 20:36:54 BNT last message repeated 5 times
Mon May 15 20:40:35 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SPARES_LOW)
Mon May 15 20:56:17 BNT last message repeated 4 times
Mon May 15 20:56:17 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201705110743.1) for host (0) was not posted to NetApp. The system will drop the message
Mon May 15 21:00:00 BNT [filer02: kern.uptime.filer:info]:   9:00pm up 13:13 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 365543 iSCSI ops
Mon May 15 21:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 11 hours.
Mon May 15 22:00:00 BNT [filer02: kern.uptime.filer:info]:  10:00pm up 14:13 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 477239 iSCSI ops
Mon May 15 22:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 10 hours.
Mon May 15 23:00:00 BNT [filer02: kern.uptime.filer:info]:  11:00pm up 15:13 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 520079 iSCSI ops
Mon May 15 23:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 9 hours.
Tue May 16 00:00:00 BNT [filer02: kern.uptime.filer:info]:  12:00am up 16:13 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 628384 iSCSI ops
Tue May 16 00:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 8 hours.
Tue May 16 00:00:31 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of nightly.0 snapshot (Snapshot operation failed: No space left on device.)
Tue May 16 00:00:36 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (PERFORMANCE SNAPSHOT)
Tue May 16 00:04:30 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (PERFORMANCE SNAPSHOT)
Tue May 16 00:08:47 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (PERFORMANCE SNAPSHOT)
Tue May 16 00:12:54 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (PERFORMANCE SNAPSHOT)
Tue May 16 00:16:37 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (PERFORMANCE SNAPSHOT)
Tue May 16 00:36:17 BNT last message repeated 5 times
Tue May 16 00:40:05 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (PERFORMANCE SNAPSHOT)
Tue May 16 00:54:53 BNT last message repeated 4 times
Tue May 16 00:54:53 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201705110743.2) for host (0) was not posted to NetApp. The system will drop the message
Tue May 16 01:00:00 BNT [filer02: kern.uptime.filer:info]:   1:00am up 17:13 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 671024 iSCSI ops
Tue May 16 01:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 7 hours.
Tue May 16 02:00:00 BNT [filer02: kern.uptime.filer:info]:   2:00am up 18:13 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 791216 iSCSI ops
Tue May 16 02:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 6 hours.
Tue May 16 03:00:00 BNT [filer02: kern.uptime.filer:info]:   3:00am up 19:13 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 830704 iSCSI ops
Tue May 16 03:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 5 hours.
Tue May 16 04:00:00 BNT [filer02: kern.uptime.filer:info]:   4:00am up 20:13 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 949979 iSCSI ops
Tue May 16 04:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 4 hours.
Tue May 16 05:00:00 BNT [filer02: kern.uptime.filer:info]:   5:00am up 21:13 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 989369 iSCSI ops
Tue May 16 05:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 3 hours.
Tue May 16 06:00:00 BNT [filer02: kern.uptime.filer:info]:   6:00am up 22:13 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 1095798 iSCSI ops
Tue May 16 06:00:01 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 2 hours.
Tue May 16 07:00:00 BNT [filer02: kern.uptime.filer:info]:   7:00am up 23:13 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 1138383 iSCSI ops
Tue May 16 07:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 1 hour.
Tue May 16 07:00:00 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] is still present in the system and should be removed.
Tue May 16 07:00:00 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] is still present in the system and should be removed.
Tue May 16 07:00:00 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] is still present in the system and should be removed.
Tue May 16 07:30:59 BNT [filer02: ses.status.psWarning:warning]: DS14-Mk2-AT shelf 2 on channel 0a power warning for Power supply 2: non-critical status; DC undervoltage fault. This module is on the rear side of the shelf, at the right.
Tue May 16 07:31:02 BNT [filer02: fci.link.break:error]: Link break detected on Fibre Channel adapter 0a.
Mon May 15 23:31:50 GMT [localhost: rc:notice]: The system was down for 20 seconds
Mon May 15 23:31:51 GMT [filer02: cf.fsm.takeoverByPartnerEnabled:notice]: Cluster monitor: takeover of filer02 by filer01 enabled
Tue May 16 07:31:10 BNT [filer02: cf.nm.nicTransitionUp:info]: replayed event: Interconnect link 0 is UP
Tue May 16 07:31:17 BNT [filer02: fci.initialization.failed:error]: replayed event: Initialization failed on Fibre Channel adapter 0b.
Tue May 16 07:31:19 BNT [filer02: nvram.battery.turned.on:info]: replayed event: The NVRAM battery is turned ON. It is turned OFF during system shutdown.
Tue May 16 07:31:22 BNT [filer02: diskown.isEnabled:info]: replayed event: software ownership has been enabled for this system
Tue May 16 07:31:26 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.2 is a local HA mailbox disk.
Tue May 16 07:31:26 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.0 is a local HA mailbox disk.
Tue May 16 07:31:26 BNT [filer02: fmmb.instStat.change:info]: replayed event: normal mailbox instance on local side.
Tue May 16 07:31:26 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.7 is a partner HA mailbox disk.
Tue May 16 07:31:26 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.3 is a partner HA mailbox disk.
Tue May 16 07:31:26 BNT [filer02: fmmb.instStat.change:info]: replayed event: normal mailbox instance on partner side.
Tue May 16 07:31:27 BNT [filer02: cf.fm.partner:info]: replayed event: Cluster monitor: partner 'filer01'
Tue May 16 07:31:27 BNT [filer02: cf.fm.timeMasterStatus:info]: replayed event: Acting as cluster time slave
Tue May 16 07:31:27 BNT [filer02: raid.assim.disk.brokenPreAssim:error]: replayed event: Broken Disk 0a.40 Shelf ? Bay ? [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] detected prior to assimilation. It should be removed.
Tue May 16 07:31:27 BNT [filer02: raid.assim.disk.brokenPreAssim:error]: replayed event: Broken Disk 0a.35 Shelf ? Bay ? [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] detected prior to assimilation. It should be removed.
Tue May 16 07:31:27 BNT [filer02: raid.assim.disk.brokenPreAssim:error]: replayed event: Broken Disk 0a.33 Shelf ? Bay ? [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] detected prior to assimilation. It should be removed.
Tue May 16 07:31:28 BNT [filer02: coredump.spare.none:info]: replayed event: No sparecore disk was found.
Tue May 16 07:31:28 BNT [filer02: raid.vol.replay.nvram:info]: replayed event: Performing raid replay on volume(s)
Tue May 16 07:31:28 BNT [filer02: raid.cksum.replay.summary:info]: replayed event: Replayed 0 checksum blocks.
Tue May 16 07:31:28 BNT [filer02: raid.stripe.replay.summary:info]: replayed event: Replayed 0 stripes.
Tue May 16 07:31:31 BNT [filer02: ses.status.psWarning:warning]: replayed event: DS12-ESAS shelf 0 on channel 0c power warning for Power supply 1: non-critical status; DC undervoltage fault. This module is on the rear side of the shelf, at the left.
Tue May 16 07:31:31 BNT [filer02: ses.status.psWarning:warning]: replayed event: DS12-ESAS shelf 0 on channel 0c power warning for Power supply 2: non-critical status; DC undervoltage fault. This module is on the rear side of the shelf, at the right.
Tue May 16 07:31:40 BNT [filer02: cf.fm.launch:info]: replayed event: Launching cluster monitor
Tue May 16 07:31:40 BNT [filer02: cf.fm.partner:info]: replayed event: Cluster monitor: partner 'filer01'
Tue May 16 07:31:47 BNT [filer02: cf.fsm.takeoverOfPartnerDisabled:notice]: replayed event: Cluster monitor: takeover of filer01 disabled (partner booting)
Tue May 16 07:31:48 BNT [filer02: cf.fsm.takeoverByPartnerEnabled:notice]: replayed event: Cluster monitor: takeover of  by filer01 enabled
Tue May 16 07:31:49 BNT [filer02: cf.fsm.takeoverOfPartnerEnabled:notice]: replayed event: Cluster monitor: takeover of filer01 enabled
Tue May 16 07:31:50 BNT [filer02: fcp.service.adapter:warning]: replayed event: FCP started but no adapter is present!
Tue May 16 07:31:50 BNT [filer02: cf.fsm.takeoverByPartnerDisabled:notice]: replayed event: Cluster monitor: takeover of  by filer01 disabled (partner booting)
Tue May 16 07:31:50 BNT [filer02: kern.syslog.msg:notice]: replayed event: The system was down for 20 seconds
Tue May 16 07:31:51 BNT [filer02: cf.fsm.takeoverByPartnerEnabled:notice]: replayed event: Cluster monitor: takeover of filer02 by filer01 enabled
Tue May 16 07:31:52 BNT [filer02: disk.failmsg:error]: Disk 0a.40 (N00V0W8L): message received.
Tue May 16 07:31:52 BNT [filer02: disk.failmsg:error]: Disk 0a.35 (N00WLAUL): message received.
Tue May 16 07:31:52 BNT [filer02: disk.failmsg:error]: Disk 0a.33 (N00W75PL): message received.
Tue May 16 07:31:52 BNT [filer02: raid.disk.unload.done:info]: Unload of Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] has completed successfully
Tue May 16 07:31:52 BNT [filer02: raid.disk.unload.done:info]: Unload of Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] has completed successfully
Tue May 16 07:31:52 BNT [filer02: raid.disk.unload.done:info]: Unload of Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] has completed successfully
Tue May 16 07:31:52 BNT [filer02: raid.fdr.failed.ok:info]: Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] successfully deleted from spare pool
Tue May 16 07:31:52 BNT [filer02: raid.fdr.failed.ok:info]: Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] successfully deleted from spare pool
Tue May 16 07:31:52 BNT [filer02: raid.fdr.failed.ok:info]: Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] successfully deleted from spare pool
Tue May 16 07:31:53 BNT [filer02: dfu.firmwareUpToDate:info]: Firmware is up-to-date on all disk drives
Tue May 16 07:31:54 BNT [filer02: perf.archive.start:info]: Performance archiver started. Sampling 22 objects and 198 counters.
Tue May 16 07:31:56 BNT [filer02: netif.linkUp:info]: Ethernet e0a: Link up.
Tue May 16 07:31:56 BNT [filer02: netif.linkUp:info]: Ethernet e0b: Link up.
Tue May 16 07:31:57 BNT [filer02: pvif.switchLink:warning]: vif1: switching to e0a
Tue May 16 07:31:58 BNT [filer02: rc:info]: relog syslog Tue May 16 07:30:59 BNT [filer02: ses.status.psWarning:warning]: DS14-Mk2-AT shelf 2 on channel 0a power warning for P
Tue May 16 07:31:58 BNT [filer02: rc:info]: relog syslog Tue May 16 07:31:02 BNT [filer02: fci.link.break:error]: Link break detected on Fibre Channel adapter 0a.
Tue May 16 07:31:58 BNT [filer02: iscsi.service.startup:info]: iSCSI service startup
Tue May 16 07:31:58 BNT [filer02: iscsi.notice:notice]: ISCSI: New session from initiator iqn.1991-05.com.microsoft:pcb-722-ad1.pcbl722dom.local at IP addr 10.72.4.1
Tue May 16 07:31:59 BNT [filer02: netif.linkDown:info]: Ethernet e0b: Link down, check cable.
Tue May 16 07:31:59 BNT [filer02: cmds.sysconf.logErr:error]: sysconfig: Unless directed by NetApp Global Services the cf.takeover.on_panic option should be set to On. .
Tue May 16 07:31:59 BNT [filer02: raid.rg.recons.missing:notice]: RAID group /aggr1/plex0/rg0 is missing 2 disk(s).
Tue May 16 07:31:59 BNT [filer02: raid.rg.recons.cantStart:warning]: The reconstruction cannot start in RAID group /aggr1/plex0/rg0: No 520 bps disk of required size available in spare pool, targeting any  spare pool
Tue May 16 07:31:59 BNT [filer02: mgr.boot.disk_done:info]: NetApp Release 7.3.2 boot complete. Last disk update written at Tue May 16 07:30:50 BNT 2017
Tue May 16 07:31:59 BNT [filer02: monitor.raid.brokenDisk:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken.
Tue May 16 07:31:59 BNT [filer02: monitor.brokenDisk.notice:notice]: When two disks are broken in raid_dp volume,  the system shuts down automatically every 24  hours to encourage you to replace the disk.  If you reboot the system it will run for another  24 hours before shutting down. (The 24 hour timeout  may be increased by altering the "raid.timeout"  value using the "options" command.)
Tue May 16 07:31:59 BNT [filer02: diskown.errorReadingOwnership:warning]: error 3 (disk failed) while reading ownership on disk 0a.24 (S/N N00SSSAL)
Tue May 16 07:31:59 BNT [filer02: diskown.errorReadingOwnership:warning]: error 3 (disk failed) while reading ownership on disk 0a.26 (S/N N00V0UWL)
Tue May 16 07:32:00 BNT [filer02: shelf.config.spha:info]: System is using single path HA attached storage only.
Tue May 16 07:32:00 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] is still present in the system and should be removed.
Tue May 16 07:32:00 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] is still present in the system and should be removed.
Tue May 16 07:32:00 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] is still present in the system and should be removed.
Tue May 16 07:32:00 BNT [filer02: raid.rg.spares.low:warning]: /aggr1/plex0/rg0
Tue May 16 07:32:00 BNT [filer02: mgr.boot.reason_ok:notice]: System rebooted after power-on.
Tue May 16 07:32:01 BNT [filer02: netif.linkUp:info]: Ethernet e0b: Link up.
Tue May 16 07:32:13 BNT [filer02: monitor.globalStatus.nonCritical:warning]: Disk on adapter 0a, shelf 2, bay 8, failed. Disk on adapter 0a, shelf 2, bay 3, failed. Disk on adapter 0a, shelf 2, bay 1, failed.  There are not enough spare disks. /vol/vol1 is full (using or reserving 100% of space and 0% of inodes, using 92% of reserve).
Tue May 16 07:32:25 BNT [filer02: nbt.nbns.registrationComplete:info]: NBT: All CIFS name registrations have completed for the local server.
Tue May 16 07:32:26 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Tue May 16 07:32:33 BNT last message repeated 2 times
Tue May 16 07:32:33 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201705110800.0) was not posted for host (0), it was dropped from queue due to lack of space
Tue May 16 07:32:40 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Tue May 16 07:32:40 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201705111200.0) was not posted for host (0), it was dropped from queue due to lack of space
Tue May 16 07:32:47 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Tue May 16 07:32:47 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201705111600.0) was not posted for host (0), it was dropped from queue due to lack of space
Tue May 16 07:33:08 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Tue May 16 07:33:08 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201705112000.0) was not posted for host (0), it was dropped from queue due to lack of space
Tue May 16 07:37:50 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Tue May 16 07:42:18 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Tue May 16 07:45:56 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Tue May 16 07:49:45 BNT [filer02: asup.post.host:info]: Autosupport (SCHEDULED SNAPSHOT CREATE FAILED) cannot connect to url support.netapp.com/asupprod/post/1.0/postAsup (Could not find hostname 'support.netapp.com', hostname lookup resolution error: Host name lookup failure)
Tue May 16 07:53:41 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Tue May 16 07:58:01 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Tue May 16 08:00:00 BNT [filer02: kern.uptime.filer:info]:   8:00am up 28 mins, 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 3712 iSCSI ops
Tue May 16 08:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 24 hours.
Tue May 16 08:00:01 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Tue May 16 08:00:16 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Tue May 16 08:00:16 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201705120000.0) was not posted for host (0), it was dropped from queue due to lack of space
Tue May 16 08:04:04 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Tue May 16 08:08:24 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Tue May 16 08:08:24 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201705120746.0) for host (0) was not posted to NetApp. The system will drop the message
Tue May 16 09:00:00 BNT [filer02: kern.uptime.filer:info]:   9:00am up  1:28 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 7772 iSCSI ops
Tue May 16 09:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 23 hours.
Tue May 16 10:00:00 BNT [filer02: kern.uptime.filer:info]:  10:00am up  2:28 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 40094 iSCSI ops
Tue May 16 10:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 22 hours.
Tue May 16 11:00:00 BNT [filer02: kern.uptime.filer:info]:  11:00am up  3:28 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 120786 iSCSI ops
Tue May 16 11:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 21 hours.
Tue May 16 12:00:00 BNT [filer02: kern.uptime.filer:info]:  12:00pm up  4:28 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 195121 iSCSI ops
Tue May 16 12:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 20 hours.
Tue May 16 12:00:01 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Tue May 16 12:00:15 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SHUTDOWN PENDING (degraded mode))
Tue May 16 12:03:47 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SHUTDOWN PENDING (degraded mode))
Tue May 16 12:08:11 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SHUTDOWN PENDING (degraded mode))
Tue May 16 12:12:34 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SHUTDOWN PENDING (degraded mode))
Tue May 16 12:16:12 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SHUTDOWN PENDING (degraded mode))
Tue May 16 12:36:24 BNT last message repeated 5 times
Tue May 16 12:40:19 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SHUTDOWN PENDING (degraded mode))
Tue May 16 12:56:52 BNT last message repeated 4 times
Tue May 16 12:56:52 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201705120747.0) for host (0) was not posted to NetApp. The system will drop the message
Tue May 16 13:00:00 BNT [filer02: kern.uptime.filer:info]:   1:00pm up  5:28 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 195123 iSCSI ops
Tue May 16 13:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 19 hours.
Tue May 16 14:00:00 BNT [filer02: kern.uptime.filer:info]:   2:00pm up  6:28 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 198493 iSCSI ops
Tue May 16 14:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 18 hours.
Tue May 16 15:00:00 BNT [filer02: kern.uptime.filer:info]:   3:00pm up  7:28 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 212564 iSCSI ops
Tue May 16 15:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 17 hours.
Tue May 16 16:00:00 BNT [filer02: kern.uptime.filer:info]:   4:00pm up  8:28 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 232478 iSCSI ops
Tue May 16 16:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 16 hours.
Tue May 16 16:00:46 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Tue May 16 16:01:00 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (REBOOT (power on))
Tue May 16 16:05:18 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (REBOOT (power on))
Tue May 16 16:09:36 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (REBOOT (power on))
Tue May 16 16:13:10 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (REBOOT (power on))
Tue May 16 16:16:45 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (REBOOT (power on))
Tue May 16 16:36:45 BNT last message repeated 5 times
Tue May 16 16:40:30 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (REBOOT (power on))
Tue May 16 16:56:45 BNT last message repeated 4 times
Tue May 16 16:56:45 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201705120747.1) for host (0) was not posted to NetApp. The system will drop the message
Tue May 16 17:00:00 BNT [filer02: kern.uptime.filer:info]:   5:00pm up  9:28 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 362221 iSCSI ops
Tue May 16 17:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 15 hours.
Tue May 16 18:00:00 BNT [filer02: kern.uptime.filer:info]:   6:00pm up 10:28 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 364972 iSCSI ops
Tue May 16 18:00:01 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 14 hours.
Tue May 16 19:00:00 BNT [filer02: kern.uptime.filer:info]:   7:00pm up 11:28 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 365220 iSCSI ops
Tue May 16 19:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 13 hours.
Tue May 16 20:00:00 BNT [filer02: kern.uptime.filer:info]:   8:00pm up 12:28 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 365220 iSCSI ops
Tue May 16 20:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 12 hours.
Tue May 16 20:00:31 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Tue May 16 20:00:45 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SPARES_LOW)
Tue May 16 20:04:42 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SPARES_LOW)
Tue May 16 20:08:57 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SPARES_LOW)
Tue May 16 20:13:00 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SPARES_LOW)
Tue May 16 20:16:40 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SPARES_LOW)
Tue May 16 20:35:16 BNT last message repeated 5 times
Tue May 16 20:39:17 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SPARES_LOW)
Tue May 16 20:54:41 BNT last message repeated 4 times
Tue May 16 20:54:41 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201705120747.2) for host (0) was not posted to NetApp. The system will drop the message
Tue May 16 21:00:00 BNT [filer02: kern.uptime.filer:info]:   9:00pm up 13:28 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 365223 iSCSI ops
Tue May 16 21:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 11 hours.
Tue May 16 22:00:00 BNT [filer02: kern.uptime.filer:info]:  10:00pm up 14:28 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 365223 iSCSI ops
Tue May 16 22:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 10 hours.
Tue May 16 23:00:00 BNT [filer02: kern.uptime.filer:info]:  11:00pm up 15:28 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 365223 iSCSI ops
Tue May 16 23:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 9 hours.
Wed May 17 00:00:00 BNT [filer02: kern.uptime.filer:info]:  12:00am up 16:28 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 365223 iSCSI ops
Wed May 17 00:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 8 hours.
Wed May 17 00:00:30 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of nightly.0 snapshot (Snapshot operation failed: No space left on device.)
Wed May 17 00:00:40 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (PERFORMANCE SNAPSHOT)
Wed May 17 00:05:00 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (PERFORMANCE SNAPSHOT)
Wed May 17 00:08:55 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (PERFORMANCE SNAPSHOT)
Wed May 17 00:12:35 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (PERFORMANCE SNAPSHOT)
Wed May 17 00:16:42 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (PERFORMANCE SNAPSHOT)
Wed May 17 00:36:49 BNT last message repeated 5 times
Wed May 17 00:40:54 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (PERFORMANCE SNAPSHOT)
Wed May 17 00:57:05 BNT last message repeated 4 times
Wed May 17 00:57:05 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201705120747.3) for host (0) was not posted to NetApp. The system will drop the message
Wed May 17 01:00:00 BNT [filer02: kern.uptime.filer:info]:   1:00am up 17:28 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 365223 iSCSI ops
Wed May 17 01:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 7 hours.
Wed May 17 02:00:00 BNT [filer02: kern.uptime.filer:info]:   2:00am up 18:28 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 365223 iSCSI ops
Wed May 17 02:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 6 hours.
Wed May 17 03:00:00 BNT [filer02: kern.uptime.filer:info]:   3:00am up 19:28 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 365223 iSCSI ops
Wed May 17 03:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 5 hours.
Wed May 17 04:00:00 BNT [filer02: kern.uptime.filer:info]:   4:00am up 20:28 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 365223 iSCSI ops
Wed May 17 04:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 4 hours.
Wed May 17 05:00:00 BNT [filer02: kern.uptime.filer:info]:   5:00am up 21:28 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 365223 iSCSI ops
Wed May 17 05:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 3 hours.
Wed May 17 06:00:00 BNT [filer02: kern.uptime.filer:info]:   6:00am up 22:28 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 365223 iSCSI ops
Wed May 17 06:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 2 hours.
Wed May 17 07:00:00 BNT [filer02: kern.uptime.filer:info]:   7:00am up 23:28 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 365223 iSCSI ops
Wed May 17 07:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 1 hour.
Wed May 17 07:00:00 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] is still present in the system and should be removed.
Wed May 17 07:00:00 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] is still present in the system and should be removed.
Wed May 17 07:00:00 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] is still present in the system and should be removed.
Wed May 17 07:32:23 BNT [filer02: fci.link.break:error]: Link break detected on Fibre Channel adapter 0a.
Wed May 17 07:32:29 BNT [filer02: fci.adapter.error:warning]: Fibre Channel adapter driver encountered error "Failed to get loop map" on adapter 0a.
Tue May 16 23:33:07 GMT [localhost: rc:notice]: The system was down for 19 seconds
Tue May 16 23:33:07 GMT [filer02: cf.fsm.takeoverOfPartnerEnabled:notice]: Cluster monitor: takeover of filer01 enabled
Wed May 17 07:32:31 BNT [filer02: cf.nm.nicTransitionUp:info]: replayed event: Interconnect link 0 is UP
Wed May 17 07:32:38 BNT [filer02: fci.initialization.failed:error]: replayed event: Initialization failed on Fibre Channel adapter 0b.
Wed May 17 07:32:40 BNT [filer02: nvram.battery.turned.on:info]: replayed event: The NVRAM battery is turned ON. It is turned OFF during system shutdown.
Wed May 17 07:33:08 BNT [filer02: cf.fsm.takeoverByPartnerEnabled:notice]: Cluster monitor: takeover of filer02 by filer01 enabled
Wed May 17 07:32:43 BNT [filer02: diskown.isEnabled:info]: replayed event: software ownership has been enabled for this system
Wed May 17 07:32:47 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.2 is a local HA mailbox disk.
Wed May 17 07:32:47 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.0 is a local HA mailbox disk.
Wed May 17 07:32:47 BNT [filer02: fmmb.instStat.change:info]: replayed event: normal mailbox instance on local side.
Wed May 17 07:32:47 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.7 is a partner HA mailbox disk.
Wed May 17 07:32:47 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.3 is a partner HA mailbox disk.
Wed May 17 07:32:47 BNT [filer02: fmmb.instStat.change:info]: replayed event: normal mailbox instance on partner side.
Wed May 17 07:32:48 BNT [filer02: cf.fm.partner:info]: replayed event: Cluster monitor: partner 'filer01'
Wed May 17 07:32:48 BNT [filer02: cf.fm.timeMasterStatus:info]: replayed event: Acting as cluster time slave
Wed May 17 07:32:48 BNT [filer02: raid.assim.disk.brokenPreAssim:error]: replayed event: Broken Disk 0a.33 Shelf ? Bay ? [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] detected prior to assimilation. It should be removed.
Wed May 17 07:32:48 BNT [filer02: raid.assim.disk.brokenPreAssim:error]: replayed event: Broken Disk 0a.40 Shelf ? Bay ? [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] detected prior to assimilation. It should be removed.
Wed May 17 07:32:48 BNT [filer02: raid.assim.disk.brokenPreAssim:error]: replayed event: Broken Disk 0a.35 Shelf ? Bay ? [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] detected prior to assimilation. It should be removed.
Wed May 17 07:32:49 BNT [filer02: coredump.spare.none:info]: replayed event: No sparecore disk was found.
Wed May 17 07:32:49 BNT [filer02: raid.vol.replay.nvram:info]: replayed event: Performing raid replay on volume(s)
Wed May 17 07:32:49 BNT [filer02: raid.cksum.replay.summary:info]: replayed event: Replayed 0 checksum blocks.
Wed May 17 07:32:49 BNT [filer02: raid.stripe.replay.summary:info]: replayed event: Replayed 0 stripes.
Wed May 17 07:32:52 BNT [filer02: ses.status.psWarning:warning]: replayed event: DS12-ESAS shelf 0 on channel 0c power warning for Power supply 1: non-critical status; DC undervoltage fault. This module is on the rear side of the shelf, at the left.
Wed May 17 07:32:52 BNT [filer02: ses.status.psWarning:warning]: replayed event: DS12-ESAS shelf 0 on channel 0c power warning for Power supply 2: non-critical status; DC undervoltage fault. This module is on the rear side of the shelf, at the right.
Wed May 17 07:33:01 BNT [filer02: cf.fm.launch:info]: replayed event: Launching cluster monitor
Wed May 17 07:33:01 BNT [filer02: cf.fm.partner:info]: replayed event: Cluster monitor: partner 'filer01'
Wed May 17 07:33:06 BNT [filer02: cf.fsm.takeoverByPartnerDisabled:notice]: replayed event: Cluster monitor: takeover of  by filer01 disabled (partner booting)
Wed May 17 07:33:06 BNT [filer02: fcp.service.adapter:warning]: replayed event: FCP started but no adapter is present!
Wed May 17 07:33:07 BNT [filer02: kern.syslog.msg:notice]: replayed event: The system was down for 19 seconds
Wed May 17 07:33:07 BNT [filer02: cf.fsm.takeoverOfPartnerEnabled:notice]: replayed event: Cluster monitor: takeover of filer01 enabled
Wed May 17 07:33:08 BNT [filer02: disk.failmsg:error]: Disk 0a.33 (N00W75PL): message received.
Wed May 17 07:33:08 BNT [filer02: disk.failmsg:error]: Disk 0a.40 (N00V0W8L): message received.
Wed May 17 07:33:08 BNT [filer02: disk.failmsg:error]: Disk 0a.35 (N00WLAUL): message received.
Wed May 17 07:33:08 BNT [filer02: raid.disk.unload.done:info]: Unload of Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] has completed successfully
Wed May 17 07:33:08 BNT [filer02: raid.disk.unload.done:info]: Unload of Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] has completed successfully
Wed May 17 07:33:08 BNT [filer02: raid.disk.unload.done:info]: Unload of Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] has completed successfully
Wed May 17 07:33:08 BNT [filer02: raid.fdr.failed.ok:info]: Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] successfully deleted from spare pool
Wed May 17 07:33:08 BNT [filer02: raid.fdr.failed.ok:info]: Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] successfully deleted from spare pool
Wed May 17 07:33:08 BNT [filer02: raid.fdr.failed.ok:info]: Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] successfully deleted from spare pool
Wed May 17 07:33:09 BNT [filer02: dfu.firmwareUpToDate:info]: Firmware is up-to-date on all disk drives
Wed May 17 07:33:10 BNT [filer02: perf.archive.start:info]: Performance archiver started. Sampling 22 objects and 198 counters.
Wed May 17 07:33:12 BNT [filer02: netif.linkUp:info]: Ethernet e0b: Link up.
Wed May 17 07:33:12 BNT [filer02: netif.linkUp:info]: Ethernet e0a: Link up.
Wed May 17 07:33:13 BNT [filer02: pvif.switchLink:warning]: vif1: switching to e0a
Wed May 17 07:33:14 BNT [filer02: rc:info]: relog syslog Wed May 17 07:32:31 BNT [filer02: fci.adapter.error:warning]: Fibre Channel adapter driver encountered error "Failed t
Wed May 17 07:33:14 BNT [filer02: rc:info]: relog syslog Wed May 17 07:32:31 BNT [filer02: fci.adapter.reset:debug]: Resetting Fibre Channel adapter 0a.
Wed May 17 07:33:14 BNT [filer02: iscsi.service.startup:info]: iSCSI service startup
Wed May 17 07:33:15 BNT [filer02: cmds.sysconf.logErr:error]: sysconfig: Unless directed by NetApp Global Services the cf.takeover.on_panic option should be set to On. .
Wed May 17 07:33:15 BNT [filer02: raid.rg.recons.missing:notice]: RAID group /aggr1/plex0/rg0 is missing 2 disk(s).
Wed May 17 07:33:15 BNT [filer02: raid.rg.recons.cantStart:warning]: The reconstruction cannot start in RAID group /aggr1/plex0/rg0: No 520 bps disk of required size available in spare pool, targeting any  spare pool
Wed May 17 07:33:15 BNT [filer02: mgr.boot.disk_done:info]: NetApp Release 7.3.2 boot complete. Last disk update written at Wed May 17 07:32:16 BNT 2017
Wed May 17 07:33:15 BNT [filer02: monitor.raid.brokenDisk:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken.
Wed May 17 07:33:15 BNT [filer02: monitor.brokenDisk.notice:notice]: When two disks are broken in raid_dp volume,  the system shuts down automatically every 24  hours to encourage you to replace the disk.  If you reboot the system it will run for another  24 hours before shutting down. (The 24 hour timeout  may be increased by altering the "raid.timeout"  value using the "options" command.)
Wed May 17 07:33:15 BNT [filer02: diskown.errorReadingOwnership:warning]: error 3 (disk failed) while reading ownership on disk 0a.24 (S/N N00SSSAL)
Wed May 17 07:33:15 BNT [filer02: diskown.errorReadingOwnership:warning]: error 3 (disk failed) while reading ownership on disk 0a.26 (S/N N00V0UWL)
Wed May 17 07:33:15 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] is still present in the system and should be removed.
Wed May 17 07:33:15 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] is still present in the system and should be removed.
Wed May 17 07:33:15 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] is still present in the system and should be removed.
Wed May 17 07:33:15 BNT [filer02: shelf.config.spha:info]: System is using single path HA attached storage only.
Wed May 17 07:33:16 BNT [filer02: mgr.boot.reason_ok:notice]: System rebooted after power-on.
Wed May 17 07:33:38 BNT [filer02: raid.rg.spares.low:warning]: /aggr1/plex0/rg0
Wed May 17 07:33:41 BNT [filer02: nbt.nbns.registrationComplete:info]: NBT: All CIFS name registrations have completed for the local server.
Wed May 17 07:33:41 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Wed May 17 07:33:47 BNT last message repeated 2 times
Wed May 17 07:33:47 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201705120800.0) was not posted for host (0), it was dropped from queue due to lack of space
Wed May 17 07:33:54 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Wed May 17 07:33:54 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201705121200.0) was not posted for host (0), it was dropped from queue due to lack of space
Wed May 17 07:34:00 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Wed May 17 07:34:00 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201705121600.0) was not posted for host (0), it was dropped from queue due to lack of space
Wed May 17 07:34:05 BNT [filer02: monitor.globalStatus.nonCritical:warning]: Disk on adapter 0a, shelf 2, bay 1, failed. Disk on adapter 0a, shelf 2, bay 8, failed. Disk on adapter 0a, shelf 2, bay 3, failed.  There are not enough spare disks. /vol/vol1 is full (using or reserving 100% of space and 0% of inodes, using 92% of reserve).
Wed May 17 07:34:24 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Wed May 17 07:34:24 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201705122000.0) was not posted for host (0), it was dropped from queue due to lack of space
Wed May 17 07:38:57 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Wed May 17 07:43:32 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Wed May 17 07:48:00 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Wed May 17 07:52:16 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Wed May 17 07:56:24 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Wed May 17 08:00:00 BNT [filer02: kern.uptime.filer:info]:   8:00am up 26 mins, 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 0 iSCSI ops
Wed May 17 08:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 24 hours.
Wed May 17 08:00:31 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Wed May 17 08:00:37 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Wed May 17 08:00:38 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201705130000.0) was not posted for host (0), it was dropped from queue due to lack of space
Wed May 17 08:04:34 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Wed May 17 08:08:48 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Wed May 17 08:12:58 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Wed May 17 08:12:58 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201705131729.0) for host (0) was not posted to NetApp. The system will drop the message
Wed May 17 08:33:43 BNT [filer02: iscsi.notice:notice]: ISCSI: New session from initiator iqn.1991-05.com.microsoft:pcb-722-ad1.pcbl722dom.local at IP addr 10.72.4.1
Wed May 17 09:00:00 BNT [filer02: kern.uptime.filer:info]:   9:00am up  1:26 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 1218 iSCSI ops
Wed May 17 09:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 23 hours.
Wed May 17 10:00:00 BNT [filer02: kern.uptime.filer:info]:  10:00am up  2:26 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 24041 iSCSI ops
Wed May 17 10:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 22 hours.
Wed May 17 11:00:00 BNT [filer02: kern.uptime.filer:info]:  11:00am up  3:26 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 46817 iSCSI ops
Wed May 17 11:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 21 hours.
Wed May 17 12:00:00 BNT [filer02: kern.uptime.filer:info]:  12:00pm up  4:26 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 50900 iSCSI ops
Wed May 17 12:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 20 hours.
Wed May 17 12:00:15 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Wed May 17 12:00:27 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SHUTDOWN PENDING (degraded mode))
Wed May 17 12:04:12 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SHUTDOWN PENDING (degraded mode))
Wed May 17 12:08:35 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SHUTDOWN PENDING (degraded mode))
Wed May 17 12:12:58 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SHUTDOWN PENDING (degraded mode))
Wed May 17 12:16:50 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SHUTDOWN PENDING (degraded mode))
Wed May 17 12:33:07 BNT last message repeated 4 times
Wed May 17 12:37:27 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SHUTDOWN PENDING (degraded mode))
Wed May 17 12:57:35 BNT last message repeated 5 times
Wed May 17 12:57:35 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201705131730.0) for host (0) was not posted to NetApp. The system will drop the message
Wed May 17 13:00:00 BNT [filer02: kern.uptime.filer:info]:   1:00pm up  5:26 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 51906 iSCSI ops
Wed May 17 13:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 19 hours.
Wed May 17 14:00:00 BNT [filer02: kern.uptime.filer:info]:   2:00pm up  6:26 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 53608 iSCSI ops
Wed May 17 14:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 18 hours.
Wed May 17 15:00:00 BNT [filer02: kern.uptime.filer:info]:   3:00pm up  7:26 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 146993 iSCSI ops
Wed May 17 15:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 17 hours.
Wed May 17 16:00:00 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Wed May 17 16:00:00 BNT [filer02: kern.uptime.filer:info]:   4:00pm up  8:27 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 175839 iSCSI ops
Wed May 17 16:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 16 hours.
Wed May 17 16:00:16 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (REBOOT)
Wed May 17 16:04:39 BNT [filer02: asup.post.sent:notice]: Cluster Notification message posted to NetApp: Cluster Notification from filer02 (REBOOT) INFO
Wed May 17 16:04:39 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SPARES_LOW)
Wed May 17 16:08:26 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SPARES_LOW)
Wed May 17 16:12:18 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SPARES_LOW)
Wed May 17 16:16:46 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SPARES_LOW)
Wed May 17 16:20:33 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SPARES_LOW)
Wed May 17 16:40:20 BNT last message repeated 5 times
Wed May 17 16:43:56 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SPARES_LOW)
Wed May 17 16:56:51 BNT last message repeated 3 times
Wed May 17 17:00:00 BNT [filer02: kern.uptime.filer:info]:   5:00pm up  9:27 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 181512 iSCSI ops
Wed May 17 17:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 15 hours.
Wed May 17 17:00:56 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SPARES_LOW)
Wed May 17 17:00:56 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201705131730.2) for host (0) was not posted to NetApp. The system will drop the message
Wed May 17 18:00:00 BNT [filer02: kern.uptime.filer:info]:   6:00pm up 10:27 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 185264 iSCSI ops
Wed May 17 18:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 14 hours.
Wed May 17 19:00:00 BNT [filer02: kern.uptime.filer:info]:   7:00pm up 11:27 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 185413 iSCSI ops
Wed May 17 19:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 13 hours.
Wed May 17 20:00:00 BNT [filer02: kern.uptime.filer:info]:   8:00pm up 12:27 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 185413 iSCSI ops
Wed May 17 20:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 12 hours.
Wed May 17 20:00:01 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Wed May 17 20:00:16 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (PERFORMANCE SNAPSHOT)
Wed May 17 20:04:30 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (PERFORMANCE SNAPSHOT)
Wed May 17 20:08:20 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (PERFORMANCE SNAPSHOT)
Wed May 17 20:12:33 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (PERFORMANCE SNAPSHOT)
Wed May 17 20:16:54 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (PERFORMANCE SNAPSHOT)
Wed May 17 20:33:03 BNT last message repeated 4 times
Wed May 17 20:37:28 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (PERFORMANCE SNAPSHOT)
Wed May 17 20:54:38 BNT last message repeated 4 times
Wed May 17 20:58:41 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (PERFORMANCE SNAPSHOT)
Wed May 17 20:58:41 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201705131730.3) for host (0) was not posted to NetApp. The system will drop the message
Wed May 17 21:00:00 BNT [filer02: kern.uptime.filer:info]:   9:00pm up 13:27 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 185413 iSCSI ops
Wed May 17 21:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 11 hours.
Wed May 17 22:00:00 BNT [filer02: kern.uptime.filer:info]:  10:00pm up 14:27 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 185413 iSCSI ops
Wed May 17 22:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 10 hours.
Wed May 17 23:00:00 BNT [filer02: kern.uptime.filer:info]:  11:00pm up 15:27 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 185413 iSCSI ops
Wed May 17 23:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 9 hours.
Thu May 18 00:00:00 BNT [filer02: kern.uptime.filer:info]:  12:00am up 16:27 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 185413 iSCSI ops
Thu May 18 00:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 8 hours.
Thu May 18 00:00:46 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of nightly.0 snapshot (Snapshot operation failed: No space left on device.)
Thu May 18 00:00:55 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (CLUSTER ERROR: DISK/SHELF COUNT MISMATCH)
Thu May 18 00:04:32 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (CLUSTER ERROR: DISK/SHELF COUNT MISMATCH)
Thu May 18 00:09:00 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (CLUSTER ERROR: DISK/SHELF COUNT MISMATCH)
Thu May 18 00:13:02 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (CLUSTER ERROR: DISK/SHELF COUNT MISMATCH)
Thu May 18 00:17:13 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (CLUSTER ERROR: DISK/SHELF COUNT MISMATCH)
Thu May 18 00:37:00 BNT last message repeated 5 times
Thu May 18 00:41:11 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (CLUSTER ERROR: DISK/SHELF COUNT MISMATCH)
Thu May 18 00:57:20 BNT last message repeated 4 times
Thu May 18 00:57:20 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201705131735.0) for host (0) was not posted to NetApp. The system will drop the message
Thu May 18 01:00:00 BNT [filer02: kern.uptime.filer:info]:   1:00am up 17:27 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 185413 iSCSI ops
Thu May 18 01:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 7 hours.
Thu May 18 02:00:00 BNT [filer02: kern.uptime.filer:info]:   2:00am up 18:27 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 185413 iSCSI ops
Thu May 18 02:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 6 hours.
Thu May 18 03:00:00 BNT [filer02: kern.uptime.filer:info]:   3:00am up 19:27 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 185413 iSCSI ops
Thu May 18 03:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 5 hours.
Thu May 18 04:00:00 BNT [filer02: kern.uptime.filer:info]:   4:00am up 20:27 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 185413 iSCSI ops
Thu May 18 04:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 4 hours.
Thu May 18 05:00:00 BNT [filer02: kern.uptime.filer:info]:   5:00am up 21:27 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 185413 iSCSI ops
Thu May 18 05:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 3 hours.
Thu May 18 06:00:00 BNT [filer02: kern.uptime.filer:info]:   6:00am up 22:27 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 185413 iSCSI ops
Thu May 18 06:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 2 hours.
Thu May 18 07:00:00 BNT [filer02: kern.uptime.filer:info]:   7:00am up 23:27 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 185413 iSCSI ops
Thu May 18 07:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 1 hour.
Thu May 18 07:00:00 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] is still present in the system and should be removed.
Thu May 18 07:00:00 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] is still present in the system and should be removed.
Thu May 18 07:00:00 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] is still present in the system and should be removed.
Thu May 18 07:50:55 BNT [filer02: fci.link.break:error]: Link break detected on Fibre Channel adapter 0a.
Thu May 18 07:51:00 BNT [filer02: fci.adapter.error:warning]: Fibre Channel adapter driver encountered error "Failed to get loop map" on adapter 0a.
Wed May 17 23:51:39 GMT [localhost: rc:notice]: The system was down for 16 seconds
Wed May 17 23:51:39 GMT [localhost: cf.fsm.takeoverOfPartnerEnabled:notice]: Cluster monitor: takeover of filer01 enabled
Thu May 18 07:51:03 BNT [filer02: cf.nm.nicTransitionUp:info]: replayed event: Interconnect link 0 is UP
Thu May 18 07:51:10 BNT [filer02: fci.initialization.failed:error]: replayed event: Initialization failed on Fibre Channel adapter 0b.
Thu May 18 07:51:12 BNT [filer02: nvram.battery.turned.on:info]: replayed event: The NVRAM battery is turned ON. It is turned OFF during system shutdown.
Thu May 18 07:51:15 BNT [filer02: diskown.isEnabled:info]: replayed event: software ownership has been enabled for this system
Thu May 18 07:51:19 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.2 is a local HA mailbox disk.
Thu May 18 07:51:19 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.0 is a local HA mailbox disk.
Thu May 18 07:51:40 BNT [filer02: cf.fsm.takeoverByPartnerEnabled:notice]: Cluster monitor: takeover of filer02 by filer01 enabled
Thu May 18 07:51:19 BNT [filer02: fmmb.instStat.change:info]: replayed event: normal mailbox instance on local side.
Thu May 18 07:51:19 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.7 is a partner HA mailbox disk.
Thu May 18 07:51:19 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.3 is a partner HA mailbox disk.
Thu May 18 07:51:19 BNT [filer02: fmmb.instStat.change:info]: replayed event: normal mailbox instance on partner side.
Thu May 18 07:51:20 BNT [filer02: cf.fm.partner:info]: replayed event: Cluster monitor: partner 'filer01'
Thu May 18 07:51:20 BNT [filer02: cf.fm.timeMasterStatus:info]: replayed event: Acting as cluster time slave
Thu May 18 07:51:20 BNT [filer02: raid.assim.disk.brokenPreAssim:error]: replayed event: Broken Disk 0a.33 Shelf ? Bay ? [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] detected prior to assimilation. It should be removed.
Thu May 18 07:51:20 BNT [filer02: raid.assim.disk.brokenPreAssim:error]: replayed event: Broken Disk 0a.40 Shelf ? Bay ? [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] detected prior to assimilation. It should be removed.
Thu May 18 07:51:20 BNT [filer02: raid.assim.disk.brokenPreAssim:error]: replayed event: Broken Disk 0a.35 Shelf ? Bay ? [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] detected prior to assimilation. It should be removed.
Thu May 18 07:51:21 BNT [filer02: coredump.spare.none:info]: replayed event: No sparecore disk was found.
Thu May 18 07:51:21 BNT [filer02: raid.vol.replay.nvram:info]: replayed event: Performing raid replay on volume(s)
Thu May 18 07:51:21 BNT [filer02: raid.cksum.replay.summary:info]: replayed event: Replayed 0 checksum blocks.
Thu May 18 07:51:21 BNT [filer02: raid.stripe.replay.summary:info]: replayed event: Replayed 0 stripes.
Thu May 18 07:51:24 BNT [filer02: ses.status.psWarning:warning]: replayed event: DS12-ESAS shelf 0 on channel 0c power warning for Power supply 1: non-critical status; DC undervoltage fault. This module is on the rear side of the shelf, at the left.
Thu May 18 07:51:24 BNT [filer02: ses.status.psWarning:warning]: replayed event: DS12-ESAS shelf 0 on channel 0c power warning for Power supply 2: non-critical status; DC undervoltage fault. This module is on the rear side of the shelf, at the right.
Thu May 18 07:51:33 BNT [filer02: cf.fm.launch:info]: replayed event: Launching cluster monitor
Thu May 18 07:51:33 BNT [filer02: cf.fm.partner:info]: replayed event: Cluster monitor: partner 'filer01'
Thu May 18 07:51:37 BNT [filer02: cf.fsm.takeoverOfPartnerDisabled:notice]: replayed event: Cluster monitor: takeover of filer01 disabled (partner booting)
Thu May 18 07:51:38 BNT [filer02: fcp.service.adapter:warning]: replayed event: FCP started but no adapter is present!
Thu May 18 07:51:39 BNT [filer02: kern.syslog.msg:notice]: replayed event: The system was down for 16 seconds
Thu May 18 07:51:39 BNT [filer02: cf.fsm.takeoverOfPartnerEnabled:notice]: replayed event: Cluster monitor: takeover of filer01 enabled
Thu May 18 07:51:40 BNT [filer02: disk.failmsg:error]: Disk 0a.33 (N00W75PL): message received.
Thu May 18 07:51:40 BNT [filer02: disk.failmsg:error]: Disk 0a.40 (N00V0W8L): message received.
Thu May 18 07:51:40 BNT [filer02: disk.failmsg:error]: Disk 0a.35 (N00WLAUL): message received.
Thu May 18 07:51:40 BNT [filer02: raid.disk.unload.done:info]: Unload of Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] has completed successfully
Thu May 18 07:51:40 BNT [filer02: raid.disk.unload.done:info]: Unload of Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] has completed successfully
Thu May 18 07:51:40 BNT [filer02: raid.disk.unload.done:info]: Unload of Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] has completed successfully
Thu May 18 07:51:40 BNT [filer02: raid.fdr.failed.ok:info]: Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] successfully deleted from spare pool
Thu May 18 07:51:40 BNT [filer02: raid.fdr.failed.ok:info]: Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] successfully deleted from spare pool
Thu May 18 07:51:40 BNT [filer02: raid.fdr.failed.ok:info]: Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] successfully deleted from spare pool
Thu May 18 07:51:40 BNT [filer02: dfu.firmwareUpToDate:info]: Firmware is up-to-date on all disk drives
Thu May 18 07:51:42 BNT [filer02: perf.archive.start:info]: Performance archiver started. Sampling 22 objects and 198 counters.
Thu May 18 07:51:44 BNT [filer02: netif.linkUp:info]: Ethernet e0a: Link up.
Thu May 18 07:51:44 BNT [filer02: netif.linkUp:info]: Ethernet e0b: Link up.
Thu May 18 07:51:45 BNT [filer02: pvif.switchLink:warning]: vif1: switching to e0a
Thu May 18 07:51:46 BNT [filer02: rc:info]: relog syslog Thu May 18 07:51:03 BNT [filer02: fci.adapter.error:warning]: Fibre Channel adapter driver encountered error "Failed t
Thu May 18 07:51:46 BNT [filer02: rc:info]: relog syslog Thu May 18 07:51:03 BNT [filer02: fci.adapter.reset:debug]: Resetting Fibre Channel adapter 0a.
Thu May 18 07:51:46 BNT [filer02: iscsi.service.startup:info]: iSCSI service startup
Thu May 18 07:51:47 BNT [filer02: cmds.sysconf.logErr:error]: sysconfig: Unless directed by NetApp Global Services the cf.takeover.on_panic option should be set to On. .
Thu May 18 07:51:47 BNT [filer02: raid.rg.recons.missing:notice]: RAID group /aggr1/plex0/rg0 is missing 2 disk(s).
Thu May 18 07:51:47 BNT [filer02: raid.rg.recons.cantStart:warning]: The reconstruction cannot start in RAID group /aggr1/plex0/rg0: No 520 bps disk of required size available in spare pool, targeting any  spare pool
Thu May 18 07:51:47 BNT [filer02: mgr.boot.disk_done:info]: NetApp Release 7.3.2 boot complete. Last disk update written at Thu May 18 07:50:50 BNT 2017
Thu May 18 07:51:47 BNT [filer02: monitor.raid.brokenDisk:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken.
Thu May 18 07:51:47 BNT [filer02: monitor.brokenDisk.notice:notice]: When two disks are broken in raid_dp volume,  the system shuts down automatically every 24  hours to encourage you to replace the disk.  If you reboot the system it will run for another  24 hours before shutting down. (The 24 hour timeout  may be increased by altering the "raid.timeout"  value using the "options" command.)
Thu May 18 07:51:47 BNT [filer02: diskown.errorReadingOwnership:warning]: error 3 (disk failed) while reading ownership on disk 0a.24 (S/N N00SSSAL)
Thu May 18 07:51:47 BNT [filer02: diskown.errorReadingOwnership:warning]: error 3 (disk failed) while reading ownership on disk 0a.26 (S/N N00V0UWL)
Thu May 18 07:51:48 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] is still present in the system and should be removed.
Thu May 18 07:51:48 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] is still present in the system and should be removed.
Thu May 18 07:51:48 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] is still present in the system and should be removed.
Thu May 18 07:51:48 BNT [filer02: shelf.config.spha:info]: System is using single path HA attached storage only.
Thu May 18 07:51:48 BNT [filer02: mgr.boot.reason_ok:notice]: System rebooted after power-on.
Thu May 18 07:52:01 BNT [filer02: raid.rg.spares.low:warning]: /aggr1/plex0/rg0
Thu May 18 07:52:02 BNT [filer02: monitor.globalStatus.nonCritical:warning]: Disk on adapter 0a, shelf 2, bay 1, failed. Disk on adapter 0a, shelf 2, bay 8, failed. Disk on adapter 0a, shelf 2, bay 3, failed.  There are not enough spare disks. /vol/vol1 is full (using or reserving 100% of space and 0% of inodes, using 92% of reserve).
Thu May 18 07:52:12 BNT [filer02: nbt.nbns.registrationComplete:info]: NBT: All CIFS name registrations have completed for the local server.
Thu May 18 07:52:15 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu May 18 07:52:28 BNT last message repeated 3 times
Thu May 18 07:52:28 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201705132000.0) was not posted for host (0), it was dropped from queue due to lack of space
Thu May 18 07:52:35 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (WEEKLY_LOG)
Thu May 18 07:52:35 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201705140000.0) was not posted for host (0), it was dropped from queue due to lack of space
Thu May 18 07:52:56 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (PERFORMANCE DATA)
Thu May 18 07:52:56 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201705140000.1) was not posted for host (0), it was dropped from queue due to lack of space
Thu May 18 07:57:22 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu May 18 08:00:02 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Thu May 18 08:00:06 BNT [filer02: kern.uptime.filer:info]:   8:00am up 8 mins, 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 0 iSCSI ops
Thu May 18 08:00:06 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 24 hours.
Thu May 18 08:00:09 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu May 18 08:00:09 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201705140000.2) was not posted for host (0), it was dropped from queue due to lack of space
Thu May 18 08:04:12 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu May 18 08:08:41 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu May 18 08:13:07 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu May 18 08:17:16 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu May 18 08:21:37 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu May 18 08:29:27 BNT last message repeated 2 times
Thu May 18 08:29:27 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201705140800.0) for host (0) was not posted to NetApp. The system will drop the message
Thu May 18 08:32:32 BNT [filer02: iscsi.notice:notice]: ISCSI: New session from initiator iqn.1991-05.com.microsoft:pcb-722-ad1.pcbl722dom.local at IP addr 10.72.4.1
Thu May 18 09:00:01 BNT [filer02: kern.uptime.filer:info]:   9:00am up  1:08 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 10 iSCSI ops
Thu May 18 09:00:01 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 23 hours.
Thu May 18 10:00:00 BNT [filer02: kern.uptime.filer:info]:  10:00am up  2:08 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 753 iSCSI ops
Thu May 18 10:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 22 hours.
Thu May 18 11:00:00 BNT [filer02: kern.uptime.filer:info]:  11:00am up  3:08 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 10834 iSCSI ops
Thu May 18 11:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 21 hours.
Thu May 18 12:00:00 BNT [filer02: kern.uptime.filer:info]:  12:00pm up  4:08 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 78443 iSCSI ops
Thu May 18 12:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 20 hours.
Thu May 18 12:00:46 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Thu May 18 12:00:59 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu May 18 12:05:10 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu May 18 12:09:28 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu May 18 12:13:58 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu May 18 12:18:26 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu May 18 12:37:57 BNT last message repeated 5 times
Thu May 18 12:42:19 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu May 18 12:57:58 BNT last message repeated 4 times
Thu May 18 12:57:58 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201705141200.0) for host (0) was not posted to NetApp. The system will drop the message
Thu May 18 13:00:00 BNT [filer02: kern.uptime.filer:info]:   1:00pm up  5:08 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 82379 iSCSI ops
Thu May 18 13:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 19 hours.
Thu May 18 14:00:00 BNT [filer02: kern.uptime.filer:info]:   2:00pm up  6:08 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 82514 iSCSI ops
Thu May 18 14:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 18 hours.
Thu May 18 15:00:00 BNT [filer02: kern.uptime.filer:info]:   3:00pm up  7:08 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 141112 iSCSI ops
Thu May 18 15:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 17 hours.
Thu May 18 16:00:00 BNT [filer02: kern.uptime.filer:info]:   4:00pm up  8:08 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 164469 iSCSI ops
Thu May 18 16:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 16 hours.
Thu May 18 16:00:45 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Thu May 18 16:01:00 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu May 18 16:05:13 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu May 18 16:09:23 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu May 18 16:13:37 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu May 18 16:17:17 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu May 18 16:37:14 BNT last message repeated 5 times
Thu May 18 16:41:04 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu May 18 16:56:31 BNT last message repeated 4 times
Thu May 18 16:56:31 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201705141600.0) for host (0) was not posted to NetApp. The system will drop the message
Thu May 18 17:00:00 BNT [filer02: kern.uptime.filer:info]:   5:00pm up  9:08 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 233456 iSCSI ops
Thu May 18 17:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 15 hours.
Thu May 18 18:00:00 BNT [filer02: kern.uptime.filer:info]:   6:00pm up 10:08 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 238024 iSCSI ops
Thu May 18 18:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 14 hours.
Thu May 18 19:00:00 BNT [filer02: kern.uptime.filer:info]:   7:00pm up 11:08 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 240955 iSCSI ops
Thu May 18 19:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 13 hours.
Thu May 18 20:00:00 BNT [filer02: kern.uptime.filer:info]:   8:00pm up 12:08 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 240955 iSCSI ops
Thu May 18 20:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 12 hours.
Thu May 18 20:00:30 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Thu May 18 20:00:49 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Thu May 18 20:04:24 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Thu May 18 20:08:41 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Thu May 18 20:12:39 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Thu May 18 20:16:59 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Thu May 18 20:36:44 BNT last message repeated 5 times
Thu May 18 20:40:31 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Thu May 18 20:57:08 BNT last message repeated 4 times
Thu May 18 20:57:08 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201705150746.0) for host (0) was not posted to NetApp. The system will drop the message
Thu May 18 21:00:00 BNT [filer02: kern.uptime.filer:info]:   9:00pm up 13:08 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 240955 iSCSI ops
Thu May 18 21:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 11 hours.
Thu May 18 22:00:00 BNT [filer02: kern.uptime.filer:info]:  10:00pm up 14:08 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 240955 iSCSI ops
Thu May 18 22:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 10 hours.
Thu May 18 23:00:00 BNT [filer02: kern.uptime.filer:info]:  11:00pm up 15:08 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 240955 iSCSI ops
Thu May 18 23:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 9 hours.
Fri May 19 00:00:00 BNT [filer02: kern.uptime.filer:info]:  12:00am up 16:08 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 240955 iSCSI ops
Fri May 19 00:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 8 hours.
Fri May 19 00:00:31 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of nightly.0 snapshot (Snapshot operation failed: No space left on device.)
Fri May 19 00:00:40 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SHUTDOWN PENDING (degraded mode))
Fri May 19 00:04:53 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SHUTDOWN PENDING (degraded mode))
Fri May 19 00:08:44 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SHUTDOWN PENDING (degraded mode))
Fri May 19 00:12:33 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SHUTDOWN PENDING (degraded mode))
Fri May 19 00:16:06 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SHUTDOWN PENDING (degraded mode))
Fri May 19 00:36:18 BNT last message repeated 5 times
Fri May 19 00:39:54 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SHUTDOWN PENDING (degraded mode))
Fri May 19 00:55:55 BNT last message repeated 4 times
Fri May 19 00:55:55 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201705150747.0) for host (0) was not posted to NetApp. The system will drop the message
Fri May 19 01:00:00 BNT [filer02: kern.uptime.filer:info]:   1:00am up 17:08 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 240955 iSCSI ops
Fri May 19 01:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 7 hours.
Fri May 19 02:00:00 BNT [filer02: kern.uptime.filer:info]:   2:00am up 18:08 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 240955 iSCSI ops
Fri May 19 02:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 6 hours.
Fri May 19 03:00:00 BNT [filer02: kern.uptime.filer:info]:   3:00am up 19:08 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 240955 iSCSI ops
Fri May 19 03:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 5 hours.
Fri May 19 04:00:00 BNT [filer02: kern.uptime.filer:info]:   4:00am up 20:08 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 240955 iSCSI ops
Fri May 19 04:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 4 hours.
Fri May 19 05:00:00 BNT [filer02: kern.uptime.filer:info]:   5:00am up 21:08 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 240955 iSCSI ops
Fri May 19 05:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 3 hours.
Fri May 19 06:00:00 BNT [filer02: kern.uptime.filer:info]:   6:00am up 22:08 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 240955 iSCSI ops
Fri May 19 06:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 2 hours.
Fri May 19 07:00:00 BNT [filer02: kern.uptime.filer:info]:   7:00am up 23:08 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 240955 iSCSI ops
Fri May 19 07:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 1 hour.
Fri May 19 07:00:00 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] is still present in the system and should be removed.
Fri May 19 07:00:00 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] is still present in the system and should be removed.
Fri May 19 07:00:00 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] is still present in the system and should be removed.
Fri May 19 07:59:52 BNT [filer02: ses.status.psWarning:warning]: DS14-Mk2-AT shelf 2 on channel 0a power warning for Power supply 2: non-critical status; DC undervoltage fault. This module is on the rear side of the shelf, at the right.
Fri May 19 07:59:55 BNT [filer02: fci.link.break:error]: Link break detected on Fibre Channel adapter 0a.
Fri May 19 08:00:00 BNT [filer02: kern.uptime.filer:info]:   8:00am up  1 day, 9 mins, 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 240955 iSCSI ops
Fri May 19 00:00:38 GMT [localhost: rc:notice]: The system was down for 13 seconds
Fri May 19 00:00:38 GMT [localhost: cf.fsm.takeoverOfPartnerEnabled:notice]: Cluster monitor: takeover of filer01 enabled
Fri May 19 08:00:39 BNT [filer02: cf.fsm.takeoverByPartnerEnabled:notice]: Cluster monitor: takeover of filer02 by filer01 enabled
Fri May 19 08:00:02 BNT [filer02: cf.nm.nicTransitionUp:info]: replayed event: Interconnect link 0 is UP
Fri May 19 08:00:09 BNT [filer02: fci.initialization.failed:error]: replayed event: Initialization failed on Fibre Channel adapter 0b.
Fri May 19 08:00:11 BNT [filer02: nvram.battery.turned.on:info]: replayed event: The NVRAM battery is turned ON. It is turned OFF during system shutdown.
Fri May 19 08:00:14 BNT [filer02: diskown.isEnabled:info]: replayed event: software ownership has been enabled for this system
Fri May 19 08:00:18 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.2 is a local HA mailbox disk.
Fri May 19 08:00:18 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.0 is a local HA mailbox disk.
Fri May 19 08:00:18 BNT [filer02: fmmb.instStat.change:info]: replayed event: normal mailbox instance on local side.
Fri May 19 08:00:18 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.7 is a partner HA mailbox disk.
Fri May 19 08:00:18 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.3 is a partner HA mailbox disk.
Fri May 19 08:00:18 BNT [filer02: fmmb.instStat.change:info]: replayed event: normal mailbox instance on partner side.
Fri May 19 08:00:19 BNT [filer02: cf.fm.partner:info]: replayed event: Cluster monitor: partner 'filer01'
Fri May 19 08:00:19 BNT [filer02: cf.fm.timeMasterStatus:info]: replayed event: Acting as cluster time slave
Fri May 19 08:00:19 BNT [filer02: raid.assim.disk.brokenPreAssim:error]: replayed event: Broken Disk 0a.40 Shelf ? Bay ? [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] detected prior to assimilation. It should be removed.
Fri May 19 08:00:19 BNT [filer02: raid.assim.disk.brokenPreAssim:error]: replayed event: Broken Disk 0a.35 Shelf ? Bay ? [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] detected prior to assimilation. It should be removed.
Fri May 19 08:00:19 BNT [filer02: raid.assim.disk.brokenPreAssim:error]: replayed event: Broken Disk 0a.33 Shelf ? Bay ? [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] detected prior to assimilation. It should be removed.
Fri May 19 08:00:20 BNT [filer02: coredump.spare.none:info]: replayed event: No sparecore disk was found.
Fri May 19 08:00:20 BNT [filer02: raid.vol.replay.nvram:info]: replayed event: Performing raid replay on volume(s)
Fri May 19 08:00:20 BNT [filer02: raid.cksum.replay.summary:info]: replayed event: Replayed 0 checksum blocks.
Fri May 19 08:00:20 BNT [filer02: raid.stripe.replay.summary:info]: replayed event: Replayed 0 stripes.
Fri May 19 08:00:23 BNT [filer02: ses.status.psWarning:warning]: replayed event: DS12-ESAS shelf 0 on channel 0c power warning for Power supply 1: non-critical status; DC undervoltage fault. This module is on the rear side of the shelf, at the left.
Fri May 19 08:00:23 BNT [filer02: ses.status.psWarning:warning]: replayed event: DS12-ESAS shelf 0 on channel 0c power warning for Power supply 2: non-critical status; DC undervoltage fault. This module is on the rear side of the shelf, at the right.
Fri May 19 08:00:32 BNT [filer02: cf.fm.launch:info]: replayed event: Launching cluster monitor
Fri May 19 08:00:32 BNT [filer02: cf.fm.partner:info]: replayed event: Cluster monitor: partner 'filer01'
Fri May 19 08:00:37 BNT [filer02: fcp.service.adapter:warning]: replayed event: FCP started but no adapter is present!
Fri May 19 08:00:38 BNT [filer02: kern.syslog.msg:notice]: replayed event: The system was down for 13 seconds
Fri May 19 08:00:38 BNT [filer02: cf.fsm.takeoverOfPartnerEnabled:notice]: replayed event: Cluster monitor: takeover of filer01 enabled
Fri May 19 08:00:39 BNT [filer02: cf.fsm.takeoverByPartnerEnabled:notice]: replayed event: Cluster monitor: takeover of filer02 by filer01 enabled
Fri May 19 08:00:39 BNT [filer02: disk.failmsg:error]: Disk 0a.40 (N00V0W8L): message received.
Fri May 19 08:00:39 BNT [filer02: disk.failmsg:error]: Disk 0a.35 (N00WLAUL): message received.
Fri May 19 08:00:39 BNT [filer02: disk.failmsg:error]: Disk 0a.33 (N00W75PL): message received.
Fri May 19 08:00:39 BNT [filer02: raid.disk.unload.done:info]: Unload of Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] has completed successfully
Fri May 19 08:00:39 BNT [filer02: raid.disk.unload.done:info]: Unload of Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] has completed successfully
Fri May 19 08:00:39 BNT [filer02: raid.disk.unload.done:info]: Unload of Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] has completed successfully
Fri May 19 08:00:39 BNT [filer02: raid.fdr.failed.ok:info]: Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] successfully deleted from spare pool
Fri May 19 08:00:39 BNT [filer02: raid.fdr.failed.ok:info]: Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] successfully deleted from spare pool
Fri May 19 08:00:39 BNT [filer02: raid.fdr.failed.ok:info]: Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] successfully deleted from spare pool
Fri May 19 08:00:40 BNT [filer02: dfu.firmwareUpToDate:info]: Firmware is up-to-date on all disk drives
Fri May 19 08:00:41 BNT [filer02: perf.archive.start:info]: Performance archiver started. Sampling 22 objects and 198 counters.
Fri May 19 08:00:43 BNT [filer02: netif.linkUp:info]: Ethernet e0a: Link up.
Fri May 19 08:00:43 BNT [filer02: netif.linkUp:info]: Ethernet e0b: Link up.
Fri May 19 08:00:44 BNT [filer02: pvif.switchLink:warning]: vif1: switching to e0a
Fri May 19 08:00:45 BNT [filer02: rc:info]: relog syslog Fri May 19 07:59:55 BNT [filer02: fci.link.break:error]: Link break detected on Fibre Channel adapter 0a.
Fri May 19 08:00:45 BNT [filer02: rc:info]: relog syslog Fri May 19 08:00:00 BNT [filer02: kern.uptime.filer:info]:   8:00am up  1 day, 9 mins, 0 NFS ops, 0 CIFS ops, 0 HTTP o
Fri May 19 08:00:46 BNT [filer02: iscsi.service.startup:info]: iSCSI service startup
Fri May 19 08:00:46 BNT [filer02: cmds.sysconf.logErr:error]: sysconfig: Unless directed by NetApp Global Services the cf.takeover.on_panic option should be set to On. .
Fri May 19 08:00:46 BNT [filer02: raid.rg.recons.missing:notice]: RAID group /aggr1/plex0/rg0 is missing 2 disk(s).
Fri May 19 08:00:46 BNT [filer02: raid.rg.recons.cantStart:warning]: The reconstruction cannot start in RAID group /aggr1/plex0/rg0: No 520 bps disk of required size available in spare pool, targeting any  spare pool
Fri May 19 08:00:46 BNT [filer02: mgr.boot.disk_done:info]: NetApp Release 7.3.2 boot complete. Last disk update written at Fri May 19 07:59:44 BNT 2017
Fri May 19 08:00:46 BNT [filer02: monitor.raid.brokenDisk:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken.
Fri May 19 08:00:46 BNT [filer02: monitor.brokenDisk.notice:notice]: When two disks are broken in raid_dp volume,  the system shuts down automatically every 24  hours to encourage you to replace the disk.  If you reboot the system it will run for another  24 hours before shutting down. (The 24 hour timeout  may be increased by altering the "raid.timeout"  value using the "options" command.)
Fri May 19 08:00:47 BNT [filer02: diskown.errorReadingOwnership:warning]: error 3 (disk failed) while reading ownership on disk 0a.24 (S/N N00SSSAL)
Fri May 19 08:00:47 BNT [filer02: diskown.errorReadingOwnership:warning]: error 3 (disk failed) while reading ownership on disk 0a.26 (S/N N00V0UWL)
Fri May 19 08:00:47 BNT [filer02: shelf.config.spha:info]: System is using single path HA attached storage only.
Fri May 19 08:00:47 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] is still present in the system and should be removed.
Fri May 19 08:00:47 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] is still present in the system and should be removed.
Fri May 19 08:00:47 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] is still present in the system and should be removed.
Fri May 19 08:00:47 BNT [filer02: mgr.boot.reason_ok:notice]: System rebooted after power-on.
Fri May 19 08:01:01 BNT [filer02: raid.rg.spares.low:warning]: /aggr1/plex0/rg0
Fri May 19 08:01:02 BNT [filer02: monitor.globalStatus.nonCritical:warning]: Disk on adapter 0a, shelf 2, bay 8, failed. Disk on adapter 0a, shelf 2, bay 3, failed. Disk on adapter 0a, shelf 2, bay 1, failed.  There are not enough spare disks. /vol/vol1 is full (using or reserving 100% of space and 0% of inodes, using 92% of reserve).
Fri May 19 08:01:12 BNT [filer02: nbt.nbns.registrationComplete:info]: NBT: All CIFS name registrations have completed for the local server.
Fri May 19 08:01:15 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (REBOOT)
Fri May 19 08:01:22 BNT last message repeated 2 times
Fri May 19 08:01:22 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201705150747.1) was not posted for host (0), it was dropped from queue due to lack of space
Fri May 19 08:01:28 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SPARES_LOW)
Fri May 19 08:01:28 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201705150747.2) was not posted for host (0), it was dropped from queue due to lack of space
Fri May 19 08:01:35 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (PERFORMANCE SNAPSHOT)
Fri May 19 08:01:35 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201705150747.3) was not posted for host (0), it was dropped from queue due to lack of space
Fri May 19 08:01:55 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (CLUSTER ERROR: DISK/SHELF COUNT MISMATCH)
Fri May 19 08:01:55 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201705150752.0) was not posted for host (0), it was dropped from queue due to lack of space
Fri May 19 08:06:27 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri May 19 08:07:36 BNT [filer02: iscsi.notice:notice]: ISCSI: New session from initiator iqn.1991-05.com.microsoft:pcb-722-ad1.pcbl722dom.local at IP addr 10.72.4.1
Fri May 19 08:10:34 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri May 19 08:14:41 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri May 19 08:18:29 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri May 19 08:22:41 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri May 19 08:26:56 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri May 19 08:38:21 BNT last message repeated 3 times
Fri May 19 08:38:21 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201705150800.0) for host (0) was not posted to NetApp. The system will drop the message
Fri May 19 09:00:00 BNT [filer02: kern.uptime.filer:info]:   9:00am up 59 mins, 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 45 iSCSI ops
Fri May 19 09:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 24 hours.
Fri May 19 10:00:00 BNT [filer02: kern.uptime.filer:info]:  10:00am up  1:59 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 45 iSCSI ops
Fri May 19 10:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 23 hours.
Fri May 19 11:00:00 BNT [filer02: kern.uptime.filer:info]:  11:00am up  2:59 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 1714 iSCSI ops
Fri May 19 11:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 22 hours.
Fri May 19 12:00:00 BNT [filer02: kern.uptime.filer:info]:  12:00pm up  3:59 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 5339 iSCSI ops
Fri May 19 12:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 21 hours.
Fri May 19 12:00:45 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Fri May 19 12:00:59 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri May 19 12:04:50 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri May 19 12:09:14 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri May 19 12:13:05 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri May 19 12:16:52 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri May 19 12:36:25 BNT last message repeated 5 times
Fri May 19 12:40:37 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri May 19 12:55:47 BNT last message repeated 4 times
Fri May 19 12:55:47 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201705151200.0) for host (0) was not posted to NetApp. The system will drop the message
Fri May 19 13:00:00 BNT [filer02: kern.uptime.filer:info]:   1:00pm up  4:59 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 5343 iSCSI ops
Fri May 19 13:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 20 hours.
Fri May 19 14:00:00 BNT [filer02: kern.uptime.filer:info]:   2:00pm up  5:59 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 6333 iSCSI ops
Fri May 19 14:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 19 hours.
Fri May 19 15:00:00 BNT [filer02: kern.uptime.filer:info]:   3:00pm up  6:59 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 169313 iSCSI ops
Fri May 19 15:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 18 hours.
Fri May 19 16:00:00 BNT [filer02: kern.uptime.filer:info]:   4:00pm up  7:59 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 175423 iSCSI ops
Fri May 19 16:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 17 hours.
Fri May 19 16:00:46 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Fri May 19 16:01:03 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri May 19 16:04:58 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri May 19 16:08:59 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri May 19 16:12:46 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri May 19 16:17:04 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri May 19 16:37:04 BNT last message repeated 5 times
Fri May 19 16:40:48 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri May 19 16:56:45 BNT last message repeated 4 times
Fri May 19 16:56:45 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201705151600.0) for host (0) was not posted to NetApp. The system will drop the message
Fri May 19 17:00:00 BNT [filer02: kern.uptime.filer:info]:   5:00pm up  8:59 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 189708 iSCSI ops
Fri May 19 17:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 16 hours.
Fri May 19 18:00:00 BNT [filer02: kern.uptime.filer:info]:   6:00pm up  9:59 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 190791 iSCSI ops
Fri May 19 18:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 15 hours.
Fri May 19 19:00:00 BNT [filer02: kern.uptime.filer:info]:   7:00pm up 10:59 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 314899 iSCSI ops
Fri May 19 19:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 14 hours.
Fri May 19 20:00:00 BNT [filer02: kern.uptime.filer:info]:   8:00pm up 11:59 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 458230 iSCSI ops
Fri May 19 20:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 13 hours.
Fri May 19 20:00:31 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Fri May 19 20:00:47 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri May 19 20:04:33 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri May 19 20:08:17 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri May 19 20:11:47 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri May 19 20:15:43 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri May 19 20:35:29 BNT last message repeated 5 times
Fri May 19 20:39:45 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri May 19 20:55:26 BNT last message repeated 4 times
Fri May 19 20:55:26 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201705152000.0) for host (0) was not posted to NetApp. The system will drop the message
Fri May 19 21:00:00 BNT [filer02: kern.uptime.filer:info]:   9:00pm up 12:59 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 565119 iSCSI ops
Fri May 19 21:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 12 hours.
Fri May 19 22:00:00 BNT [filer02: kern.uptime.filer:info]:  10:00pm up 13:59 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 1574638 iSCSI ops
Fri May 19 22:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 11 hours.
Fri May 19 23:00:00 BNT [filer02: kern.uptime.filer:info]:  11:00pm up 14:59 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 2720266 iSCSI ops
Fri May 19 23:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 10 hours.
Sat May 20 00:00:00 BNT [filer02: kern.uptime.filer:info]:  12:00am up 15:59 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 3920816 iSCSI ops
Sat May 20 00:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 9 hours.
Sat May 20 00:00:15 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of nightly.0 snapshot (Snapshot operation failed: No space left on device.)
Sat May 20 00:00:31 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Sat May 20 00:04:33 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Sat May 20 00:08:42 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Sat May 20 00:12:59 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Sat May 20 00:17:08 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Sat May 20 00:36:54 BNT last message repeated 5 times
Sat May 20 00:41:13 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Sat May 20 00:57:38 BNT last message repeated 4 times
Sat May 20 00:57:38 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201705160000.0) for host (0) was not posted to NetApp. The system will drop the message
Sat May 20 01:00:00 BNT [filer02: kern.uptime.filer:info]:   1:00am up 16:59 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 4693397 iSCSI ops
Sat May 20 01:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 8 hours.
Sat May 20 02:00:00 BNT [filer02: kern.uptime.filer:info]:   2:00am up 17:59 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 4968417 iSCSI ops
Sat May 20 02:00:01 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 7 hours.
Sat May 20 03:00:00 BNT [filer02: kern.uptime.filer:info]:   3:00am up 18:59 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 5273069 iSCSI ops
Sat May 20 03:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 6 hours.
Sat May 20 04:00:00 BNT [filer02: kern.uptime.filer:info]:   4:00am up 19:59 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 5582859 iSCSI ops
Sat May 20 04:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 5 hours.
Sat May 20 05:00:00 BNT [filer02: kern.uptime.filer:info]:   5:00am up 20:59 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 5842385 iSCSI ops
Sat May 20 05:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 4 hours.
Sat May 20 06:00:00 BNT [filer02: kern.uptime.filer:info]:   6:00am up 21:59 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 5915829 iSCSI ops
Sat May 20 06:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 3 hours.
Sat May 20 07:00:00 BNT [filer02: kern.uptime.filer:info]:   7:00am up 22:59 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 5915829 iSCSI ops
Sat May 20 07:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 2 hours.
Sat May 20 08:00:00 BNT [filer02: kern.uptime.filer:info]:   8:00am up  1 day,  0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 5915829 iSCSI ops
Sat May 20 08:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 1 hour.
Sat May 20 08:00:00 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] is still present in the system and should be removed.
Sat May 20 08:00:00 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] is still present in the system and should be removed.
Sat May 20 08:00:00 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] is still present in the system and should be removed.
Sat May 20 08:00:01 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Sat May 20 08:00:18 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Sat May 20 08:03:58 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Sat May 20 08:08:24 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Sat May 20 08:12:05 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Sat May 20 08:15:56 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Sat May 20 08:35:53 BNT last message repeated 5 times
Sat May 20 08:39:47 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Sat May 20 08:55:12 BNT last message repeated 4 times
Sat May 20 08:55:12 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201705160731.0) for host (0) was not posted to NetApp. The system will drop the message
Sat May 20 09:00:00 BNT [filer02: kern.uptime.filer:info]:   9:00am up  1 day,  1:00 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 5915829 iSCSI ops
Sat May 20 09:00:00 BNT [filer02: monitor.brokenDisk.notice:notice]: When two disks are broken in raid_dp volume,  the system shuts down automatically every 24  hours to encourage you to replace the disk.  If you reboot the system it will run for another  24 hours before shutting down. (The 24 hour timeout  may be increased by altering the "raid.timeout"  value using the "options" command.)
Sat May 20 09:00:00 BNT [filer02: monitor.shutdown.brokenDisk:EMERGENCY]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system now.
Sat May 20 09:00:03 BNT [filer02: cf.fsm.takeoverOfPartnerDisabled:notice]: Cluster monitor: takeover of filer01 disabled (local halt in progress)
Sat May 20 09:00:06 BNT [filer02: kern.shutdown:notice]: System shut down because : "BROKEN DISK".
Sun May 21 23:49:59 GMT [localhost: rc:notice]: The system was down for 168583 seconds
Mon May 22 07:49:29 BNT [filer02: cf.nm.nicTransitionUp:info]: replayed event: Interconnect link 0 is UP
Mon May 22 07:49:36 BNT [filer02: fci.initialization.failed:error]: replayed event: Initialization failed on Fibre Channel adapter 0b.
Mon May 22 07:49:38 BNT [filer02: nvram.battery.turned.on:info]: replayed event: The NVRAM battery is turned ON. It is turned OFF during system shutdown.
Mon May 22 07:49:41 BNT [filer02: diskown.isEnabled:info]: replayed event: software ownership has been enabled for this system
Mon May 22 07:49:45 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.2 is a local HA mailbox disk.
Mon May 22 07:49:45 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.0 is a local HA mailbox disk.
Mon May 22 07:49:45 BNT [filer02: fmmb.instStat.change:info]: replayed event: normal mailbox instance on local side.
Mon May 22 07:49:45 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.7 is a partner HA mailbox disk.
Mon May 22 07:49:45 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.3 is a partner HA mailbox disk.
Mon May 22 07:49:45 BNT [filer02: fmmb.instStat.change:info]: replayed event: normal mailbox instance on partner side.
Mon May 22 07:49:45 BNT [filer02: cf.fm.partner:info]: replayed event: Cluster monitor: partner 'filer01'
Mon May 22 07:49:45 BNT [filer02: cf.fm.timeMasterStatus:info]: replayed event: Acting as cluster time slave
Mon May 22 07:49:45 BNT [filer02: raid.assim.disk.brokenPreAssim:error]: replayed event: Broken Disk 0a.40 Shelf ? Bay ? [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] detected prior to assimilation. It should be removed.
Mon May 22 07:49:45 BNT [filer02: raid.assim.disk.brokenPreAssim:error]: replayed event: Broken Disk 0a.35 Shelf ? Bay ? [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] detected prior to assimilation. It should be removed.
Mon May 22 07:49:45 BNT [filer02: raid.assim.disk.brokenPreAssim:error]: replayed event: Broken Disk 0a.33 Shelf ? Bay ? [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] detected prior to assimilation. It should be removed.
Mon May 22 07:49:46 BNT [filer02: coredump.spare.none:info]: replayed event: No sparecore disk was found.
Mon May 22 07:49:47 BNT [filer02: raid.cksum.replay.summary:info]: replayed event: Replayed 0 checksum blocks.
Mon May 22 07:49:47 BNT [filer02: raid.stripe.replay.summary:info]: replayed event: Replayed 0 stripes.
Mon May 22 07:49:48 BNT [filer02: cf.fm.launch:info]: replayed event: Launching cluster monitor
Mon May 22 07:49:48 BNT [filer02: cf.fm.partner:info]: replayed event: Cluster monitor: partner 'filer01'
Mon May 22 07:49:49 BNT [filer02: ses.status.psWarning:warning]: replayed event: DS12-ESAS shelf 0 on channel 0c power warning for Power supply 1: non-critical status; DC undervoltage fault. This module is on the rear side of the shelf, at the left.
Mon May 22 07:49:49 BNT [filer02: ses.status.psWarning:warning]: replayed event: DS12-ESAS shelf 0 on channel 0c power warning for Power supply 2: non-critical status; DC undervoltage fault. This module is on the rear side of the shelf, at the right.
Mon May 22 07:49:50 BNT [filer02: cf.fsm.takeoverOfPartnerDisabled:notice]: replayed event: Cluster monitor: takeover of filer01 disabled (partner booting)
Mon May 22 07:49:59 BNT [filer02: fcp.service.adapter:warning]: replayed event: FCP started but no adapter is present!
Mon May 22 07:49:59 BNT [filer02: kern.syslog.msg:notice]: replayed event: The system was down for 168583 seconds
Mon May 22 07:50:00 BNT [filer02: disk.failmsg:error]: Disk 0a.40 (N00V0W8L): message received.
Mon May 22 07:50:00 BNT [filer02: disk.failmsg:error]: Disk 0a.35 (N00WLAUL): message received.
Mon May 22 07:50:00 BNT [filer02: disk.failmsg:error]: Disk 0a.33 (N00W75PL): message received.
Mon May 22 07:50:00 BNT [filer02: raid.disk.unload.done:info]: Unload of Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] has completed successfully
Mon May 22 07:50:00 BNT [filer02: raid.disk.unload.done:info]: Unload of Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] has completed successfully
Mon May 22 07:50:00 BNT [filer02: raid.disk.unload.done:info]: Unload of Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] has completed successfully
Mon May 22 07:50:00 BNT [filer02: raid.fdr.failed.ok:info]: Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] successfully deleted from spare pool
Mon May 22 07:50:00 BNT [filer02: raid.fdr.failed.ok:info]: Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] successfully deleted from spare pool
Mon May 22 07:50:01 BNT [filer02: raid.fdr.failed.ok:info]: Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] successfully deleted from spare pool
Mon May 22 07:50:01 BNT [filer02: dfu.firmwareUpToDate:info]: Firmware is up-to-date on all disk drives
Mon May 22 07:50:02 BNT [filer02: cf.fsm.takeoverOfPartnerEnabled:notice]: Cluster monitor: takeover of filer01 enabled
Mon May 22 07:50:02 BNT [filer02: cf.fsm.takeoverByPartnerEnabled:notice]: Cluster monitor: takeover of filer02 by filer01 enabled
Mon May 22 07:50:03 BNT [filer02: perf.archive.start:info]: Performance archiver started. Sampling 22 objects and 198 counters.
Mon May 22 07:50:05 BNT [filer02: netif.linkUp:info]: Ethernet e0a: Link up.
Mon May 22 07:50:05 BNT [filer02: netif.linkUp:info]: Ethernet e0b: Link up.
Mon May 22 07:50:06 BNT [filer02: pvif.switchLink:warning]: vif1: switching to e0a
Mon May 22 07:50:07 BNT [filer02: iscsi.service.startup:info]: iSCSI service startup
Mon May 22 07:50:08 BNT [filer02: cmds.sysconf.logErr:error]: sysconfig: Unless directed by NetApp Global Services the cf.takeover.on_panic option should be set to On. .
Mon May 22 07:50:08 BNT [filer02: raid.rg.recons.missing:notice]: RAID group /aggr1/plex0/rg0 is missing 2 disk(s).
Mon May 22 07:50:08 BNT [filer02: raid.rg.recons.cantStart:warning]: The reconstruction cannot start in RAID group /aggr1/plex0/rg0: No 520 bps disk of required size available in spare pool, targeting any  spare pool
Mon May 22 07:50:08 BNT [filer02: mgr.boot.disk_done:info]: NetApp Release 7.3.2 boot complete. Last disk update written at Sat May 20 09:00:08 BNT 2017
Mon May 22 07:50:08 BNT [filer02: monitor.raid.brokenDisk:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken.
Mon May 22 07:50:08 BNT [filer02: monitor.brokenDisk.notice:notice]: When two disks are broken in raid_dp volume,  the system shuts down automatically every 24  hours to encourage you to replace the disk.  If you reboot the system it will run for another  24 hours before shutting down. (The 24 hour timeout  may be increased by altering the "raid.timeout"  value using the "options" command.)
Mon May 22 07:50:08 BNT [filer02: diskown.errorReadingOwnership:warning]: error 3 (disk failed) while reading ownership on disk 0a.24 (S/N N00SSSAL)
Mon May 22 07:50:08 BNT [filer02: diskown.errorReadingOwnership:warning]: error 3 (disk failed) while reading ownership on disk 0a.26 (S/N N00V0UWL)
Mon May 22 07:50:08 BNT [filer02: mgr.boot.reason_ok:notice]: System rebooted.
Mon May 22 07:50:08 BNT [filer02: shelf.config.spha:info]: System is using single path HA attached storage only.
Mon May 22 07:50:09 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] is still present in the system and should be removed.
Mon May 22 07:50:09 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] is still present in the system and should be removed.
Mon May 22 07:50:09 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] is still present in the system and should be removed.
Mon May 22 07:50:12 BNT [filer02: iscsi.notice:notice]: ISCSI: New session from initiator iqn.1991-05.com.microsoft:pcb-722-ad1.pcbl722dom.local at IP addr 10.72.4.1
Mon May 22 07:50:29 BNT [filer02: raid.rg.spares.low:warning]: /aggr1/plex0/rg0
Mon May 22 07:50:34 BNT [filer02: nbt.nbns.registrationComplete:info]: NBT: All CIFS name registrations have completed for the local server.
Mon May 22 07:50:42 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SHUTDOWN PENDING (degraded mode))
Mon May 22 07:50:51 BNT last message repeated 2 times
Mon May 22 07:50:51 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201705160732.0) was not posted for host (0), it was dropped from queue due to lack of space
Mon May 22 07:50:58 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SPARES_LOW)
Mon May 22 07:50:58 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201705160732.1) was not posted for host (0), it was dropped from queue due to lack of space
Mon May 22 07:51:05 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (REBOOT (power on))
Mon May 22 07:51:05 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201705160732.2) was not posted for host (0), it was dropped from queue due to lack of space
Mon May 22 07:51:06 BNT [filer02: monitor.globalStatus.nonCritical:warning]: Disk on adapter 0a, shelf 2, bay 8, failed. Disk on adapter 0a, shelf 2, bay 3, failed. Disk on adapter 0a, shelf 2, bay 1, failed.  There are not enough spare disks. /vol/vol1 is full (using or reserving 100% of space and 0% of inodes, using 92% of reserve).
Mon May 22 07:51:17 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (PERFORMANCE SNAPSHOT)
Mon May 22 07:51:17 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201705160733.0) was not posted for host (0), it was dropped from queue due to lack of space
Mon May 22 07:55:48 BNT [filer02: cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 0a.33 (016F3AB5:B475F4EC:10505E83:02202755:00000000:00000000:00000000:00000000:00000000:00000000) has recently changed or the node (filer02) is missing the disk.
Mon May 22 07:55:48 BNT [filer02: cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 0a.35 (9115483F:9DD43BC4:27172773:99F7D8B4:00000000:00000000:00000000:00000000:00000000:00000000) has recently changed or the node (filer02) is missing the disk.
Mon May 22 07:55:48 BNT [filer02: cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 0a.40 (072AEB65:C662A9CF:E58B9917:9BFA787B:00000000:00000000:00000000:00000000:00000000:00000000) has recently changed or the node (filer02) is missing the disk.
Mon May 22 07:55:49 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Mon May 22 07:55:58 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Mon May 22 07:55:58 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201705160800.0) was not posted for host (0), it was dropped from queue due to lack of space
Mon May 22 08:00:00 BNT [filer02: kern.uptime.filer:info]:   8:00am up 9 mins, 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 2697 iSCSI ops
Mon May 22 08:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 24 hours.
Mon May 22 08:00:16 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Mon May 22 08:00:30 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Mon May 22 08:00:30 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201705161200.0) was not posted for host (0), it was dropped from queue due to lack of space
Mon May 22 08:04:02 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Mon May 22 08:07:59 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Mon May 22 08:11:39 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Mon May 22 08:15:22 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Mon May 22 08:18:55 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Mon May 22 08:22:36 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Mon May 22 08:22:36 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201705161600.0) for host (0) was not posted to NetApp. The system will drop the message
Mon May 22 08:57:03 BNT [filer02: cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 0a.33 (016F3AB5:B475F4EC:10505E83:02202755:00000000:00000000:00000000:00000000:00000000:00000000) has recently changed or the node (filer02) is missing the disk.
Mon May 22 08:57:03 BNT [filer02: cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 0a.35 (9115483F:9DD43BC4:27172773:99F7D8B4:00000000:00000000:00000000:00000000:00000000:00000000) has recently changed or the node (filer02) is missing the disk.
Mon May 22 08:57:03 BNT [filer02: cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 0a.40 (072AEB65:C662A9CF:E58B9917:9BFA787B:00000000:00000000:00000000:00000000:00000000:00000000) has recently changed or the node (filer02) is missing the disk.
Mon May 22 09:00:00 BNT [filer02: kern.uptime.filer:info]:   9:00am up  1:09 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 9101 iSCSI ops
Mon May 22 09:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 23 hours.
Mon May 22 10:00:00 BNT [filer02: kern.uptime.filer:info]:  10:00am up  2:09 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 128774 iSCSI ops
Mon May 22 10:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 22 hours.
Mon May 22 11:00:00 BNT [filer02: kern.uptime.filer:info]:  11:00am up  3:09 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 197735 iSCSI ops
Mon May 22 11:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 21 hours.
Mon May 22 12:00:00 BNT [filer02: kern.uptime.filer:info]:  12:00pm up  4:09 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 367202 iSCSI ops
Mon May 22 12:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 20 hours.
Mon May 22 12:00:16 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Mon May 22 12:00:30 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Mon May 22 12:04:48 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Mon May 22 12:08:23 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Mon May 22 12:12:19 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Mon May 22 12:16:39 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Mon May 22 12:33:24 BNT last message repeated 4 times
Mon May 22 12:37:48 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Mon May 22 12:57:53 BNT last message repeated 5 times
Mon May 22 12:57:53 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201705162000.0) for host (0) was not posted to NetApp. The system will drop the message
Mon May 22 13:00:00 BNT [filer02: kern.uptime.filer:info]:   1:00pm up  5:09 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 368248 iSCSI ops
Mon May 22 13:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 19 hours.
Mon May 22 14:00:00 BNT [filer02: kern.uptime.filer:info]:   2:00pm up  6:09 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 369024 iSCSI ops
Mon May 22 14:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 18 hours.
Mon May 22 15:00:00 BNT [filer02: kern.uptime.filer:info]:   3:00pm up  7:09 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 427793 iSCSI ops
Mon May 22 15:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 17 hours.
Mon May 22 16:00:00 BNT [filer02: kern.uptime.filer:info]:   4:00pm up  8:10 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 460486 iSCSI ops
Mon May 22 16:00:01 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 16 hours.
Mon May 22 16:00:01 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Mon May 22 16:00:18 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Mon May 22 16:03:51 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Mon May 22 16:08:15 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Mon May 22 16:11:49 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Mon May 22 16:15:22 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Mon May 22 16:31:40 BNT last message repeated 4 times
Mon May 22 16:35:49 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Mon May 22 16:55:56 BNT last message repeated 5 times
Mon May 22 16:55:56 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201705170000.0) for host (0) was not posted to NetApp. The system will drop the message
Mon May 22 17:00:00 BNT [filer02: kern.uptime.filer:info]:   5:00pm up  9:10 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 478832 iSCSI ops
Mon May 22 17:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 15 hours.
Mon May 22 18:00:00 BNT [filer02: kern.uptime.filer:info]:   6:00pm up 10:10 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 492516 iSCSI ops
Mon May 22 18:00:01 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 14 hours.
Mon May 22 19:00:00 BNT [filer02: kern.uptime.filer:info]:   7:00pm up 11:10 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 492538 iSCSI ops
Mon May 22 19:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 13 hours.
Mon May 22 20:00:00 BNT [filer02: kern.uptime.filer:info]:   8:00pm up 12:10 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 492538 iSCSI ops
Mon May 22 20:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 12 hours.
Mon May 22 20:00:00 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Mon May 22 20:38:48 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Mon May 22 20:53:55 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201705170733.0) for host (0) was not posted to NetApp. The system will drop the message
Mon May 22 21:00:00 BNT [filer02: kern.uptime.filer:info]:   9:00pm up 13:10 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 492538 iSCSI ops
Mon May 22 21:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 11 hours.
Mon May 22 22:00:00 BNT [filer02: kern.uptime.filer:info]:  10:00pm up 14:10 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 492538 iSCSI ops
Mon May 22 22:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 10 hours.
Mon May 22 23:00:01 BNT [filer02: kern.uptime.filer:info]:  11:00pm up 15:10 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 492538 iSCSI ops
Mon May 22 23:00:01 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 9 hours.
Tue May 23 00:00:00 BNT [filer02: kern.uptime.filer:info]:  12:00am up 16:10 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 492538 iSCSI ops
Tue May 23 00:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 8 hours.
Tue May 23 00:00:45 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of nightly.0 snapshot (Snapshot operation failed: No space left on device.)
Tue May 23 00:16:34 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SHUTDOWN PENDING (degraded mode))
Tue May 23 00:58:10 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201705170733.1) for host (0) was not posted to NetApp. The system will drop the message
Tue May 23 01:00:00 BNT [filer02: kern.uptime.filer:info]:   1:00am up 17:10 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 492538 iSCSI ops
Tue May 23 01:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 7 hours.
Tue May 23 02:00:00 BNT [filer02: kern.uptime.filer:info]:   2:00am up 18:10 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 492538 iSCSI ops
Tue May 23 02:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 6 hours.
Tue May 23 03:00:00 BNT [filer02: kern.uptime.filer:info]:   3:00am up 19:10 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 492538 iSCSI ops
Tue May 23 03:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 5 hours.
Tue May 23 04:00:00 BNT [filer02: kern.uptime.filer:info]:   4:00am up 20:10 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 492538 iSCSI ops
Tue May 23 04:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 4 hours.
Tue May 23 05:00:00 BNT [filer02: kern.uptime.filer:info]:   5:00am up 21:10 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 492538 iSCSI ops
Tue May 23 05:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 3 hours.
Tue May 23 06:00:00 BNT [filer02: kern.uptime.filer:info]:   6:00am up 22:10 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 492538 iSCSI ops
Tue May 23 06:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 2 hours.
Tue May 23 07:00:01 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] is still present in the system and should be removed.
Tue May 23 07:00:01 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] is still present in the system and should be removed.
Tue May 23 07:00:01 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] is still present in the system and should be removed.
Tue May 23 07:43:32 BNT [filer02: fci.link.break:error]: Link break detected on Fibre Channel adapter 0a.
Tue May 23 07:43:37 BNT [filer02: fci.adapter.error:warning]: Fibre Channel adapter driver encountered error "Failed to get loop map" on adapter 0a.
Mon May 22 23:44:15 GMT [localhost: rc:notice]: The system was down for 18 seconds
Tue May 23 07:43:39 BNT [filer02: cf.nm.nicTransitionUp:info]: replayed event: Interconnect link 0 is UP
Tue May 23 07:43:46 BNT [filer02: fci.initialization.failed:error]: replayed event: Initialization failed on Fibre Channel adapter 0b.
Tue May 23 07:43:48 BNT [filer02: nvram.battery.turned.on:info]: replayed event: The NVRAM battery is turned ON. It is turned OFF during system shutdown.
Tue May 23 07:43:51 BNT [filer02: diskown.isEnabled:info]: replayed event: software ownership has been enabled for this system
Tue May 23 07:43:55 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.2 is a local HA mailbox disk.
Tue May 23 07:43:55 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.0 is a local HA mailbox disk.
Tue May 23 07:43:55 BNT [filer02: fmmb.instStat.change:info]: replayed event: normal mailbox instance on local side.
Tue May 23 07:43:55 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.7 is a partner HA mailbox disk.
Tue May 23 07:43:55 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.3 is a partner HA mailbox disk.
Tue May 23 07:43:55 BNT [filer02: fmmb.instStat.change:info]: replayed event: normal mailbox instance on partner side.
Tue May 23 07:43:56 BNT [filer02: cf.fm.partner:info]: replayed event: Cluster monitor: partner 'filer01'
Tue May 23 07:43:56 BNT [filer02: cf.fm.timeMasterStatus:info]: replayed event: Acting as cluster time slave
Tue May 23 07:43:56 BNT [filer02: raid.assim.disk.brokenPreAssim:error]: replayed event: Broken Disk 0a.40 Shelf ? Bay ? [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] detected prior to assimilation. It should be removed.
Tue May 23 07:43:56 BNT [filer02: raid.assim.disk.brokenPreAssim:error]: replayed event: Broken Disk 0a.35 Shelf ? Bay ? [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] detected prior to assimilation. It should be removed.
Tue May 23 07:43:56 BNT [filer02: raid.assim.disk.brokenPreAssim:error]: replayed event: Broken Disk 0a.33 Shelf ? Bay ? [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] detected prior to assimilation. It should be removed.
Tue May 23 07:43:57 BNT [filer02: coredump.spare.none:info]: replayed event: No sparecore disk was found.
Tue May 23 07:43:57 BNT [filer02: raid.vol.replay.nvram:info]: replayed event: Performing raid replay on volume(s)
Tue May 23 07:43:57 BNT [filer02: raid.cksum.replay.summary:info]: replayed event: Replayed 0 checksum blocks.
Tue May 23 07:43:57 BNT [filer02: raid.stripe.replay.summary:info]: replayed event: Replayed 0 stripes.
Tue May 23 07:44:00 BNT [filer02: ses.status.psWarning:warning]: replayed event: DS12-ESAS shelf 0 on channel 0c power warning for Power supply 1: non-critical status; DC undervoltage fault. This module is on the rear side of the shelf, at the left.
Tue May 23 07:44:00 BNT [filer02: ses.status.psWarning:warning]: replayed event: DS12-ESAS shelf 0 on channel 0c power warning for Power supply 2: non-critical status; DC undervoltage fault. This module is on the rear side of the shelf, at the right.
Tue May 23 07:44:09 BNT [filer02: cf.fm.launch:info]: replayed event: Launching cluster monitor
Tue May 23 07:44:09 BNT [filer02: cf.fm.partner:info]: replayed event: Cluster monitor: partner 'filer01'
Tue May 23 07:44:15 BNT [filer02: fcp.service.adapter:warning]: replayed event: FCP started but no adapter is present!
Tue May 23 07:44:15 BNT [filer02: cf.fsm.takeoverOfPartnerEnabled:notice]: replayed event: Cluster monitor: takeover of filer01 enabled
Tue May 23 07:44:15 BNT [filer02: cf.fsm.takeoverByPartnerDisabled:notice]: replayed event: Cluster monitor: takeover of  by filer01 disabled (partner booting)
Tue May 23 07:44:15 BNT [filer02: kern.syslog.msg:notice]: replayed event: The system was down for 18 seconds
Tue May 23 07:44:17 BNT [filer02: disk.failmsg:error]: Disk 0a.40 (N00V0W8L): message received.
Tue May 23 07:44:17 BNT [filer02: disk.failmsg:error]: Disk 0a.35 (N00WLAUL): message received.
Tue May 23 07:44:17 BNT [filer02: disk.failmsg:error]: Disk 0a.33 (N00W75PL): message received.
Tue May 23 07:44:17 BNT [filer02: raid.disk.unload.done:info]: Unload of Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] has completed successfully
Tue May 23 07:44:17 BNT [filer02: raid.disk.unload.done:info]: Unload of Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] has completed successfully
Tue May 23 07:44:17 BNT [filer02: raid.disk.unload.done:info]: Unload of Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] has completed successfully
Tue May 23 07:44:17 BNT [filer02: raid.fdr.failed.ok:info]: Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] successfully deleted from spare pool
Tue May 23 07:44:17 BNT [filer02: raid.fdr.failed.ok:info]: Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] successfully deleted from spare pool
Tue May 23 07:44:17 BNT [filer02: raid.fdr.failed.ok:info]: Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] successfully deleted from spare pool
Tue May 23 07:44:17 BNT [filer02: cf.fsm.takeoverByPartnerEnabled:notice]: Cluster monitor: takeover of filer02 by filer01 enabled
Tue May 23 07:44:18 BNT [filer02: dfu.firmwareUpToDate:info]: Firmware is up-to-date on all disk drives
Tue May 23 07:44:19 BNT [filer02: perf.archive.start:info]: Performance archiver started. Sampling 22 objects and 198 counters.
Tue May 23 07:44:21 BNT [filer02: netif.linkUp:info]: Ethernet e0b: Link up.
Tue May 23 07:44:21 BNT [filer02: netif.linkUp:info]: Ethernet e0a: Link up.
Tue May 23 07:44:22 BNT [filer02: pvif.switchLink:warning]: vif1: switching to e0a
Tue May 23 07:44:23 BNT [filer02: rc:info]: relog syslog Tue May 23 07:43:40 BNT [filer02: fci.adapter.error:warning]: Fibre Channel adapter driver encountered error "Failed t
Tue May 23 07:44:23 BNT [filer02: rc:info]: relog syslog Tue May 23 07:43:40 BNT [filer02: fci.adapter.reset:debug]: Resetting Fibre Channel adapter 0a.
Tue May 23 07:44:23 BNT [filer02: iscsi.service.startup:info]: iSCSI service startup
Tue May 23 07:44:23 BNT [filer02: netif.linkDown:info]: Ethernet e0b: Link down, check cable.
Tue May 23 07:44:24 BNT [filer02: netif.linkDown:info]: Ethernet e0a: Link down, check cable.
Tue May 23 07:44:24 BNT [filer02: pvif.allLinksDown:CRITICAL]: vif1: all links down
Tue May 23 07:44:24 BNT [filer02: cmds.sysconf.logErr:error]: sysconfig: Unless directed by NetApp Global Services the cf.takeover.on_panic option should be set to On. .
Tue May 23 07:44:24 BNT [filer02: raid.rg.recons.missing:notice]: RAID group /aggr1/plex0/rg0 is missing 2 disk(s).
Tue May 23 07:44:24 BNT [filer02: raid.rg.recons.cantStart:warning]: The reconstruction cannot start in RAID group /aggr1/plex0/rg0: No 520 bps disk of required size available in spare pool, targeting any  spare pool
Tue May 23 07:44:24 BNT [filer02: mgr.boot.disk_done:info]: NetApp Release 7.3.2 boot complete. Last disk update written at Tue May 23 07:43:21 BNT 2017
Tue May 23 07:44:24 BNT [filer02: monitor.raid.brokenDisk:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken.
Tue May 23 07:44:24 BNT [filer02: monitor.brokenDisk.notice:notice]: When two disks are broken in raid_dp volume,  the system shuts down automatically every 24  hours to encourage you to replace the disk.  If you reboot the system it will run for another  24 hours before shutting down. (The 24 hour timeout  may be increased by altering the "raid.timeout"  value using the "options" command.)
Tue May 23 07:44:25 BNT [filer02: diskown.errorReadingOwnership:warning]: error 3 (disk failed) while reading ownership on disk 0a.24 (S/N N00SSSAL)
Tue May 23 07:44:25 BNT [filer02: diskown.errorReadingOwnership:warning]: error 3 (disk failed) while reading ownership on disk 0a.26 (S/N N00V0UWL)
Tue May 23 07:44:25 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] is still present in the system and should be removed.
Tue May 23 07:44:25 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] is still present in the system and should be removed.
Tue May 23 07:44:25 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] is still present in the system and should be removed.
Tue May 23 07:44:25 BNT [filer02: shelf.config.spha:info]: System is using single path HA attached storage only.
Tue May 23 07:44:25 BNT [filer02: mgr.boot.reason_ok:notice]: System rebooted after power-on.
Tue May 23 07:44:26 BNT [filer02: netif.linkUp:info]: Ethernet e0b: Link up.
Tue May 23 07:44:27 BNT [filer02: pvif.switchLink:warning]: vif1: switching to e0b
Tue May 23 07:44:30 BNT [filer02: netif.linkUp:info]: Ethernet e0a: Link up.
Tue May 23 07:44:46 BNT [filer02: raid.rg.spares.low:warning]: /aggr1/plex0/rg0
Tue May 23 07:44:49 BNT [filer02: nbt.nbns.registrationComplete:info]: NBT: All CIFS name registrations have completed for the local server.
Tue May 23 07:44:53 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (REBOOT (power on))
Tue May 23 07:45:00 BNT last message repeated 2 times
Tue May 23 07:45:00 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201705170733.2) was not posted for host (0), it was dropped from queue due to lack of space
Tue May 23 07:45:04 BNT [filer02: monitor.globalStatus.nonCritical:warning]: Disk on adapter 0a, shelf 2, bay 8, failed. Disk on adapter 0a, shelf 2, bay 3, failed. Disk on adapter 0a, shelf 2, bay 1, failed.  There are not enough spare disks. /vol/vol1 is full (using or reserving 100% of space and 0% of inodes, using 92% of reserve).
Tue May 23 07:45:07 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SPARES_LOW)
Tue May 23 07:45:07 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201705170733.3) was not posted for host (0), it was dropped from queue due to lack of space
Tue May 23 07:45:13 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (PERFORMANCE SNAPSHOT)
Tue May 23 07:45:13 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201705170734.0) was not posted for host (0), it was dropped from queue due to lack of space
Tue May 23 07:45:33 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Tue May 23 07:45:33 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201705170800.0) was not posted for host (0), it was dropped from queue due to lack of space
Tue May 23 07:50:24 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Tue May 23 07:54:40 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Tue May 23 07:58:53 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Tue May 23 08:00:17 BNT [filer02: kern.uptime.filer:info]:   8:00am up 15 mins, 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 0 iSCSI ops
Tue May 23 08:00:17 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 24 hours.
Tue May 23 08:00:31 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Tue May 23 08:00:38 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Tue May 23 08:00:38 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201705171200.0) was not posted for host (0), it was dropped from queue due to lack of space
Tue May 23 08:04:30 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Tue May 23 08:06:19 BNT [filer02: iscsi.notice:notice]: ISCSI: New session from initiator iqn.1991-05.com.microsoft:pcb-722-ad1.pcbl722dom.local at IP addr 10.72.4.1
Tue May 23 08:08:05 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Tue May 23 08:11:45 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Tue May 23 08:15:20 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Tue May 23 08:19:10 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Tue May 23 08:19:10 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201705171600.0) for host (0) was not posted to NetApp. The system will drop the message
Tue May 23 09:00:00 BNT [filer02: kern.uptime.filer:info]:   9:00am up  1:15 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 1152 iSCSI ops
Tue May 23 09:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 23 hours.
Tue May 23 10:00:01 BNT [filer02: kern.uptime.filer:info]:  10:00am up  2:15 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 1686 iSCSI ops
Tue May 23 10:00:03 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 22 hours.
Tue May 23 11:00:00 BNT [filer02: kern.uptime.filer:info]:  11:00am up  3:15 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 3592 iSCSI ops
Tue May 23 11:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 21 hours.
Tue May 23 12:00:00 BNT [filer02: kern.uptime.filer:info]:  12:00pm up  4:15 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 347475 iSCSI ops
Tue May 23 12:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 20 hours.
Tue May 23 12:00:31 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Tue May 23 12:00:46 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Tue May 23 12:04:31 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Tue May 23 12:08:15 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Tue May 23 12:12:13 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Tue May 23 12:15:57 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Tue May 23 12:32:05 BNT last message repeated 4 times
Tue May 23 12:36:29 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Tue May 23 12:55:59 BNT last message repeated 5 times
Tue May 23 12:55:59 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201705172000.0) for host (0) was not posted to NetApp. The system will drop the message
Tue May 23 13:00:00 BNT [filer02: kern.uptime.filer:info]:   1:00pm up  5:15 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 785590 iSCSI ops
Tue May 23 13:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 19 hours.
Tue May 23 14:00:00 BNT [filer02: kern.uptime.filer:info]:   2:00pm up  6:15 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 792560 iSCSI ops
Tue May 23 14:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 18 hours.
Tue May 23 15:00:00 BNT [filer02: kern.uptime.filer:info]:   3:00pm up  7:15 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 801759 iSCSI ops
Tue May 23 15:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 17 hours.
Tue May 23 16:00:00 BNT [filer02: kern.uptime.filer:info]:   4:00pm up  8:15 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 816565 iSCSI ops
Tue May 23 16:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 16 hours.
Tue May 23 16:00:16 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Tue May 23 16:00:35 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Tue May 23 16:04:39 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Tue May 23 16:08:38 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Tue May 23 16:12:20 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Tue May 23 16:16:41 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Tue May 23 16:32:33 BNT last message repeated 4 times
Tue May 23 16:36:53 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Tue May 23 16:51:55 BNT last message repeated 4 times
Tue May 23 16:56:06 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201705180000.0) for host (0) was not posted to NetApp. The system will drop the message
Tue May 23 17:00:00 BNT [filer02: kern.uptime.filer:info]:   5:00pm up  9:15 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 817236 iSCSI ops
Tue May 23 17:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 15 hours.
Tue May 23 18:00:00 BNT [filer02: kern.uptime.filer:info]:   6:00pm up 10:15 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 821236 iSCSI ops
Tue May 23 18:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 14 hours.
Tue May 23 19:00:00 BNT [filer02: kern.uptime.filer:info]:   7:00pm up 11:15 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 821236 iSCSI ops
Tue May 23 19:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 13 hours.
Tue May 23 20:00:00 BNT [filer02: kern.uptime.filer:info]:   8:00pm up 12:15 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 821236 iSCSI ops
Tue May 23 20:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 12 hours.
Tue May 23 20:00:01 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Tue May 23 20:15:42 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Tue May 23 20:55:09 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Tue May 23 20:55:09 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201705180751.0) for host (0) was not posted to NetApp. The system will drop the message
Tue May 23 21:00:00 BNT [filer02: kern.uptime.filer:info]:   9:00pm up 13:16 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 821236 iSCSI ops
Tue May 23 21:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 11 hours.
Tue May 23 22:00:00 BNT [filer02: kern.uptime.filer:info]:  10:00pm up 14:16 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 821236 iSCSI ops
Tue May 23 22:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 10 hours.
Tue May 23 23:00:00 BNT [filer02: kern.uptime.filer:info]:  11:00pm up 15:16 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 821236 iSCSI ops
Tue May 23 23:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 9 hours.
Wed May 24 00:00:00 BNT [filer02: kern.uptime.filer:info]:  12:00am up 16:16 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 821236 iSCSI ops
Wed May 24 00:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 8 hours.
Wed May 24 00:00:00 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of nightly.0 snapshot (Snapshot operation failed: No space left on device.)
Wed May 24 00:55:25 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201705180752.0) for host (0) was not posted to NetApp. The system will drop the message
Wed May 24 01:00:00 BNT [filer02: kern.uptime.filer:info]:   1:00am up 17:16 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 821236 iSCSI ops
Wed May 24 01:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 7 hours.
Wed May 24 02:00:00 BNT [filer02: kern.uptime.filer:info]:   2:00am up 18:16 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 821236 iSCSI ops
Wed May 24 02:00:01 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 6 hours.
Wed May 24 03:00:00 BNT [filer02: kern.uptime.filer:info]:   3:00am up 19:16 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 821236 iSCSI ops
Wed May 24 03:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 5 hours.
Wed May 24 04:00:00 BNT [filer02: kern.uptime.filer:info]:   4:00am up 20:16 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 821236 iSCSI ops
Wed May 24 04:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 4 hours.
Wed May 24 05:00:00 BNT [filer02: kern.uptime.filer:info]:   5:00am up 21:16 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 821236 iSCSI ops
Wed May 24 05:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 3 hours.
Wed May 24 06:00:00 BNT [filer02: kern.uptime.filer:info]:   6:00am up 22:16 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 821236 iSCSI ops
Wed May 24 06:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 2 hours.
Wed May 24 07:00:00 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] is still present in the system and should be removed.
Wed May 24 07:00:00 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] is still present in the system and should be removed.
Wed May 24 07:00:00 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] is still present in the system and should be removed.
Wed May 24 08:00:00 BNT [filer02: kern.uptime.filer:info]:   8:00am up  1 day, 16 mins, 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 821236 iSCSI ops
Wed May 24 08:00:00 BNT [filer02: monitor.brokenDisk.notice:notice]: When two disks are broken in raid_dp volume,  the system shuts down automatically every 24  hours to encourage you to replace the disk.  If you reboot the system it will run for another  24 hours before shutting down. (The 24 hour timeout  may be increased by altering the "raid.timeout"  value using the "options" command.)
Wed May 24 08:00:00 BNT [filer02: monitor.shutdown.brokenDisk:EMERGENCY]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system now.
Wed May 24 08:00:03 BNT [filer02: cf.fsm.takeoverOfPartnerDisabled:notice]: Cluster monitor: takeover of filer01 disabled (local halt in progress)
Wed May 24 08:00:06 BNT [filer02: kern.shutdown:notice]: System shut down because : "BROKEN DISK".
Wed May 24 00:02:24 GMT [localhost: rc:notice]: The system was down for 134 seconds
Wed May 24 08:01:58 BNT [filer02: cf.nm.nicTransitionUp:info]: replayed event: Interconnect link 0 is UP
Wed May 24 08:02:05 BNT [filer02: fci.initialization.failed:error]: replayed event: Initialization failed on Fibre Channel adapter 0b.
Wed May 24 08:02:07 BNT [filer02: nvram.battery.turned.on:info]: replayed event: The NVRAM battery is turned ON. It is turned OFF during system shutdown.
Wed May 24 08:02:10 BNT [filer02: diskown.isEnabled:info]: replayed event: software ownership has been enabled for this system
Wed May 24 08:02:14 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.2 is a local HA mailbox disk.
Wed May 24 08:02:14 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.0 is a local HA mailbox disk.
Wed May 24 08:02:14 BNT [filer02: fmmb.instStat.change:info]: replayed event: normal mailbox instance on local side.
Wed May 24 08:02:14 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.7 is a partner HA mailbox disk.
Wed May 24 08:02:14 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.3 is a partner HA mailbox disk.
Wed May 24 08:02:14 BNT [filer02: fmmb.instStat.change:info]: replayed event: normal mailbox instance on partner side.
Wed May 24 08:02:14 BNT [filer02: cf.fm.partner:info]: replayed event: Cluster monitor: partner 'filer01'
Wed May 24 08:02:14 BNT [filer02: cf.fm.timeMasterStatus:info]: replayed event: Acting as cluster time slave
Wed May 24 08:02:15 BNT [filer02: raid.assim.disk.brokenPreAssim:error]: replayed event: Broken Disk 0a.40 Shelf ? Bay ? [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] detected prior to assimilation. It should be removed.
Wed May 24 08:02:15 BNT [filer02: raid.assim.disk.brokenPreAssim:error]: replayed event: Broken Disk 0a.35 Shelf ? Bay ? [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] detected prior to assimilation. It should be removed.
Wed May 24 08:02:15 BNT [filer02: raid.assim.disk.brokenPreAssim:error]: replayed event: Broken Disk 0a.33 Shelf ? Bay ? [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] detected prior to assimilation. It should be removed.
Wed May 24 08:02:15 BNT [filer02: coredump.spare.none:info]: replayed event: No sparecore disk was found.
Wed May 24 08:02:16 BNT [filer02: raid.cksum.replay.summary:info]: replayed event: Replayed 0 checksum blocks.
Wed May 24 08:02:16 BNT [filer02: raid.stripe.replay.summary:info]: replayed event: Replayed 0 stripes.
Wed May 24 08:02:17 BNT [filer02: cf.fm.launch:info]: replayed event: Launching cluster monitor
Wed May 24 08:02:18 BNT [filer02: cf.fm.partner:info]: replayed event: Cluster monitor: partner 'filer01'
Wed May 24 08:02:19 BNT [filer02: cf.fsm.takeoverOfPartnerDisabled:notice]: replayed event: Cluster monitor: takeover of filer01 disabled (partner booting)
Wed May 24 08:02:24 BNT [filer02: fcp.service.adapter:warning]: replayed event: FCP started but no adapter is present!
Wed May 24 08:02:24 BNT [filer02: kern.syslog.msg:notice]: replayed event: The system was down for 134 seconds
Wed May 24 08:02:25 BNT [filer02: disk.failmsg:error]: Disk 0a.40 (N00V0W8L): message received.
Wed May 24 08:02:25 BNT [filer02: disk.failmsg:error]: Disk 0a.35 (N00WLAUL): message received.
Wed May 24 08:02:25 BNT [filer02: disk.failmsg:error]: Disk 0a.33 (N00W75PL): message received.
Wed May 24 08:02:25 BNT [filer02: raid.disk.unload.done:info]: Unload of Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] has completed successfully
Wed May 24 08:02:25 BNT [filer02: raid.disk.unload.done:info]: Unload of Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] has completed successfully
Wed May 24 08:02:25 BNT [filer02: raid.disk.unload.done:info]: Unload of Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] has completed successfully
Wed May 24 08:02:26 BNT [filer02: raid.fdr.failed.ok:info]: Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] successfully deleted from spare pool
Wed May 24 08:02:26 BNT [filer02: raid.fdr.failed.ok:info]: Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] successfully deleted from spare pool
Wed May 24 08:02:26 BNT [filer02: raid.fdr.failed.ok:info]: Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] successfully deleted from spare pool
Wed May 24 08:02:26 BNT [filer02: dfu.firmwareUpToDate:info]: Firmware is up-to-date on all disk drives
Wed May 24 08:02:27 BNT [filer02: cf.fsm.takeoverOfPartnerEnabled:notice]: Cluster monitor: takeover of filer01 enabled
Wed May 24 08:02:27 BNT [filer02: cf.fsm.takeoverByPartnerEnabled:notice]: Cluster monitor: takeover of filer02 by filer01 enabled
Wed May 24 08:02:28 BNT [filer02: perf.archive.start:info]: Performance archiver started. Sampling 22 objects and 198 counters.
Wed May 24 08:02:30 BNT [filer02: netif.linkUp:info]: Ethernet e0b: Link up.
Wed May 24 08:02:30 BNT [filer02: netif.linkUp:info]: Ethernet e0a: Link up.
Wed May 24 08:02:31 BNT [filer02: pvif.switchLink:warning]: vif1: switching to e0a
Wed May 24 08:02:32 BNT [filer02: iscsi.service.startup:info]: iSCSI service startup
Wed May 24 08:02:33 BNT [filer02: cmds.sysconf.logErr:error]: sysconfig: Unless directed by NetApp Global Services the cf.takeover.on_panic option should be set to On. .
Wed May 24 08:02:33 BNT [filer02: raid.rg.recons.missing:notice]: RAID group /aggr1/plex0/rg0 is missing 2 disk(s).
Wed May 24 08:02:33 BNT [filer02: raid.rg.recons.cantStart:warning]: The reconstruction cannot start in RAID group /aggr1/plex0/rg0: No 520 bps disk of required size available in spare pool, targeting any  spare pool
Wed May 24 08:02:33 BNT [filer02: mgr.boot.disk_done:info]: NetApp Release 7.3.2 boot complete. Last disk update written at Wed May 24 08:00:07 BNT 2017
Wed May 24 08:02:33 BNT [filer02: monitor.raid.brokenDisk:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken.
Wed May 24 08:02:33 BNT [filer02: monitor.brokenDisk.notice:notice]: When two disks are broken in raid_dp volume,  the system shuts down automatically every 24  hours to encourage you to replace the disk.  If you reboot the system it will run for another  24 hours before shutting down. (The 24 hour timeout  may be increased by altering the "raid.timeout"  value using the "options" command.)
Wed May 24 08:02:34 BNT [filer02: diskown.errorReadingOwnership:warning]: error 3 (disk failed) while reading ownership on disk 0a.26 (S/N N00V0UWL)
Wed May 24 08:02:34 BNT [filer02: diskown.errorReadingOwnership:warning]: error 3 (disk failed) while reading ownership on disk 0a.24 (S/N N00SSSAL)
Wed May 24 08:02:34 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] is still present in the system and should be removed.
Wed May 24 08:02:34 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] is still present in the system and should be removed.
Wed May 24 08:02:34 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] is still present in the system and should be removed.
Wed May 24 08:02:34 BNT [filer02: shelf.config.spha:info]: System is using single path HA attached storage only.
Wed May 24 08:02:34 BNT [filer02: mgr.boot.reason_ok:notice]: System rebooted.
Wed May 24 08:02:36 BNT [filer02: iscsi.notice:notice]: ISCSI: New session from initiator iqn.1991-05.com.microsoft:pcb-722-ad1.pcbl722dom.local at IP addr 10.72.4.1
Wed May 24 08:02:56 BNT [filer02: raid.rg.spares.low:warning]: /aggr1/plex0/rg0
Wed May 24 08:02:58 BNT [filer02: nbt.nbns.registrationComplete:info]: NBT: All CIFS name registrations have completed for the local server.
Wed May 24 08:03:03 BNT [filer02: monitor.globalStatus.nonCritical:warning]: Disk on adapter 0a, shelf 2, bay 8, failed. Disk on adapter 0a, shelf 2, bay 3, failed. Disk on adapter 0a, shelf 2, bay 1, failed.  There are not enough spare disks. /vol/vol1 is full (using or reserving 100% of space and 0% of inodes, using 92% of reserve).
Wed May 24 08:03:03 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (REBOOT (power on))
Wed May 24 08:03:11 BNT last message repeated 2 times
Wed May 24 08:03:11 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201705180752.1) was not posted for host (0), it was dropped from queue due to lack of space
Wed May 24 08:03:18 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SPARES_LOW)
Wed May 24 08:03:18 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201705180752.2) was not posted for host (0), it was dropped from queue due to lack of space
Wed May 24 08:03:25 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (PERFORMANCE SNAPSHOT)
Wed May 24 08:03:25 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201705180752.3) was not posted for host (0), it was dropped from queue due to lack of space
Wed May 24 08:03:42 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Wed May 24 08:03:42 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201705180800.0) was not posted for host (0), it was dropped from queue due to lack of space
Wed May 24 08:08:13 BNT [filer02: cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 0a.33 (016F3AB5:B475F4EC:10505E83:02202755:00000000:00000000:00000000:00000000:00000000:00000000) has recently changed or the node (filer02) is missing the disk.
Wed May 24 08:08:13 BNT [filer02: cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 0a.35 (9115483F:9DD43BC4:27172773:99F7D8B4:00000000:00000000:00000000:00000000:00000000:00000000) has recently changed or the node (filer02) is missing the disk.
Wed May 24 08:08:13 BNT [filer02: cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 0a.40 (072AEB65:C662A9CF:E58B9917:9BFA787B:00000000:00000000:00000000:00000000:00000000:00000000) has recently changed or the node (filer02) is missing the disk.
Wed May 24 08:08:13 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Wed May 24 08:08:20 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Wed May 24 08:08:20 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201705181200.0) was not posted for host (0), it was dropped from queue due to lack of space
Wed May 24 08:12:37 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Wed May 24 08:17:05 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Wed May 24 08:21:00 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Wed May 24 08:25:22 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Wed May 24 08:29:05 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Wed May 24 08:36:33 BNT last message repeated 2 times
Wed May 24 08:36:33 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201705181600.0) for host (0) was not posted to NetApp. The system will drop the message
Wed May 24 09:00:00 BNT [filer02: kern.uptime.filer:info]:   9:00am up 57 mins, 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 2851 iSCSI ops
Wed May 24 09:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 24 hours.
Wed May 24 09:09:29 BNT [filer02: cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 0a.33 (016F3AB5:B475F4EC:10505E83:02202755:00000000:00000000:00000000:00000000:00000000:00000000) has recently changed or the node (filer02) is missing the disk.
Wed May 24 09:09:29 BNT [filer02: cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 0a.35 (9115483F:9DD43BC4:27172773:99F7D8B4:00000000:00000000:00000000:00000000:00000000:00000000) has recently changed or the node (filer02) is missing the disk.
Wed May 24 09:09:29 BNT [filer02: cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 0a.40 (072AEB65:C662A9CF:E58B9917:9BFA787B:00000000:00000000:00000000:00000000:00000000:00000000) has recently changed or the node (filer02) is missing the disk.
Wed May 24 10:00:00 BNT [filer02: kern.uptime.filer:info]:  10:00am up  1:57 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 24249 iSCSI ops
Wed May 24 10:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 23 hours.
Wed May 24 11:00:00 BNT [filer02: kern.uptime.filer:info]:  11:00am up  2:57 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 72359 iSCSI ops
Wed May 24 11:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 22 hours.
Wed May 24 12:00:00 BNT [filer02: kern.uptime.filer:info]:  12:00pm up  3:57 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 93755 iSCSI ops
Wed May 24 12:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 21 hours.
Wed May 24 12:00:45 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Wed May 24 12:01:07 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Wed May 24 12:05:08 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Wed May 24 12:09:30 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Wed May 24 12:13:19 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Wed May 24 12:17:04 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Wed May 24 12:34:01 BNT last message repeated 4 times
Wed May 24 12:37:56 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Wed May 24 12:54:22 BNT last message repeated 4 times
Wed May 24 12:58:40 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Wed May 24 12:58:40 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201705182000.0) for host (0) was not posted to NetApp. The system will drop the message
Wed May 24 13:00:00 BNT [filer02: kern.uptime.filer:info]:   1:00pm up  4:57 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 94324 iSCSI ops
Wed May 24 13:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 20 hours.
Wed May 24 14:00:00 BNT [filer02: kern.uptime.filer:info]:   2:00pm up  5:57 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 95527 iSCSI ops
Wed May 24 14:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 19 hours.
Wed May 24 15:00:00 BNT [filer02: kern.uptime.filer:info]:   3:00pm up  6:57 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 99045 iSCSI ops
Wed May 24 15:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 18 hours.
Wed May 24 16:00:00 BNT [filer02: kern.uptime.filer:info]:   4:00pm up  7:57 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 155111 iSCSI ops
Wed May 24 16:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 17 hours.
Wed May 24 16:00:30 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Wed May 24 16:00:50 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Wed May 24 16:04:37 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Wed May 24 16:08:53 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Wed May 24 16:29:02 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Wed May 24 16:56:43 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201705190000.0) for host (0) was not posted to NetApp. The system will drop the message
Wed May 24 17:00:00 BNT [filer02: kern.uptime.filer:info]:   5:00pm up  8:57 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 162355 iSCSI ops
Wed May 24 17:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 16 hours.
Wed May 24 18:00:00 BNT [filer02: kern.uptime.filer:info]:   6:00pm up  9:57 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 234844 iSCSI ops
Wed May 24 18:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 15 hours.
Wed May 24 19:00:00 BNT [filer02: kern.uptime.filer:info]:   7:00pm up 10:57 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 234844 iSCSI ops
Wed May 24 19:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 14 hours.
Wed May 24 20:00:00 BNT [filer02: kern.uptime.filer:info]:   8:00pm up 11:57 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 234846 iSCSI ops
Wed May 24 20:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 13 hours.
Wed May 24 20:00:31 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Wed May 24 20:47:42 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Wed May 24 20:55:24 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201705190800.0) for host (0) was not posted to NetApp. The system will drop the message
Wed May 24 21:00:00 BNT [filer02: kern.uptime.filer:info]:   9:00pm up 12:57 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 234846 iSCSI ops
Wed May 24 21:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 12 hours.
Wed May 24 22:00:00 BNT [filer02: kern.uptime.filer:info]:  10:00pm up 13:57 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 234846 iSCSI ops
Wed May 24 22:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 11 hours.
Wed May 24 23:00:00 BNT [filer02: kern.uptime.filer:info]:  11:00pm up 14:57 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 234846 iSCSI ops
Wed May 24 23:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 10 hours.
Thu May 25 00:00:00 BNT [filer02: kern.uptime.filer:info]:  12:00am up 15:57 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 234846 iSCSI ops
Thu May 25 00:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 9 hours.
Thu May 25 00:00:16 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of nightly.0 snapshot (Snapshot operation failed: No space left on device.)
Thu May 25 00:12:29 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SHUTDOWN PENDING (degraded mode))
Thu May 25 00:32:29 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SHUTDOWN PENDING (degraded mode))
Thu May 25 00:56:05 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201705190801.0) for host (0) was not posted to NetApp. The system will drop the message
Thu May 25 01:00:00 BNT [filer02: kern.uptime.filer:info]:   1:00am up 16:57 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 234846 iSCSI ops
Thu May 25 01:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 8 hours.
Thu May 25 02:00:00 BNT [filer02: kern.uptime.filer:info]:   2:00am up 17:57 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 234846 iSCSI ops
Thu May 25 02:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 7 hours.
Thu May 25 03:00:00 BNT [filer02: kern.uptime.filer:info]:   3:00am up 18:57 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 234846 iSCSI ops
Thu May 25 03:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 6 hours.
Thu May 25 04:00:00 BNT [filer02: kern.uptime.filer:info]:   4:00am up 19:57 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 234846 iSCSI ops
Thu May 25 04:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 5 hours.
Thu May 25 05:00:00 BNT [filer02: kern.uptime.filer:info]:   5:00am up 20:57 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 234846 iSCSI ops
Thu May 25 05:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 4 hours.
Thu May 25 06:00:00 BNT [filer02: kern.uptime.filer:info]:   6:00am up 21:58 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 234846 iSCSI ops
Thu May 25 06:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 3 hours.
Thu May 25 07:00:00 BNT [filer02: kern.uptime.filer:info]:   7:00am up 22:58 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 234846 iSCSI ops
Thu May 25 07:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 2 hours.
Thu May 25 07:45:26 BNT [filer02: ses.status.psWarning:warning]: DS14-Mk2-AT shelf 1 on channel 0a power warning for Power supply 2: non-critical status; DC undervoltage fault. This module is on the rear side of the shelf, at the right.
Thu May 25 07:45:29 BNT [filer02: fci.link.break:error]: Link break detected on Fibre Channel adapter 0a.
Wed May 24 23:46:11 GMT [localhost: rc:notice]: The system was down for 18 seconds
Wed May 24 23:46:11 GMT [filer02: cf.fsm.takeoverOfPartnerEnabled:notice]: Cluster monitor: takeover of filer01 enabled
Thu May 25 07:45:35 BNT [filer02: cf.nm.nicTransitionUp:info]: replayed event: Interconnect link 0 is UP
Thu May 25 07:45:42 BNT [filer02: fci.initialization.failed:error]: replayed event: Initialization failed on Fibre Channel adapter 0b.
Thu May 25 07:45:44 BNT [filer02: nvram.battery.turned.on:info]: replayed event: The NVRAM battery is turned ON. It is turned OFF during system shutdown.
Thu May 25 07:46:12 BNT [filer02: cf.fsm.takeoverByPartnerEnabled:notice]: Cluster monitor: takeover of filer02 by filer01 enabled
Thu May 25 07:45:47 BNT [filer02: diskown.isEnabled:info]: replayed event: software ownership has been enabled for this system
Thu May 25 07:45:51 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.2 is a local HA mailbox disk.
Thu May 25 07:45:51 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.0 is a local HA mailbox disk.
Thu May 25 07:45:51 BNT [filer02: fmmb.instStat.change:info]: replayed event: normal mailbox instance on local side.
Thu May 25 07:45:51 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.7 is a partner HA mailbox disk.
Thu May 25 07:45:51 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.3 is a partner HA mailbox disk.
Thu May 25 07:45:51 BNT [filer02: fmmb.instStat.change:info]: replayed event: normal mailbox instance on partner side.
Thu May 25 07:45:52 BNT [filer02: cf.fm.partner:info]: replayed event: Cluster monitor: partner 'filer01'
Thu May 25 07:45:52 BNT [filer02: cf.fm.timeMasterStatus:info]: replayed event: Acting as cluster time slave
Thu May 25 07:45:52 BNT [filer02: raid.assim.disk.brokenPreAssim:error]: replayed event: Broken Disk 0a.40 Shelf ? Bay ? [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] detected prior to assimilation. It should be removed.
Thu May 25 07:45:52 BNT [filer02: raid.assim.disk.brokenPreAssim:error]: replayed event: Broken Disk 0a.35 Shelf ? Bay ? [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] detected prior to assimilation. It should be removed.
Thu May 25 07:45:52 BNT [filer02: raid.assim.disk.brokenPreAssim:error]: replayed event: Broken Disk 0a.33 Shelf ? Bay ? [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] detected prior to assimilation. It should be removed.
Thu May 25 07:45:52 BNT [filer02: coredump.spare.none:info]: replayed event: No sparecore disk was found.
Thu May 25 07:45:53 BNT [filer02: raid.vol.replay.nvram:info]: replayed event: Performing raid replay on volume(s)
Thu May 25 07:45:53 BNT [filer02: raid.cksum.replay.summary:info]: replayed event: Replayed 0 checksum blocks.
Thu May 25 07:45:53 BNT [filer02: raid.stripe.replay.summary:info]: replayed event: Replayed 0 stripes.
Thu May 25 07:45:56 BNT [filer02: ses.status.psWarning:warning]: replayed event: DS12-ESAS shelf 0 on channel 0c power warning for Power supply 1: non-critical status; DC undervoltage fault. This module is on the rear side of the shelf, at the left.
Thu May 25 07:45:56 BNT [filer02: ses.status.psWarning:warning]: replayed event: DS12-ESAS shelf 0 on channel 0c power warning for Power supply 2: non-critical status; DC undervoltage fault. This module is on the rear side of the shelf, at the right.
Thu May 25 07:46:05 BNT [filer02: cf.fm.launch:info]: replayed event: Launching cluster monitor
Thu May 25 07:46:05 BNT [filer02: cf.fm.partner:info]: replayed event: Cluster monitor: partner 'filer01'
Thu May 25 07:46:10 BNT [filer02: cf.fsm.takeoverByPartnerDisabled:notice]: replayed event: Cluster monitor: takeover of  by filer01 disabled (partner booting)
Thu May 25 07:46:10 BNT [filer02: fcp.service.adapter:warning]: replayed event: FCP started but no adapter is present!
Thu May 25 07:46:11 BNT [filer02: kern.syslog.msg:notice]: replayed event: The system was down for 18 seconds
Thu May 25 07:46:11 BNT [filer02: cf.fsm.takeoverOfPartnerEnabled:notice]: replayed event: Cluster monitor: takeover of filer01 enabled
Thu May 25 07:46:12 BNT [filer02: disk.failmsg:error]: Disk 0a.40 (N00V0W8L): message received.
Thu May 25 07:46:12 BNT [filer02: disk.failmsg:error]: Disk 0a.35 (N00WLAUL): message received.
Thu May 25 07:46:12 BNT [filer02: disk.failmsg:error]: Disk 0a.33 (N00W75PL): message received.
Thu May 25 07:46:12 BNT [filer02: raid.disk.unload.done:info]: Unload of Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] has completed successfully
Thu May 25 07:46:12 BNT [filer02: raid.disk.unload.done:info]: Unload of Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] has completed successfully
Thu May 25 07:46:12 BNT [filer02: raid.disk.unload.done:info]: Unload of Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] has completed successfully
Thu May 25 07:46:12 BNT [filer02: raid.fdr.failed.ok:info]: Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] successfully deleted from spare pool
Thu May 25 07:46:12 BNT [filer02: raid.fdr.failed.ok:info]: Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] successfully deleted from spare pool
Thu May 25 07:46:12 BNT [filer02: raid.fdr.failed.ok:info]: Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] successfully deleted from spare pool
Thu May 25 07:46:13 BNT [filer02: dfu.firmwareUpToDate:info]: Firmware is up-to-date on all disk drives
Thu May 25 07:46:14 BNT [filer02: perf.archive.start:info]: Performance archiver started. Sampling 22 objects and 198 counters.
Thu May 25 07:46:16 BNT [filer02: netif.linkUp:info]: Ethernet e0b: Link up.
Thu May 25 07:46:16 BNT [filer02: netif.linkUp:info]: Ethernet e0a: Link up.
Thu May 25 07:46:17 BNT [filer02: pvif.switchLink:warning]: vif1: switching to e0a
Thu May 25 07:46:18 BNT [filer02: rc:info]: relog syslog Thu May 25 07:45:26 BNT [filer02: ses.status.psWarning:warning]: DS14-Mk2-AT shelf 1 on channel 0a power warning for P
Thu May 25 07:46:18 BNT [filer02: rc:info]: relog syslog Thu May 25 07:45:29 BNT [filer02: fci.link.break:error]: Link break detected on Fibre Channel adapter 0a.
Thu May 25 07:46:18 BNT [filer02: iscsi.service.startup:info]: iSCSI service startup
Thu May 25 07:46:19 BNT [filer02: cmds.sysconf.logErr:error]: sysconfig: Unless directed by NetApp Global Services the cf.takeover.on_panic option should be set to On. .
Thu May 25 07:46:19 BNT [filer02: raid.rg.recons.missing:notice]: RAID group /aggr1/plex0/rg0 is missing 2 disk(s).
Thu May 25 07:46:19 BNT [filer02: raid.rg.recons.cantStart:warning]: The reconstruction cannot start in RAID group /aggr1/plex0/rg0: No 520 bps disk of required size available in spare pool, targeting any  spare pool
Thu May 25 07:46:19 BNT [filer02: mgr.boot.disk_done:info]: NetApp Release 7.3.2 boot complete. Last disk update written at Thu May 25 07:45:18 BNT 2017
Thu May 25 07:46:19 BNT [filer02: monitor.raid.brokenDisk:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken.
Thu May 25 07:46:19 BNT [filer02: monitor.brokenDisk.notice:notice]: When two disks are broken in raid_dp volume,  the system shuts down automatically every 24  hours to encourage you to replace the disk.  If you reboot the system it will run for another  24 hours before shutting down. (The 24 hour timeout  may be increased by altering the "raid.timeout"  value using the "options" command.)
Thu May 25 07:46:19 BNT [filer02: diskown.errorReadingOwnership:warning]: error 3 (disk failed) while reading ownership on disk 0a.24 (S/N N00SSSAL)
Thu May 25 07:46:19 BNT [filer02: diskown.errorReadingOwnership:warning]: error 3 (disk failed) while reading ownership on disk 0a.26 (S/N N00V0UWL)
Thu May 25 07:46:19 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] is still present in the system and should be removed.
Thu May 25 07:46:19 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] is still present in the system and should be removed.
Thu May 25 07:46:19 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] is still present in the system and should be removed.
Thu May 25 07:46:19 BNT [filer02: shelf.config.spha:info]: System is using single path HA attached storage only.
Thu May 25 07:46:20 BNT [filer02: mgr.boot.reason_ok:notice]: System rebooted after power-on.
Thu May 25 07:46:39 BNT [filer02: raid.rg.spares.low:warning]: /aggr1/plex0/rg0
Thu May 25 07:46:45 BNT [filer02: nbt.nbns.registrationComplete:info]: NBT: All CIFS name registrations have completed for the local server.
Thu May 25 07:46:46 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (REBOOT (power on))
Thu May 25 07:46:54 BNT last message repeated 2 times
Thu May 25 07:46:54 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201705190801.1) was not posted for host (0), it was dropped from queue due to lack of space
Thu May 25 07:47:00 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SPARES_LOW)
Thu May 25 07:47:00 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201705190801.2) was not posted for host (0), it was dropped from queue due to lack of space
Thu May 25 07:47:05 BNT [filer02: monitor.globalStatus.nonCritical:warning]: Disk on adapter 0a, shelf 2, bay 8, failed. Disk on adapter 0a, shelf 2, bay 3, failed. Disk on adapter 0a, shelf 2, bay 1, failed.  There are not enough spare disks. /vol/vol1 is full (using or reserving 100% of space and 0% of inodes, using 92% of reserve).
Thu May 25 07:47:07 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (PERFORMANCE SNAPSHOT)
Thu May 25 07:47:07 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201705190801.3) was not posted for host (0), it was dropped from queue due to lack of space
Thu May 25 07:47:28 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu May 25 07:47:28 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201705191200.0) was not posted for host (0), it was dropped from queue due to lack of space
Thu May 25 07:52:06 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu May 25 07:56:42 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu May 25 08:00:00 BNT [filer02: kern.uptime.filer:info]:   8:00am up 13 mins, 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 0 iSCSI ops
Thu May 25 08:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 24 hours.
Thu May 25 08:00:20 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu May 25 08:00:30 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Thu May 25 08:00:38 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu May 25 08:00:38 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201705191600.0) was not posted for host (0), it was dropped from queue due to lack of space
Thu May 25 08:04:28 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu May 25 08:08:30 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu May 25 08:12:12 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu May 25 08:14:26 BNT [filer02: iscsi.notice:notice]: ISCSI: New session from initiator iqn.1991-05.com.microsoft:pcb-722-ad1.pcbl722dom.local at IP addr 10.72.4.1
Thu May 25 08:15:56 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu May 25 08:20:17 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu May 25 08:20:17 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201705192000.0) for host (0) was not posted to NetApp. The system will drop the message
Thu May 25 09:00:00 BNT [filer02: kern.uptime.filer:info]:   9:00am up  1:13 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 1131 iSCSI ops
Thu May 25 09:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 23 hours.
Thu May 25 10:00:00 BNT [filer02: kern.uptime.filer:info]:  10:00am up  2:13 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 7858 iSCSI ops
Thu May 25 10:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 22 hours.
Thu May 25 11:00:00 BNT [filer02: kern.uptime.filer:info]:  11:00am up  3:13 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 100999 iSCSI ops
Thu May 25 11:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 21 hours.
Thu May 25 12:00:00 BNT [filer02: kern.uptime.filer:info]:  12:00pm up  4:13 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 401904 iSCSI ops
Thu May 25 12:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 20 hours.
Thu May 25 12:00:15 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Thu May 25 12:00:34 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu May 25 12:04:13 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu May 25 12:07:50 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu May 25 12:11:25 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu May 25 12:14:57 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu May 25 12:34:42 BNT last message repeated 5 times
Thu May 25 12:39:03 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu May 25 12:55:03 BNT last message repeated 4 times
Thu May 25 12:55:03 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201705200000.0) for host (0) was not posted to NetApp. The system will drop the message
Thu May 25 13:00:00 BNT [filer02: kern.uptime.filer:info]:   1:00pm up  5:13 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 402655 iSCSI ops
Thu May 25 13:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 19 hours.
Thu May 25 14:00:00 BNT [filer02: kern.uptime.filer:info]:   2:00pm up  6:13 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 402671 iSCSI ops
Thu May 25 14:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 18 hours.
Thu May 25 15:00:00 BNT [filer02: kern.uptime.filer:info]:   3:00pm up  7:13 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 779959 iSCSI ops
Thu May 25 15:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 17 hours.
Thu May 25 16:00:00 BNT [filer02: kern.uptime.filer:info]:   4:00pm up  8:13 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 980403 iSCSI ops
Thu May 25 16:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 16 hours.
Thu May 25 16:00:16 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Thu May 25 16:00:39 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu May 25 16:04:13 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu May 25 16:07:55 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu May 25 16:12:19 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu May 25 16:16:11 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu May 25 16:35:28 BNT last message repeated 5 times
Thu May 25 16:39:43 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu May 25 16:51:17 BNT last message repeated 3 times
Thu May 25 16:55:16 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201705200800.0) for host (0) was not posted to NetApp. The system will drop the message
Thu May 25 17:00:00 BNT [filer02: kern.uptime.filer:info]:   5:00pm up  9:13 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 1127544 iSCSI ops
Thu May 25 17:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 15 hours.
Thu May 25 18:00:00 BNT [filer02: kern.uptime.filer:info]:   6:00pm up 10:13 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 1404283 iSCSI ops
Thu May 25 18:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 14 hours.
Thu May 25 19:00:00 BNT [filer02: kern.uptime.filer:info]:   7:00pm up 11:14 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 1404465 iSCSI ops
Thu May 25 19:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 13 hours.
Thu May 25 20:00:00 BNT [filer02: kern.uptime.filer:info]:   8:00pm up 12:14 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 1405451 iSCSI ops
Thu May 25 20:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 12 hours.
Thu May 25 20:00:01 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Thu May 25 20:04:20 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Thu May 25 20:56:38 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201705220750.0) for host (0) was not posted to NetApp. The system will drop the message
Thu May 25 21:00:00 BNT [filer02: kern.uptime.filer:info]:   9:00pm up 13:14 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 1405619 iSCSI ops
Thu May 25 21:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 11 hours.
Thu May 25 22:00:00 BNT [filer02: kern.uptime.filer:info]:  10:00pm up 14:14 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 1405619 iSCSI ops
Thu May 25 22:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 10 hours.
Thu May 25 23:00:00 BNT [filer02: kern.uptime.filer:info]:  11:00pm up 15:14 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 1405619 iSCSI ops
Thu May 25 23:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 9 hours.
Fri May 26 00:00:00 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of nightly.0 snapshot (Snapshot operation failed: No space left on device.)
Fri May 26 00:00:00 BNT [filer02: kern.uptime.filer:info]:  12:00am up 16:14 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 1405619 iSCSI ops
Fri May 26 00:00:01 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 8 hours.
Fri May 26 00:04:28 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SHUTDOWN PENDING (degraded mode))
Fri May 26 00:56:13 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201705220750.1) for host (0) was not posted to NetApp. The system will drop the message
Fri May 26 01:00:00 BNT [filer02: kern.uptime.filer:info]:   1:00am up 17:14 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 1405619 iSCSI ops
Fri May 26 01:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 7 hours.
Fri May 26 02:00:00 BNT [filer02: kern.uptime.filer:info]:   2:00am up 18:14 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 1405619 iSCSI ops
Fri May 26 02:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 6 hours.
Fri May 26 03:00:00 BNT [filer02: kern.uptime.filer:info]:   3:00am up 19:14 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 1405619 iSCSI ops
Fri May 26 03:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 5 hours.
Fri May 26 04:00:00 BNT [filer02: kern.uptime.filer:info]:   4:00am up 20:14 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 1405619 iSCSI ops
Fri May 26 04:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 4 hours.
Fri May 26 05:00:00 BNT [filer02: kern.uptime.filer:info]:   5:00am up 21:14 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 1405619 iSCSI ops
Fri May 26 05:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 3 hours.
Fri May 26 06:00:00 BNT [filer02: kern.uptime.filer:info]:   6:00am up 22:14 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 1405619 iSCSI ops
Fri May 26 06:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 2 hours.
Fri May 26 07:00:00 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] is still present in the system and should be removed.
Fri May 26 07:00:00 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] is still present in the system and should be removed.
Fri May 26 07:00:00 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] is still present in the system and should be removed.
Fri May 26 08:00:00 BNT [filer02: monitor.brokenDisk.notice:notice]: When two disks are broken in raid_dp volume,  the system shuts down automatically every 24  hours to encourage you to replace the disk.  If you reboot the system it will run for another  24 hours before shutting down. (The 24 hour timeout  may be increased by altering the "raid.timeout"  value using the "options" command.)
Fri May 26 08:00:00 BNT [filer02: monitor.shutdown.brokenDisk:EMERGENCY]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system now.
Fri May 26 08:00:04 BNT [filer02: cf.fsm.takeoverOfPartnerDisabled:notice]: Cluster monitor: takeover of filer01 disabled (local halt in progress)
Fri May 26 08:00:06 BNT [filer02: kern.shutdown:notice]: System shut down because : "BROKEN DISK".
Fri May 26 00:08:43 GMT [localhost: rc:notice]: The system was down for 509 seconds
Fri May 26 08:08:17 BNT [filer02: cf.nm.nicTransitionUp:info]: replayed event: Interconnect link 0 is UP
Fri May 26 08:08:24 BNT [filer02: fci.initialization.failed:error]: replayed event: Initialization failed on Fibre Channel adapter 0b.
Fri May 26 08:08:26 BNT [filer02: nvram.battery.turned.on:info]: replayed event: The NVRAM battery is turned ON. It is turned OFF during system shutdown.
Fri May 26 08:08:29 BNT [filer02: diskown.isEnabled:info]: replayed event: software ownership has been enabled for this system
Fri May 26 08:08:33 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.2 is a local HA mailbox disk.
Fri May 26 08:08:33 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.0 is a local HA mailbox disk.
Fri May 26 08:08:33 BNT [filer02: fmmb.instStat.change:info]: replayed event: normal mailbox instance on local side.
Fri May 26 08:08:33 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.7 is a partner HA mailbox disk.
Fri May 26 08:08:33 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.3 is a partner HA mailbox disk.
Fri May 26 08:08:33 BNT [filer02: fmmb.instStat.change:info]: replayed event: normal mailbox instance on partner side.
Fri May 26 08:08:33 BNT [filer02: cf.fm.partner:info]: replayed event: Cluster monitor: partner 'filer01'
Fri May 26 08:08:33 BNT [filer02: cf.fm.timeMasterStatus:info]: replayed event: Acting as cluster time slave
Fri May 26 08:08:34 BNT [filer02: raid.assim.disk.brokenPreAssim:error]: replayed event: Broken Disk 0a.40 Shelf ? Bay ? [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] detected prior to assimilation. It should be removed.
Fri May 26 08:08:34 BNT [filer02: raid.assim.disk.brokenPreAssim:error]: replayed event: Broken Disk 0a.35 Shelf ? Bay ? [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] detected prior to assimilation. It should be removed.
Fri May 26 08:08:34 BNT [filer02: raid.assim.disk.brokenPreAssim:error]: replayed event: Broken Disk 0a.33 Shelf ? Bay ? [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] detected prior to assimilation. It should be removed.
Fri May 26 08:08:34 BNT [filer02: coredump.spare.none:info]: replayed event: No sparecore disk was found.
Fri May 26 08:08:35 BNT [filer02: raid.cksum.replay.summary:info]: replayed event: Replayed 0 checksum blocks.
Fri May 26 08:08:35 BNT [filer02: raid.stripe.replay.summary:info]: replayed event: Replayed 0 stripes.
Fri May 26 08:08:36 BNT [filer02: cf.fm.launch:info]: replayed event: Launching cluster monitor
Fri May 26 08:08:37 BNT [filer02: cf.fm.partner:info]: replayed event: Cluster monitor: partner 'filer01'
Fri May 26 08:08:38 BNT [filer02: cf.fsm.takeoverOfPartnerDisabled:notice]: replayed event: Cluster monitor: takeover of filer01 disabled (partner booting)
Fri May 26 08:08:43 BNT [filer02: fcp.service.adapter:warning]: replayed event: FCP started but no adapter is present!
Fri May 26 08:08:43 BNT [filer02: kern.syslog.msg:notice]: replayed event: The system was down for 509 seconds
Fri May 26 08:08:44 BNT [filer02: disk.failmsg:error]: Disk 0a.40 (N00V0W8L): message received.
Fri May 26 08:08:44 BNT [filer02: disk.failmsg:error]: Disk 0a.35 (N00WLAUL): message received.
Fri May 26 08:08:44 BNT [filer02: disk.failmsg:error]: Disk 0a.33 (N00W75PL): message received.
Fri May 26 08:08:44 BNT [filer02: raid.disk.unload.done:info]: Unload of Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] has completed successfully
Fri May 26 08:08:44 BNT [filer02: raid.disk.unload.done:info]: Unload of Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] has completed successfully
Fri May 26 08:08:44 BNT [filer02: raid.disk.unload.done:info]: Unload of Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] has completed successfully
Fri May 26 08:08:44 BNT [filer02: raid.fdr.failed.ok:info]: Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] successfully deleted from spare pool
Fri May 26 08:08:44 BNT [filer02: raid.fdr.failed.ok:info]: Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] successfully deleted from spare pool
Fri May 26 08:08:45 BNT [filer02: raid.fdr.failed.ok:info]: Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] successfully deleted from spare pool
Fri May 26 08:08:45 BNT [filer02: dfu.firmwareUpToDate:info]: Firmware is up-to-date on all disk drives
Fri May 26 08:08:46 BNT [filer02: cf.fsm.takeoverOfPartnerEnabled:notice]: Cluster monitor: takeover of filer01 enabled
Fri May 26 08:08:46 BNT [filer02: cf.fsm.takeoverByPartnerEnabled:notice]: Cluster monitor: takeover of filer02 by filer01 enabled
Fri May 26 08:08:47 BNT [filer02: perf.archive.start:info]: Performance archiver started. Sampling 22 objects and 198 counters.
Fri May 26 08:08:49 BNT [filer02: netif.linkUp:info]: Ethernet e0a: Link up.
Fri May 26 08:08:49 BNT [filer02: netif.linkUp:info]: Ethernet e0b: Link up.
Fri May 26 08:08:50 BNT [filer02: pvif.switchLink:warning]: vif1: switching to e0a
Fri May 26 08:08:51 BNT [filer02: iscsi.service.startup:info]: iSCSI service startup
Fri May 26 08:08:52 BNT [filer02: cmds.sysconf.logErr:error]: sysconfig: Unless directed by NetApp Global Services the cf.takeover.on_panic option should be set to On. .
Fri May 26 08:08:52 BNT [filer02: raid.rg.recons.missing:notice]: RAID group /aggr1/plex0/rg0 is missing 2 disk(s).
Fri May 26 08:08:52 BNT [filer02: raid.rg.recons.cantStart:warning]: The reconstruction cannot start in RAID group /aggr1/plex0/rg0: No 520 bps disk of required size available in spare pool, targeting any  spare pool
Fri May 26 08:08:52 BNT [filer02: mgr.boot.disk_done:info]: NetApp Release 7.3.2 boot complete. Last disk update written at Fri May 26 08:00:08 BNT 2017
Fri May 26 08:08:52 BNT [filer02: monitor.raid.brokenDisk:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken.
Fri May 26 08:08:52 BNT [filer02: monitor.brokenDisk.notice:notice]: When two disks are broken in raid_dp volume,  the system shuts down automatically every 24  hours to encourage you to replace the disk.  If you reboot the system it will run for another  24 hours before shutting down. (The 24 hour timeout  may be increased by altering the "raid.timeout"  value using the "options" command.)
Fri May 26 08:08:52 BNT [filer02: netif.linkDown:info]: Ethernet e0b: Link down, check cable.
Fri May 26 08:08:53 BNT [filer02: diskown.errorReadingOwnership:warning]: error 3 (disk failed) while reading ownership on disk 0a.24 (S/N N00SSSAL)
Fri May 26 08:08:53 BNT [filer02: diskown.errorReadingOwnership:warning]: error 3 (disk failed) while reading ownership on disk 0a.26 (S/N N00V0UWL)
Fri May 26 08:08:53 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] is still present in the system and should be removed.
Fri May 26 08:08:53 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] is still present in the system and should be removed.
Fri May 26 08:08:53 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] is still present in the system and should be removed.
Fri May 26 08:08:53 BNT [filer02: shelf.config.spha:info]: System is using single path HA attached storage only.
Fri May 26 08:08:53 BNT [filer02: mgr.boot.reason_ok:notice]: System rebooted.
Fri May 26 08:08:55 BNT [filer02: iscsi.notice:notice]: ISCSI: New session from initiator iqn.1991-05.com.microsoft:pcb-722-ad1.pcbl722dom.local at IP addr 10.72.4.1
Fri May 26 08:08:58 BNT [filer02: netif.linkUp:info]: Ethernet e0b: Link up.
Fri May 26 08:09:00 BNT [filer02: raid.rg.spares.low:warning]: /aggr1/plex0/rg0
Fri May 26 08:09:07 BNT [filer02: monitor.globalStatus.nonCritical:warning]: Disk on adapter 0a, shelf 2, bay 8, failed. Disk on adapter 0a, shelf 2, bay 3, failed. Disk on adapter 0a, shelf 2, bay 1, failed.  There are not enough spare disks. /vol/vol1 is full (using or reserving 100% of space and 0% of inodes, using 92% of reserve).
Fri May 26 08:09:17 BNT [filer02: nbt.nbns.registrationComplete:info]: NBT: All CIFS name registrations have completed for the local server.
Fri May 26 08:09:23 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (REBOOT)
Fri May 26 08:09:31 BNT last message repeated 2 times
Fri May 26 08:09:31 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201705220750.2) was not posted for host (0), it was dropped from queue due to lack of space
Fri May 26 08:09:38 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SPARES_LOW)
Fri May 26 08:09:38 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201705220750.3) was not posted for host (0), it was dropped from queue due to lack of space
Fri May 26 08:09:46 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (PERFORMANCE SNAPSHOT)
Fri May 26 08:09:46 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201705220751.0) was not posted for host (0), it was dropped from queue due to lack of space
Fri May 26 08:10:01 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (CLUSTER ERROR: DISK/SHELF COUNT MISMATCH)
Fri May 26 08:10:01 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201705220755.0) was not posted for host (0), it was dropped from queue due to lack of space
Fri May 26 08:14:32 BNT [filer02: cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 0a.33 (016F3AB5:B475F4EC:10505E83:02202755:00000000:00000000:00000000:00000000:00000000:00000000) has recently changed or the node (filer02) is missing the disk.
Fri May 26 08:14:32 BNT [filer02: cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 0a.35 (9115483F:9DD43BC4:27172773:99F7D8B4:00000000:00000000:00000000:00000000:00000000:00000000) has recently changed or the node (filer02) is missing the disk.
Fri May 26 08:14:32 BNT [filer02: cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 0a.40 (072AEB65:C662A9CF:E58B9917:9BFA787B:00000000:00000000:00000000:00000000:00000000:00000000) has recently changed or the node (filer02) is missing the disk.
Fri May 26 08:14:32 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri May 26 08:14:39 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri May 26 08:14:39 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201705220800.0) was not posted for host (0), it was dropped from queue due to lack of space
Fri May 26 08:18:31 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri May 26 08:22:44 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri May 26 08:26:53 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri May 26 08:31:10 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri May 26 08:34:45 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri May 26 08:42:08 BNT last message repeated 2 times
Fri May 26 08:42:08 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201705221200.0) for host (0) was not posted to NetApp. The system will drop the message
Fri May 26 09:00:00 BNT [filer02: kern.uptime.filer:info]:   9:00am up 50 mins, 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 4430 iSCSI ops
Fri May 26 09:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 24 hours.
Fri May 26 09:15:48 BNT [filer02: cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 0a.33 (016F3AB5:B475F4EC:10505E83:02202755:00000000:00000000:00000000:00000000:00000000:00000000) has recently changed or the node (filer02) is missing the disk.
Fri May 26 09:15:48 BNT [filer02: cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 0a.35 (9115483F:9DD43BC4:27172773:99F7D8B4:00000000:00000000:00000000:00000000:00000000:00000000) has recently changed or the node (filer02) is missing the disk.
Fri May 26 09:15:48 BNT [filer02: cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 0a.40 (072AEB65:C662A9CF:E58B9917:9BFA787B:00000000:00000000:00000000:00000000:00000000:00000000) has recently changed or the node (filer02) is missing the disk.
Fri May 26 10:00:00 BNT [filer02: kern.uptime.filer:info]:  10:00am up  1:50 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 8195 iSCSI ops
Fri May 26 10:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 23 hours.
Fri May 26 11:00:00 BNT [filer02: kern.uptime.filer:info]:  11:00am up  2:51 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 31922 iSCSI ops
Fri May 26 11:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 22 hours.
Fri May 26 12:00:00 BNT [filer02: kern.uptime.filer:info]:  12:00pm up  3:51 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 150966 iSCSI ops
Fri May 26 12:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 21 hours.
Fri May 26 12:00:00 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Fri May 26 12:00:23 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri May 26 12:04:36 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri May 26 12:08:54 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri May 26 12:13:02 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri May 26 12:17:03 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri May 26 12:33:25 BNT last message repeated 4 times
Fri May 26 12:37:49 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri May 26 12:57:02 BNT last message repeated 5 times
Fri May 26 12:57:02 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201705221600.0) for host (0) was not posted to NetApp. The system will drop the message
Fri May 26 13:00:00 BNT [filer02: kern.uptime.filer:info]:   1:00pm up  4:51 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 162141 iSCSI ops
Fri May 26 13:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 20 hours.
Fri May 26 14:00:00 BNT [filer02: kern.uptime.filer:info]:   2:00pm up  5:51 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 162880 iSCSI ops
Fri May 26 14:00:01 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 19 hours.
Fri May 26 15:00:00 BNT [filer02: kern.uptime.filer:info]:   3:00pm up  6:51 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 189268 iSCSI ops
Fri May 26 15:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 18 hours.
Fri May 26 16:00:00 BNT [filer02: kern.uptime.filer:info]:   4:00pm up  7:51 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 193455 iSCSI ops
Fri May 26 16:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 17 hours.
Fri May 26 16:00:01 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Fri May 26 16:00:24 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri May 26 16:04:47 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri May 26 16:09:14 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri May 26 16:12:50 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri May 26 16:21:01 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri May 26 16:36:06 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri May 26 16:56:00 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201705222000.0) for host (0) was not posted to NetApp. The system will drop the message
Fri May 26 17:00:00 BNT [filer02: kern.uptime.filer:info]:   5:00pm up  8:51 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 275917 iSCSI ops
Fri May 26 17:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 16 hours.
Fri May 26 18:00:00 BNT [filer02: kern.uptime.filer:info]:   6:00pm up  9:51 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 277256 iSCSI ops
Fri May 26 18:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 15 hours.
Fri May 26 19:00:00 BNT [filer02: kern.uptime.filer:info]:   7:00pm up 10:51 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 277256 iSCSI ops
Fri May 26 19:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 14 hours.
Fri May 26 20:00:00 BNT [filer02: kern.uptime.filer:info]:   8:00pm up 11:51 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 277256 iSCSI ops
Fri May 26 20:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 13 hours.
Fri May 26 20:00:46 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Fri May 26 20:56:56 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201705230000.0) for host (0) was not posted to NetApp. The system will drop the message
Fri May 26 21:00:00 BNT [filer02: kern.uptime.filer:info]:   9:00pm up 12:51 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 277256 iSCSI ops
Fri May 26 21:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 12 hours.
Fri May 26 22:00:00 BNT [filer02: kern.uptime.filer:info]:  10:00pm up 13:51 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 277256 iSCSI ops
Fri May 26 22:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 11 hours.
Fri May 26 23:00:00 BNT [filer02: kern.uptime.filer:info]:  11:00pm up 14:51 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 277256 iSCSI ops
Fri May 26 23:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 10 hours.
Sat May 27 00:00:00 BNT [filer02: kern.uptime.filer:info]:  12:00am up 15:51 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 277256 iSCSI ops
Sat May 27 00:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 9 hours.
Sat May 27 00:00:45 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of nightly.0 snapshot (Snapshot operation failed: No space left on device.)
Sat May 27 00:54:01 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Sat May 27 00:58:20 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201705230744.0) for host (0) was not posted to NetApp. The system will drop the message
Sat May 27 01:00:00 BNT [filer02: kern.uptime.filer:info]:   1:00am up 16:51 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 277256 iSCSI ops
Sat May 27 01:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 8 hours.
Sat May 27 02:00:00 BNT [filer02: kern.uptime.filer:info]:   2:00am up 17:51 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 277256 iSCSI ops
Sat May 27 02:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 7 hours.
Sat May 27 03:00:00 BNT [filer02: kern.uptime.filer:info]:   3:00am up 18:51 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 277256 iSCSI ops
Sat May 27 03:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 6 hours.
Sat May 27 04:00:00 BNT [filer02: kern.uptime.filer:info]:   4:00am up 19:51 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 277256 iSCSI ops
Sat May 27 04:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 5 hours.
Sat May 27 05:00:00 BNT [filer02: kern.uptime.filer:info]:   5:00am up 20:51 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 277256 iSCSI ops
Sat May 27 05:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 4 hours.
Sat May 27 06:00:00 BNT [filer02: kern.uptime.filer:info]:   6:00am up 21:51 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 277256 iSCSI ops
Sat May 27 06:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 3 hours.
Sat May 27 07:00:00 BNT [filer02: kern.uptime.filer:info]:   7:00am up 22:51 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 277256 iSCSI ops
Sat May 27 07:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 2 hours.
Sat May 27 08:00:00 BNT [filer02: kern.uptime.filer:info]:   8:00am up 23:51 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 277256 iSCSI ops
Sat May 27 08:00:00 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] is still present in the system and should be removed.
Sat May 27 08:00:00 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] is still present in the system and should be removed.
Sat May 27 08:00:00 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] is still present in the system and should be removed.
Sat May 27 08:00:16 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Sat May 27 08:40:59 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SHUTDOWN PENDING (degraded mode))
Sat May 27 08:57:00 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201705230744.1) for host (0) was not posted to NetApp. The system will drop the message
Sat May 27 09:00:00 BNT [filer02: monitor.brokenDisk.notice:notice]: When two disks are broken in raid_dp volume,  the system shuts down automatically every 24  hours to encourage you to replace the disk.  If you reboot the system it will run for another  24 hours before shutting down. (The 24 hour timeout  may be increased by altering the "raid.timeout"  value using the "options" command.)
Sat May 27 09:00:00 BNT [filer02: monitor.shutdown.brokenDisk:EMERGENCY]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system now.
Sat May 27 09:00:04 BNT [filer02: cf.fsm.takeoverOfPartnerDisabled:notice]: Cluster monitor: takeover of filer01 disabled (local halt in progress)
Sat May 27 09:00:06 BNT [filer02: kern.shutdown:notice]: System shut down because : "BROKEN DISK".
Mon May 29 23:43:14 GMT [localhost: rc:notice]: The system was down for 254579 seconds
Tue May 30 07:42:48 BNT [filer02: cf.nm.nicTransitionUp:info]: replayed event: Interconnect link 0 is UP
Tue May 30 07:42:55 BNT [filer02: fci.initialization.failed:error]: replayed event: Initialization failed on Fibre Channel adapter 0b.
Tue May 30 07:42:57 BNT [filer02: nvram.battery.turned.on:info]: replayed event: The NVRAM battery is turned ON. It is turned OFF during system shutdown.
Tue May 30 07:43:00 BNT [filer02: diskown.isEnabled:info]: replayed event: software ownership has been enabled for this system
Tue May 30 07:43:04 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.2 is a local HA mailbox disk.
Tue May 30 07:43:04 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.0 is a local HA mailbox disk.
Tue May 30 07:43:04 BNT [filer02: fmmb.instStat.change:info]: replayed event: normal mailbox instance on local side.
Tue May 30 07:43:04 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.7 is a partner HA mailbox disk.
Tue May 30 07:43:04 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.3 is a partner HA mailbox disk.
Tue May 30 07:43:04 BNT [filer02: fmmb.instStat.change:info]: replayed event: normal mailbox instance on partner side.
Tue May 30 07:43:04 BNT [filer02: cf.fm.partner:info]: replayed event: Cluster monitor: partner 'filer01'
Tue May 30 07:43:04 BNT [filer02: cf.fm.timeMasterStatus:info]: replayed event: Acting as cluster time slave
Tue May 30 07:43:05 BNT [filer02: raid.assim.disk.brokenPreAssim:error]: replayed event: Broken Disk 0a.40 Shelf ? Bay ? [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] detected prior to assimilation. It should be removed.
Tue May 30 07:43:05 BNT [filer02: raid.assim.disk.brokenPreAssim:error]: replayed event: Broken Disk 0a.35 Shelf ? Bay ? [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] detected prior to assimilation. It should be removed.
Tue May 30 07:43:05 BNT [filer02: raid.assim.disk.brokenPreAssim:error]: replayed event: Broken Disk 0a.33 Shelf ? Bay ? [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] detected prior to assimilation. It should be removed.
Tue May 30 07:43:05 BNT [filer02: coredump.spare.none:info]: replayed event: No sparecore disk was found.
Tue May 30 07:43:06 BNT [filer02: raid.cksum.replay.summary:info]: replayed event: Replayed 0 checksum blocks.
Tue May 30 07:43:06 BNT [filer02: raid.stripe.replay.summary:info]: replayed event: Replayed 0 stripes.
Tue May 30 07:43:08 BNT [filer02: cf.fm.launch:info]: replayed event: Launching cluster monitor
Tue May 30 07:43:08 BNT [filer02: cf.fm.partner:info]: replayed event: Cluster monitor: partner 'filer01'
Tue May 30 07:43:09 BNT [filer02: cf.fsm.takeoverOfPartnerDisabled:notice]: replayed event: Cluster monitor: takeover of filer01 disabled (partner booting)
Tue May 30 07:43:14 BNT [filer02: fcp.service.adapter:warning]: replayed event: FCP started but no adapter is present!
Tue May 30 07:43:14 BNT [filer02: kern.syslog.msg:notice]: replayed event: The system was down for 254579 seconds
Tue May 30 07:43:15 BNT [filer02: disk.failmsg:error]: Disk 0a.40 (N00V0W8L): message received.
Tue May 30 07:43:15 BNT [filer02: disk.failmsg:error]: Disk 0a.35 (N00WLAUL): message received.
Tue May 30 07:43:15 BNT [filer02: disk.failmsg:error]: Disk 0a.33 (N00W75PL): message received.
Tue May 30 07:43:15 BNT [filer02: raid.disk.unload.done:info]: Unload of Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] has completed successfully
Tue May 30 07:43:15 BNT [filer02: raid.disk.unload.done:info]: Unload of Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] has completed successfully
Tue May 30 07:43:15 BNT [filer02: raid.disk.unload.done:info]: Unload of Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] has completed successfully
Tue May 30 07:43:15 BNT [filer02: raid.fdr.failed.ok:info]: Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] successfully deleted from spare pool
Tue May 30 07:43:16 BNT [filer02: raid.fdr.failed.ok:info]: Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] successfully deleted from spare pool
Tue May 30 07:43:16 BNT [filer02: raid.fdr.failed.ok:info]: Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] successfully deleted from spare pool
Tue May 30 07:43:16 BNT [filer02: dfu.firmwareUpToDate:info]: Firmware is up-to-date on all disk drives
Tue May 30 07:43:17 BNT [filer02: cf.fsm.takeoverOfPartnerEnabled:notice]: Cluster monitor: takeover of filer01 enabled
Tue May 30 07:43:17 BNT [filer02: cf.fsm.takeoverByPartnerEnabled:notice]: Cluster monitor: takeover of filer02 by filer01 enabled
Tue May 30 07:43:18 BNT [filer02: perf.archive.start:info]: Performance archiver started. Sampling 22 objects and 198 counters.
Tue May 30 07:43:20 BNT [filer02: netif.linkUp:info]: Ethernet e0b: Link up.
Tue May 30 07:43:20 BNT [filer02: netif.linkUp:info]: Ethernet e0a: Link up.
Tue May 30 07:43:20 BNT [filer02: pvif.switchLink:warning]: vif1: switching to e0a
Tue May 30 07:43:22 BNT [filer02: iscsi.service.startup:info]: iSCSI service startup
Tue May 30 07:43:22 BNT [filer02: iscsi.notice:notice]: ISCSI: New session from initiator iqn.1991-05.com.microsoft:pcb-722-ad1.pcbl722dom.local at IP addr 10.72.4.1
Tue May 30 07:43:23 BNT [filer02: cmds.sysconf.logErr:error]: sysconfig: Unless directed by NetApp Global Services the cf.takeover.on_panic option should be set to On. .
Tue May 30 07:43:23 BNT [filer02: raid.rg.recons.missing:notice]: RAID group /aggr1/plex0/rg0 is missing 2 disk(s).
Tue May 30 07:43:23 BNT [filer02: raid.rg.recons.cantStart:warning]: The reconstruction cannot start in RAID group /aggr1/plex0/rg0: No 520 bps disk of required size available in spare pool, targeting any  spare pool
Tue May 30 07:43:23 BNT [filer02: mgr.boot.disk_done:info]: NetApp Release 7.3.2 boot complete. Last disk update written at Sat May 27 09:00:08 BNT 2017
Tue May 30 07:43:23 BNT [filer02: monitor.raid.brokenDisk:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken.
Tue May 30 07:43:23 BNT [filer02: monitor.brokenDisk.notice:notice]: When two disks are broken in raid_dp volume,  the system shuts down automatically every 24  hours to encourage you to replace the disk.  If you reboot the system it will run for another  24 hours before shutting down. (The 24 hour timeout  may be increased by altering the "raid.timeout"  value using the "options" command.)
Tue May 30 07:43:24 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] is still present in the system and should be removed.
Tue May 30 07:43:24 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] is still present in the system and should be removed.
Tue May 30 07:43:24 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] is still present in the system and should be removed.
Tue May 30 07:43:24 BNT [filer02: shelf.config.spha:info]: System is using single path HA attached storage only.
Tue May 30 07:43:24 BNT [filer02: mgr.boot.reason_ok:notice]: System rebooted.
Tue May 30 07:43:37 BNT [filer02: diskown.errorReadingOwnership:warning]: error 3 (disk failed) while reading ownership on disk 0a.24 (S/N N00SSSAL)
Tue May 30 07:43:45 BNT [filer02: raid.rg.spares.low:warning]: /aggr1/plex0/rg0
Tue May 30 07:43:48 BNT [filer02: nbt.nbns.registrationComplete:info]: NBT: All CIFS name registrations have completed for the local server.
Tue May 30 07:43:54 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (REBOOT (power on))
Tue May 30 07:44:02 BNT last message repeated 2 times
Tue May 30 07:44:02 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201705230745.0) was not posted for host (0), it was dropped from queue due to lack of space
Tue May 30 07:44:04 BNT [filer02: monitor.globalStatus.nonCritical:warning]: Disk on adapter 0a, shelf 2, bay 8, failed. Disk on adapter 0a, shelf 2, bay 3, failed. Disk on adapter 0a, shelf 2, bay 1, failed.  There are not enough spare disks. /vol/vol1 is full (using or reserving 100% of space and 0% of inodes, using 92% of reserve).
Tue May 30 07:44:09 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SPARES_LOW)
Tue May 30 07:44:09 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201705230745.1) was not posted for host (0), it was dropped from queue due to lack of space
Tue May 30 07:44:16 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (PERFORMANCE SNAPSHOT)
Tue May 30 07:44:16 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201705230745.2) was not posted for host (0), it was dropped from queue due to lack of space
Tue May 30 07:44:32 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Tue May 30 07:44:32 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201705230800.0) was not posted for host (0), it was dropped from queue due to lack of space
Tue May 30 07:49:12 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Tue May 30 07:49:14 BNT [filer02: cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 0a.26 (DB493CEF:613B738F:C6775410:52ABD6E0:00000000:00000000:00000000:00000000:00000000:00000000) has recently changed or the node (filer01) is missing the disk.
Tue May 30 07:49:14 BNT [filer02: cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 0a.33 (016F3AB5:B475F4EC:10505E83:02202755:00000000:00000000:00000000:00000000:00000000:00000000) has recently changed or the node (filer02) is missing the disk.
Tue May 30 07:49:14 BNT [filer02: cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 0a.35 (9115483F:9DD43BC4:27172773:99F7D8B4:00000000:00000000:00000000:00000000:00000000:00000000) has recently changed or the node (filer02) is missing the disk.
Tue May 30 07:49:14 BNT [filer02: cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 0a.40 (072AEB65:C662A9CF:E58B9917:9BFA787B:00000000:00000000:00000000:00000000:00000000:00000000) has recently changed or the node (filer02) is missing the disk.
Tue May 30 07:49:14 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Tue May 30 07:49:21 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Tue May 30 07:49:21 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201705231200.0) was not posted for host (0), it was dropped from queue due to lack of space
Tue May 30 07:53:55 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Tue May 30 07:57:44 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Tue May 30 08:00:00 BNT [filer02: kern.uptime.filer:info]:   8:00am up 16 mins, 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 2696 iSCSI ops
Tue May 30 08:00:01 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 24 hours.
Tue May 30 08:00:46 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Tue May 30 08:00:55 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Tue May 30 08:00:55 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201705231600.0) was not posted for host (0), it was dropped from queue due to lack of space
Tue May 30 08:04:34 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Tue May 30 08:08:43 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Tue May 30 08:13:02 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Tue May 30 08:13:02 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201705232000.0) for host (0) was not posted to NetApp. The system will drop the message
Tue May 30 08:50:30 BNT [filer02: cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 0a.26 (DB493CEF:613B738F:C6775410:52ABD6E0:00000000:00000000:00000000:00000000:00000000:00000000) has recently changed or the node (filer01) is missing the disk.
Tue May 30 08:50:30 BNT [filer02: cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 0a.33 (016F3AB5:B475F4EC:10505E83:02202755:00000000:00000000:00000000:00000000:00000000:00000000) has recently changed or the node (filer02) is missing the disk.
Tue May 30 08:50:30 BNT [filer02: cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 0a.35 (9115483F:9DD43BC4:27172773:99F7D8B4:00000000:00000000:00000000:00000000:00000000:00000000) has recently changed or the node (filer02) is missing the disk.
Tue May 30 08:50:30 BNT [filer02: cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 0a.40 (072AEB65:C662A9CF:E58B9917:9BFA787B:00000000:00000000:00000000:00000000:00000000:00000000) has recently changed or the node (filer02) is missing the disk.
Tue May 30 09:00:00 BNT [filer02: kern.uptime.filer:info]:   9:00am up  1:16 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 4140 iSCSI ops
Tue May 30 09:00:01 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 23 hours.
Tue May 30 10:00:00 BNT [filer02: kern.uptime.filer:info]:  10:00am up  2:16 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 8902 iSCSI ops
Tue May 30 10:00:01 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 22 hours.
Tue May 30 11:00:00 BNT [filer02: kern.uptime.filer:info]:  11:00am up  3:16 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 58642 iSCSI ops
Tue May 30 11:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 21 hours.
Tue May 30 12:00:00 BNT [filer02: kern.uptime.filer:info]:  12:00pm up  4:16 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 174860 iSCSI ops
Tue May 30 12:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 20 hours.
Tue May 30 12:00:31 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Tue May 30 12:00:53 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Tue May 30 12:04:30 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Tue May 30 12:08:10 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Tue May 30 12:12:15 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Tue May 30 12:15:45 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Tue May 30 12:32:05 BNT last message repeated 4 times
Tue May 30 12:36:24 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Tue May 30 12:52:32 BNT last message repeated 4 times
Tue May 30 12:56:41 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Tue May 30 12:56:41 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201705240000.0) for host (0) was not posted to NetApp. The system will drop the message
Tue May 30 13:00:00 BNT [filer02: kern.uptime.filer:info]:   1:00pm up  5:16 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 225509 iSCSI ops
Tue May 30 13:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 19 hours.
Tue May 30 14:00:00 BNT [filer02: kern.uptime.filer:info]:   2:00pm up  6:16 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 229068 iSCSI ops
Tue May 30 14:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 18 hours.
Tue May 30 15:00:00 BNT [filer02: kern.uptime.filer:info]:   3:00pm up  7:16 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 234491 iSCSI ops
Tue May 30 15:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 17 hours.
Tue May 30 16:00:00 BNT [filer02: kern.uptime.filer:info]:   4:00pm up  8:16 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 241613 iSCSI ops
Tue May 30 16:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 16 hours.
Tue May 30 16:00:30 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Tue May 30 16:20:24 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Tue May 30 16:45:16 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Tue May 30 16:56:23 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201705240802.0) for host (0) was not posted to NetApp. The system will drop the message
Tue May 30 17:00:00 BNT [filer02: kern.uptime.filer:info]:   5:00pm up  9:16 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 241613 iSCSI ops
Tue May 30 17:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 15 hours.
Tue May 30 18:00:00 BNT [filer02: kern.uptime.filer:info]:   6:00pm up 10:16 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 241613 iSCSI ops
Tue May 30 18:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 14 hours.
Tue May 30 19:00:00 BNT [filer02: kern.uptime.filer:info]:   7:00pm up 11:16 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 241613 iSCSI ops
Tue May 30 19:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 13 hours.
Tue May 30 20:00:00 BNT [filer02: kern.uptime.filer:info]:   8:00pm up 12:16 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 241613 iSCSI ops
Tue May 30 20:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 12 hours.
Tue May 30 20:00:15 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Tue May 30 20:55:52 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201705240803.0) for host (0) was not posted to NetApp. The system will drop the message
Tue May 30 21:00:00 BNT [filer02: kern.uptime.filer:info]:   9:00pm up 13:16 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 241613 iSCSI ops
Tue May 30 21:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 11 hours.
Tue May 30 22:00:00 BNT [filer02: kern.uptime.filer:info]:  10:00pm up 14:16 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 241613 iSCSI ops
Tue May 30 22:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 10 hours.
Tue May 30 23:00:00 BNT [filer02: kern.uptime.filer:info]:  11:00pm up 15:16 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 241613 iSCSI ops
Tue May 30 23:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 9 hours.
Wed May 31 00:00:00 BNT [filer02: kern.uptime.filer:info]:  12:00am up 16:16 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 241613 iSCSI ops
Wed May 31 00:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 8 hours.
Wed May 31 00:00:16 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of nightly.0 snapshot (Snapshot operation failed: No space left on device.)
Wed May 31 00:21:29 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (REBOOT)
Wed May 31 00:58:02 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201705240803.1) for host (0) was not posted to NetApp. The system will drop the message
Wed May 31 01:00:00 BNT [filer02: kern.uptime.filer:info]:   1:00am up 17:16 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 241613 iSCSI ops
Wed May 31 01:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 7 hours.
Wed May 31 02:00:00 BNT [filer02: kern.uptime.filer:info]:   2:00am up 18:17 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 241613 iSCSI ops
Wed May 31 02:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 6 hours.
Wed May 31 03:00:00 BNT [filer02: kern.uptime.filer:info]:   3:00am up 19:17 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 241613 iSCSI ops
Wed May 31 03:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 5 hours.
Wed May 31 04:00:00 BNT [filer02: kern.uptime.filer:info]:   4:00am up 20:17 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 241613 iSCSI ops
Wed May 31 04:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 4 hours.
Wed May 31 05:00:00 BNT [filer02: kern.uptime.filer:info]:   5:00am up 21:17 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 241613 iSCSI ops
Wed May 31 05:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 3 hours.
Wed May 31 06:00:00 BNT [filer02: kern.uptime.filer:info]:   6:00am up 22:17 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 241613 iSCSI ops
Wed May 31 06:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 2 hours.
Wed May 31 07:00:00 BNT [filer02: kern.uptime.filer:info]:   7:00am up 23:17 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 241613 iSCSI ops
Wed May 31 07:00:00 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] is still present in the system and should be removed.
Wed May 31 07:00:00 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] is still present in the system and should be removed.
Wed May 31 07:00:00 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] is still present in the system and should be removed.
Wed May 31 08:00:00 BNT [filer02: monitor.brokenDisk.notice:notice]: When two disks are broken in raid_dp volume,  the system shuts down automatically every 24  hours to encourage you to replace the disk.  If you reboot the system it will run for another  24 hours before shutting down. (The 24 hour timeout  may be increased by altering the "raid.timeout"  value using the "options" command.)
Wed May 31 08:00:00 BNT [filer02: monitor.shutdown.brokenDisk:EMERGENCY]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system now.
Wed May 31 08:00:03 BNT [filer02: cf.fsm.takeoverOfPartnerDisabled:notice]: Cluster monitor: takeover of filer01 disabled (local halt in progress)
Wed May 31 08:00:06 BNT [filer02: kern.shutdown:notice]: System shut down because : "BROKEN DISK".
Wed May 31 23:42:27 GMT [localhost: rc:notice]: The system was down for 85337 seconds
Thu Jun  1 07:42:01 BNT [filer02: cf.nm.nicTransitionUp:info]: replayed event: Interconnect link 0 is UP
Thu Jun  1 07:42:08 BNT [filer02: fci.initialization.failed:error]: replayed event: Initialization failed on Fibre Channel adapter 0b.
Thu Jun  1 07:42:10 BNT [filer02: nvram.battery.turned.on:info]: replayed event: The NVRAM battery is turned ON. It is turned OFF during system shutdown.
Thu Jun  1 07:42:13 BNT [filer02: diskown.isEnabled:info]: replayed event: software ownership has been enabled for this system
Thu Jun  1 07:42:17 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.2 is a local HA mailbox disk.
Thu Jun  1 07:42:17 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.0 is a local HA mailbox disk.
Thu Jun  1 07:42:17 BNT [filer02: fmmb.instStat.change:info]: replayed event: normal mailbox instance on local side.
Thu Jun  1 07:42:17 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.7 is a partner HA mailbox disk.
Thu Jun  1 07:42:17 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.3 is a partner HA mailbox disk.
Thu Jun  1 07:42:17 BNT [filer02: fmmb.instStat.change:info]: replayed event: normal mailbox instance on partner side.
Thu Jun  1 07:42:17 BNT [filer02: cf.fm.partner:info]: replayed event: Cluster monitor: partner 'filer01'
Thu Jun  1 07:42:17 BNT [filer02: cf.fm.timeMasterStatus:info]: replayed event: Acting as cluster time slave
Thu Jun  1 07:42:17 BNT [filer02: raid.assim.disk.brokenPreAssim:error]: replayed event: Broken Disk 0a.40 Shelf ? Bay ? [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] detected prior to assimilation. It should be removed.
Thu Jun  1 07:42:17 BNT [filer02: raid.assim.disk.brokenPreAssim:error]: replayed event: Broken Disk 0a.35 Shelf ? Bay ? [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] detected prior to assimilation. It should be removed.
Thu Jun  1 07:42:17 BNT [filer02: raid.assim.disk.brokenPreAssim:error]: replayed event: Broken Disk 0a.33 Shelf ? Bay ? [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] detected prior to assimilation. It should be removed.
Thu Jun  1 07:42:18 BNT [filer02: coredump.spare.none:info]: replayed event: No sparecore disk was found.
Thu Jun  1 07:42:19 BNT [filer02: raid.cksum.replay.summary:info]: replayed event: Replayed 0 checksum blocks.
Thu Jun  1 07:42:19 BNT [filer02: raid.stripe.replay.summary:info]: replayed event: Replayed 0 stripes.
Thu Jun  1 07:42:21 BNT [filer02: cf.fm.launch:info]: replayed event: Launching cluster monitor
Thu Jun  1 07:42:21 BNT [filer02: cf.fm.partner:info]: replayed event: Cluster monitor: partner 'filer01'
Thu Jun  1 07:42:21 BNT [filer02: cf.fsm.takeoverOfPartnerDisabled:notice]: replayed event: Cluster monitor: takeover of filer01 disabled (partner booting)
Thu Jun  1 07:42:27 BNT [filer02: fcp.service.adapter:warning]: replayed event: FCP started but no adapter is present!
Thu Jun  1 07:42:27 BNT [filer02: kern.syslog.msg:notice]: replayed event: The system was down for 85337 seconds
Thu Jun  1 07:42:28 BNT [filer02: disk.failmsg:error]: Disk 0a.40 (N00V0W8L): message received.
Thu Jun  1 07:42:28 BNT [filer02: disk.failmsg:error]: Disk 0a.35 (N00WLAUL): message received.
Thu Jun  1 07:42:28 BNT [filer02: disk.failmsg:error]: Disk 0a.33 (N00W75PL): message received.
Thu Jun  1 07:42:28 BNT [filer02: raid.disk.unload.done:info]: Unload of Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] has completed successfully
Thu Jun  1 07:42:28 BNT [filer02: raid.disk.unload.done:info]: Unload of Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] has completed successfully
Thu Jun  1 07:42:28 BNT [filer02: raid.disk.unload.done:info]: Unload of Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] has completed successfully
Thu Jun  1 07:42:28 BNT [filer02: raid.fdr.failed.ok:info]: Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] successfully deleted from spare pool
Thu Jun  1 07:42:28 BNT [filer02: raid.fdr.failed.ok:info]: Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] successfully deleted from spare pool
Thu Jun  1 07:42:28 BNT [filer02: raid.fdr.failed.ok:info]: Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] successfully deleted from spare pool
Thu Jun  1 07:42:29 BNT [filer02: dfu.firmwareUpToDate:info]: Firmware is up-to-date on all disk drives
Thu Jun  1 07:42:29 BNT [filer02: cf.fsm.takeoverOfPartnerEnabled:notice]: Cluster monitor: takeover of filer01 enabled
Thu Jun  1 07:42:29 BNT [filer02: cf.fsm.takeoverByPartnerEnabled:notice]: Cluster monitor: takeover of filer02 by filer01 enabled
Thu Jun  1 07:42:30 BNT [filer02: perf.archive.start:info]: Performance archiver started. Sampling 22 objects and 198 counters.
Thu Jun  1 07:42:33 BNT [filer02: netif.linkUp:info]: Ethernet e0b: Link up.
Thu Jun  1 07:42:33 BNT [filer02: netif.linkUp:info]: Ethernet e0a: Link up.
Thu Jun  1 07:42:33 BNT [filer02: pvif.switchLink:warning]: vif1: switching to e0a
Thu Jun  1 07:42:35 BNT [filer02: iscsi.service.startup:info]: iSCSI service startup
Thu Jun  1 07:42:35 BNT [filer02: iscsi.notice:notice]: ISCSI: New session from initiator iqn.1991-05.com.microsoft:pcb-722-ad1.pcbl722dom.local at IP addr 10.72.4.1
Thu Jun  1 07:42:35 BNT [filer02: cmds.sysconf.logErr:error]: sysconfig: Unless directed by NetApp Global Services the cf.takeover.on_panic option should be set to On. .
Thu Jun  1 07:42:35 BNT [filer02: raid.rg.recons.missing:notice]: RAID group /aggr1/plex0/rg0 is missing 2 disk(s).
Thu Jun  1 07:42:35 BNT [filer02: raid.rg.recons.cantStart:warning]: The reconstruction cannot start in RAID group /aggr1/plex0/rg0: No 520 bps disk of required size available in spare pool, targeting any  spare pool
Thu Jun  1 07:42:35 BNT [filer02: mgr.boot.disk_done:info]: NetApp Release 7.3.2 boot complete. Last disk update written at Wed May 31 08:00:07 BNT 2017
Thu Jun  1 07:42:36 BNT [filer02: monitor.raid.brokenDisk:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken.
Thu Jun  1 07:42:36 BNT [filer02: monitor.brokenDisk.notice:notice]: When two disks are broken in raid_dp volume,  the system shuts down automatically every 24  hours to encourage you to replace the disk.  If you reboot the system it will run for another  24 hours before shutting down. (The 24 hour timeout  may be increased by altering the "raid.timeout"  value using the "options" command.)
Thu Jun  1 07:42:36 BNT [filer02: diskown.errorReadingOwnership:warning]: error 3 (disk failed) while reading ownership on disk 0a.24 (S/N N00SSSAL)
Thu Jun  1 07:42:36 BNT [filer02: diskown.errorReadingOwnership:warning]: error 3 (disk failed) while reading ownership on disk 0a.26 (S/N N00V0UWL)
Thu Jun  1 07:42:36 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] is still present in the system and should be removed.
Thu Jun  1 07:42:36 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] is still present in the system and should be removed.
Thu Jun  1 07:42:36 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] is still present in the system and should be removed.
Thu Jun  1 07:42:36 BNT [filer02: shelf.config.spha:info]: System is using single path HA attached storage only.
Thu Jun  1 07:42:36 BNT [filer02: mgr.boot.reason_ok:notice]: System rebooted.
Thu Jun  1 07:42:57 BNT [filer02: raid.rg.spares.low:warning]: /aggr1/plex0/rg0
Thu Jun  1 07:43:01 BNT [filer02: nbt.nbns.registrationComplete:info]: NBT: All CIFS name registrations have completed for the local server.
Thu Jun  1 07:43:02 BNT [filer02: monitor.globalStatus.nonCritical:warning]: Disk on adapter 0a, shelf 2, bay 8, failed. Disk on adapter 0a, shelf 2, bay 3, failed. Disk on adapter 0a, shelf 2, bay 1, failed.  There are not enough spare disks. /vol/vol1 is full (using or reserving 100% of space and 0% of inodes, using 92% of reserve).
Thu Jun  1 07:43:05 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SPARES_LOW)
Thu Jun  1 07:43:14 BNT last message repeated 2 times
Thu Jun  1 07:43:14 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201705240803.2) was not posted for host (0), it was dropped from queue due to lack of space
Thu Jun  1 07:43:21 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (PERFORMANCE SNAPSHOT)
Thu Jun  1 07:43:21 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201705240803.3) was not posted for host (0), it was dropped from queue due to lack of space
Thu Jun  1 07:43:29 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (CLUSTER ERROR: DISK/SHELF COUNT MISMATCH)
Thu Jun  1 07:43:29 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201705240808.0) was not posted for host (0), it was dropped from queue due to lack of space
Thu Jun  1 07:43:44 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu Jun  1 07:43:44 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201705241200.0) was not posted for host (0), it was dropped from queue due to lack of space
Thu Jun  1 07:48:15 BNT [filer02: cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 0a.33 (016F3AB5:B475F4EC:10505E83:02202755:00000000:00000000:00000000:00000000:00000000:00000000) has recently changed or the node (filer02) is missing the disk.
Thu Jun  1 07:48:15 BNT [filer02: cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 0a.35 (9115483F:9DD43BC4:27172773:99F7D8B4:00000000:00000000:00000000:00000000:00000000:00000000) has recently changed or the node (filer02) is missing the disk.
Thu Jun  1 07:48:15 BNT [filer02: cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 0a.40 (072AEB65:C662A9CF:E58B9917:9BFA787B:00000000:00000000:00000000:00000000:00000000:00000000) has recently changed or the node (filer02) is missing the disk.
Thu Jun  1 07:48:16 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu Jun  1 07:48:23 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu Jun  1 07:48:23 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201705241600.0) was not posted for host (0), it was dropped from queue due to lack of space
Thu Jun  1 07:52:31 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu Jun  1 07:56:59 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu Jun  1 08:00:00 BNT [filer02: kern.uptime.filer:info]:   8:00am up 17 mins, 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 2691 iSCSI ops
Thu Jun  1 08:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 24 hours.
Thu Jun  1 08:00:00 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Thu Jun  1 08:00:10 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu Jun  1 08:00:10 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201705242000.0) was not posted for host (0), it was dropped from queue due to lack of space
Thu Jun  1 08:04:34 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu Jun  1 08:08:13 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu Jun  1 08:12:31 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu Jun  1 08:16:33 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu Jun  1 08:16:33 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201705250000.0) for host (0) was not posted to NetApp. The system will drop the message
Thu Jun  1 08:49:33 BNT [filer02: cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 0a.33 (016F3AB5:B475F4EC:10505E83:02202755:00000000:00000000:00000000:00000000:00000000:00000000) has recently changed or the node (filer02) is missing the disk.
Thu Jun  1 08:49:33 BNT [filer02: cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 0a.35 (9115483F:9DD43BC4:27172773:99F7D8B4:00000000:00000000:00000000:00000000:00000000:00000000) has recently changed or the node (filer02) is missing the disk.
Thu Jun  1 08:49:33 BNT [filer02: cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 0a.40 (072AEB65:C662A9CF:E58B9917:9BFA787B:00000000:00000000:00000000:00000000:00000000:00000000) has recently changed or the node (filer02) is missing the disk.
Thu Jun  1 09:00:00 BNT [filer02: kern.uptime.filer:info]:   9:00am up  1:17 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 100450 iSCSI ops
Thu Jun  1 09:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 23 hours.
Thu Jun  1 10:00:00 BNT [filer02: kern.uptime.filer:info]:  10:00am up  2:17 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 621873 iSCSI ops
Thu Jun  1 10:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 22 hours.
Thu Jun  1 11:00:00 BNT [filer02: kern.uptime.filer:info]:  11:00am up  3:17 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 770366 iSCSI ops
Thu Jun  1 11:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 21 hours.
Thu Jun  1 12:00:00 BNT [filer02: kern.uptime.filer:info]:  12:00pm up  4:17 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 808053 iSCSI ops
Thu Jun  1 12:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 20 hours.
Thu Jun  1 12:00:45 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Thu Jun  1 12:01:09 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Thu Jun  1 12:05:25 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Thu Jun  1 12:09:52 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Thu Jun  1 12:13:44 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Thu Jun  1 12:17:51 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Thu Jun  1 12:32:42 BNT last message repeated 4 times
Thu Jun  1 12:36:25 BNT [filer02: telnet_0:info]: root logged in from host: pcb-722-ad1.pcbl722dom.local
Thu Jun  1 12:36:45 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Thu Jun  1 12:41:00 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Thu Jun  1 12:45:15 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Thu Jun  1 12:49:44 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Thu Jun  1 12:53:36 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Thu Jun  1 12:57:25 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Thu Jun  1 12:57:25 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201705250746.0) for host (0) was not posted to NetApp. The system will drop the message
Thu Jun  1 13:00:00 BNT [filer02: kern.uptime.filer:info]:   1:00pm up  5:17 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 834394 iSCSI ops
Thu Jun  1 13:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 19 hours.
Thu Jun  1 14:00:00 BNT [filer02: kern.uptime.filer:info]:   2:00pm up  6:17 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 1174903 iSCSI ops
Thu Jun  1 14:00:01 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 18 hours.
Thu Jun  1 15:00:00 BNT [filer02: kern.uptime.filer:info]:   3:00pm up  7:17 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 1684470 iSCSI ops
Thu Jun  1 15:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 17 hours.
Thu Jun  1 16:00:00 BNT [filer02: kern.uptime.filer:info]:   4:00pm up  8:17 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 1684603 iSCSI ops
Thu Jun  1 16:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 16 hours.
Thu Jun  1 16:00:46 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Thu Jun  1 16:04:51 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SHUTDOWN PENDING (degraded mode))
Thu Jun  1 16:24:21 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SHUTDOWN PENDING (degraded mode))
Thu Jun  1 16:32:27 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SHUTDOWN PENDING (degraded mode))
Thu Jun  1 16:57:27 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201705250746.1) for host (0) was not posted to NetApp. The system will drop the message
Thu Jun  1 17:00:00 BNT [filer02: kern.uptime.filer:info]:   5:00pm up  9:17 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 1684603 iSCSI ops
Thu Jun  1 17:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 15 hours.
Thu Jun  1 18:00:00 BNT [filer02: kern.uptime.filer:info]:   6:00pm up 10:17 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 1684603 iSCSI ops
Thu Jun  1 18:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 14 hours.
Thu Jun  1 19:00:00 BNT [filer02: kern.uptime.filer:info]:   7:00pm up 11:17 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 1684603 iSCSI ops
Thu Jun  1 19:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 13 hours.
Thu Jun  1 20:00:00 BNT [filer02: kern.uptime.filer:info]:   8:00pm up 12:17 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 1684603 iSCSI ops
Thu Jun  1 20:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 12 hours.
Thu Jun  1 20:00:31 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Thu Jun  1 20:24:28 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (REBOOT (power on))
Thu Jun  1 20:58:02 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201705250746.2) for host (0) was not posted to NetApp. The system will drop the message
Thu Jun  1 21:00:00 BNT [filer02: kern.uptime.filer:info]:   9:00pm up 13:17 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 1684603 iSCSI ops
Thu Jun  1 21:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 11 hours.
Thu Jun  1 22:00:00 BNT [filer02: kern.uptime.filer:info]:  10:00pm up 14:17 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 1684603 iSCSI ops
Thu Jun  1 22:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 10 hours.
Thu Jun  1 23:00:00 BNT [filer02: kern.uptime.filer:info]:  11:00pm up 15:17 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 1684603 iSCSI ops
Thu Jun  1 23:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 9 hours.
Fri Jun  2 00:00:00 BNT [filer02: kern.uptime.filer:info]:  12:00am up 16:17 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 1684603 iSCSI ops
Fri Jun  2 00:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 8 hours.
Fri Jun  2 00:00:30 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of nightly.0 snapshot (Snapshot operation failed: No space left on device.)
Fri Jun  2 00:17:50 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SPARES_LOW)
Fri Jun  2 00:59:39 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201705250747.0) for host (0) was not posted to NetApp. The system will drop the message
Fri Jun  2 01:00:00 BNT [filer02: kern.uptime.filer:info]:   1:00am up 17:17 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 1684603 iSCSI ops
Fri Jun  2 01:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 7 hours.
Fri Jun  2 02:00:00 BNT [filer02: kern.uptime.filer:info]:   2:00am up 18:17 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 1684603 iSCSI ops
Fri Jun  2 02:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 6 hours.
Fri Jun  2 03:00:00 BNT [filer02: kern.uptime.filer:info]:   3:00am up 19:17 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 1684603 iSCSI ops
Fri Jun  2 03:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 5 hours.
Fri Jun  2 04:00:00 BNT [filer02: kern.uptime.filer:info]:   4:00am up 20:17 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 1684603 iSCSI ops
Fri Jun  2 04:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 4 hours.
Fri Jun  2 05:00:00 BNT [filer02: kern.uptime.filer:info]:   5:00am up 21:17 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 1684603 iSCSI ops
Fri Jun  2 05:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 3 hours.
Fri Jun  2 06:00:00 BNT [filer02: kern.uptime.filer:info]:   6:00am up 22:17 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 1684603 iSCSI ops
Fri Jun  2 06:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 2 hours.
Fri Jun  2 07:00:01 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] is still present in the system and should be removed.
Fri Jun  2 07:00:01 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] is still present in the system and should be removed.
Fri Jun  2 07:00:01 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] is still present in the system and should be removed.
Fri Jun  2 08:00:00 BNT [filer02: monitor.brokenDisk.notice:notice]: When two disks are broken in raid_dp volume,  the system shuts down automatically every 24  hours to encourage you to replace the disk.  If you reboot the system it will run for another  24 hours before shutting down. (The 24 hour timeout  may be increased by altering the "raid.timeout"  value using the "options" command.)
Fri Jun  2 08:00:00 BNT [filer02: monitor.shutdown.brokenDisk:EMERGENCY]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system now.
Fri Jun  2 08:00:00 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Fri Jun  2 08:00:03 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol0: skipping creation of hourly.0 snapshot (Snapshot operation not allowed during giveback)
Fri Jun  2 08:00:04 BNT [filer02: cf.fsm.takeoverOfPartnerDisabled:notice]: Cluster monitor: takeover of filer01 disabled (local halt in progress)
Fri Jun  2 08:00:07 BNT [filer02: kern.shutdown:notice]: System shut down because : "BROKEN DISK".
Fri Jun  2 00:08:09 GMT [localhost: rc:notice]: The system was down for 477 seconds
Fri Jun  2 08:07:42 BNT [filer02: cf.nm.nicTransitionUp:info]: replayed event: Interconnect link 0 is UP
Fri Jun  2 08:07:49 BNT [filer02: fci.initialization.failed:error]: replayed event: Initialization failed on Fibre Channel adapter 0b.
Fri Jun  2 08:07:51 BNT [filer02: nvram.battery.turned.on:info]: replayed event: The NVRAM battery is turned ON. It is turned OFF during system shutdown.
Fri Jun  2 08:07:54 BNT [filer02: diskown.isEnabled:info]: replayed event: software ownership has been enabled for this system
Fri Jun  2 08:07:57 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.2 is a local HA mailbox disk.
Fri Jun  2 08:07:57 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.0 is a local HA mailbox disk.
Fri Jun  2 08:07:57 BNT [filer02: fmmb.instStat.change:info]: replayed event: normal mailbox instance on local side.
Fri Jun  2 08:07:58 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.7 is a partner HA mailbox disk.
Fri Jun  2 08:07:58 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.3 is a partner HA mailbox disk.
Fri Jun  2 08:07:58 BNT [filer02: fmmb.instStat.change:info]: replayed event: normal mailbox instance on partner side.
Fri Jun  2 08:07:58 BNT [filer02: cf.fm.partner:info]: replayed event: Cluster monitor: partner 'filer01'
Fri Jun  2 08:07:58 BNT [filer02: cf.fm.timeMasterStatus:info]: replayed event: Acting as cluster time slave
Fri Jun  2 08:07:59 BNT [filer02: raid.assim.disk.brokenPreAssim:error]: replayed event: Broken Disk 0a.40 Shelf ? Bay ? [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] detected prior to assimilation. It should be removed.
Fri Jun  2 08:07:59 BNT [filer02: raid.assim.disk.brokenPreAssim:error]: replayed event: Broken Disk 0a.35 Shelf ? Bay ? [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] detected prior to assimilation. It should be removed.
Fri Jun  2 08:07:59 BNT [filer02: raid.assim.disk.brokenPreAssim:error]: replayed event: Broken Disk 0a.33 Shelf ? Bay ? [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] detected prior to assimilation. It should be removed.
Fri Jun  2 08:07:59 BNT [filer02: coredump.spare.none:info]: replayed event: No sparecore disk was found.
Fri Jun  2 08:08:00 BNT [filer02: raid.cksum.replay.summary:info]: replayed event: Replayed 0 checksum blocks.
Fri Jun  2 08:08:00 BNT [filer02: raid.stripe.replay.summary:info]: replayed event: Replayed 0 stripes.
Fri Jun  2 08:08:02 BNT [filer02: cf.fm.launch:info]: replayed event: Launching cluster monitor
Fri Jun  2 08:08:02 BNT [filer02: cf.fm.partner:info]: replayed event: Cluster monitor: partner 'filer01'
Fri Jun  2 08:08:02 BNT [filer02: cf.fsm.takeoverOfPartnerDisabled:notice]: replayed event: Cluster monitor: takeover of filer01 disabled (partner booting)
Fri Jun  2 08:08:08 BNT [filer02: fcp.service.adapter:warning]: replayed event: FCP started but no adapter is present!
Fri Jun  2 08:08:09 BNT [filer02: kern.syslog.msg:notice]: replayed event: The system was down for 477 seconds
Fri Jun  2 08:08:10 BNT [filer02: disk.failmsg:error]: Disk 0a.40 (N00V0W8L): message received.
Fri Jun  2 08:08:10 BNT [filer02: disk.failmsg:error]: Disk 0a.35 (N00WLAUL): message received.
Fri Jun  2 08:08:10 BNT [filer02: disk.failmsg:error]: Disk 0a.33 (N00W75PL): message received.
Fri Jun  2 08:08:10 BNT [filer02: raid.disk.unload.done:info]: Unload of Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] has completed successfully
Fri Jun  2 08:08:10 BNT [filer02: raid.disk.unload.done:info]: Unload of Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] has completed successfully
Fri Jun  2 08:08:10 BNT [filer02: raid.disk.unload.done:info]: Unload of Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] has completed successfully
Fri Jun  2 08:08:10 BNT [filer02: raid.fdr.failed.ok:info]: Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] successfully deleted from spare pool
Fri Jun  2 08:08:10 BNT [filer02: raid.fdr.failed.ok:info]: Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] successfully deleted from spare pool
Fri Jun  2 08:08:10 BNT [filer02: raid.fdr.failed.ok:info]: Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] successfully deleted from spare pool
Fri Jun  2 08:08:10 BNT [filer02: dfu.firmwareUpToDate:info]: Firmware is up-to-date on all disk drives
Fri Jun  2 08:08:11 BNT [filer02: cf.fsm.takeoverOfPartnerEnabled:notice]: Cluster monitor: takeover of filer01 enabled
Fri Jun  2 08:08:11 BNT [filer02: cf.fsm.takeoverByPartnerEnabled:notice]: Cluster monitor: takeover of filer02 by filer01 enabled
Fri Jun  2 08:08:12 BNT [filer02: perf.archive.start:info]: Performance archiver started. Sampling 22 objects and 198 counters.
Fri Jun  2 08:08:14 BNT [filer02: netif.linkUp:info]: Ethernet e0b: Link up.
Fri Jun  2 08:08:14 BNT [filer02: netif.linkUp:info]: Ethernet e0a: Link up.
Fri Jun  2 08:08:15 BNT [filer02: pvif.switchLink:warning]: vif1: switching to e0a
Fri Jun  2 08:08:16 BNT [filer02: iscsi.service.startup:info]: iSCSI service startup
Fri Jun  2 08:08:16 BNT [filer02: iscsi.notice:notice]: ISCSI: New session from initiator iqn.1991-05.com.microsoft:pcb-722-ad1.pcbl722dom.local at IP addr 10.72.4.1
Fri Jun  2 08:08:17 BNT [filer02: cmds.sysconf.logErr:error]: sysconfig: Unless directed by NetApp Global Services the cf.takeover.on_panic option should be set to On. .
Fri Jun  2 08:08:17 BNT [filer02: raid.rg.recons.missing:notice]: RAID group /aggr1/plex0/rg0 is missing 2 disk(s).
Fri Jun  2 08:08:17 BNT [filer02: raid.rg.recons.cantStart:warning]: The reconstruction cannot start in RAID group /aggr1/plex0/rg0: No 520 bps disk of required size available in spare pool, targeting any  spare pool
Fri Jun  2 08:08:17 BNT [filer02: mgr.boot.disk_done:info]: NetApp Release 7.3.2 boot complete. Last disk update written at Fri Jun  2 08:00:09 BNT 2017
Fri Jun  2 08:08:17 BNT [filer02: monitor.raid.brokenDisk:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken.
Fri Jun  2 08:08:17 BNT [filer02: monitor.brokenDisk.notice:notice]: When two disks are broken in raid_dp volume,  the system shuts down automatically every 24  hours to encourage you to replace the disk.  If you reboot the system it will run for another  24 hours before shutting down. (The 24 hour timeout  may be increased by altering the "raid.timeout"  value using the "options" command.)
Fri Jun  2 08:08:17 BNT [filer02: diskown.errorReadingOwnership:warning]: error 3 (disk failed) while reading ownership on disk 0a.24 (S/N N00SSSAL)
Fri Jun  2 08:08:17 BNT [filer02: diskown.errorReadingOwnership:warning]: error 3 (disk failed) while reading ownership on disk 0a.26 (S/N N00V0UWL)
Fri Jun  2 08:08:17 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] is still present in the system and should be removed.
Fri Jun  2 08:08:17 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] is still present in the system and should be removed.
Fri Jun  2 08:08:17 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] is still present in the system and should be removed.
Fri Jun  2 08:08:17 BNT [filer02: shelf.config.spha:info]: System is using single path HA attached storage only.
Fri Jun  2 08:08:17 BNT [filer02: mgr.boot.reason_ok:notice]: System rebooted.
Fri Jun  2 08:08:39 BNT [filer02: raid.rg.spares.low:warning]: /aggr1/plex0/rg0
Fri Jun  2 08:08:42 BNT [filer02: nbt.nbns.registrationComplete:info]: NBT: All CIFS name registrations have completed for the local server.
Fri Jun  2 08:08:47 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (PERFORMANCE SNAPSHOT)
Fri Jun  2 08:08:56 BNT last message repeated 2 times
Fri Jun  2 08:08:56 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201705250747.1) was not posted for host (0), it was dropped from queue due to lack of space
Fri Jun  2 08:09:03 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jun  2 08:09:03 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201705250800.0) was not posted for host (0), it was dropped from queue due to lack of space
Fri Jun  2 08:09:05 BNT [filer02: monitor.globalStatus.nonCritical:warning]: Disk on adapter 0a, shelf 2, bay 8, failed. Disk on adapter 0a, shelf 2, bay 3, failed. Disk on adapter 0a, shelf 2, bay 1, failed.  There are not enough spare disks. /vol/vol1 is full (using or reserving 100% of space and 0% of inodes, using 92% of reserve).
Fri Jun  2 08:09:11 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jun  2 08:09:11 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201705251200.0) was not posted for host (0), it was dropped from queue due to lack of space
Fri Jun  2 08:09:25 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jun  2 08:09:25 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201705251600.0) was not posted for host (0), it was dropped from queue due to lack of space
Fri Jun  2 08:13:57 BNT [filer02: cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 0a.33 (016F3AB5:B475F4EC:10505E83:02202755:00000000:00000000:00000000:00000000:00000000:00000000) has recently changed or the node (filer02) is missing the disk.
Fri Jun  2 08:13:57 BNT [filer02: cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 0a.35 (9115483F:9DD43BC4:27172773:99F7D8B4:00000000:00000000:00000000:00000000:00000000:00000000) has recently changed or the node (filer02) is missing the disk.
Fri Jun  2 08:13:57 BNT [filer02: cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 0a.40 (072AEB65:C662A9CF:E58B9917:9BFA787B:00000000:00000000:00000000:00000000:00000000:00000000) has recently changed or the node (filer02) is missing the disk.
Fri Jun  2 08:13:58 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jun  2 08:14:05 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jun  2 08:14:05 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201705252000.0) was not posted for host (0), it was dropped from queue due to lack of space
Fri Jun  2 08:18:23 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jun  2 08:22:32 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jun  2 08:26:54 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jun  2 08:31:12 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jun  2 08:35:33 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jun  2 08:44:02 BNT last message repeated 2 times
Fri Jun  2 08:44:02 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201705260000.0) for host (0) was not posted to NetApp. The system will drop the message
Fri Jun  2 09:00:00 BNT [filer02: kern.uptime.filer:info]:   9:00am up 51 mins, 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 3118 iSCSI ops
Fri Jun  2 09:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 24 hours.
Fri Jun  2 09:15:16 BNT [filer02: cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 0a.33 (016F3AB5:B475F4EC:10505E83:02202755:00000000:00000000:00000000:00000000:00000000:00000000) has recently changed or the node (filer02) is missing the disk.
Fri Jun  2 09:15:16 BNT [filer02: cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 0a.35 (9115483F:9DD43BC4:27172773:99F7D8B4:00000000:00000000:00000000:00000000:00000000:00000000) has recently changed or the node (filer02) is missing the disk.
Fri Jun  2 09:15:16 BNT [filer02: cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 0a.40 (072AEB65:C662A9CF:E58B9917:9BFA787B:00000000:00000000:00000000:00000000:00000000:00000000) has recently changed or the node (filer02) is missing the disk.
Fri Jun  2 10:00:00 BNT [filer02: kern.uptime.filer:info]:  10:00am up  1:51 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 16757 iSCSI ops
Fri Jun  2 10:00:01 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 23 hours.
Fri Jun  2 11:00:00 BNT [filer02: kern.uptime.filer:info]:  11:00am up  2:51 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 225151 iSCSI ops
Fri Jun  2 11:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 22 hours.
Fri Jun  2 12:00:00 BNT [filer02: kern.uptime.filer:info]:  12:00pm up  3:51 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 508835 iSCSI ops
Fri Jun  2 12:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 21 hours.
Fri Jun  2 12:00:30 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Fri Jun  2 12:00:56 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Fri Jun  2 12:04:42 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Fri Jun  2 12:09:06 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Fri Jun  2 12:13:19 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Fri Jun  2 12:17:25 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Fri Jun  2 12:33:59 BNT last message repeated 4 times
Fri Jun  2 12:37:46 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Fri Jun  2 12:54:02 BNT last message repeated 4 times
Fri Jun  2 12:58:18 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Fri Jun  2 12:58:18 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201705260808.0) for host (0) was not posted to NetApp. The system will drop the message
Fri Jun  2 13:00:00 BNT [filer02: kern.uptime.filer:info]:   1:00pm up  4:51 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 514202 iSCSI ops
Fri Jun  2 13:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 20 hours.
Fri Jun  2 14:00:00 BNT [filer02: kern.uptime.filer:info]:   2:00pm up  5:51 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 514710 iSCSI ops
Fri Jun  2 14:00:01 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 19 hours.
Fri Jun  2 15:00:00 BNT [filer02: kern.uptime.filer:info]:   3:00pm up  6:51 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 514710 iSCSI ops
Fri Jun  2 15:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 18 hours.
Fri Jun  2 16:00:00 BNT [filer02: kern.uptime.filer:info]:   4:00pm up  7:51 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 514710 iSCSI ops
Fri Jun  2 16:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 17 hours.
Fri Jun  2 16:00:30 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Fri Jun  2 16:00:55 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SHUTDOWN PENDING (degraded mode))
Fri Jun  2 16:05:18 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SHUTDOWN PENDING (degraded mode))
Fri Jun  2 16:08:53 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SHUTDOWN PENDING (degraded mode))
Fri Jun  2 16:12:40 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SHUTDOWN PENDING (degraded mode))
Fri Jun  2 16:55:49 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201705260809.0) for host (0) was not posted to NetApp. The system will drop the message
Fri Jun  2 17:00:00 BNT [filer02: kern.uptime.filer:info]:   5:00pm up  8:51 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 514714 iSCSI ops
Fri Jun  2 17:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 16 hours.
Fri Jun  2 18:00:00 BNT [filer02: kern.uptime.filer:info]:   6:00pm up  9:51 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 514714 iSCSI ops
Fri Jun  2 18:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 15 hours.
Fri Jun  2 19:00:00 BNT [filer02: kern.uptime.filer:info]:   7:00pm up 10:51 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 514714 iSCSI ops
Fri Jun  2 19:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 14 hours.
Fri Jun  2 20:00:00 BNT [filer02: kern.uptime.filer:info]:   8:00pm up 11:51 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 514714 iSCSI ops
Fri Jun  2 20:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 13 hours.
Fri Jun  2 20:00:15 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Fri Jun  2 20:00:42 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (REBOOT)
Fri Jun  2 20:42:16 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (REBOOT)
Fri Jun  2 20:57:38 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201705260809.1) for host (0) was not posted to NetApp. The system will drop the message
Fri Jun  2 21:00:00 BNT [filer02: kern.uptime.filer:info]:   9:00pm up 12:51 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 514714 iSCSI ops
Fri Jun  2 21:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 12 hours.
Fri Jun  2 22:00:00 BNT [filer02: kern.uptime.filer:info]:  10:00pm up 13:51 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 514714 iSCSI ops
Fri Jun  2 22:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 11 hours.
Fri Jun  2 23:00:00 BNT [filer02: kern.uptime.filer:info]:  11:00pm up 14:51 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 514714 iSCSI ops
Fri Jun  2 23:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 10 hours.
Sat Jun  3 00:00:00 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of nightly.0 snapshot (Snapshot operation failed: No space left on device.)
Sat Jun  3 00:00:00 BNT [filer02: kern.uptime.filer:info]:  12:00am up 15:52 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 514714 iSCSI ops
Sat Jun  3 00:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 9 hours.
Sat Jun  3 00:08:02 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SPARES_LOW)
Sat Jun  3 00:27:26 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SPARES_LOW)
Sat Jun  3 00:54:59 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201705260809.2) for host (0) was not posted to NetApp. The system will drop the message
Sat Jun  3 01:00:00 BNT [filer02: kern.uptime.filer:info]:   1:00am up 16:52 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 514714 iSCSI ops
Sat Jun  3 01:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 8 hours.
Sat Jun  3 02:00:00 BNT [filer02: kern.uptime.filer:info]:   2:00am up 17:52 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 514714 iSCSI ops
Sat Jun  3 02:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 7 hours.
Sat Jun  3 03:00:00 BNT [filer02: kern.uptime.filer:info]:   3:00am up 18:52 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 514714 iSCSI ops
Sat Jun  3 03:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 6 hours.
Sat Jun  3 04:00:00 BNT [filer02: kern.uptime.filer:info]:   4:00am up 19:52 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 514714 iSCSI ops
Sat Jun  3 04:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 5 hours.
Sat Jun  3 05:00:00 BNT [filer02: kern.uptime.filer:info]:   5:00am up 20:52 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 514714 iSCSI ops
Sat Jun  3 05:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 4 hours.
Sat Jun  3 06:00:00 BNT [filer02: kern.uptime.filer:info]:   6:00am up 21:52 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 514714 iSCSI ops
Sat Jun  3 06:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 3 hours.
Sat Jun  3 07:00:00 BNT [filer02: kern.uptime.filer:info]:   7:00am up 22:52 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 514714 iSCSI ops
Sat Jun  3 07:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 2 hours.
Sat Jun  3 08:00:00 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] is still present in the system and should be removed.
Sat Jun  3 08:00:00 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] is still present in the system and should be removed.
Sat Jun  3 08:00:00 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] is still present in the system and should be removed.
Sat Jun  3 08:00:46 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Sat Jun  3 08:10:08 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (PERFORMANCE SNAPSHOT)
Sat Jun  3 08:59:24 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (PERFORMANCE SNAPSHOT)
Sat Jun  3 08:59:24 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201705260810.0) for host (0) was not posted to NetApp. The system will drop the message
Sat Jun  3 09:00:00 BNT [filer02: monitor.brokenDisk.notice:notice]: When two disks are broken in raid_dp volume,  the system shuts down automatically every 24  hours to encourage you to replace the disk.  If you reboot the system it will run for another  24 hours before shutting down. (The 24 hour timeout  may be increased by altering the "raid.timeout"  value using the "options" command.)
Sat Jun  3 09:00:00 BNT [filer02: monitor.shutdown.brokenDisk:EMERGENCY]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system now.
Sat Jun  3 09:00:03 BNT [filer02: cf.fsm.takeoverOfPartnerDisabled:notice]: Cluster monitor: takeover of filer01 disabled (local halt in progress)
Sat Jun  3 09:00:06 BNT [filer02: kern.shutdown:notice]: System shut down because : "BROKEN DISK".
Sun Jun  4 23:43:48 GMT [localhost: rc:notice]: The system was down for 168216 seconds
Mon Jun  5 07:43:22 BNT [filer02: cf.nm.nicTransitionUp:info]: replayed event: Interconnect link 0 is UP
Mon Jun  5 07:43:29 BNT [filer02: fci.initialization.failed:error]: replayed event: Initialization failed on Fibre Channel adapter 0b.
Mon Jun  5 07:43:31 BNT [filer02: nvram.battery.turned.on:info]: replayed event: The NVRAM battery is turned ON. It is turned OFF during system shutdown.
Mon Jun  5 07:43:34 BNT [filer02: diskown.isEnabled:info]: replayed event: software ownership has been enabled for this system
Mon Jun  5 07:43:38 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.2 is a local HA mailbox disk.
Mon Jun  5 07:43:38 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.0 is a local HA mailbox disk.
Mon Jun  5 07:43:38 BNT [filer02: fmmb.instStat.change:info]: replayed event: normal mailbox instance on local side.
Mon Jun  5 07:43:38 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.7 is a partner HA mailbox disk.
Mon Jun  5 07:43:38 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.3 is a partner HA mailbox disk.
Mon Jun  5 07:43:38 BNT [filer02: fmmb.instStat.change:info]: replayed event: normal mailbox instance on partner side.
Mon Jun  5 07:43:38 BNT [filer02: cf.fm.partner:info]: replayed event: Cluster monitor: partner 'filer01'
Mon Jun  5 07:43:38 BNT [filer02: cf.fm.timeMasterStatus:info]: replayed event: Acting as cluster time slave
Mon Jun  5 07:43:39 BNT [filer02: raid.assim.disk.brokenPreAssim:error]: replayed event: Broken Disk 0a.40 Shelf ? Bay ? [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] detected prior to assimilation. It should be removed.
Mon Jun  5 07:43:39 BNT [filer02: raid.assim.disk.brokenPreAssim:error]: replayed event: Broken Disk 0a.35 Shelf ? Bay ? [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] detected prior to assimilation. It should be removed.
Mon Jun  5 07:43:39 BNT [filer02: raid.assim.disk.brokenPreAssim:error]: replayed event: Broken Disk 0a.33 Shelf ? Bay ? [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] detected prior to assimilation. It should be removed.
Mon Jun  5 07:43:39 BNT [filer02: coredump.spare.none:info]: replayed event: No sparecore disk was found.
Mon Jun  5 07:43:40 BNT [filer02: raid.cksum.replay.summary:info]: replayed event: Replayed 0 checksum blocks.
Mon Jun  5 07:43:40 BNT [filer02: raid.stripe.replay.summary:info]: replayed event: Replayed 0 stripes.
Mon Jun  5 07:43:42 BNT [filer02: cf.fm.launch:info]: replayed event: Launching cluster monitor
Mon Jun  5 07:43:42 BNT [filer02: cf.fm.partner:info]: replayed event: Cluster monitor: partner 'filer01'
Mon Jun  5 07:43:42 BNT [filer02: cf.fsm.takeoverOfPartnerDisabled:notice]: replayed event: Cluster monitor: takeover of filer01 disabled (partner booting)
Mon Jun  5 07:43:48 BNT [filer02: fcp.service.adapter:warning]: replayed event: FCP started but no adapter is present!
Mon Jun  5 07:43:48 BNT [filer02: kern.syslog.msg:notice]: replayed event: The system was down for 168216 seconds
Mon Jun  5 07:43:50 BNT [filer02: disk.failmsg:error]: Disk 0a.40 (N00V0W8L): message received.
Mon Jun  5 07:43:50 BNT [filer02: disk.failmsg:error]: Disk 0a.35 (N00WLAUL): message received.
Mon Jun  5 07:43:50 BNT [filer02: disk.failmsg:error]: Disk 0a.33 (N00W75PL): message received.
Mon Jun  5 07:43:50 BNT [filer02: raid.disk.unload.done:info]: Unload of Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] has completed successfully
Mon Jun  5 07:43:50 BNT [filer02: raid.disk.unload.done:info]: Unload of Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] has completed successfully
Mon Jun  5 07:43:50 BNT [filer02: raid.disk.unload.done:info]: Unload of Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] has completed successfully
Mon Jun  5 07:43:50 BNT [filer02: raid.fdr.failed.ok:info]: Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] successfully deleted from spare pool
Mon Jun  5 07:43:50 BNT [filer02: raid.fdr.failed.ok:info]: Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] successfully deleted from spare pool
Mon Jun  5 07:43:50 BNT [filer02: raid.fdr.failed.ok:info]: Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] successfully deleted from spare pool
Mon Jun  5 07:43:50 BNT [filer02: dfu.firmwareUpToDate:info]: Firmware is up-to-date on all disk drives
Mon Jun  5 07:43:50 BNT [filer02: cf.fsm.takeoverOfPartnerEnabled:notice]: Cluster monitor: takeover of filer01 enabled
Mon Jun  5 07:43:50 BNT [filer02: cf.fsm.takeoverByPartnerEnabled:notice]: Cluster monitor: takeover of filer02 by filer01 enabled
Mon Jun  5 07:43:52 BNT [filer02: perf.archive.start:info]: Performance archiver started. Sampling 22 objects and 198 counters.
Mon Jun  5 07:43:54 BNT [filer02: netif.linkUp:info]: Ethernet e0a: Link up.
Mon Jun  5 07:43:54 BNT [filer02: netif.linkUp:info]: Ethernet e0b: Link up.
Mon Jun  5 07:43:55 BNT [filer02: pvif.switchLink:warning]: vif1: switching to e0a
Mon Jun  5 07:43:56 BNT [filer02: iscsi.service.startup:info]: iSCSI service startup
Mon Jun  5 07:43:57 BNT [filer02: cmds.sysconf.logErr:error]: sysconfig: Unless directed by NetApp Global Services the cf.takeover.on_panic option should be set to On. .
Mon Jun  5 07:43:57 BNT [filer02: raid.rg.recons.missing:notice]: RAID group /aggr1/plex0/rg0 is missing 2 disk(s).
Mon Jun  5 07:43:57 BNT [filer02: raid.rg.recons.cantStart:warning]: The reconstruction cannot start in RAID group /aggr1/plex0/rg0: No 520 bps disk of required size available in spare pool, targeting any  spare pool
Mon Jun  5 07:43:57 BNT [filer02: mgr.boot.disk_done:info]: NetApp Release 7.3.2 boot complete. Last disk update written at Sat Jun  3 09:00:08 BNT 2017
Mon Jun  5 07:43:57 BNT [filer02: monitor.raid.brokenDisk:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken.
Mon Jun  5 07:43:57 BNT [filer02: monitor.brokenDisk.notice:notice]: When two disks are broken in raid_dp volume,  the system shuts down automatically every 24  hours to encourage you to replace the disk.  If you reboot the system it will run for another  24 hours before shutting down. (The 24 hour timeout  may be increased by altering the "raid.timeout"  value using the "options" command.)
Mon Jun  5 07:43:58 BNT [filer02: shelf.config.spha:info]: System is using single path HA attached storage only.
Mon Jun  5 07:43:58 BNT [filer02: diskown.errorReadingOwnership:warning]: error 3 (disk failed) while reading ownership on disk 0a.24 (S/N N00SSSAL)
Mon Jun  5 07:43:58 BNT [filer02: diskown.errorReadingOwnership:warning]: error 3 (disk failed) while reading ownership on disk 0a.26 (S/N N00V0UWL)
Mon Jun  5 07:43:58 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] is still present in the system and should be removed.
Mon Jun  5 07:43:58 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] is still present in the system and should be removed.
Mon Jun  5 07:43:58 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] is still present in the system and should be removed.
Mon Jun  5 07:43:58 BNT [filer02: mgr.boot.reason_ok:notice]: System rebooted.
Mon Jun  5 07:44:00 BNT [filer02: raid.rg.spares.low:warning]: /aggr1/plex0/rg0
Mon Jun  5 07:44:00 BNT [filer02: iscsi.notice:notice]: ISCSI: New session from initiator iqn.1991-05.com.microsoft:pcb-722-ad1.pcbl722dom.local at IP addr 10.72.4.1
Mon Jun  5 07:44:12 BNT [filer02: monitor.globalStatus.nonCritical:warning]: Disk on adapter 0a, shelf 2, bay 8, failed. Disk on adapter 0a, shelf 2, bay 3, failed. Disk on adapter 0a, shelf 2, bay 1, failed.  There are not enough spare disks. /vol/vol1 is full (using or reserving 100% of space and 0% of inodes, using 92% of reserve).
Mon Jun  5 07:44:22 BNT [filer02: nbt.nbns.registrationComplete:info]: NBT: All CIFS name registrations have completed for the local server.
Mon Jun  5 07:44:28 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (CLUSTER ERROR: DISK/SHELF COUNT MISMATCH)
Mon Jun  5 07:44:37 BNT last message repeated 2 times
Mon Jun  5 07:44:37 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201705260814.0) was not posted for host (0), it was dropped from queue due to lack of space
Mon Jun  5 07:44:44 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Mon Jun  5 07:44:44 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201705261200.0) was not posted for host (0), it was dropped from queue due to lack of space
Mon Jun  5 07:44:51 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Mon Jun  5 07:44:51 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201705261600.0) was not posted for host (0), it was dropped from queue due to lack of space
Mon Jun  5 07:45:06 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Mon Jun  5 07:45:06 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201705262000.0) was not posted for host (0), it was dropped from queue due to lack of space
Mon Jun  5 07:49:37 BNT [filer02: cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 0a.33 (016F3AB5:B475F4EC:10505E83:02202755:00000000:00000000:00000000:00000000:00000000:00000000) has recently changed or the node (filer02) is missing the disk.
Mon Jun  5 07:49:37 BNT [filer02: cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 0a.35 (9115483F:9DD43BC4:27172773:99F7D8B4:00000000:00000000:00000000:00000000:00000000:00000000) has recently changed or the node (filer02) is missing the disk.
Mon Jun  5 07:49:37 BNT [filer02: cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 0a.40 (072AEB65:C662A9CF:E58B9917:9BFA787B:00000000:00000000:00000000:00000000:00000000:00000000) has recently changed or the node (filer02) is missing the disk.
Mon Jun  5 07:49:37 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Mon Jun  5 07:49:45 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Mon Jun  5 07:49:45 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201705270000.0) was not posted for host (0), it was dropped from queue due to lack of space
Mon Jun  5 07:54:20 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Mon Jun  5 07:58:25 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Mon Jun  5 08:00:00 BNT [filer02: kern.uptime.filer:info]:   8:00am up 15 mins, 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 2693 iSCSI ops
Mon Jun  5 08:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 24 hours.
Mon Jun  5 08:00:16 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Mon Jun  5 08:00:25 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Mon Jun  5 08:00:25 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201705270800.0) was not posted for host (0), it was dropped from queue due to lack of space
Mon Jun  5 08:04:26 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Mon Jun  5 08:07:59 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Mon Jun  5 08:12:15 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Mon Jun  5 08:16:23 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Mon Jun  5 08:16:23 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201705300743.0) for host (0) was not posted to NetApp. The system will drop the message
Mon Jun  5 08:50:55 BNT [filer02: cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 0a.33 (016F3AB5:B475F4EC:10505E83:02202755:00000000:00000000:00000000:00000000:00000000:00000000) has recently changed or the node (filer02) is missing the disk.
Mon Jun  5 08:50:55 BNT [filer02: cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 0a.35 (9115483F:9DD43BC4:27172773:99F7D8B4:00000000:00000000:00000000:00000000:00000000:00000000) has recently changed or the node (filer02) is missing the disk.
Mon Jun  5 08:50:55 BNT [filer02: cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 0a.40 (072AEB65:C662A9CF:E58B9917:9BFA787B:00000000:00000000:00000000:00000000:00000000:00000000) has recently changed or the node (filer02) is missing the disk.
Mon Jun  5 09:00:00 BNT [filer02: kern.uptime.filer:info]:   9:00am up  1:15 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 9542 iSCSI ops
Mon Jun  5 09:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 23 hours.
Mon Jun  5 10:00:00 BNT [filer02: kern.uptime.filer:info]:  10:00am up  2:15 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 75862 iSCSI ops
Mon Jun  5 10:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 22 hours.
Mon Jun  5 11:00:00 BNT [filer02: kern.uptime.filer:info]:  11:00am up  3:15 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 126442 iSCSI ops
Mon Jun  5 11:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 21 hours.
Mon Jun  5 12:00:00 BNT [filer02: kern.uptime.filer:info]:  12:00pm up  4:15 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 340844 iSCSI ops
Mon Jun  5 12:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 20 hours.
Mon Jun  5 12:00:15 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Mon Jun  5 12:00:39 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SHUTDOWN PENDING (degraded mode))
Mon Jun  5 12:04:14 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SHUTDOWN PENDING (degraded mode))
Mon Jun  5 12:08:16 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SHUTDOWN PENDING (degraded mode))
Mon Jun  5 12:12:21 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SHUTDOWN PENDING (degraded mode))
Mon Jun  5 12:16:30 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SHUTDOWN PENDING (degraded mode))
Mon Jun  5 12:35:31 BNT last message repeated 5 times
Mon Jun  5 12:39:52 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SHUTDOWN PENDING (degraded mode))
Mon Jun  5 12:54:40 BNT last message repeated 4 times
Mon Jun  5 12:54:40 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201705300743.1) for host (0) was not posted to NetApp. The system will drop the message
Mon Jun  5 13:00:00 BNT [filer02: kern.uptime.filer:info]:   1:00pm up  5:15 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 370691 iSCSI ops
Mon Jun  5 13:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 19 hours.
Mon Jun  5 14:00:00 BNT [filer02: kern.uptime.filer:info]:   2:00pm up  6:16 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 532257 iSCSI ops
Mon Jun  5 14:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 18 hours.
Mon Jun  5 15:00:00 BNT [filer02: kern.uptime.filer:info]:   3:00pm up  7:16 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 659587 iSCSI ops
Mon Jun  5 15:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 17 hours.
Mon Jun  5 16:00:00 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Mon Jun  5 16:00:00 BNT [filer02: kern.uptime.filer:info]:   4:00pm up  8:16 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 660995 iSCSI ops
Mon Jun  5 16:00:01 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 16 hours.
Mon Jun  5 16:28:51 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (REBOOT)
Mon Jun  5 16:32:36 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (REBOOT)
Mon Jun  5 16:56:22 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201705300744.0) for host (0) was not posted to NetApp. The system will drop the message
Mon Jun  5 17:00:00 BNT [filer02: kern.uptime.filer:info]:   5:00pm up  9:16 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 660995 iSCSI ops
Mon Jun  5 17:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 15 hours.
Mon Jun  5 18:00:00 BNT [filer02: kern.uptime.filer:info]:   6:00pm up 10:16 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 660995 iSCSI ops
Mon Jun  5 18:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 14 hours.
Mon Jun  5 19:00:00 BNT [filer02: kern.uptime.filer:info]:   7:00pm up 11:16 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 660995 iSCSI ops
Mon Jun  5 19:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 13 hours.
Mon Jun  5 20:00:00 BNT [filer02: kern.uptime.filer:info]:   8:00pm up 12:16 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 660995 iSCSI ops
Mon Jun  5 20:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 12 hours.
Mon Jun  5 20:00:01 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Mon Jun  5 20:04:05 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SPARES_LOW)
Mon Jun  5 20:28:17 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SPARES_LOW)
Mon Jun  5 20:57:01 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201705300744.1) for host (0) was not posted to NetApp. The system will drop the message
Mon Jun  5 21:00:00 BNT [filer02: kern.uptime.filer:info]:   9:00pm up 13:16 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 660995 iSCSI ops
Mon Jun  5 21:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 11 hours.
Mon Jun  5 22:00:00 BNT [filer02: kern.uptime.filer:info]:  10:00pm up 14:16 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 660995 iSCSI ops
Mon Jun  5 22:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 10 hours.
Mon Jun  5 23:00:00 BNT [filer02: kern.uptime.filer:info]:  11:00pm up 15:16 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 660995 iSCSI ops
Mon Jun  5 23:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 9 hours.
Tue Jun  6 00:00:00 BNT [filer02: kern.uptime.filer:info]:  12:00am up 16:16 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 660995 iSCSI ops
Tue Jun  6 00:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 8 hours.
Tue Jun  6 00:00:46 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of nightly.0 snapshot (Snapshot operation failed: No space left on device.)
Tue Jun  6 00:37:03 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (PERFORMANCE SNAPSHOT)
Tue Jun  6 00:56:50 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201705300744.2) for host (0) was not posted to NetApp. The system will drop the message
Tue Jun  6 01:00:00 BNT [filer02: kern.uptime.filer:info]:   1:00am up 17:16 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 660995 iSCSI ops
Tue Jun  6 01:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 7 hours.
Tue Jun  6 02:00:00 BNT [filer02: kern.uptime.filer:info]:   2:00am up 18:16 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 660995 iSCSI ops
Tue Jun  6 02:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 6 hours.
Tue Jun  6 03:00:00 BNT [filer02: kern.uptime.filer:info]:   3:00am up 19:16 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 660995 iSCSI ops
Tue Jun  6 03:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 5 hours.
Tue Jun  6 04:00:00 BNT [filer02: kern.uptime.filer:info]:   4:00am up 20:16 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 660995 iSCSI ops
Tue Jun  6 04:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 4 hours.
Tue Jun  6 05:00:00 BNT [filer02: kern.uptime.filer:info]:   5:00am up 21:16 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 660995 iSCSI ops
Tue Jun  6 05:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 3 hours.
Tue Jun  6 06:00:00 BNT [filer02: kern.uptime.filer:info]:   6:00am up 22:16 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 660995 iSCSI ops
Tue Jun  6 06:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 2 hours.
Tue Jun  6 07:00:00 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] is still present in the system and should be removed.
Tue Jun  6 07:00:00 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] is still present in the system and should be removed.
Tue Jun  6 07:00:00 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] is still present in the system and should be removed.
Tue Jun  6 07:36:07 BNT [filer02: fci.link.break:error]: Link break detected on Fibre Channel adapter 0a.
Tue Jun  6 07:36:13 BNT [filer02: fci.adapter.error:warning]: Fibre Channel adapter driver encountered error "Failed to get loop map" on adapter 0a.
Mon Jun  5 23:36:49 GMT [localhost: rc:notice]: The system was down for 20 seconds
Mon Jun  5 23:36:49 GMT [localhost: cf.fsm.takeoverOfPartnerEnabled:notice]: Cluster monitor: takeover of filer01 enabled
Tue Jun  6 07:36:50 BNT [filer02: cf.fsm.takeoverByPartnerEnabled:notice]: Cluster monitor: takeover of filer02 by filer01 enabled
Tue Jun  6 07:36:13 BNT [filer02: cf.nm.nicTransitionUp:info]: replayed event: Interconnect link 0 is UP
Tue Jun  6 07:36:20 BNT [filer02: fci.initialization.failed:error]: replayed event: Initialization failed on Fibre Channel adapter 0b.
Tue Jun  6 07:36:22 BNT [filer02: nvram.battery.turned.on:info]: replayed event: The NVRAM battery is turned ON. It is turned OFF during system shutdown.
Tue Jun  6 07:36:25 BNT [filer02: diskown.isEnabled:info]: replayed event: software ownership has been enabled for this system
Tue Jun  6 07:36:29 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.2 is a local HA mailbox disk.
Tue Jun  6 07:36:29 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.0 is a local HA mailbox disk.
Tue Jun  6 07:36:29 BNT [filer02: fmmb.instStat.change:info]: replayed event: normal mailbox instance on local side.
Tue Jun  6 07:36:29 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.7 is a partner HA mailbox disk.
Tue Jun  6 07:36:29 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.3 is a partner HA mailbox disk.
Tue Jun  6 07:36:29 BNT [filer02: fmmb.instStat.change:info]: replayed event: normal mailbox instance on partner side.
Tue Jun  6 07:36:30 BNT [filer02: cf.fm.partner:info]: replayed event: Cluster monitor: partner 'filer01'
Tue Jun  6 07:36:30 BNT [filer02: cf.fm.timeMasterStatus:info]: replayed event: Acting as cluster time slave
Tue Jun  6 07:36:30 BNT [filer02: raid.assim.disk.brokenPreAssim:error]: replayed event: Broken Disk 0a.40 Shelf ? Bay ? [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] detected prior to assimilation. It should be removed.
Tue Jun  6 07:36:30 BNT [filer02: raid.assim.disk.brokenPreAssim:error]: replayed event: Broken Disk 0a.33 Shelf ? Bay ? [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] detected prior to assimilation. It should be removed.
Tue Jun  6 07:36:30 BNT [filer02: raid.assim.disk.brokenPreAssim:error]: replayed event: Broken Disk 0a.35 Shelf ? Bay ? [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] detected prior to assimilation. It should be removed.
Tue Jun  6 07:36:31 BNT [filer02: coredump.spare.none:info]: replayed event: No sparecore disk was found.
Tue Jun  6 07:36:31 BNT [filer02: raid.vol.replay.nvram:info]: replayed event: Performing raid replay on volume(s)
Tue Jun  6 07:36:31 BNT [filer02: raid.cksum.replay.summary:info]: replayed event: Replayed 0 checksum blocks.
Tue Jun  6 07:36:31 BNT [filer02: raid.stripe.replay.summary:info]: replayed event: Replayed 0 stripes.
Tue Jun  6 07:36:34 BNT [filer02: ses.status.psWarning:warning]: replayed event: DS12-ESAS shelf 0 on channel 0c power warning for Power supply 1: non-critical status; DC undervoltage fault. This module is on the rear side of the shelf, at the left.
Tue Jun  6 07:36:34 BNT [filer02: ses.status.psWarning:warning]: replayed event: DS12-ESAS shelf 0 on channel 0c power warning for Power supply 2: non-critical status; DC undervoltage fault. This module is on the rear side of the shelf, at the right.
Tue Jun  6 07:36:43 BNT [filer02: cf.fm.launch:info]: replayed event: Launching cluster monitor
Tue Jun  6 07:36:43 BNT [filer02: cf.fm.partner:info]: replayed event: Cluster monitor: partner 'filer01'
Tue Jun  6 07:36:47 BNT [filer02: cf.fsm.takeoverOfPartnerDisabled:notice]: replayed event: Cluster monitor: takeover of filer01 disabled (partner booting)
Tue Jun  6 07:36:48 BNT [filer02: fcp.service.adapter:warning]: replayed event: FCP started but no adapter is present!
Tue Jun  6 07:36:49 BNT [filer02: kern.syslog.msg:notice]: replayed event: The system was down for 20 seconds
Tue Jun  6 07:36:49 BNT [filer02: cf.fsm.takeoverOfPartnerEnabled:notice]: replayed event: Cluster monitor: takeover of filer01 enabled
Tue Jun  6 07:36:50 BNT [filer02: cf.fsm.takeoverByPartnerEnabled:notice]: replayed event: Cluster monitor: takeover of filer02 by filer01 enabled
Tue Jun  6 07:36:50 BNT [filer02: disk.failmsg:error]: Disk 0a.40 (N00V0W8L): message received.
Tue Jun  6 07:36:50 BNT [filer02: disk.failmsg:error]: Disk 0a.33 (N00W75PL): message received.
Tue Jun  6 07:36:50 BNT [filer02: disk.failmsg:error]: Disk 0a.35 (N00WLAUL): message received.
Tue Jun  6 07:36:50 BNT [filer02: raid.disk.unload.done:info]: Unload of Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] has completed successfully
Tue Jun  6 07:36:50 BNT [filer02: raid.disk.unload.done:info]: Unload of Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] has completed successfully
Tue Jun  6 07:36:50 BNT [filer02: raid.disk.unload.done:info]: Unload of Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] has completed successfully
Tue Jun  6 07:36:50 BNT [filer02: raid.fdr.failed.ok:info]: Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] successfully deleted from spare pool
Tue Jun  6 07:36:50 BNT [filer02: raid.fdr.failed.ok:info]: Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] successfully deleted from spare pool
Tue Jun  6 07:36:50 BNT [filer02: raid.fdr.failed.ok:info]: Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] successfully deleted from spare pool
Tue Jun  6 07:36:51 BNT [filer02: dfu.firmwareUpToDate:info]: Firmware is up-to-date on all disk drives
Tue Jun  6 07:36:52 BNT [filer02: perf.archive.start:info]: Performance archiver started. Sampling 22 objects and 198 counters.
Tue Jun  6 07:36:54 BNT [filer02: netif.linkUp:info]: Ethernet e0a: Link up.
Tue Jun  6 07:36:54 BNT [filer02: netif.linkUp:info]: Ethernet e0b: Link up.
Tue Jun  6 07:36:55 BNT [filer02: pvif.switchLink:warning]: vif1: switching to e0a
Tue Jun  6 07:36:56 BNT [filer02: rc:info]: relog syslog Tue Jun  6 07:36:15 BNT [filer02: fci.adapter.error:warning]: Fibre Channel adapter driver encountered error "Failed t
Tue Jun  6 07:36:56 BNT [filer02: rc:info]: relog syslog Tue Jun  6 07:36:15 BNT [filer02: fci.adapter.reset:debug]: Resetting Fibre Channel adapter 0a.
Tue Jun  6 07:36:57 BNT [filer02: iscsi.service.startup:info]: iSCSI service startup
Tue Jun  6 07:36:57 BNT [filer02: cmds.sysconf.logErr:error]: sysconfig: Unless directed by NetApp Global Services the cf.takeover.on_panic option should be set to On. .
Tue Jun  6 07:36:57 BNT [filer02: raid.rg.recons.missing:notice]: RAID group /aggr1/plex0/rg0 is missing 2 disk(s).
Tue Jun  6 07:36:57 BNT [filer02: raid.rg.recons.cantStart:warning]: The reconstruction cannot start in RAID group /aggr1/plex0/rg0: No 520 bps disk of required size available in spare pool, targeting any  spare pool
Tue Jun  6 07:36:57 BNT [filer02: mgr.boot.disk_done:info]: NetApp Release 7.3.2 boot complete. Last disk update written at Tue Jun  6 07:36:04 BNT 2017
Tue Jun  6 07:36:57 BNT [filer02: monitor.raid.brokenDisk:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken.
Tue Jun  6 07:36:57 BNT [filer02: monitor.brokenDisk.notice:notice]: When two disks are broken in raid_dp volume,  the system shuts down automatically every 24  hours to encourage you to replace the disk.  If you reboot the system it will run for another  24 hours before shutting down. (The 24 hour timeout  may be increased by altering the "raid.timeout"  value using the "options" command.)
Tue Jun  6 07:36:58 BNT [filer02: diskown.errorReadingOwnership:warning]: error 3 (disk failed) while reading ownership on disk 0a.24 (S/N N00SSSAL)
Tue Jun  6 07:36:58 BNT [filer02: diskown.errorReadingOwnership:warning]: error 3 (disk failed) while reading ownership on disk 0a.26 (S/N N00V0UWL)
Tue Jun  6 07:36:58 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] is still present in the system and should be removed.
Tue Jun  6 07:36:58 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] is still present in the system and should be removed.
Tue Jun  6 07:36:58 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] is still present in the system and should be removed.
Tue Jun  6 07:36:58 BNT [filer02: shelf.config.spha:info]: System is using single path HA attached storage only.
Tue Jun  6 07:36:58 BNT [filer02: mgr.boot.reason_ok:notice]: System rebooted after power-on.
Tue Jun  6 07:37:00 BNT [filer02: raid.rg.spares.low:warning]: /aggr1/plex0/rg0
Tue Jun  6 07:37:12 BNT [filer02: monitor.globalStatus.nonCritical:warning]: Disk on adapter 0a, shelf 2, bay 8, failed. Disk on adapter 0a, shelf 2, bay 1, failed. Disk on adapter 0a, shelf 2, bay 3, failed.  There are not enough spare disks. /vol/vol1 is full (using or reserving 100% of space and 0% of inodes, using 92% of reserve).
Tue Jun  6 07:37:23 BNT [filer02: nbt.nbns.registrationComplete:info]: NBT: All CIFS name registrations have completed for the local server.
Tue Jun  6 07:37:27 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (CLUSTER ERROR: DISK/SHELF COUNT MISMATCH)
Tue Jun  6 07:37:35 BNT last message repeated 2 times
Tue Jun  6 07:37:35 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201705300749.0) was not posted for host (0), it was dropped from queue due to lack of space
Tue Jun  6 07:37:43 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Tue Jun  6 07:37:43 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201705300800.0) was not posted for host (0), it was dropped from queue due to lack of space
Tue Jun  6 07:37:50 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Tue Jun  6 07:37:50 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201705301200.0) was not posted for host (0), it was dropped from queue due to lack of space
Tue Jun  6 07:38:06 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Tue Jun  6 07:38:06 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201705301600.0) was not posted for host (0), it was dropped from queue due to lack of space
Tue Jun  6 07:42:20 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Tue Jun  6 07:46:32 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Tue Jun  6 07:50:48 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Tue Jun  6 07:55:10 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Tue Jun  6 07:55:59 BNT [filer02: iscsi.notice:notice]: ISCSI: New session from initiator iqn.1991-05.com.microsoft:pcb-722-ad1.pcbl722dom.local at IP addr 10.72.4.1
Tue Jun  6 07:59:15 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Tue Jun  6 08:00:00 BNT [filer02: kern.uptime.filer:info]:   8:00am up 22 mins, 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 10 iSCSI ops
Tue Jun  6 08:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 24 hours.
Tue Jun  6 08:00:16 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Tue Jun  6 08:00:23 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Tue Jun  6 08:00:23 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201705302000.0) was not posted for host (0), it was dropped from queue due to lack of space
Tue Jun  6 08:04:24 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Tue Jun  6 08:08:34 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Tue Jun  6 08:12:57 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Tue Jun  6 08:12:57 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201705310000.0) for host (0) was not posted to NetApp. The system will drop the message
Tue Jun  6 09:00:00 BNT [filer02: kern.uptime.filer:info]:   9:00am up  1:22 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 137 iSCSI ops
Tue Jun  6 09:00:01 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 23 hours.
Tue Jun  6 10:00:00 BNT [filer02: kern.uptime.filer:info]:  10:00am up  2:23 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 25324 iSCSI ops
Tue Jun  6 10:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 22 hours.
Tue Jun  6 11:00:00 BNT [filer02: kern.uptime.filer:info]:  11:00am up  3:23 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 54121 iSCSI ops
Tue Jun  6 11:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 21 hours.
Tue Jun  6 12:00:00 BNT [filer02: kern.uptime.filer:info]:  12:00pm up  4:23 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 80445 iSCSI ops
Tue Jun  6 12:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 20 hours.
Tue Jun  6 12:00:00 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Tue Jun  6 12:00:27 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Tue Jun  6 12:04:39 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Tue Jun  6 12:08:32 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Tue Jun  6 12:12:55 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Tue Jun  6 12:17:03 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Tue Jun  6 12:36:27 BNT last message repeated 5 times
Tue Jun  6 12:40:51 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Tue Jun  6 12:55:58 BNT last message repeated 4 times
Tue Jun  6 12:55:58 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201706010742.0) for host (0) was not posted to NetApp. The system will drop the message
Tue Jun  6 13:00:00 BNT [filer02: kern.uptime.filer:info]:   1:00pm up  5:23 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 184841 iSCSI ops
Tue Jun  6 13:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 19 hours.
Tue Jun  6 14:00:00 BNT [filer02: kern.uptime.filer:info]:   2:00pm up  6:23 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 248267 iSCSI ops
Tue Jun  6 14:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 18 hours.
Tue Jun  6 15:00:00 BNT [filer02: kern.uptime.filer:info]:   3:00pm up  7:23 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 252651 iSCSI ops
Tue Jun  6 15:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 17 hours.
Tue Jun  6 16:00:00 BNT [filer02: kern.uptime.filer:info]:   4:00pm up  8:23 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 255820 iSCSI ops
Tue Jun  6 16:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 16 hours.
Tue Jun  6 16:00:01 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Tue Jun  6 16:00:29 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SHUTDOWN PENDING (degraded mode))
Tue Jun  6 16:04:33 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SHUTDOWN PENDING (degraded mode))
Tue Jun  6 16:08:28 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SHUTDOWN PENDING (degraded mode))
Tue Jun  6 16:12:27 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SHUTDOWN PENDING (degraded mode))
Tue Jun  6 16:16:26 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SHUTDOWN PENDING (degraded mode))
Tue Jun  6 16:33:13 BNT last message repeated 4 times
Tue Jun  6 16:37:21 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SHUTDOWN PENDING (degraded mode))
Tue Jun  6 16:48:54 BNT last message repeated 3 times
Tue Jun  6 16:56:14 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201706010743.0) for host (0) was not posted to NetApp. The system will drop the message
Tue Jun  6 17:00:00 BNT [filer02: kern.uptime.filer:info]:   5:00pm up  9:23 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 257294 iSCSI ops
Tue Jun  6 17:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 15 hours.
Tue Jun  6 18:00:00 BNT [filer02: kern.uptime.filer:info]:   6:00pm up 10:23 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 257294 iSCSI ops
Tue Jun  6 18:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 14 hours.
Tue Jun  6 19:00:00 BNT [filer02: kern.uptime.filer:info]:   7:00pm up 11:23 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 257294 iSCSI ops
Tue Jun  6 19:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 13 hours.
Tue Jun  6 20:00:00 BNT [filer02: kern.uptime.filer:info]:   8:00pm up 12:23 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 257294 iSCSI ops
Tue Jun  6 20:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 12 hours.
Tue Jun  6 20:00:46 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Tue Jun  6 20:05:13 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (REBOOT)
Tue Jun  6 20:29:21 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (REBOOT)
Tue Jun  6 20:33:28 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (REBOOT)
Tue Jun  6 20:57:27 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (REBOOT)
Tue Jun  6 20:57:27 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201706010743.1) for host (0) was not posted to NetApp. The system will drop the message
Tue Jun  6 21:00:00 BNT [filer02: kern.uptime.filer:info]:   9:00pm up 13:23 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 257294 iSCSI ops
Tue Jun  6 21:00:01 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 11 hours.
Tue Jun  6 22:00:00 BNT [filer02: kern.uptime.filer:info]:  10:00pm up 14:23 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 257294 iSCSI ops
Tue Jun  6 22:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 10 hours.
Tue Jun  6 23:00:00 BNT [filer02: kern.uptime.filer:info]:  11:00pm up 15:23 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 257294 iSCSI ops
Tue Jun  6 23:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 9 hours.
Wed Jun  7 00:00:00 BNT [filer02: kern.uptime.filer:info]:  12:00am up 16:23 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 257294 iSCSI ops
Wed Jun  7 00:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 8 hours.
Wed Jun  7 00:00:31 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of nightly.0 snapshot (Snapshot operation failed: No space left on device.)
Wed Jun  7 00:28:56 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SPARES_LOW)
Wed Jun  7 00:44:55 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SPARES_LOW)
Wed Jun  7 00:56:07 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201706010743.2) for host (0) was not posted to NetApp. The system will drop the message
Wed Jun  7 01:00:00 BNT [filer02: kern.uptime.filer:info]:   1:00am up 17:23 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 257294 iSCSI ops
Wed Jun  7 01:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 7 hours.
Wed Jun  7 02:00:00 BNT [filer02: kern.uptime.filer:info]:   2:00am up 18:23 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 257294 iSCSI ops
Wed Jun  7 02:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 6 hours.
Wed Jun  7 03:00:00 BNT [filer02: kern.uptime.filer:info]:   3:00am up 19:23 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 257294 iSCSI ops
Wed Jun  7 03:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 5 hours.
Wed Jun  7 04:00:00 BNT [filer02: kern.uptime.filer:info]:   4:00am up 20:23 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 257294 iSCSI ops
Wed Jun  7 04:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 4 hours.
Wed Jun  7 05:00:00 BNT [filer02: kern.uptime.filer:info]:   5:00am up 21:23 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 257294 iSCSI ops
Wed Jun  7 05:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 3 hours.
Wed Jun  7 06:00:00 BNT [filer02: kern.uptime.filer:info]:   6:00am up 22:23 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 257294 iSCSI ops
Wed Jun  7 06:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 2 hours.
Wed Jun  7 07:00:00 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] is still present in the system and should be removed.
Wed Jun  7 07:00:00 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] is still present in the system and should be removed.
Wed Jun  7 07:00:00 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] is still present in the system and should be removed.
Wed Jun  7 07:38:13 BNT [filer02: ses.status.psWarning:warning]: DS14-Mk2-AT shelf 2 on channel 0a power warning for Power supply 2: non-critical status; DC undervoltage fault. This module is on the rear side of the shelf, at the right.
Wed Jun  7 07:38:13 BNT [filer02: ses.status.psWarning:warning]: DS14-Mk2-AT shelf 1 on channel 0a power warning for Power supply 2: non-critical status; DC undervoltage fault. This module is on the rear side of the shelf, at the right.
Wed Jun  7 07:38:17 BNT [filer02: fci.link.break:error]: Link break detected on Fibre Channel adapter 0a.
Wed Jun  7 07:38:22 BNT [filer02: fci.adapter.error:warning]: Fibre Channel adapter driver encountered error "Failed to get loop map" on adapter 0a.
Tue Jun  6 23:38:58 GMT [localhost: rc:notice]: The system was down for 19 seconds
Tue Jun  6 23:38:59 GMT [localhost: cf.fsm.takeoverByPartnerEnabled:notice]: Cluster monitor: takeover of  by filer01 enabled
Wed Jun  7 07:38:23 BNT [filer02: cf.nm.nicTransitionUp:info]: replayed event: Interconnect link 0 is UP
Wed Jun  7 07:38:30 BNT [filer02: fci.initialization.failed:error]: replayed event: Initialization failed on Fibre Channel adapter 0b.
Wed Jun  7 07:38:32 BNT [filer02: nvram.battery.turned.on:info]: replayed event: The NVRAM battery is turned ON. It is turned OFF during system shutdown.
Wed Jun  7 07:38:35 BNT [filer02: diskown.isEnabled:info]: replayed event: software ownership has been enabled for this system
Wed Jun  7 07:38:39 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.2 is a local HA mailbox disk.
Wed Jun  7 07:38:39 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.0 is a local HA mailbox disk.
Wed Jun  7 07:38:39 BNT [filer02: fmmb.instStat.change:info]: replayed event: normal mailbox instance on local side.
Wed Jun  7 07:38:39 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.7 is a partner HA mailbox disk.
Wed Jun  7 07:38:39 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.3 is a partner HA mailbox disk.
Wed Jun  7 07:38:39 BNT [filer02: fmmb.instStat.change:info]: replayed event: normal mailbox instance on partner side.
Wed Jun  7 07:38:40 BNT [filer02: cf.fm.partner:info]: replayed event: Cluster monitor: partner 'filer01'
Wed Jun  7 07:38:40 BNT [filer02: cf.fm.timeMasterStatus:info]: replayed event: Acting as cluster time slave
Wed Jun  7 07:38:40 BNT [filer02: raid.assim.disk.brokenPreAssim:error]: replayed event: Broken Disk 0a.35 Shelf ? Bay ? [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] detected prior to assimilation. It should be removed.
Wed Jun  7 07:38:40 BNT [filer02: raid.assim.disk.brokenPreAssim:error]: replayed event: Broken Disk 0a.33 Shelf ? Bay ? [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] detected prior to assimilation. It should be removed.
Wed Jun  7 07:38:40 BNT [filer02: raid.assim.disk.brokenPreAssim:error]: replayed event: Broken Disk 0a.40 Shelf ? Bay ? [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] detected prior to assimilation. It should be removed.
Wed Jun  7 07:38:41 BNT [filer02: coredump.spare.none:info]: replayed event: No sparecore disk was found.
Wed Jun  7 07:38:41 BNT [filer02: raid.vol.replay.nvram:info]: replayed event: Performing raid replay on volume(s)
Wed Jun  7 07:38:41 BNT [filer02: raid.cksum.replay.summary:info]: replayed event: Replayed 0 checksum blocks.
Wed Jun  7 07:38:41 BNT [filer02: raid.stripe.replay.summary:info]: replayed event: Replayed 0 stripes.
Wed Jun  7 07:38:44 BNT [filer02: ses.status.psWarning:warning]: replayed event: DS12-ESAS shelf 0 on channel 0c power warning for Power supply 1: non-critical status; DC undervoltage fault. This module is on the rear side of the shelf, at the left.
Wed Jun  7 07:38:44 BNT [filer02: ses.status.psWarning:warning]: replayed event: DS12-ESAS shelf 0 on channel 0c power warning for Power supply 2: non-critical status; DC undervoltage fault. This module is on the rear side of the shelf, at the right.
Wed Jun  7 07:38:53 BNT [filer02: cf.fm.launch:info]: replayed event: Launching cluster monitor
Wed Jun  7 07:38:53 BNT [filer02: cf.fm.partner:info]: replayed event: Cluster monitor: partner 'filer01'
Wed Jun  7 07:38:57 BNT [filer02: cf.fsm.takeoverOfPartnerDisabled:notice]: replayed event: Cluster monitor: takeover of filer01 disabled (partner booting)
Wed Jun  7 07:38:58 BNT [filer02: fcp.service.adapter:warning]: replayed event: FCP started but no adapter is present!
Wed Jun  7 07:38:58 BNT [filer02: cf.fsm.takeoverOfPartnerEnabled:notice]: replayed event: Cluster monitor: takeover of filer01 enabled
Wed Jun  7 07:38:58 BNT [filer02: cf.fsm.takeoverByPartnerDisabled:notice]: replayed event: Cluster monitor: takeover of  by filer01 disabled (partner booting)
Wed Jun  7 07:38:58 BNT [filer02: kern.syslog.msg:notice]: replayed event: The system was down for 19 seconds
Wed Jun  7 07:38:59 BNT [filer02: cf.fsm.takeoverByPartnerEnabled:notice]: replayed event: Cluster monitor: takeover of  by filer01 enabled
Wed Jun  7 07:39:00 BNT [filer02: disk.failmsg:error]: Disk 0a.35 (N00WLAUL): message received.
Wed Jun  7 07:39:00 BNT [filer02: disk.failmsg:error]: Disk 0a.33 (N00W75PL): message received.
Wed Jun  7 07:39:00 BNT [filer02: disk.failmsg:error]: Disk 0a.40 (N00V0W8L): message received.
Wed Jun  7 07:39:00 BNT [filer02: raid.disk.unload.done:info]: Unload of Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] has completed successfully
Wed Jun  7 07:39:00 BNT [filer02: raid.disk.unload.done:info]: Unload of Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] has completed successfully
Wed Jun  7 07:39:00 BNT [filer02: raid.disk.unload.done:info]: Unload of Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] has completed successfully
Wed Jun  7 07:39:00 BNT [filer02: raid.fdr.failed.ok:info]: Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] successfully deleted from spare pool
Wed Jun  7 07:39:00 BNT [filer02: raid.fdr.failed.ok:info]: Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] successfully deleted from spare pool
Wed Jun  7 07:39:00 BNT [filer02: raid.fdr.failed.ok:info]: Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] successfully deleted from spare pool
Wed Jun  7 07:39:00 BNT [filer02: dfu.firmwareUpToDate:info]: Firmware is up-to-date on all disk drives
Wed Jun  7 07:39:02 BNT [filer02: perf.archive.start:info]: Performance archiver started. Sampling 22 objects and 198 counters.
Wed Jun  7 07:39:04 BNT [filer02: netif.linkUp:info]: Ethernet e0b: Link up.
Wed Jun  7 07:39:04 BNT [filer02: netif.linkUp:info]: Ethernet e0a: Link up.
Wed Jun  7 07:39:05 BNT [filer02: pvif.switchLink:warning]: vif1: switching to e0a
Wed Jun  7 07:39:06 BNT [filer02: rc:info]: relog syslog Wed Jun  7 07:38:25 BNT [filer02: fci.adapter.error:warning]: Fibre Channel adapter driver encountered error "Failed t
Wed Jun  7 07:39:06 BNT [filer02: rc:info]: relog syslog Wed Jun  7 07:38:25 BNT [filer02: fci.adapter.reset:debug]: Resetting Fibre Channel adapter 0a.
Wed Jun  7 07:39:06 BNT [filer02: iscsi.service.startup:info]: iSCSI service startup
Wed Jun  7 07:39:07 BNT [filer02: cmds.sysconf.logErr:error]: sysconfig: Unless directed by NetApp Global Services the cf.takeover.on_panic option should be set to On. .
Wed Jun  7 07:39:07 BNT [filer02: raid.rg.recons.missing:notice]: RAID group /aggr1/plex0/rg0 is missing 2 disk(s).
Wed Jun  7 07:39:07 BNT [filer02: raid.rg.recons.cantStart:warning]: The reconstruction cannot start in RAID group /aggr1/plex0/rg0: No 520 bps disk of required size available in spare pool, targeting any  spare pool
Wed Jun  7 07:39:07 BNT [filer02: mgr.boot.disk_done:info]: NetApp Release 7.3.2 boot complete. Last disk update written at Wed Jun  7 07:38:12 BNT 2017
Wed Jun  7 07:39:07 BNT [filer02: monitor.raid.brokenDisk:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken.
Wed Jun  7 07:39:07 BNT [filer02: monitor.brokenDisk.notice:notice]: When two disks are broken in raid_dp volume,  the system shuts down automatically every 24  hours to encourage you to replace the disk.  If you reboot the system it will run for another  24 hours before shutting down. (The 24 hour timeout  may be increased by altering the "raid.timeout"  value using the "options" command.)
Wed Jun  7 07:39:07 BNT [filer02: diskown.errorReadingOwnership:warning]: error 3 (disk failed) while reading ownership on disk 0a.24 (S/N N00SSSAL)
Wed Jun  7 07:39:07 BNT [filer02: diskown.errorReadingOwnership:warning]: error 3 (disk failed) while reading ownership on disk 0a.26 (S/N N00V0UWL)
Wed Jun  7 07:39:07 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] is still present in the system and should be removed.
Wed Jun  7 07:39:07 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] is still present in the system and should be removed.
Wed Jun  7 07:39:07 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] is still present in the system and should be removed.
Wed Jun  7 07:39:07 BNT [filer02: shelf.config.spha:info]: System is using single path HA attached storage only.
Wed Jun  7 07:39:08 BNT [filer02: mgr.boot.reason_ok:notice]: System rebooted after power-on.
Wed Jun  7 07:39:30 BNT [filer02: raid.rg.spares.low:warning]: /aggr1/plex0/rg0
Wed Jun  7 07:39:33 BNT [filer02: nbt.nbns.registrationComplete:info]: NBT: All CIFS name registrations have completed for the local server.
Wed Jun  7 07:39:38 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (PERFORMANCE SNAPSHOT)
Wed Jun  7 07:39:46 BNT last message repeated 2 times
Wed Jun  7 07:39:46 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201706010743.3) was not posted for host (0), it was dropped from queue due to lack of space
Wed Jun  7 07:39:53 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (CLUSTER ERROR: DISK/SHELF COUNT MISMATCH)
Wed Jun  7 07:39:53 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201706010748.0) was not posted for host (0), it was dropped from queue due to lack of space
Wed Jun  7 07:40:01 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Wed Jun  7 07:40:01 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201706010800.0) was not posted for host (0), it was dropped from queue due to lack of space
Wed Jun  7 07:40:06 BNT [filer02: monitor.globalStatus.nonCritical:warning]: Disk on adapter 0a, shelf 2, bay 3, failed. Disk on adapter 0a, shelf 2, bay 1, failed. Disk on adapter 0a, shelf 2, bay 8, failed.  There are not enough spare disks. /vol/vol1 is full (using or reserving 100% of space and 0% of inodes, using 92% of reserve).
Wed Jun  7 07:40:16 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Wed Jun  7 07:40:16 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201706011200.0) was not posted for host (0), it was dropped from queue due to lack of space
Wed Jun  7 07:44:55 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Wed Jun  7 07:48:56 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Wed Jun  7 07:53:03 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Wed Jun  7 07:57:07 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Wed Jun  7 08:00:00 BNT [filer02: kern.uptime.filer:info]:   8:00am up 20 mins, 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 0 iSCSI ops
Wed Jun  7 08:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 24 hours.
Wed Jun  7 08:00:16 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Wed Jun  7 08:00:24 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Wed Jun  7 08:00:24 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201706011600.0) was not posted for host (0), it was dropped from queue due to lack of space
Wed Jun  7 08:04:06 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Wed Jun  7 08:04:12 BNT [filer02: iscsi.notice:notice]: ISCSI: New session from initiator iqn.1991-05.com.microsoft:pcb-722-ad1.pcbl722dom.local at IP addr 10.72.4.1
Wed Jun  7 08:08:34 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Wed Jun  7 08:12:22 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Wed Jun  7 08:16:40 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Wed Jun  7 08:16:40 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201706012000.0) for host (0) was not posted to NetApp. The system will drop the message
Wed Jun  7 09:00:00 BNT [filer02: kern.uptime.filer:info]:   9:00am up  1:20 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 4530 iSCSI ops
Wed Jun  7 09:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 23 hours.
Wed Jun  7 10:00:00 BNT [filer02: kern.uptime.filer:info]:  10:00am up  2:20 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 31171 iSCSI ops
Wed Jun  7 10:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 22 hours.
Wed Jun  7 11:00:00 BNT [filer02: kern.uptime.filer:info]:  11:00am up  3:20 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 175875 iSCSI ops
Wed Jun  7 11:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 21 hours.
Wed Jun  7 12:00:00 BNT [filer02: kern.uptime.filer:info]:  12:00pm up  4:20 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 312540 iSCSI ops
Wed Jun  7 12:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 20 hours.
Wed Jun  7 12:00:16 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Wed Jun  7 12:00:38 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Wed Jun  7 12:04:43 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Wed Jun  7 12:08:39 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Wed Jun  7 12:12:14 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Wed Jun  7 12:15:50 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Wed Jun  7 12:35:32 BNT last message repeated 5 times
Wed Jun  7 12:39:27 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Wed Jun  7 12:54:25 BNT last message repeated 4 times
Wed Jun  7 12:54:25 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201706020000.0) for host (0) was not posted to NetApp. The system will drop the message
Wed Jun  7 13:00:00 BNT [filer02: kern.uptime.filer:info]:   1:00pm up  5:20 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 729790 iSCSI ops
Wed Jun  7 13:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 19 hours.
Wed Jun  7 14:00:00 BNT [filer02: kern.uptime.filer:info]:   2:00pm up  6:20 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 1217911 iSCSI ops
Wed Jun  7 14:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 18 hours.
Wed Jun  7 15:00:00 BNT [filer02: kern.uptime.filer:info]:   3:00pm up  7:21 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 1275673 iSCSI ops
Wed Jun  7 15:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 17 hours.
Wed Jun  7 16:00:00 BNT [filer02: kern.uptime.filer:info]:   4:00pm up  8:21 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 1287071 iSCSI ops
Wed Jun  7 16:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 16 hours.
Wed Jun  7 16:00:01 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Wed Jun  7 16:00:28 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Wed Jun  7 16:04:30 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Wed Jun  7 16:08:30 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Wed Jun  7 16:12:01 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Wed Jun  7 16:15:42 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Wed Jun  7 16:35:12 BNT last message repeated 5 times
Wed Jun  7 16:39:03 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Wed Jun  7 16:47:20 BNT last message repeated 2 times
Wed Jun  7 16:55:55 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201706020808.0) for host (0) was not posted to NetApp. The system will drop the message
Wed Jun  7 17:00:00 BNT [filer02: kern.uptime.filer:info]:   5:00pm up  9:21 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 1287108 iSCSI ops
Wed Jun  7 17:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 15 hours.
Wed Jun  7 18:00:00 BNT [filer02: kern.uptime.filer:info]:   6:00pm up 10:21 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 1287108 iSCSI ops
Wed Jun  7 18:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 14 hours.
Wed Jun  7 19:00:00 BNT [filer02: kern.uptime.filer:info]:   7:00pm up 11:21 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 1287108 iSCSI ops
Wed Jun  7 19:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 13 hours.
Wed Jun  7 20:00:00 BNT [filer02: kern.uptime.filer:info]:   8:00pm up 12:21 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 1287108 iSCSI ops
Wed Jun  7 20:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 12 hours.
Wed Jun  7 20:00:00 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Wed Jun  7 20:20:12 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SHUTDOWN PENDING (degraded mode))
Wed Jun  7 20:56:12 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201706020808.1) for host (0) was not posted to NetApp. The system will drop the message
Wed Jun  7 21:00:00 BNT [filer02: kern.uptime.filer:info]:   9:00pm up 13:21 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 1287108 iSCSI ops
Wed Jun  7 21:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 11 hours.
Wed Jun  7 22:00:00 BNT [filer02: kern.uptime.filer:info]:  10:00pm up 14:21 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 1287108 iSCSI ops
Wed Jun  7 22:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 10 hours.
Wed Jun  7 23:00:00 BNT [filer02: kern.uptime.filer:info]:  11:00pm up 15:21 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 1287108 iSCSI ops
Wed Jun  7 23:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 9 hours.
Thu Jun  8 00:00:00 BNT [filer02: kern.uptime.filer:info]:  12:00am up 16:21 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 1287108 iSCSI ops
Thu Jun  8 00:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 8 hours.
Thu Jun  8 00:00:45 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of nightly.0 snapshot (Snapshot operation failed: No space left on device.)
Thu Jun  8 00:05:28 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (REBOOT)
Thu Jun  8 00:36:00 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (REBOOT)
Thu Jun  8 00:55:26 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201706020808.2) for host (0) was not posted to NetApp. The system will drop the message
Thu Jun  8 01:00:00 BNT [filer02: kern.uptime.filer:info]:   1:00am up 17:21 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 1287108 iSCSI ops
Thu Jun  8 01:00:01 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 7 hours.
Thu Jun  8 02:00:00 BNT [filer02: kern.uptime.filer:info]:   2:00am up 18:21 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 1287108 iSCSI ops
Thu Jun  8 02:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 6 hours.
Thu Jun  8 03:00:00 BNT [filer02: kern.uptime.filer:info]:   3:00am up 19:21 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 1287108 iSCSI ops
Thu Jun  8 03:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 5 hours.
Thu Jun  8 04:00:00 BNT [filer02: kern.uptime.filer:info]:   4:00am up 20:21 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 1287108 iSCSI ops
Thu Jun  8 04:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 4 hours.
Thu Jun  8 05:00:00 BNT [filer02: kern.uptime.filer:info]:   5:00am up 21:21 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 1287108 iSCSI ops
Thu Jun  8 05:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 3 hours.
Thu Jun  8 06:00:00 BNT [filer02: kern.uptime.filer:info]:   6:00am up 22:21 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 1287108 iSCSI ops
Thu Jun  8 06:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 2 hours.
Thu Jun  8 07:00:00 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] is still present in the system and should be removed.
Thu Jun  8 07:00:00 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] is still present in the system and should be removed.
Thu Jun  8 07:00:00 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] is still present in the system and should be removed.
Thu Jun  8 07:45:54 BNT [filer02: ses.status.psWarning:warning]: DS14-Mk2-AT shelf 2 on channel 0a power warning for Power supply 2: non-critical status; DC undervoltage fault. This module is on the rear side of the shelf, at the right.
Thu Jun  8 07:45:58 BNT [filer02: fci.link.break:error]: Link break detected on Fibre Channel adapter 0a.
Wed Jun  7 23:46:36 GMT [localhost: rc:notice]: The system was down for 18 seconds
Thu Jun  8 07:46:00 BNT [filer02: cf.nm.nicTransitionUp:info]: replayed event: Interconnect link 0 is UP
Thu Jun  8 07:46:07 BNT [filer02: fci.initialization.failed:error]: replayed event: Initialization failed on Fibre Channel adapter 0b.
Thu Jun  8 07:46:09 BNT [filer02: nvram.battery.turned.on:info]: replayed event: The NVRAM battery is turned ON. It is turned OFF during system shutdown.
Thu Jun  8 07:46:37 BNT [filer02: cf.fsm.takeoverByPartnerEnabled:notice]: Cluster monitor: takeover of filer02 by filer01 enabled
Thu Jun  8 07:46:12 BNT [filer02: diskown.isEnabled:info]: replayed event: software ownership has been enabled for this system
Thu Jun  8 07:46:16 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.2 is a local HA mailbox disk.
Thu Jun  8 07:46:16 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.0 is a local HA mailbox disk.
Thu Jun  8 07:46:16 BNT [filer02: fmmb.instStat.change:info]: replayed event: normal mailbox instance on local side.
Thu Jun  8 07:46:16 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.7 is a partner HA mailbox disk.
Thu Jun  8 07:46:16 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.3 is a partner HA mailbox disk.
Thu Jun  8 07:46:16 BNT [filer02: fmmb.instStat.change:info]: replayed event: normal mailbox instance on partner side.
Thu Jun  8 07:46:17 BNT [filer02: cf.fm.partner:info]: replayed event: Cluster monitor: partner 'filer01'
Thu Jun  8 07:46:17 BNT [filer02: cf.fm.timeMasterStatus:info]: replayed event: Acting as cluster time slave
Thu Jun  8 07:46:17 BNT [filer02: raid.assim.disk.brokenPreAssim:error]: replayed event: Broken Disk 0a.40 Shelf ? Bay ? [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] detected prior to assimilation. It should be removed.
Thu Jun  8 07:46:17 BNT [filer02: raid.assim.disk.brokenPreAssim:error]: replayed event: Broken Disk 0a.33 Shelf ? Bay ? [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] detected prior to assimilation. It should be removed.
Thu Jun  8 07:46:17 BNT [filer02: raid.assim.disk.brokenPreAssim:error]: replayed event: Broken Disk 0a.35 Shelf ? Bay ? [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] detected prior to assimilation. It should be removed.
Thu Jun  8 07:46:18 BNT [filer02: coredump.spare.none:info]: replayed event: No sparecore disk was found.
Thu Jun  8 07:46:18 BNT [filer02: raid.vol.replay.nvram:info]: replayed event: Performing raid replay on volume(s)
Thu Jun  8 07:46:18 BNT [filer02: raid.cksum.replay.summary:info]: replayed event: Replayed 0 checksum blocks.
Thu Jun  8 07:46:18 BNT [filer02: raid.stripe.replay.summary:info]: replayed event: Replayed 0 stripes.
Thu Jun  8 07:46:30 BNT [filer02: cf.fm.launch:info]: replayed event: Launching cluster monitor
Thu Jun  8 07:46:30 BNT [filer02: cf.fm.partner:info]: replayed event: Cluster monitor: partner 'filer01'
Thu Jun  8 07:46:34 BNT [filer02: cf.fsm.takeoverOfPartnerDisabled:notice]: replayed event: Cluster monitor: takeover of filer01 disabled (partner booting)
Thu Jun  8 07:46:35 BNT [filer02: cf.fsm.takeoverOfPartnerEnabled:notice]: replayed event: Cluster monitor: takeover of filer01 enabled
Thu Jun  8 07:46:35 BNT [filer02: cf.fsm.takeoverByPartnerDisabled:notice]: replayed event: Cluster monitor: takeover of  by filer01 disabled (partner booting)
Thu Jun  8 07:46:35 BNT [filer02: fcp.service.adapter:warning]: replayed event: FCP started but no adapter is present!
Thu Jun  8 07:46:36 BNT [filer02: kern.syslog.msg:notice]: replayed event: The system was down for 18 seconds
Thu Jun  8 07:46:37 BNT [filer02: disk.failmsg:error]: Disk 0a.40 (N00V0W8L): message received.
Thu Jun  8 07:46:37 BNT [filer02: disk.failmsg:error]: Disk 0a.33 (N00W75PL): message received.
Thu Jun  8 07:46:37 BNT [filer02: disk.failmsg:error]: Disk 0a.35 (N00WLAUL): message received.
Thu Jun  8 07:46:37 BNT [filer02: raid.disk.unload.done:info]: Unload of Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] has completed successfully
Thu Jun  8 07:46:37 BNT [filer02: raid.disk.unload.done:info]: Unload of Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] has completed successfully
Thu Jun  8 07:46:37 BNT [filer02: raid.disk.unload.done:info]: Unload of Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] has completed successfully
Thu Jun  8 07:46:37 BNT [filer02: raid.fdr.failed.ok:info]: Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] successfully deleted from spare pool
Thu Jun  8 07:46:37 BNT [filer02: raid.fdr.failed.ok:info]: Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] successfully deleted from spare pool
Thu Jun  8 07:46:37 BNT [filer02: raid.fdr.failed.ok:info]: Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] successfully deleted from spare pool
Thu Jun  8 07:46:38 BNT [filer02: dfu.firmwareUpToDate:info]: Firmware is up-to-date on all disk drives
Thu Jun  8 07:46:39 BNT [filer02: perf.archive.start:info]: Performance archiver started. Sampling 22 objects and 198 counters.
Thu Jun  8 07:46:41 BNT [filer02: netif.linkUp:info]: Ethernet e0b: Link up.
Thu Jun  8 07:46:41 BNT [filer02: netif.linkUp:info]: Ethernet e0a: Link up.
Thu Jun  8 07:46:42 BNT [filer02: pvif.switchLink:warning]: vif1: switching to e0a
Thu Jun  8 07:46:43 BNT [filer02: rc:info]: relog syslog Thu Jun  8 07:45:54 BNT [filer02: ses.status.psWarning:warning]: DS14-Mk2-AT shelf 2 on channel 0a power warning for P
Thu Jun  8 07:46:43 BNT [filer02: rc:info]: relog syslog Thu Jun  8 07:45:58 BNT [filer02: fci.link.break:error]: Link break detected on Fibre Channel adapter 0a.
Thu Jun  8 07:46:43 BNT [filer02: iscsi.service.startup:info]: iSCSI service startup
Thu Jun  8 07:46:43 BNT [filer02: netif.linkDown:info]: Ethernet e0a: Link down, check cable.
Thu Jun  8 07:46:44 BNT [filer02: cmds.sysconf.logErr:error]: sysconfig: Unless directed by NetApp Global Services the cf.takeover.on_panic option should be set to On. .
Thu Jun  8 07:46:44 BNT [filer02: raid.rg.recons.missing:notice]: RAID group /aggr1/plex0/rg0 is missing 2 disk(s).
Thu Jun  8 07:46:44 BNT [filer02: raid.rg.recons.cantStart:warning]: The reconstruction cannot start in RAID group /aggr1/plex0/rg0: No 520 bps disk of required size available in spare pool, targeting any  spare pool
Thu Jun  8 07:46:44 BNT [filer02: mgr.boot.disk_done:info]: NetApp Release 7.3.2 boot complete. Last disk update written at Thu Jun  8 07:45:52 BNT 2017
Thu Jun  8 07:46:44 BNT [filer02: monitor.raid.brokenDisk:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken.
Thu Jun  8 07:46:44 BNT [filer02: monitor.brokenDisk.notice:notice]: When two disks are broken in raid_dp volume,  the system shuts down automatically every 24  hours to encourage you to replace the disk.  If you reboot the system it will run for another  24 hours before shutting down. (The 24 hour timeout  may be increased by altering the "raid.timeout"  value using the "options" command.)
Thu Jun  8 07:46:45 BNT [filer02: shelf.config.spha:info]: System is using single path HA attached storage only.
Thu Jun  8 07:46:45 BNT [filer02: diskown.errorReadingOwnership:warning]: error 3 (disk failed) while reading ownership on disk 0a.24 (S/N N00SSSAL)
Thu Jun  8 07:46:45 BNT [filer02: diskown.errorReadingOwnership:warning]: error 3 (disk failed) while reading ownership on disk 0a.26 (S/N N00V0UWL)
Thu Jun  8 07:46:45 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] is still present in the system and should be removed.
Thu Jun  8 07:46:45 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] is still present in the system and should be removed.
Thu Jun  8 07:46:45 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] is still present in the system and should be removed.
Thu Jun  8 07:46:45 BNT [filer02: mgr.boot.reason_ok:notice]: System rebooted after power-on.
Thu Jun  8 07:46:46 BNT [filer02: netif.linkUp:info]: Ethernet e0a: Link up.
Thu Jun  8 07:47:01 BNT [filer02: raid.rg.spares.low:warning]: /aggr1/plex0/rg0
Thu Jun  8 07:47:01 BNT [filer02: monitor.globalStatus.nonCritical:warning]: Disk on adapter 0a, shelf 2, bay 8, failed. Disk on adapter 0a, shelf 2, bay 1, failed. Disk on adapter 0a, shelf 2, bay 3, failed.  There are not enough spare disks. /vol/vol1 is full (using or reserving 100% of space and 0% of inodes, using 92% of reserve).
Thu Jun  8 07:47:10 BNT [filer02: nbt.nbns.registrationComplete:info]: NBT: All CIFS name registrations have completed for the local server.
Thu Jun  8 07:47:15 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SPARES_LOW)
Thu Jun  8 07:47:23 BNT last message repeated 2 times
Thu Jun  8 07:47:23 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201706020809.0) was not posted for host (0), it was dropped from queue due to lack of space
Thu Jun  8 07:47:31 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (PERFORMANCE SNAPSHOT)
Thu Jun  8 07:47:31 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201706020809.1) was not posted for host (0), it was dropped from queue due to lack of space
Thu Jun  8 07:47:38 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (CLUSTER ERROR: DISK/SHELF COUNT MISMATCH)
Thu Jun  8 07:47:38 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201706020813.0) was not posted for host (0), it was dropped from queue due to lack of space
Thu Jun  8 07:47:53 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu Jun  8 07:47:54 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201706021200.0) was not posted for host (0), it was dropped from queue due to lack of space
Thu Jun  8 07:51:22 BNT [filer02: iscsi.notice:notice]: ISCSI: New session from initiator iqn.1991-05.com.microsoft:pcb-722-ad1.pcbl722dom.local at IP addr 10.72.4.1
Thu Jun  8 07:51:54 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu Jun  8 07:56:47 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu Jun  8 08:00:00 BNT [filer02: kern.uptime.filer:info]:   8:00am up 13 mins, 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 10 iSCSI ops
Thu Jun  8 08:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 24 hours.
Thu Jun  8 08:00:01 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Thu Jun  8 08:00:10 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu Jun  8 08:00:11 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201706021600.0) was not posted for host (0), it was dropped from queue due to lack of space
Thu Jun  8 08:04:24 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu Jun  8 08:07:56 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu Jun  8 08:12:09 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu Jun  8 08:15:56 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu Jun  8 08:19:37 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu Jun  8 08:24:05 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu Jun  8 08:24:05 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201706022000.0) for host (0) was not posted to NetApp. The system will drop the message
Thu Jun  8 09:00:01 BNT [filer02: kern.uptime.filer:info]:   9:00am up  1:13 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 5592 iSCSI ops
Thu Jun  8 09:00:01 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 23 hours.
Thu Jun  8 10:00:00 BNT [filer02: kern.uptime.filer:info]:  10:00am up  2:13 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 125569 iSCSI ops
Thu Jun  8 10:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 22 hours.
Thu Jun  8 11:00:00 BNT [filer02: kern.uptime.filer:info]:  11:00am up  3:13 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 227290 iSCSI ops
Thu Jun  8 11:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 21 hours.
Thu Jun  8 12:00:00 BNT [filer02: kern.uptime.filer:info]:  12:00pm up  4:13 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 268033 iSCSI ops
Thu Jun  8 12:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 20 hours.
Thu Jun  8 12:00:45 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Thu Jun  8 12:01:11 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu Jun  8 12:04:45 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu Jun  8 12:08:22 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu Jun  8 12:12:48 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu Jun  8 12:16:30 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu Jun  8 12:33:09 BNT last message repeated 4 times
Thu Jun  8 12:37:24 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu Jun  8 12:56:57 BNT last message repeated 5 times
Thu Jun  8 12:56:57 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201706030000.0) for host (0) was not posted to NetApp. The system will drop the message
Thu Jun  8 13:00:00 BNT [filer02: kern.uptime.filer:info]:   1:00pm up  5:13 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 333048 iSCSI ops
Thu Jun  8 13:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 19 hours.
Thu Jun  8 14:00:00 BNT [filer02: kern.uptime.filer:info]:   2:00pm up  6:13 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 338331 iSCSI ops
Thu Jun  8 14:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 18 hours.
Thu Jun  8 15:00:00 BNT [filer02: kern.uptime.filer:info]:   3:00pm up  7:13 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 361030 iSCSI ops
Thu Jun  8 15:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 17 hours.
Thu Jun  8 16:00:00 BNT [filer02: kern.uptime.filer:info]:   4:00pm up  8:13 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 367856 iSCSI ops
Thu Jun  8 16:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 16 hours.
Thu Jun  8 16:00:46 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Thu Jun  8 16:01:13 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu Jun  8 16:04:49 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu Jun  8 16:08:50 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu Jun  8 16:12:59 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu Jun  8 16:17:13 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu Jun  8 16:37:06 BNT last message repeated 5 times
Thu Jun  8 16:40:58 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu Jun  8 16:53:07 BNT last message repeated 3 times
Thu Jun  8 16:57:33 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201706030800.0) for host (0) was not posted to NetApp. The system will drop the message
Thu Jun  8 17:00:00 BNT [filer02: kern.uptime.filer:info]:   5:00pm up  9:13 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 376059 iSCSI ops
Thu Jun  8 17:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 15 hours.
Thu Jun  8 18:00:00 BNT [filer02: kern.uptime.filer:info]:   6:00pm up 10:13 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 376059 iSCSI ops
Thu Jun  8 18:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 14 hours.
Thu Jun  8 19:00:00 BNT [filer02: kern.uptime.filer:info]:   7:00pm up 11:13 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 376059 iSCSI ops
Thu Jun  8 19:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 13 hours.
Thu Jun  8 20:00:00 BNT [filer02: kern.uptime.filer:info]:   8:00pm up 12:13 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 376059 iSCSI ops
Thu Jun  8 20:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 12 hours.
Thu Jun  8 20:00:31 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Thu Jun  8 20:13:38 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Thu Jun  8 20:58:33 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201706050743.0) for host (0) was not posted to NetApp. The system will drop the message
Thu Jun  8 21:00:00 BNT [filer02: kern.uptime.filer:info]:   9:00pm up 13:13 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 376059 iSCSI ops
Thu Jun  8 21:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 11 hours.
Thu Jun  8 22:00:00 BNT [filer02: kern.uptime.filer:info]:  10:00pm up 14:13 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 376059 iSCSI ops
Thu Jun  8 22:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 10 hours.
Thu Jun  8 23:00:00 BNT [filer02: kern.uptime.filer:info]:  11:00pm up 15:13 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 376059 iSCSI ops
Thu Jun  8 23:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 9 hours.
Fri Jun  9 00:00:00 BNT [filer02: kern.uptime.filer:info]:  12:00am up 16:13 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 376059 iSCSI ops
Fri Jun  9 00:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 8 hours.
Fri Jun  9 00:00:30 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of nightly.0 snapshot (Snapshot operation failed: No space left on device.)
Fri Jun  9 00:27:37 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SHUTDOWN PENDING (degraded mode))
Fri Jun  9 00:35:34 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SHUTDOWN PENDING (degraded mode))
Fri Jun  9 00:55:30 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201706050744.0) for host (0) was not posted to NetApp. The system will drop the message
Fri Jun  9 01:00:00 BNT [filer02: kern.uptime.filer:info]:   1:00am up 17:13 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 376059 iSCSI ops
Fri Jun  9 01:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 7 hours.
Fri Jun  9 02:00:00 BNT [filer02: kern.uptime.filer:info]:   2:00am up 18:13 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 376059 iSCSI ops
Fri Jun  9 02:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 6 hours.
Fri Jun  9 03:00:00 BNT [filer02: kern.uptime.filer:info]:   3:00am up 19:13 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 376059 iSCSI ops
Fri Jun  9 03:00:01 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 5 hours.
Fri Jun  9 04:00:00 BNT [filer02: kern.uptime.filer:info]:   4:00am up 20:13 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 376059 iSCSI ops
Fri Jun  9 04:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 4 hours.
Fri Jun  9 05:00:00 BNT [filer02: kern.uptime.filer:info]:   5:00am up 21:13 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 376059 iSCSI ops
Fri Jun  9 05:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 3 hours.
Fri Jun  9 06:00:00 BNT [filer02: kern.uptime.filer:info]:   6:00am up 22:13 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 376059 iSCSI ops
Fri Jun  9 06:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 2 hours.
Fri Jun  9 07:00:00 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] is still present in the system and should be removed.
Fri Jun  9 07:00:00 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] is still present in the system and should be removed.
Fri Jun  9 07:00:00 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] is still present in the system and should be removed.
Fri Jun  9 08:00:00 BNT [filer02: monitor.brokenDisk.notice:notice]: When two disks are broken in raid_dp volume,  the system shuts down automatically every 24  hours to encourage you to replace the disk.  If you reboot the system it will run for another  24 hours before shutting down. (The 24 hour timeout  may be increased by altering the "raid.timeout"  value using the "options" command.)
Fri Jun  9 08:00:00 BNT [filer02: monitor.shutdown.brokenDisk:EMERGENCY]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system now.
Fri Jun  9 08:00:03 BNT [filer02: cf.fsm.takeoverOfPartnerDisabled:notice]: Cluster monitor: takeover of filer01 disabled (local halt in progress)
Fri Jun  9 08:00:06 BNT [filer02: kern.shutdown:notice]: System shut down because : "BROKEN DISK".
Fri Jun  9 00:00:55 GMT [localhost: rc:notice]: The system was down for 44 seconds
Fri Jun  9 08:00:30 BNT [filer02: cf.nm.nicTransitionUp:info]: replayed event: Interconnect link 0 is UP
Fri Jun  9 08:00:37 BNT [filer02: fci.initialization.failed:error]: replayed event: Initialization failed on Fibre Channel adapter 0b.
Fri Jun  9 08:00:39 BNT [filer02: nvram.battery.turned.on:info]: replayed event: The NVRAM battery is turned ON. It is turned OFF during system shutdown.
Fri Jun  9 08:00:42 BNT [filer02: diskown.isEnabled:info]: replayed event: software ownership has been enabled for this system
Fri Jun  9 08:00:46 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.2 is a local HA mailbox disk.
Fri Jun  9 08:00:46 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.0 is a local HA mailbox disk.
Fri Jun  9 08:00:46 BNT [filer02: fmmb.instStat.change:info]: replayed event: normal mailbox instance on local side.
Fri Jun  9 08:00:46 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.7 is a partner HA mailbox disk.
Fri Jun  9 08:00:46 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.3 is a partner HA mailbox disk.
Fri Jun  9 08:00:46 BNT [filer02: fmmb.instStat.change:info]: replayed event: normal mailbox instance on partner side.
Fri Jun  9 08:00:46 BNT [filer02: cf.fm.partner:info]: replayed event: Cluster monitor: partner 'filer01'
Fri Jun  9 08:00:46 BNT [filer02: cf.fm.timeMasterStatus:info]: replayed event: Acting as cluster time slave
Fri Jun  9 08:00:47 BNT [filer02: raid.assim.disk.brokenPreAssim:error]: replayed event: Broken Disk 0a.40 Shelf ? Bay ? [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] detected prior to assimilation. It should be removed.
Fri Jun  9 08:00:47 BNT [filer02: raid.assim.disk.brokenPreAssim:error]: replayed event: Broken Disk 0a.35 Shelf ? Bay ? [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] detected prior to assimilation. It should be removed.
Fri Jun  9 08:00:47 BNT [filer02: raid.assim.disk.brokenPreAssim:error]: replayed event: Broken Disk 0a.33 Shelf ? Bay ? [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] detected prior to assimilation. It should be removed.
Fri Jun  9 08:00:47 BNT [filer02: coredump.spare.none:info]: replayed event: No sparecore disk was found.
Fri Jun  9 08:00:48 BNT [filer02: raid.cksum.replay.summary:info]: replayed event: Replayed 0 checksum blocks.
Fri Jun  9 08:00:48 BNT [filer02: raid.stripe.replay.summary:info]: replayed event: Replayed 0 stripes.
Fri Jun  9 08:00:50 BNT [filer02: cf.fm.launch:info]: replayed event: Launching cluster monitor
Fri Jun  9 08:00:50 BNT [filer02: cf.fm.partner:info]: replayed event: Cluster monitor: partner 'filer01'
Fri Jun  9 08:00:50 BNT [filer02: cf.fsm.takeoverOfPartnerDisabled:notice]: replayed event: Cluster monitor: takeover of filer01 disabled (partner booting)
Fri Jun  9 08:00:55 BNT [filer02: fcp.service.adapter:warning]: replayed event: FCP started but no adapter is present!
Fri Jun  9 08:00:55 BNT [filer02: kern.syslog.msg:notice]: replayed event: The system was down for 44 seconds
Fri Jun  9 08:00:57 BNT [filer02: disk.failmsg:error]: Disk 0a.40 (N00V0W8L): message received.
Fri Jun  9 08:00:57 BNT [filer02: disk.failmsg:error]: Disk 0a.35 (N00WLAUL): message received.
Fri Jun  9 08:00:57 BNT [filer02: disk.failmsg:error]: Disk 0a.33 (N00W75PL): message received.
Fri Jun  9 08:00:57 BNT [filer02: raid.disk.unload.done:info]: Unload of Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] has completed successfully
Fri Jun  9 08:00:57 BNT [filer02: raid.disk.unload.done:info]: Unload of Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] has completed successfully
Fri Jun  9 08:00:57 BNT [filer02: raid.disk.unload.done:info]: Unload of Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] has completed successfully
Fri Jun  9 08:00:57 BNT [filer02: raid.fdr.failed.ok:info]: Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] successfully deleted from spare pool
Fri Jun  9 08:00:57 BNT [filer02: raid.fdr.failed.ok:info]: Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] successfully deleted from spare pool
Fri Jun  9 08:00:57 BNT [filer02: raid.fdr.failed.ok:info]: Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] successfully deleted from spare pool
Fri Jun  9 08:00:57 BNT [filer02: cf.fsm.takeoverOfPartnerEnabled:notice]: Cluster monitor: takeover of filer01 enabled
Fri Jun  9 08:00:57 BNT [filer02: cf.fsm.takeoverByPartnerEnabled:notice]: Cluster monitor: takeover of filer02 by filer01 enabled
Fri Jun  9 08:00:57 BNT [filer02: dfu.firmwareUpToDate:info]: Firmware is up-to-date on all disk drives
Fri Jun  9 08:00:59 BNT [filer02: perf.archive.start:info]: Performance archiver started. Sampling 22 objects and 198 counters.
Fri Jun  9 08:01:01 BNT [filer02: netif.linkUp:info]: Ethernet e0b: Link up.
Fri Jun  9 08:01:01 BNT [filer02: netif.linkUp:info]: Ethernet e0a: Link up.
Fri Jun  9 08:01:02 BNT [filer02: pvif.switchLink:warning]: vif1: switching to e0a
Fri Jun  9 08:01:03 BNT [filer02: iscsi.service.startup:info]: iSCSI service startup
Fri Jun  9 08:01:03 BNT [filer02: iscsi.notice:notice]: ISCSI: New session from initiator iqn.1991-05.com.microsoft:pcb-722-ad1.pcbl722dom.local at IP addr 10.72.4.1
Fri Jun  9 08:01:04 BNT [filer02: cmds.sysconf.logErr:error]: sysconfig: Unless directed by NetApp Global Services the cf.takeover.on_panic option should be set to On. .
Fri Jun  9 08:01:04 BNT [filer02: raid.rg.recons.missing:notice]: RAID group /aggr1/plex0/rg0 is missing 2 disk(s).
Fri Jun  9 08:01:04 BNT [filer02: raid.rg.recons.cantStart:warning]: The reconstruction cannot start in RAID group /aggr1/plex0/rg0: No 520 bps disk of required size available in spare pool, targeting any  spare pool
Fri Jun  9 08:01:04 BNT [filer02: mgr.boot.disk_done:info]: NetApp Release 7.3.2 boot complete. Last disk update written at Fri Jun  9 08:00:08 BNT 2017
Fri Jun  9 08:01:04 BNT [filer02: monitor.raid.brokenDisk:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken.
Fri Jun  9 08:01:04 BNT [filer02: monitor.brokenDisk.notice:notice]: When two disks are broken in raid_dp volume,  the system shuts down automatically every 24  hours to encourage you to replace the disk.  If you reboot the system it will run for another  24 hours before shutting down. (The 24 hour timeout  may be increased by altering the "raid.timeout"  value using the "options" command.)
Fri Jun  9 08:01:04 BNT [filer02: diskown.errorReadingOwnership:warning]: error 3 (disk failed) while reading ownership on disk 0a.24 (S/N N00SSSAL)
Fri Jun  9 08:01:04 BNT [filer02: diskown.errorReadingOwnership:warning]: error 3 (disk failed) while reading ownership on disk 0a.26 (S/N N00V0UWL)
Fri Jun  9 08:01:04 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] is still present in the system and should be removed.
Fri Jun  9 08:01:04 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] is still present in the system and should be removed.
Fri Jun  9 08:01:04 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] is still present in the system and should be removed.
Fri Jun  9 08:01:04 BNT [filer02: shelf.config.spha:info]: System is using single path HA attached storage only.
Fri Jun  9 08:01:04 BNT [filer02: mgr.boot.reason_ok:notice]: System rebooted.
Fri Jun  9 08:01:25 BNT [filer02: raid.rg.spares.low:warning]: /aggr1/plex0/rg0
Fri Jun  9 08:01:29 BNT [filer02: nbt.nbns.registrationComplete:info]: NBT: All CIFS name registrations have completed for the local server.
Fri Jun  9 08:01:33 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (REBOOT)
Fri Jun  9 08:01:42 BNT last message repeated 2 times
Fri Jun  9 08:01:42 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201706050744.1) was not posted for host (0), it was dropped from queue due to lack of space
Fri Jun  9 08:01:51 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SPARES_LOW)
Fri Jun  9 08:01:51 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201706050744.2) was not posted for host (0), it was dropped from queue due to lack of space
Fri Jun  9 08:01:59 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (PERFORMANCE SNAPSHOT)
Fri Jun  9 08:01:59 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201706050745.0) was not posted for host (0), it was dropped from queue due to lack of space
Fri Jun  9 08:02:06 BNT [filer02: monitor.globalStatus.nonCritical:warning]: Disk on adapter 0a, shelf 2, bay 8, failed. Disk on adapter 0a, shelf 2, bay 3, failed. Disk on adapter 0a, shelf 2, bay 1, failed.  There are not enough spare disks. /vol/vol1 is full (using or reserving 100% of space and 0% of inodes, using 92% of reserve).
Fri Jun  9 08:02:13 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (CLUSTER ERROR: DISK/SHELF COUNT MISMATCH)
Fri Jun  9 08:02:13 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201706050749.0) was not posted for host (0), it was dropped from queue due to lack of space
Fri Jun  9 08:06:44 BNT [filer02: cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 0a.33 (016F3AB5:B475F4EC:10505E83:02202755:00000000:00000000:00000000:00000000:00000000:00000000) has recently changed or the node (filer02) is missing the disk.
Fri Jun  9 08:06:44 BNT [filer02: cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 0a.35 (9115483F:9DD43BC4:27172773:99F7D8B4:00000000:00000000:00000000:00000000:00000000:00000000) has recently changed or the node (filer02) is missing the disk.
Fri Jun  9 08:06:44 BNT [filer02: cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 0a.40 (072AEB65:C662A9CF:E58B9917:9BFA787B:00000000:00000000:00000000:00000000:00000000:00000000) has recently changed or the node (filer02) is missing the disk.
Fri Jun  9 08:06:44 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jun  9 08:06:52 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jun  9 08:06:52 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201706050800.0) was not posted for host (0), it was dropped from queue due to lack of space
Fri Jun  9 08:11:23 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jun  9 08:15:46 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jun  9 08:19:31 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jun  9 08:23:15 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jun  9 08:27:00 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jun  9 08:34:21 BNT last message repeated 2 times
Fri Jun  9 08:34:21 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201706051200.0) for host (0) was not posted to NetApp. The system will drop the message
Fri Jun  9 09:00:00 BNT [filer02: kern.uptime.filer:info]:   9:00am up 58 mins, 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 11457 iSCSI ops
Fri Jun  9 09:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 24 hours.
Fri Jun  9 09:08:04 BNT [filer02: cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 0a.33 (016F3AB5:B475F4EC:10505E83:02202755:00000000:00000000:00000000:00000000:00000000:00000000) has recently changed or the node (filer02) is missing the disk.
Fri Jun  9 09:08:04 BNT [filer02: cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 0a.35 (9115483F:9DD43BC4:27172773:99F7D8B4:00000000:00000000:00000000:00000000:00000000:00000000) has recently changed or the node (filer02) is missing the disk.
Fri Jun  9 09:08:04 BNT [filer02: cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 0a.40 (072AEB65:C662A9CF:E58B9917:9BFA787B:00000000:00000000:00000000:00000000:00000000:00000000) has recently changed or the node (filer02) is missing the disk.
Fri Jun  9 10:00:00 BNT [filer02: kern.uptime.filer:info]:  10:00am up  1:58 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 399237 iSCSI ops
Fri Jun  9 10:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 23 hours.
Fri Jun  9 11:00:00 BNT [filer02: kern.uptime.filer:info]:  11:00am up  2:58 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 415327 iSCSI ops
Fri Jun  9 11:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 22 hours.
Fri Jun  9 12:00:00 BNT [filer02: kern.uptime.filer:info]:  12:00pm up  3:58 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 662231 iSCSI ops
Fri Jun  9 12:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 21 hours.
Fri Jun  9 12:00:16 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Fri Jun  9 12:00:42 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jun  9 12:05:03 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jun  9 12:08:43 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jun  9 12:12:51 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jun  9 12:16:40 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jun  9 12:33:10 BNT last message repeated 4 times
Fri Jun  9 12:37:06 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jun  9 12:53:41 BNT last message repeated 4 times
Fri Jun  9 12:57:48 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jun  9 12:57:48 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201706051600.0) for host (0) was not posted to NetApp. The system will drop the message
Fri Jun  9 13:00:00 BNT [filer02: kern.uptime.filer:info]:   1:00pm up  4:58 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 677867 iSCSI ops
Fri Jun  9 13:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 20 hours.
Fri Jun  9 14:00:00 BNT [filer02: kern.uptime.filer:info]:   2:00pm up  5:58 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 677897 iSCSI ops
Fri Jun  9 14:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 19 hours.
Fri Jun  9 15:00:00 BNT [filer02: kern.uptime.filer:info]:   3:00pm up  6:58 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 677921 iSCSI ops
Fri Jun  9 15:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 18 hours.
Fri Jun  9 16:00:00 BNT [filer02: kern.uptime.filer:info]:   4:00pm up  7:58 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 677986 iSCSI ops
Fri Jun  9 16:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 17 hours.
Fri Jun  9 16:00:16 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Fri Jun  9 16:00:41 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jun  9 16:04:41 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jun  9 16:08:55 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jun  9 16:41:17 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jun  9 16:52:51 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jun  9 16:56:24 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201706052000.0) for host (0) was not posted to NetApp. The system will drop the message
Fri Jun  9 17:00:00 BNT [filer02: kern.uptime.filer:info]:   5:00pm up  8:58 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 677986 iSCSI ops
Fri Jun  9 17:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 16 hours.
Fri Jun  9 18:00:00 BNT [filer02: kern.uptime.filer:info]:   6:00pm up  9:58 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 677986 iSCSI ops
Fri Jun  9 18:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 15 hours.
Fri Jun  9 19:00:00 BNT [filer02: kern.uptime.filer:info]:   7:00pm up 10:59 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 677986 iSCSI ops
Fri Jun  9 19:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 14 hours.
Fri Jun  9 20:00:00 BNT [filer02: kern.uptime.filer:info]:   8:00pm up 11:59 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 677986 iSCSI ops
Fri Jun  9 20:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 13 hours.
Fri Jun  9 20:00:01 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Fri Jun  9 20:56:00 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201706060000.0) for host (0) was not posted to NetApp. The system will drop the message
Fri Jun  9 21:00:00 BNT [filer02: kern.uptime.filer:info]:   9:00pm up 12:59 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 677986 iSCSI ops
Fri Jun  9 21:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 12 hours.
Fri Jun  9 22:00:00 BNT [filer02: kern.uptime.filer:info]:  10:00pm up 13:59 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 677986 iSCSI ops
Fri Jun  9 22:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 11 hours.
Fri Jun  9 23:00:00 BNT [filer02: kern.uptime.filer:info]:  11:00pm up 14:59 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 677986 iSCSI ops
Fri Jun  9 23:00:01 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 10 hours.
Sat Jun 10 00:00:00 BNT [filer02: kern.uptime.filer:info]:  12:00am up 15:59 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 677986 iSCSI ops
Sat Jun 10 00:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 9 hours.
Sat Jun 10 00:00:00 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of nightly.0 snapshot (Snapshot operation failed: No space left on device.)
Sat Jun 10 00:03:52 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Sat Jun 10 00:55:02 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Sat Jun 10 00:55:02 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201706060736.0) for host (0) was not posted to NetApp. The system will drop the message
Sat Jun 10 01:00:00 BNT [filer02: kern.uptime.filer:info]:   1:00am up 16:59 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 677986 iSCSI ops
Sat Jun 10 01:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 8 hours.
Sat Jun 10 02:00:00 BNT [filer02: kern.uptime.filer:info]:   2:00am up 17:59 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 677986 iSCSI ops
Sat Jun 10 02:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 7 hours.
Sat Jun 10 03:00:00 BNT [filer02: kern.uptime.filer:info]:   3:00am up 18:59 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 677986 iSCSI ops
Sat Jun 10 03:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 6 hours.
Sat Jun 10 04:00:00 BNT [filer02: kern.uptime.filer:info]:   4:00am up 19:59 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 677986 iSCSI ops
Sat Jun 10 04:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 5 hours.
Sat Jun 10 05:00:00 BNT [filer02: kern.uptime.filer:info]:   5:00am up 20:59 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 677986 iSCSI ops
Sat Jun 10 05:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 4 hours.
Sat Jun 10 06:00:00 BNT [filer02: kern.uptime.filer:info]:   6:00am up 21:59 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 677986 iSCSI ops
Sat Jun 10 06:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 3 hours.
Sat Jun 10 07:00:00 BNT [filer02: kern.uptime.filer:info]:   7:00am up 22:59 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 677986 iSCSI ops
Sat Jun 10 07:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 2 hours.
Sat Jun 10 08:00:01 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] is still present in the system and should be removed.
Sat Jun 10 08:00:01 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] is still present in the system and should be removed.
Sat Jun 10 08:00:01 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] is still present in the system and should be removed.
Sat Jun 10 08:00:46 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Sat Jun 10 08:59:12 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201706060737.0) for host (0) was not posted to NetApp. The system will drop the message
Sat Jun 10 09:00:00 BNT [filer02: monitor.brokenDisk.notice:notice]: When two disks are broken in raid_dp volume,  the system shuts down automatically every 24  hours to encourage you to replace the disk.  If you reboot the system it will run for another  24 hours before shutting down. (The 24 hour timeout  may be increased by altering the "raid.timeout"  value using the "options" command.)
Sat Jun 10 09:00:00 BNT [filer02: monitor.shutdown.brokenDisk:EMERGENCY]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system now.
Sat Jun 10 09:00:03 BNT [filer02: cf.fsm.takeoverOfPartnerDisabled:notice]: Cluster monitor: takeover of filer01 disabled (local halt in progress)
Sat Jun 10 09:00:06 BNT [filer02: kern.shutdown:notice]: System shut down because : "BROKEN DISK".
Sat Jun 10 01:30:13 GMT [localhost: rc:notice]: The system was down for 1802 seconds
Sat Jun 10 09:29:47 BNT [filer02: cf.nm.nicTransitionUp:info]: replayed event: Interconnect link 0 is UP
Sat Jun 10 09:29:54 BNT [filer02: fci.initialization.failed:error]: replayed event: Initialization failed on Fibre Channel adapter 0b.
Sat Jun 10 09:29:56 BNT [filer02: nvram.battery.turned.on:info]: replayed event: The NVRAM battery is turned ON. It is turned OFF during system shutdown.
Sat Jun 10 09:29:59 BNT [filer02: diskown.isEnabled:info]: replayed event: software ownership has been enabled for this system
Sat Jun 10 09:30:03 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.2 is a local HA mailbox disk.
Sat Jun 10 09:30:03 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.0 is a local HA mailbox disk.
Sat Jun 10 09:30:03 BNT [filer02: fmmb.instStat.change:info]: replayed event: normal mailbox instance on local side.
Sat Jun 10 09:30:03 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.7 is a partner HA mailbox disk.
Sat Jun 10 09:30:03 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.3 is a partner HA mailbox disk.
Sat Jun 10 09:30:03 BNT [filer02: fmmb.instStat.change:info]: replayed event: normal mailbox instance on partner side.
Sat Jun 10 09:30:03 BNT [filer02: cf.fm.partner:info]: replayed event: Cluster monitor: partner 'filer01'
Sat Jun 10 09:30:03 BNT [filer02: cf.fm.timeMasterStatus:info]: replayed event: Acting as cluster time slave
Sat Jun 10 09:30:04 BNT [filer02: raid.assim.disk.brokenPreAssim:error]: replayed event: Broken Disk 0a.40 Shelf ? Bay ? [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] detected prior to assimilation. It should be removed.
Sat Jun 10 09:30:04 BNT [filer02: raid.assim.disk.brokenPreAssim:error]: replayed event: Broken Disk 0a.35 Shelf ? Bay ? [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] detected prior to assimilation. It should be removed.
Sat Jun 10 09:30:04 BNT [filer02: raid.assim.disk.brokenPreAssim:error]: replayed event: Broken Disk 0a.33 Shelf ? Bay ? [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] detected prior to assimilation. It should be removed.
Sat Jun 10 09:30:04 BNT [filer02: coredump.spare.none:info]: replayed event: No sparecore disk was found.
Sat Jun 10 09:30:05 BNT [filer02: raid.cksum.replay.summary:info]: replayed event: Replayed 0 checksum blocks.
Sat Jun 10 09:30:05 BNT [filer02: raid.stripe.replay.summary:info]: replayed event: Replayed 0 stripes.
Sat Jun 10 09:30:07 BNT [filer02: cf.fm.launch:info]: replayed event: Launching cluster monitor
Sat Jun 10 09:30:07 BNT [filer02: cf.fm.partner:info]: replayed event: Cluster monitor: partner 'filer01'
Sat Jun 10 09:30:07 BNT [filer02: cf.fsm.takeoverOfPartnerDisabled:notice]: replayed event: Cluster monitor: takeover of filer01 disabled (partner booting)
Sat Jun 10 09:30:08 BNT [filer02: ses.status.psWarning:warning]: replayed event: DS12-ESAS shelf 0 on channel 0c power warning for Power supply 1: non-critical status; DC undervoltage fault. This module is on the rear side of the shelf, at the left.
Sat Jun 10 09:30:08 BNT [filer02: ses.status.psWarning:warning]: replayed event: DS12-ESAS shelf 0 on channel 0c power warning for Power supply 2: non-critical status; DC undervoltage fault. This module is on the rear side of the shelf, at the right.
Sat Jun 10 09:30:13 BNT [filer02: fcp.service.adapter:warning]: replayed event: FCP started but no adapter is present!
Sat Jun 10 09:30:13 BNT [filer02: kern.syslog.msg:notice]: replayed event: The system was down for 1802 seconds
Sat Jun 10 09:30:14 BNT [filer02: disk.failmsg:error]: Disk 0a.40 (N00V0W8L): message received.
Sat Jun 10 09:30:14 BNT [filer02: disk.failmsg:error]: Disk 0a.35 (N00WLAUL): message received.
Sat Jun 10 09:30:14 BNT [filer02: disk.failmsg:error]: Disk 0a.33 (N00W75PL): message received.
Sat Jun 10 09:30:14 BNT [filer02: raid.disk.unload.done:info]: Unload of Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] has completed successfully
Sat Jun 10 09:30:14 BNT [filer02: raid.disk.unload.done:info]: Unload of Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] has completed successfully
Sat Jun 10 09:30:14 BNT [filer02: raid.disk.unload.done:info]: Unload of Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] has completed successfully
Sat Jun 10 09:30:14 BNT [filer02: raid.fdr.failed.ok:info]: Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] successfully deleted from spare pool
Sat Jun 10 09:30:14 BNT [filer02: raid.fdr.failed.ok:info]: Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] successfully deleted from spare pool
Sat Jun 10 09:30:14 BNT [filer02: raid.fdr.failed.ok:info]: Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] successfully deleted from spare pool
Sat Jun 10 09:30:15 BNT [filer02: dfu.firmwareUpToDate:info]: Firmware is up-to-date on all disk drives
Sat Jun 10 09:30:15 BNT [filer02: cf.fsm.takeoverOfPartnerEnabled:notice]: Cluster monitor: takeover of filer01 enabled
Sat Jun 10 09:30:15 BNT [filer02: cf.fsm.takeoverByPartnerEnabled:notice]: Cluster monitor: takeover of filer02 by filer01 enabled
Sat Jun 10 09:30:16 BNT [filer02: perf.archive.start:info]: Performance archiver started. Sampling 22 objects and 198 counters.
Sat Jun 10 09:30:18 BNT [filer02: netif.linkUp:info]: Ethernet e0a: Link up.
Sat Jun 10 09:30:19 BNT [filer02: netif.linkUp:info]: Ethernet e0b: Link up.
Sat Jun 10 09:30:19 BNT [filer02: pvif.switchLink:warning]: vif1: switching to e0a
Sat Jun 10 09:30:21 BNT [filer02: iscsi.service.startup:info]: iSCSI service startup
Sat Jun 10 09:30:21 BNT [filer02: iscsi.notice:notice]: ISCSI: New session from initiator iqn.1991-05.com.microsoft:pcb-722-ad1.pcbl722dom.local at IP addr 10.72.4.1
Sat Jun 10 09:30:21 BNT [filer02: cmds.sysconf.logErr:error]: sysconfig: Unless directed by NetApp Global Services the cf.takeover.on_panic option should be set to On. .
Sat Jun 10 09:30:21 BNT [filer02: raid.rg.recons.missing:notice]: RAID group /aggr1/plex0/rg0 is missing 2 disk(s).
Sat Jun 10 09:30:21 BNT [filer02: raid.rg.recons.cantStart:warning]: The reconstruction cannot start in RAID group /aggr1/plex0/rg0: No 520 bps disk of required size available in spare pool, targeting any  spare pool
Sat Jun 10 09:30:21 BNT [filer02: mgr.boot.disk_done:info]: NetApp Release 7.3.2 boot complete. Last disk update written at Sat Jun 10 09:00:08 BNT 2017
Sat Jun 10 09:30:21 BNT [filer02: monitor.raid.brokenDisk:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken.
Sat Jun 10 09:30:21 BNT [filer02: monitor.brokenDisk.notice:notice]: When two disks are broken in raid_dp volume,  the system shuts down automatically every 24  hours to encourage you to replace the disk.  If you reboot the system it will run for another  24 hours before shutting down. (The 24 hour timeout  may be increased by altering the "raid.timeout"  value using the "options" command.)
Sat Jun 10 09:30:22 BNT [filer02: diskown.errorReadingOwnership:warning]: error 3 (disk failed) while reading ownership on disk 0a.24 (S/N N00SSSAL)
Sat Jun 10 09:30:22 BNT [filer02: diskown.errorReadingOwnership:warning]: error 3 (disk failed) while reading ownership on disk 0a.26 (S/N N00V0UWL)
Sat Jun 10 09:30:22 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] is still present in the system and should be removed.
Sat Jun 10 09:30:22 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] is still present in the system and should be removed.
Sat Jun 10 09:30:22 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] is still present in the system and should be removed.
Sat Jun 10 09:30:22 BNT [filer02: shelf.config.spha:info]: System is using single path HA attached storage only.
Sat Jun 10 09:30:22 BNT [filer02: mgr.boot.reason_ok:notice]: System rebooted.
Sat Jun 10 09:30:44 BNT [filer02: raid.rg.spares.low:warning]: /aggr1/plex0/rg0
Sat Jun 10 09:30:47 BNT [filer02: nbt.nbns.registrationComplete:info]: NBT: All CIFS name registrations have completed for the local server.
Sat Jun 10 09:30:55 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (REBOOT (power on))
Sat Jun 10 09:31:03 BNT last message repeated 2 times
Sat Jun 10 09:31:03 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201706060737.1) was not posted for host (0), it was dropped from queue due to lack of space
Sat Jun 10 09:31:05 BNT [filer02: monitor.globalStatus.nonCritical:warning]: Disk on adapter 0a, shelf 2, bay 8, failed. Disk on adapter 0a, shelf 2, bay 3, failed. Disk on adapter 0a, shelf 2, bay 1, failed.  There are not enough spare disks. /vol/vol1 is full (using or reserving 100% of space and 0% of inodes, using 92% of reserve).
Sat Jun 10 09:31:11 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SPARES_LOW)
Sat Jun 10 09:31:11 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201706060737.2) was not posted for host (0), it was dropped from queue due to lack of space
Sat Jun 10 09:31:19 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (PERFORMANCE SNAPSHOT)
Sat Jun 10 09:31:19 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201706060738.0) was not posted for host (0), it was dropped from queue due to lack of space
Sat Jun 10 09:31:30 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Sat Jun 10 09:31:30 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201706060800.0) was not posted for host (0), it was dropped from queue due to lack of space
Sat Jun 10 09:35:48 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Sat Jun 10 09:36:02 BNT [filer02: cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 0a.33 (016F3AB5:B475F4EC:10505E83:02202755:00000000:00000000:00000000:00000000:00000000:00000000) has recently changed or the node (filer02) is missing the disk.
Sat Jun 10 09:36:02 BNT [filer02: cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 0a.35 (9115483F:9DD43BC4:27172773:99F7D8B4:00000000:00000000:00000000:00000000:00000000:00000000) has recently changed or the node (filer02) is missing the disk.
Sat Jun 10 09:36:02 BNT [filer02: cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 0a.40 (072AEB65:C662A9CF:E58B9917:9BFA787B:00000000:00000000:00000000:00000000:00000000:00000000) has recently changed or the node (filer02) is missing the disk.
Sat Jun 10 09:36:02 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Sat Jun 10 09:36:10 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Sat Jun 10 09:36:10 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201706061200.0) was not posted for host (0), it was dropped from queue due to lack of space
Sat Jun 10 09:40:15 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Sat Jun 10 09:44:37 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Sat Jun 10 09:48:36 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Sat Jun 10 09:52:13 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Sat Jun 10 09:56:23 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Sat Jun 10 09:59:54 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Sat Jun 10 09:59:54 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201706061600.0) for host (0) was not posted to NetApp. The system will drop the message
Sat Jun 10 10:00:00 BNT [filer02: kern.uptime.filer:info]:  10:00am up 29 mins, 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 2695 iSCSI ops
Sat Jun 10 10:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 24 hours.
Sat Jun 10 10:37:22 BNT [filer02: cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 0a.33 (016F3AB5:B475F4EC:10505E83:02202755:00000000:00000000:00000000:00000000:00000000:00000000) has recently changed or the node (filer02) is missing the disk.
Sat Jun 10 10:37:22 BNT [filer02: cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 0a.35 (9115483F:9DD43BC4:27172773:99F7D8B4:00000000:00000000:00000000:00000000:00000000:00000000) has recently changed or the node (filer02) is missing the disk.
Sat Jun 10 10:37:22 BNT [filer02: cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 0a.40 (072AEB65:C662A9CF:E58B9917:9BFA787B:00000000:00000000:00000000:00000000:00000000:00000000) has recently changed or the node (filer02) is missing the disk.
Sat Jun 10 11:00:00 BNT [filer02: kern.uptime.filer:info]:  11:00am up  1:29 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 19111 iSCSI ops
Sat Jun 10 11:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 23 hours.
Sat Jun 10 12:00:00 BNT [filer02: kern.uptime.filer:info]:  12:00pm up  2:29 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 202995 iSCSI ops
Sat Jun 10 12:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 22 hours.
Sat Jun 10 12:00:46 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Sat Jun 10 12:01:12 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Sat Jun 10 12:04:45 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Sat Jun 10 12:08:17 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Sat Jun 10 12:12:42 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Sat Jun 10 12:16:12 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Sat Jun 10 12:32:43 BNT last message repeated 4 times
Sat Jun 10 12:37:03 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Sat Jun 10 12:53:20 BNT last message repeated 4 times
Sat Jun 10 12:57:19 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Sat Jun 10 12:57:19 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201706062000.0) for host (0) was not posted to NetApp. The system will drop the message
Sat Jun 10 13:00:00 BNT [filer02: kern.uptime.filer:info]:   1:00pm up  3:29 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 223716 iSCSI ops
Sat Jun 10 13:00:01 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 21 hours.
Sat Jun 10 14:00:00 BNT [filer02: kern.uptime.filer:info]:   2:00pm up  4:29 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 361788 iSCSI ops
Sat Jun 10 14:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 20 hours.
Sat Jun 10 15:00:00 BNT [filer02: kern.uptime.filer:info]:   3:00pm up  5:29 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 587837 iSCSI ops
Sat Jun 10 15:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 19 hours.
Sat Jun 10 16:00:00 BNT [filer02: kern.uptime.filer:info]:   4:00pm up  6:29 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 590209 iSCSI ops
Sat Jun 10 16:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 18 hours.
Sat Jun 10 16:00:31 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Sat Jun 10 16:01:02 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Sat Jun 10 16:05:03 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Sat Jun 10 16:09:25 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Sat Jun 10 16:13:07 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Sat Jun 10 16:16:52 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Sat Jun 10 16:33:58 BNT last message repeated 4 times
Sat Jun 10 16:38:11 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Sat Jun 10 16:46:01 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Sat Jun 10 16:58:07 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201706070000.0) for host (0) was not posted to NetApp. The system will drop the message
Sat Jun 10 17:00:00 BNT [filer02: kern.uptime.filer:info]:   5:00pm up  7:29 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 590209 iSCSI ops
Sat Jun 10 17:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 17 hours.
Sat Jun 10 18:00:00 BNT [filer02: kern.uptime.filer:info]:   6:00pm up  8:29 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 590209 iSCSI ops
Sat Jun 10 18:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 16 hours.
Sat Jun 10 19:00:00 BNT [filer02: kern.uptime.filer:info]:   7:00pm up  9:29 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 590209 iSCSI ops
Sat Jun 10 19:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 15 hours.
Sat Jun 10 20:00:00 BNT [filer02: kern.uptime.filer:info]:   8:00pm up 10:29 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 590209 iSCSI ops
Sat Jun 10 20:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 14 hours.
Sat Jun 10 20:00:30 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Sat Jun 10 20:55:40 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201706070739.0) for host (0) was not posted to NetApp. The system will drop the message
Sat Jun 10 21:00:00 BNT [filer02: kern.uptime.filer:info]:   9:00pm up 11:29 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 590209 iSCSI ops
Sat Jun 10 21:00:01 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 13 hours.
Sat Jun 10 22:00:00 BNT [filer02: kern.uptime.filer:info]:  10:00pm up 12:29 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 590209 iSCSI ops
Sat Jun 10 22:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 12 hours.
Sat Jun 10 23:00:00 BNT [filer02: kern.uptime.filer:info]:  11:00pm up 13:29 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 590209 iSCSI ops
Sat Jun 10 23:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 11 hours.
Sun Jun 11 00:00:00 BNT [filer02: kern.uptime.filer:info]:  12:00am up 14:29 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 590209 iSCSI ops
Sun Jun 11 00:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 10 hours.
Sun Jun 11 00:00:00 BNT [filer02: kern.log.rotate:notice]: System filer02 (ID 0142227438) is running NetApp Release 7.3.2
Sun Jun 11 00:00:15 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of nightly.0 snapshot (Snapshot operation failed: No space left on device.)
Sun Jun 11 00:00:24 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201706070739.1) was not posted for host (0), it was dropped from queue due to lack of space
Sun Jun 11 00:00:25 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201706070739.2) was not posted for host (0), it was dropped from queue due to lack of space
Sun Jun 11 00:24:17 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SPARES_LOW)
Sun Jun 11 00:28:00 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SPARES_LOW)
Sun Jun 11 00:47:25 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SPARES_LOW)
Sun Jun 11 00:51:49 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SPARES_LOW)
Sun Jun 11 00:55:47 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SPARES_LOW)
Sun Jun 11 00:55:47 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201706070739.3) for host (0) was not posted to NetApp. The system will drop the message
Sun Jun 11 01:00:00 BNT [filer02: kern.uptime.filer:info]:   1:00am up 15:29 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 590209 iSCSI ops
Sun Jun 11 01:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 9 hours.
Sun Jun 11 01:00:02 BNT [filer02: raid.rg.scrub.start:notice]: /aggr0/plex0/rg0: starting scrub
Sun Jun 11 02:00:00 BNT [filer02: kern.uptime.filer:info]:   2:00am up 16:29 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 590209 iSCSI ops
Sun Jun 11 02:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 8 hours.
Sun Jun 11 02:59:42 BNT [filer02: raid.rg.scrub.done:notice]: /aggr0/plex0/rg0: scrub completed in 1:59:39.82
Sun Jun 11 02:59:42 BNT [filer02: raid.rg.scrub.summary.pi:notice]: Scrub found 0 parity inconsistencies in /aggr0/plex0/rg0.
Sun Jun 11 02:59:42 BNT [filer02: raid.rg.scrub.summary.cksum:notice]: Scrub found 0 checksum errors in /aggr0/plex0/rg0.
Sun Jun 11 02:59:42 BNT [filer02: raid.rg.scrub.summary.media:notice]: Scrub found 0 media errors in /aggr0/plex0/rg0.
Sun Jun 11 03:00:00 BNT [filer02: kern.uptime.filer:info]:   3:00am up 17:29 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 590209 iSCSI ops
Sun Jun 11 03:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 7 hours.
Sun Jun 11 04:00:00 BNT [filer02: kern.uptime.filer:info]:   4:00am up 18:30 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 590209 iSCSI ops
Sun Jun 11 04:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 6 hours.
Sun Jun 11 05:00:00 BNT [filer02: kern.uptime.filer:info]:   5:00am up 19:30 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 590209 iSCSI ops
Sun Jun 11 05:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 5 hours.
Sun Jun 11 06:00:00 BNT [filer02: kern.uptime.filer:info]:   6:00am up 20:30 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 590209 iSCSI ops
Sun Jun 11 06:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 4 hours.
Sun Jun 11 07:00:00 BNT [filer02: kern.uptime.filer:info]:   7:00am up 21:30 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 590210 iSCSI ops
Sun Jun 11 07:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 3 hours.
Sun Jun 11 08:00:00 BNT [filer02: kern.uptime.filer:info]:   8:00am up 22:30 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 590210 iSCSI ops
Sun Jun 11 08:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 2 hours.
Sun Jun 11 08:00:01 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Sun Jun 11 08:00:11 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (PERFORMANCE SNAPSHOT)
Sun Jun 11 08:04:07 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (PERFORMANCE SNAPSHOT)
Sun Jun 11 08:08:20 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (PERFORMANCE SNAPSHOT)
Sun Jun 11 08:12:48 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (PERFORMANCE SNAPSHOT)
Sun Jun 11 08:16:31 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (PERFORMANCE SNAPSHOT)
Sun Jun 11 08:36:25 BNT last message repeated 5 times
Sun Jun 11 08:40:43 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (PERFORMANCE SNAPSHOT)
Sun Jun 11 08:57:11 BNT last message repeated 4 times
Sun Jun 11 08:57:11 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201706070740.0) for host (0) was not posted to NetApp. The system will drop the message
Sun Jun 11 09:00:00 BNT [filer02: kern.uptime.filer:info]:   9:00am up 23:30 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 590210 iSCSI ops
Sun Jun 11 09:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 1 hour.
Sun Jun 11 09:00:00 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] is still present in the system and should be removed.
Sun Jun 11 09:00:00 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] is still present in the system and should be removed.
Sun Jun 11 09:00:00 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] is still present in the system and should be removed.
Sun Jun 11 10:00:00 BNT [filer02: kern.uptime.filer:info]:  10:00am up  1 day, 30 mins, 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 590210 iSCSI ops
Sun Jun 11 10:00:00 BNT [filer02: monitor.brokenDisk.notice:notice]: When two disks are broken in raid_dp volume,  the system shuts down automatically every 24  hours to encourage you to replace the disk.  If you reboot the system it will run for another  24 hours before shutting down. (The 24 hour timeout  may be increased by altering the "raid.timeout"  value using the "options" command.)
Sun Jun 11 10:00:00 BNT [filer02: monitor.shutdown.brokenDisk:EMERGENCY]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system now.
Sun Jun 11 10:00:03 BNT [filer02: cf.fsm.takeoverOfPartnerDisabled:notice]: Cluster monitor: takeover of filer01 disabled (local halt in progress)
Sun Jun 11 10:00:06 BNT [filer02: kern.shutdown:notice]: System shut down because : "BROKEN DISK".
Tue Jun 13 00:05:53 GMT [localhost: rc:notice]: The system was down for 165942 seconds
Tue Jun 13 08:05:27 BNT [filer02: cf.nm.nicTransitionUp:info]: replayed event: Interconnect link 0 is UP
Tue Jun 13 08:05:34 BNT [filer02: fci.initialization.failed:error]: replayed event: Initialization failed on Fibre Channel adapter 0b.
Tue Jun 13 08:05:36 BNT [filer02: nvram.battery.turned.on:info]: replayed event: The NVRAM battery is turned ON. It is turned OFF during system shutdown.
Tue Jun 13 08:05:39 BNT [filer02: diskown.isEnabled:info]: replayed event: software ownership has been enabled for this system
Tue Jun 13 08:05:43 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.2 is a local HA mailbox disk.
Tue Jun 13 08:05:43 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.0 is a local HA mailbox disk.
Tue Jun 13 08:05:43 BNT [filer02: fmmb.instStat.change:info]: replayed event: normal mailbox instance on local side.
Tue Jun 13 08:05:43 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.7 is a partner HA mailbox disk.
Tue Jun 13 08:05:43 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.3 is a partner HA mailbox disk.
Tue Jun 13 08:05:43 BNT [filer02: fmmb.instStat.change:info]: replayed event: normal mailbox instance on partner side.
Tue Jun 13 08:05:43 BNT [filer02: cf.fm.partner:info]: replayed event: Cluster monitor: partner 'filer01'
Tue Jun 13 08:05:43 BNT [filer02: cf.fm.timeMasterStatus:info]: replayed event: Acting as cluster time slave
Tue Jun 13 08:05:44 BNT [filer02: raid.assim.disk.brokenPreAssim:error]: replayed event: Broken Disk 0a.40 Shelf ? Bay ? [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] detected prior to assimilation. It should be removed.
Tue Jun 13 08:05:44 BNT [filer02: raid.assim.disk.brokenPreAssim:error]: replayed event: Broken Disk 0a.35 Shelf ? Bay ? [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] detected prior to assimilation. It should be removed.
Tue Jun 13 08:05:44 BNT [filer02: raid.assim.disk.brokenPreAssim:error]: replayed event: Broken Disk 0a.33 Shelf ? Bay ? [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] detected prior to assimilation. It should be removed.
Tue Jun 13 08:05:44 BNT [filer02: coredump.spare.none:info]: replayed event: No sparecore disk was found.
Tue Jun 13 08:05:45 BNT [filer02: raid.cksum.replay.summary:info]: replayed event: Replayed 0 checksum blocks.
Tue Jun 13 08:05:45 BNT [filer02: raid.stripe.replay.summary:info]: replayed event: Replayed 0 stripes.
Tue Jun 13 08:05:47 BNT [filer02: cf.fm.launch:info]: replayed event: Launching cluster monitor
Tue Jun 13 08:05:47 BNT [filer02: cf.fm.partner:info]: replayed event: Cluster monitor: partner 'filer01'
Tue Jun 13 08:05:47 BNT [filer02: cf.fsm.takeoverOfPartnerDisabled:notice]: replayed event: Cluster monitor: takeover of filer01 disabled (partner booting)
Tue Jun 13 08:05:52 BNT [filer02: fcp.service.adapter:warning]: replayed event: FCP started but no adapter is present!
Tue Jun 13 08:05:53 BNT [filer02: kern.syslog.msg:notice]: replayed event: The system was down for 165942 seconds
Tue Jun 13 08:05:54 BNT [filer02: disk.failmsg:error]: Disk 0a.40 (N00V0W8L): message received.
Tue Jun 13 08:05:54 BNT [filer02: disk.failmsg:error]: Disk 0a.35 (N00WLAUL): message received.
Tue Jun 13 08:05:54 BNT [filer02: disk.failmsg:error]: Disk 0a.33 (N00W75PL): message received.
Tue Jun 13 08:05:54 BNT [filer02: raid.disk.unload.done:info]: Unload of Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] has completed successfully
Tue Jun 13 08:05:54 BNT [filer02: raid.disk.unload.done:info]: Unload of Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] has completed successfully
Tue Jun 13 08:05:54 BNT [filer02: raid.disk.unload.done:info]: Unload of Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] has completed successfully
Tue Jun 13 08:05:54 BNT [filer02: raid.fdr.failed.ok:info]: Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] successfully deleted from spare pool
Tue Jun 13 08:05:54 BNT [filer02: raid.fdr.failed.ok:info]: Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] successfully deleted from spare pool
Tue Jun 13 08:05:54 BNT [filer02: raid.fdr.failed.ok:info]: Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] successfully deleted from spare pool
Tue Jun 13 08:05:55 BNT [filer02: dfu.firmwareUpToDate:info]: Firmware is up-to-date on all disk drives
Tue Jun 13 08:05:56 BNT [filer02: cf.fsm.takeoverOfPartnerEnabled:notice]: Cluster monitor: takeover of filer01 enabled
Tue Jun 13 08:05:56 BNT [filer02: cf.fsm.takeoverByPartnerEnabled:notice]: Cluster monitor: takeover of filer02 by filer01 enabled
Tue Jun 13 08:05:56 BNT [filer02: perf.archive.start:info]: Performance archiver started. Sampling 22 objects and 198 counters.
Tue Jun 13 08:05:58 BNT [filer02: netif.linkUp:info]: Ethernet e0a: Link up.
Tue Jun 13 08:05:58 BNT [filer02: netif.linkUp:info]: Ethernet e0b: Link up.
Tue Jun 13 08:05:59 BNT [filer02: pvif.switchLink:warning]: vif1: switching to e0a
Tue Jun 13 08:06:00 BNT [filer02: iscsi.service.startup:info]: iSCSI service startup
Tue Jun 13 08:06:00 BNT [filer02: iscsi.notice:notice]: ISCSI: New session from initiator iqn.1991-05.com.microsoft:pcb-722-ad1.pcbl722dom.local at IP addr 10.72.4.1
Tue Jun 13 08:06:01 BNT [filer02: cmds.sysconf.logErr:error]: sysconfig: Unless directed by NetApp Global Services the cf.takeover.on_panic option should be set to On. .
Tue Jun 13 08:06:01 BNT [filer02: raid.rg.recons.missing:notice]: RAID group /aggr1/plex0/rg0 is missing 2 disk(s).
Tue Jun 13 08:06:01 BNT [filer02: raid.rg.recons.cantStart:warning]: The reconstruction cannot start in RAID group /aggr1/plex0/rg0: No 520 bps disk of required size available in spare pool, targeting any  spare pool
Tue Jun 13 08:06:01 BNT [filer02: mgr.boot.disk_done:info]: NetApp Release 7.3.2 boot complete. Last disk update written at Sun Jun 11 10:00:07 BNT 2017
Tue Jun 13 08:06:01 BNT [filer02: monitor.raid.brokenDisk:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken.
Tue Jun 13 08:06:01 BNT [filer02: monitor.brokenDisk.notice:notice]: When two disks are broken in raid_dp volume,  the system shuts down automatically every 24  hours to encourage you to replace the disk.  If you reboot the system it will run for another  24 hours before shutting down. (The 24 hour timeout  may be increased by altering the "raid.timeout"  value using the "options" command.)
Tue Jun 13 08:06:02 BNT [filer02: shelf.config.spha:info]: System is using single path HA attached storage only.
Tue Jun 13 08:06:02 BNT [filer02: diskown.errorReadingOwnership:warning]: error 3 (disk failed) while reading ownership on disk 0a.24 (S/N N00SSSAL)
Tue Jun 13 08:06:02 BNT [filer02: diskown.errorReadingOwnership:warning]: error 3 (disk failed) while reading ownership on disk 0a.26 (S/N N00V0UWL)
Tue Jun 13 08:06:02 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] is still present in the system and should be removed.
Tue Jun 13 08:06:02 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] is still present in the system and should be removed.
Tue Jun 13 08:06:02 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] is still present in the system and should be removed.
Tue Jun 13 08:06:02 BNT [filer02: mgr.boot.reason_ok:notice]: System rebooted.
Tue Jun 13 08:06:24 BNT [filer02: raid.rg.spares.low:warning]: /aggr1/plex0/rg0
Tue Jun 13 08:06:27 BNT [filer02: nbt.nbns.registrationComplete:info]: NBT: All CIFS name registrations have completed for the local server.
Tue Jun 13 08:06:27 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Tue Jun 13 08:06:34 BNT last message repeated 2 times
Tue Jun 13 08:06:34 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201706070800.0) was not posted for host (0), it was dropped from queue due to lack of space
Tue Jun 13 08:06:41 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Tue Jun 13 08:06:41 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201706071200.0) was not posted for host (0), it was dropped from queue due to lack of space
Tue Jun 13 08:06:47 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Tue Jun 13 08:06:47 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201706071600.0) was not posted for host (0), it was dropped from queue due to lack of space
Tue Jun 13 08:07:07 BNT [filer02: monitor.globalStatus.nonCritical:warning]: Disk on adapter 0a, shelf 2, bay 8, failed. Disk on adapter 0a, shelf 2, bay 3, failed. Disk on adapter 0a, shelf 2, bay 1, failed.  There are not enough spare disks. /vol/vol1 is full (using or reserving 100% of space and 0% of inodes, using 92% of reserve).
Tue Jun 13 08:07:10 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Tue Jun 13 08:07:10 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201706072000.0) was not posted for host (0), it was dropped from queue due to lack of space
Tue Jun 13 08:11:16 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Tue Jun 13 08:11:41 BNT [filer02: cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 0a.33 (016F3AB5:B475F4EC:10505E83:02202755:00000000:00000000:00000000:00000000:00000000:00000000) has recently changed or the node (filer02) is missing the disk.
Tue Jun 13 08:11:41 BNT [filer02: cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 0a.35 (9115483F:9DD43BC4:27172773:99F7D8B4:00000000:00000000:00000000:00000000:00000000:00000000) has recently changed or the node (filer02) is missing the disk.
Tue Jun 13 08:11:41 BNT [filer02: cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 0a.40 (072AEB65:C662A9CF:E58B9917:9BFA787B:00000000:00000000:00000000:00000000:00000000:00000000) has recently changed or the node (filer02) is missing the disk.
Tue Jun 13 08:11:42 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Tue Jun 13 08:11:48 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Tue Jun 13 08:11:48 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201706080000.0) was not posted for host (0), it was dropped from queue due to lack of space
Tue Jun 13 08:16:39 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Tue Jun 13 08:20:32 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Tue Jun 13 08:24:14 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Tue Jun 13 08:28:29 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Tue Jun 13 08:32:51 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Tue Jun 13 08:36:41 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Tue Jun 13 08:36:41 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201706080746.0) for host (0) was not posted to NetApp. The system will drop the message
Tue Jun 13 09:00:00 BNT [filer02: kern.uptime.filer:info]:   9:00am up 53 mins, 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 5603 iSCSI ops
Tue Jun 13 09:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 24 hours.
Tue Jun 13 09:13:02 BNT [filer02: cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 0a.33 (016F3AB5:B475F4EC:10505E83:02202755:00000000:00000000:00000000:00000000:00000000:00000000) has recently changed or the node (filer02) is missing the disk.
Tue Jun 13 09:13:02 BNT [filer02: cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 0a.35 (9115483F:9DD43BC4:27172773:99F7D8B4:00000000:00000000:00000000:00000000:00000000:00000000) has recently changed or the node (filer02) is missing the disk.
Tue Jun 13 09:13:02 BNT [filer02: cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 0a.40 (072AEB65:C662A9CF:E58B9917:9BFA787B:00000000:00000000:00000000:00000000:00000000:00000000) has recently changed or the node (filer02) is missing the disk.
Tue Jun 13 10:00:00 BNT [filer02: kern.uptime.filer:info]:  10:00am up  1:53 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 156424 iSCSI ops
Tue Jun 13 10:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 23 hours.
Tue Jun 13 11:00:00 BNT [filer02: kern.uptime.filer:info]:  11:00am up  2:53 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 600457 iSCSI ops
Tue Jun 13 11:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 22 hours.
Tue Jun 13 12:00:00 BNT [filer02: kern.uptime.filer:info]:  12:00pm up  3:53 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 631559 iSCSI ops
Tue Jun 13 12:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 21 hours.
Tue Jun 13 12:00:15 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Tue Jun 13 12:00:27 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SHUTDOWN PENDING (degraded mode))
Tue Jun 13 12:04:13 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SHUTDOWN PENDING (degraded mode))
Tue Jun 13 12:08:11 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SHUTDOWN PENDING (degraded mode))
Tue Jun 13 12:12:27 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SHUTDOWN PENDING (degraded mode))
Tue Jun 13 12:16:52 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SHUTDOWN PENDING (degraded mode))
Tue Jun 13 12:36:38 BNT last message repeated 5 times
Tue Jun 13 12:40:19 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SHUTDOWN PENDING (degraded mode))
Tue Jun 13 12:57:00 BNT last message repeated 4 times
Tue Jun 13 12:57:00 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201706080747.0) for host (0) was not posted to NetApp. The system will drop the message
Tue Jun 13 13:00:00 BNT [filer02: kern.uptime.filer:info]:   1:00pm up  4:53 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 633593 iSCSI ops
Tue Jun 13 13:00:01 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 20 hours.
Tue Jun 13 14:00:00 BNT [filer02: kern.uptime.filer:info]:   2:00pm up  5:53 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 642544 iSCSI ops
Tue Jun 13 14:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 19 hours.
Tue Jun 13 15:00:00 BNT [filer02: kern.uptime.filer:info]:   3:00pm up  6:53 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 728108 iSCSI ops
Tue Jun 13 15:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 18 hours.
Tue Jun 13 16:00:00 BNT [filer02: kern.uptime.filer:info]:   4:00pm up  7:53 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 1049790 iSCSI ops
Tue Jun 13 16:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 17 hours.
Tue Jun 13 16:00:16 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Tue Jun 13 16:00:30 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (REBOOT (power on))
Tue Jun 13 16:04:26 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (REBOOT (power on))
Tue Jun 13 16:08:00 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (REBOOT (power on))
Tue Jun 13 16:12:15 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (REBOOT (power on))
Tue Jun 13 16:16:03 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (REBOOT (power on))
Tue Jun 13 16:35:34 BNT last message repeated 5 times
Tue Jun 13 16:39:23 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (REBOOT (power on))
Tue Jun 13 16:55:36 BNT last message repeated 4 times
Tue Jun 13 16:55:36 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201706080747.1) for host (0) was not posted to NetApp. The system will drop the message
Tue Jun 13 17:00:00 BNT [filer02: kern.uptime.filer:info]:   5:00pm up  8:53 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 1482650 iSCSI ops
Tue Jun 13 17:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 16 hours.
Tue Jun 13 18:00:00 BNT [filer02: kern.uptime.filer:info]:   6:00pm up  9:54 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 1663871 iSCSI ops
Tue Jun 13 18:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 15 hours.
Tue Jun 13 19:00:00 BNT [filer02: kern.uptime.filer:info]:   7:00pm up 10:54 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 1663871 iSCSI ops
Tue Jun 13 19:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 14 hours.
Tue Jun 13 20:00:00 BNT [filer02: kern.uptime.filer:info]:   8:00pm up 11:54 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 1663871 iSCSI ops
Tue Jun 13 20:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 13 hours.
Tue Jun 13 20:00:01 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Tue Jun 13 20:00:15 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SPARES_LOW)
Tue Jun 13 20:03:53 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SPARES_LOW)
Tue Jun 13 20:07:55 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SPARES_LOW)
Tue Jun 13 20:11:26 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SPARES_LOW)
Tue Jun 13 20:15:36 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SPARES_LOW)
Tue Jun 13 20:35:53 BNT last message repeated 5 times
Tue Jun 13 20:40:21 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SPARES_LOW)
Tue Jun 13 20:55:48 BNT last message repeated 4 times
Tue Jun 13 20:55:48 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201706080747.2) for host (0) was not posted to NetApp. The system will drop the message
Tue Jun 13 21:00:00 BNT [filer02: kern.uptime.filer:info]:   9:00pm up 12:54 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 1663873 iSCSI ops
Tue Jun 13 21:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 12 hours.
Tue Jun 13 22:00:00 BNT [filer02: kern.uptime.filer:info]:  10:00pm up 13:54 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 1663873 iSCSI ops
Tue Jun 13 22:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 11 hours.
Tue Jun 13 23:00:00 BNT [filer02: kern.uptime.filer:info]:  11:00pm up 14:54 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 1663873 iSCSI ops
Tue Jun 13 23:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 10 hours.
Wed Jun 14 00:00:00 BNT [filer02: kern.uptime.filer:info]:  12:00am up 15:54 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 1663873 iSCSI ops
Wed Jun 14 00:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 9 hours.
Wed Jun 14 00:00:46 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of nightly.0 snapshot (Snapshot operation failed: No space left on device.)
Wed Jun 14 00:00:54 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (PERFORMANCE SNAPSHOT)
Wed Jun 14 00:04:59 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (PERFORMANCE SNAPSHOT)
Wed Jun 14 00:09:10 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (PERFORMANCE SNAPSHOT)
Wed Jun 14 00:12:55 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (PERFORMANCE SNAPSHOT)
Wed Jun 14 00:16:46 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (PERFORMANCE SNAPSHOT)
Wed Jun 14 00:36:33 BNT last message repeated 5 times
Wed Jun 14 00:40:27 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (PERFORMANCE SNAPSHOT)
Wed Jun 14 00:56:53 BNT last message repeated 4 times
Wed Jun 14 00:56:53 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201706080747.3) for host (0) was not posted to NetApp. The system will drop the message
Wed Jun 14 01:00:00 BNT [filer02: kern.uptime.filer:info]:   1:00am up 16:54 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 1663873 iSCSI ops
Wed Jun 14 01:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 8 hours.
Wed Jun 14 02:00:00 BNT [filer02: kern.uptime.filer:info]:   2:00am up 17:54 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 1663873 iSCSI ops
Wed Jun 14 02:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 7 hours.
Wed Jun 14 03:00:00 BNT [filer02: kern.uptime.filer:info]:   3:00am up 18:54 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 1663873 iSCSI ops
Wed Jun 14 03:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 6 hours.
Wed Jun 14 04:00:00 BNT [filer02: kern.uptime.filer:info]:   4:00am up 19:54 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 1663950 iSCSI ops
Wed Jun 14 04:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 5 hours.
Wed Jun 14 05:00:00 BNT [filer02: kern.uptime.filer:info]:   5:00am up 20:54 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 1663950 iSCSI ops
Wed Jun 14 05:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 4 hours.
Wed Jun 14 06:00:00 BNT [filer02: kern.uptime.filer:info]:   6:00am up 21:54 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 1663950 iSCSI ops
Wed Jun 14 06:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 3 hours.
Wed Jun 14 07:00:00 BNT [filer02: kern.uptime.filer:info]:   7:00am up 22:54 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 1663950 iSCSI ops
Wed Jun 14 07:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 2 hours.
Wed Jun 14 07:46:45 BNT [filer02: ses.status.psWarning:warning]: DS14-Mk2-AT shelf 2 on channel 0a power warning for Power supply 2: non-critical status; DC undervoltage fault. This module is on the rear side of the shelf, at the right.
Wed Jun 14 07:46:51 BNT [filer02: fci.link.break:error]: Link break detected on Fibre Channel adapter 0a.
Wed Jun 14 07:46:55 BNT [filer02: fci.adapter.error:warning]: Fibre Channel adapter driver encountered error "Failed to get loop map" on adapter 0a.
Tue Jun 13 23:47:30 GMT [localhost: rc:notice]: The system was down for 16 seconds
Tue Jun 13 23:47:31 GMT [filer02: cf.fsm.takeoverOfPartnerEnabled:notice]: Cluster monitor: takeover of filer01 enabled
Tue Jun 13 23:47:31 GMT [filer02: cf.fsm.takeoverByPartnerEnabled:notice]: Cluster monitor: takeover of filer02 by filer01 enabled
Wed Jun 14 07:46:55 BNT [filer02: cf.nm.nicTransitionUp:info]: replayed event: Interconnect link 0 is UP
Wed Jun 14 07:47:02 BNT [filer02: fci.initialization.failed:error]: replayed event: Initialization failed on Fibre Channel adapter 0b.
Wed Jun 14 07:47:04 BNT [filer02: nvram.battery.turned.on:info]: replayed event: The NVRAM battery is turned ON. It is turned OFF during system shutdown.
Wed Jun 14 07:47:07 BNT [filer02: diskown.isEnabled:info]: replayed event: software ownership has been enabled for this system
Wed Jun 14 07:47:11 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.2 is a local HA mailbox disk.
Wed Jun 14 07:47:11 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.0 is a local HA mailbox disk.
Wed Jun 14 07:47:11 BNT [filer02: fmmb.instStat.change:info]: replayed event: normal mailbox instance on local side.
Wed Jun 14 07:47:11 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.7 is a partner HA mailbox disk.
Wed Jun 14 07:47:11 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.3 is a partner HA mailbox disk.
Wed Jun 14 07:47:11 BNT [filer02: fmmb.instStat.change:info]: replayed event: normal mailbox instance on partner side.
Wed Jun 14 07:47:12 BNT [filer02: cf.fm.partner:info]: replayed event: Cluster monitor: partner 'filer01'
Wed Jun 14 07:47:12 BNT [filer02: cf.fm.timeMasterStatus:info]: replayed event: Acting as cluster time slave
Wed Jun 14 07:47:12 BNT [filer02: raid.assim.disk.brokenPreAssim:error]: replayed event: Broken Disk 0a.40 Shelf ? Bay ? [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] detected prior to assimilation. It should be removed.
Wed Jun 14 07:47:12 BNT [filer02: raid.assim.disk.brokenPreAssim:error]: replayed event: Broken Disk 0a.35 Shelf ? Bay ? [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] detected prior to assimilation. It should be removed.
Wed Jun 14 07:47:12 BNT [filer02: raid.assim.disk.brokenPreAssim:error]: replayed event: Broken Disk 0a.33 Shelf ? Bay ? [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] detected prior to assimilation. It should be removed.
Wed Jun 14 07:47:13 BNT [filer02: coredump.spare.none:info]: replayed event: No sparecore disk was found.
Wed Jun 14 07:47:13 BNT [filer02: raid.vol.replay.nvram:info]: replayed event: Performing raid replay on volume(s)
Wed Jun 14 07:47:13 BNT [filer02: raid.cksum.replay.summary:info]: replayed event: Replayed 0 checksum blocks.
Wed Jun 14 07:47:13 BNT [filer02: raid.stripe.replay.summary:info]: replayed event: Replayed 0 stripes.
Wed Jun 14 07:47:25 BNT [filer02: cf.fm.launch:info]: replayed event: Launching cluster monitor
Wed Jun 14 07:47:25 BNT [filer02: cf.fm.partner:info]: replayed event: Cluster monitor: partner 'filer01'
Wed Jun 14 07:47:29 BNT [filer02: cf.fsm.takeoverOfPartnerDisabled:notice]: replayed event: Cluster monitor: takeover of filer01 disabled (partner booting)
Wed Jun 14 07:47:30 BNT [filer02: fcp.service.adapter:warning]: replayed event: FCP started but no adapter is present!
Wed Jun 14 07:47:30 BNT [filer02: cf.fsm.takeoverByPartnerDisabled:notice]: replayed event: Cluster monitor: takeover of  by filer01 disabled (partner booting)
Wed Jun 14 07:47:30 BNT [filer02: kern.syslog.msg:notice]: replayed event: The system was down for 16 seconds
Wed Jun 14 07:47:31 BNT [filer02: cf.fsm.takeoverOfPartnerEnabled:notice]: replayed event: Cluster monitor: takeover of filer01 enabled
Wed Jun 14 07:47:31 BNT [filer02: cf.fsm.takeoverByPartnerEnabled:notice]: replayed event: Cluster monitor: takeover of filer02 by filer01 enabled
Wed Jun 14 07:47:32 BNT [filer02: disk.failmsg:error]: Disk 0a.40 (N00V0W8L): message received.
Wed Jun 14 07:47:32 BNT [filer02: disk.failmsg:error]: Disk 0a.35 (N00WLAUL): message received.
Wed Jun 14 07:47:32 BNT [filer02: disk.failmsg:error]: Disk 0a.33 (N00W75PL): message received.
Wed Jun 14 07:47:32 BNT [filer02: raid.disk.unload.done:info]: Unload of Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] has completed successfully
Wed Jun 14 07:47:32 BNT [filer02: raid.disk.unload.done:info]: Unload of Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] has completed successfully
Wed Jun 14 07:47:32 BNT [filer02: raid.disk.unload.done:info]: Unload of Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] has completed successfully
Wed Jun 14 07:47:32 BNT [filer02: raid.fdr.failed.ok:info]: Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] successfully deleted from spare pool
Wed Jun 14 07:47:32 BNT [filer02: raid.fdr.failed.ok:info]: Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] successfully deleted from spare pool
Wed Jun 14 07:47:32 BNT [filer02: raid.fdr.failed.ok:info]: Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] successfully deleted from spare pool
Wed Jun 14 07:47:32 BNT [filer02: dfu.firmwareUpToDate:info]: Firmware is up-to-date on all disk drives
Wed Jun 14 07:47:34 BNT [filer02: perf.archive.start:info]: Performance archiver started. Sampling 22 objects and 198 counters.
Wed Jun 14 07:47:36 BNT [filer02: netif.linkUp:info]: Ethernet e0b: Link up.
Wed Jun 14 07:47:36 BNT [filer02: netif.linkUp:info]: Ethernet e0a: Link up.
Wed Jun 14 07:47:37 BNT [filer02: pvif.switchLink:warning]: vif1: switching to e0a
Wed Jun 14 07:47:38 BNT [filer02: rc:info]: relog syslog Wed Jun 14 07:46:56 BNT [filer02: fci.adapter.error:warning]: Fibre Channel adapter driver encountered error "Failed t
Wed Jun 14 07:47:38 BNT [filer02: rc:info]: relog syslog Wed Jun 14 07:46:56 BNT [filer02: fci.adapter.reset:debug]: Resetting Fibre Channel adapter 0a.
Wed Jun 14 07:47:38 BNT [filer02: iscsi.service.startup:info]: iSCSI service startup
Wed Jun 14 07:47:39 BNT [filer02: cmds.sysconf.logErr:error]: sysconfig: Unless directed by NetApp Global Services the cf.takeover.on_panic option should be set to On. .
Wed Jun 14 07:47:39 BNT [filer02: raid.rg.recons.missing:notice]: RAID group /aggr1/plex0/rg0 is missing 2 disk(s).
Wed Jun 14 07:47:39 BNT [filer02: raid.rg.recons.cantStart:warning]: The reconstruction cannot start in RAID group /aggr1/plex0/rg0: No 520 bps disk of required size available in spare pool, targeting any  spare pool
Wed Jun 14 07:47:39 BNT [filer02: mgr.boot.disk_done:info]: NetApp Release 7.3.2 boot complete. Last disk update written at Wed Jun 14 07:46:43 BNT 2017
Wed Jun 14 07:47:39 BNT [filer02: monitor.raid.brokenDisk:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken.
Wed Jun 14 07:47:39 BNT [filer02: monitor.brokenDisk.notice:notice]: When two disks are broken in raid_dp volume,  the system shuts down automatically every 24  hours to encourage you to replace the disk.  If you reboot the system it will run for another  24 hours before shutting down. (The 24 hour timeout  may be increased by altering the "raid.timeout"  value using the "options" command.)
Wed Jun 14 07:47:39 BNT [filer02: diskown.errorReadingOwnership:warning]: error 3 (disk failed) while reading ownership on disk 0a.24 (S/N N00SSSAL)
Wed Jun 14 07:47:39 BNT [filer02: diskown.errorReadingOwnership:warning]: error 3 (disk failed) while reading ownership on disk 0a.26 (S/N N00V0UWL)
Wed Jun 14 07:47:40 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] is still present in the system and should be removed.
Wed Jun 14 07:47:40 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] is still present in the system and should be removed.
Wed Jun 14 07:47:40 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] is still present in the system and should be removed.
Wed Jun 14 07:47:40 BNT [filer02: shelf.config.spha:info]: System is using single path HA attached storage only.
Wed Jun 14 07:47:40 BNT [filer02: mgr.boot.reason_ok:notice]: System rebooted after power-on.
Wed Jun 14 07:47:50 BNT [filer02: iscsi.notice:notice]: ISCSI: New session from initiator iqn.1991-05.com.microsoft:pcb-722-ad1.pcbl722dom.local at IP addr 10.72.4.1
Wed Jun 14 07:48:02 BNT [filer02: raid.rg.spares.low:warning]: /aggr1/plex0/rg0
Wed Jun 14 07:48:02 BNT [filer02: monitor.globalStatus.nonCritical:warning]: Disk on adapter 0a, shelf 2, bay 8, failed. Disk on adapter 0a, shelf 2, bay 3, failed. Disk on adapter 0a, shelf 2, bay 1, failed.  There are not enough spare disks. /vol/vol1 is full (using or reserving 100% of space and 0% of inodes, using 92% of reserve).
Wed Jun 14 07:48:04 BNT [filer02: nbt.nbns.registrationComplete:info]: NBT: All CIFS name registrations have completed for the local server.
Wed Jun 14 07:48:07 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Wed Jun 14 07:48:14 BNT last message repeated 2 times
Wed Jun 14 07:48:14 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201706080800.0) was not posted for host (0), it was dropped from queue due to lack of space
Wed Jun 14 07:48:20 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Wed Jun 14 07:48:20 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201706081200.0) was not posted for host (0), it was dropped from queue due to lack of space
Wed Jun 14 07:48:26 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Wed Jun 14 07:48:26 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201706081600.0) was not posted for host (0), it was dropped from queue due to lack of space
Wed Jun 14 07:48:48 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Wed Jun 14 07:48:48 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201706082000.0) was not posted for host (0), it was dropped from queue due to lack of space
Wed Jun 14 07:53:44 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Wed Jun 14 07:58:30 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Wed Jun 14 08:00:00 BNT [filer02: kern.uptime.filer:info]:   8:00am up 12 mins, 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 10 iSCSI ops
Wed Jun 14 08:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 24 hours.
Wed Jun 14 08:00:00 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Wed Jun 14 08:00:07 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Wed Jun 14 08:00:07 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201706090000.0) was not posted for host (0), it was dropped from queue due to lack of space
Wed Jun 14 08:04:12 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Wed Jun 14 08:08:42 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Wed Jun 14 08:12:28 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Wed Jun 14 08:16:24 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Wed Jun 14 08:20:24 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Wed Jun 14 08:24:46 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Wed Jun 14 08:24:46 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201706090801.0) for host (0) was not posted to NetApp. The system will drop the message
Wed Jun 14 09:00:00 BNT [filer02: kern.uptime.filer:info]:   9:00am up  1:12 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 2991 iSCSI ops
Wed Jun 14 09:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 23 hours.
Wed Jun 14 10:00:00 BNT [filer02: kern.uptime.filer:info]:  10:00am up  2:12 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 211623 iSCSI ops
Wed Jun 14 10:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 22 hours.
Wed Jun 14 11:00:00 BNT [filer02: kern.uptime.filer:info]:  11:00am up  3:12 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 617176 iSCSI ops
Wed Jun 14 11:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 21 hours.
Wed Jun 14 12:00:00 BNT [filer02: kern.uptime.filer:info]:  12:00pm up  4:12 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 617731 iSCSI ops
Wed Jun 14 12:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 20 hours.
Wed Jun 14 12:00:45 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Wed Jun 14 12:00:56 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SHUTDOWN PENDING (degraded mode))
Wed Jun 14 12:05:23 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SHUTDOWN PENDING (degraded mode))
Wed Jun 14 12:09:15 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SHUTDOWN PENDING (degraded mode))
Wed Jun 14 12:13:27 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SHUTDOWN PENDING (degraded mode))
Wed Jun 14 12:17:00 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SHUTDOWN PENDING (degraded mode))
Wed Jun 14 12:37:17 BNT last message repeated 5 times
Wed Jun 14 12:41:17 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SHUTDOWN PENDING (degraded mode))
Wed Jun 14 12:57:54 BNT last message repeated 4 times
Wed Jun 14 12:57:54 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201706090801.1) for host (0) was not posted to NetApp. The system will drop the message
Wed Jun 14 13:00:00 BNT [filer02: kern.uptime.filer:info]:   1:00pm up  5:12 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 618179 iSCSI ops
Wed Jun 14 13:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 19 hours.
Wed Jun 14 14:00:00 BNT [filer02: kern.uptime.filer:info]:   2:00pm up  6:12 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 622330 iSCSI ops
Wed Jun 14 14:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 18 hours.
Wed Jun 14 15:00:00 BNT [filer02: kern.uptime.filer:info]:   3:00pm up  7:12 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 622350 iSCSI ops
Wed Jun 14 15:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 17 hours.
Wed Jun 14 16:00:00 BNT [filer02: kern.uptime.filer:info]:   4:00pm up  8:12 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 622566 iSCSI ops
Wed Jun 14 16:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 16 hours.
Wed Jun 14 16:00:46 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Wed Jun 14 16:01:00 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (REBOOT)
Wed Jun 14 16:05:10 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (REBOOT)
Wed Jun 14 16:09:14 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (REBOOT)
Wed Jun 14 16:13:18 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (REBOOT)
Wed Jun 14 16:17:13 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (REBOOT)
Wed Jun 14 16:33:35 BNT last message repeated 4 times
Wed Jun 14 16:37:58 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (REBOOT)
Wed Jun 14 16:57:35 BNT last message repeated 5 times
Wed Jun 14 16:57:35 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201706090801.2) for host (0) was not posted to NetApp. The system will drop the message
Wed Jun 14 17:00:00 BNT [filer02: kern.uptime.filer:info]:   5:00pm up  9:12 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 622567 iSCSI ops
Wed Jun 14 17:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 15 hours.
Wed Jun 14 18:00:00 BNT [filer02: kern.uptime.filer:info]:   6:00pm up 10:12 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 622567 iSCSI ops
Wed Jun 14 18:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 14 hours.
Wed Jun 14 19:00:00 BNT [filer02: kern.uptime.filer:info]:   7:00pm up 11:12 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 622567 iSCSI ops
Wed Jun 14 19:00:01 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 13 hours.
Wed Jun 14 20:00:00 BNT [filer02: kern.uptime.filer:info]:   8:00pm up 12:12 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 622567 iSCSI ops
Wed Jun 14 20:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 12 hours.
Wed Jun 14 20:00:31 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Wed Jun 14 20:00:45 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SPARES_LOW)
Wed Jun 14 20:04:28 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SPARES_LOW)
Wed Jun 14 20:08:05 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SPARES_LOW)
Wed Jun 14 20:11:38 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SPARES_LOW)
Wed Jun 14 20:16:07 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SPARES_LOW)
Wed Jun 14 20:36:19 BNT last message repeated 5 times
Wed Jun 14 20:40:04 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SPARES_LOW)
Wed Jun 14 20:56:29 BNT last message repeated 4 times
Wed Jun 14 20:56:29 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201706090801.3) for host (0) was not posted to NetApp. The system will drop the message
Wed Jun 14 21:00:00 BNT [filer02: kern.uptime.filer:info]:   9:00pm up 13:12 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 622567 iSCSI ops
Wed Jun 14 21:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 11 hours.
Wed Jun 14 22:00:00 BNT [filer02: kern.uptime.filer:info]:  10:00pm up 14:12 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 622567 iSCSI ops
Wed Jun 14 22:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 10 hours.
Wed Jun 14 23:00:00 BNT [filer02: kern.uptime.filer:info]:  11:00pm up 15:12 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 622567 iSCSI ops
Wed Jun 14 23:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 9 hours.
Thu Jun 15 00:00:00 BNT [filer02: kern.uptime.filer:info]:  12:00am up 16:12 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 622567 iSCSI ops
Thu Jun 15 00:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 8 hours.
Thu Jun 15 00:00:16 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of nightly.0 snapshot (Snapshot operation failed: No space left on device.)
Thu Jun 15 00:00:24 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (PERFORMANCE SNAPSHOT)
Thu Jun 15 00:04:52 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (PERFORMANCE SNAPSHOT)
Thu Jun 15 00:08:49 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (PERFORMANCE SNAPSHOT)
Thu Jun 15 00:12:36 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (PERFORMANCE SNAPSHOT)
Thu Jun 15 00:16:31 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (PERFORMANCE SNAPSHOT)
Thu Jun 15 00:36:06 BNT last message repeated 5 times
Thu Jun 15 00:39:44 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (PERFORMANCE SNAPSHOT)
Thu Jun 15 00:56:27 BNT last message repeated 4 times
Thu Jun 15 00:56:27 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201706090802.0) for host (0) was not posted to NetApp. The system will drop the message
Thu Jun 15 01:00:00 BNT [filer02: kern.uptime.filer:info]:   1:00am up 17:12 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 622567 iSCSI ops
Thu Jun 15 01:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 7 hours.
Thu Jun 15 02:00:00 BNT [filer02: kern.uptime.filer:info]:   2:00am up 18:12 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 622567 iSCSI ops
Thu Jun 15 02:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 6 hours.
Thu Jun 15 03:00:00 BNT [filer02: kern.uptime.filer:info]:   3:00am up 19:12 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 622567 iSCSI ops
Thu Jun 15 03:00:01 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 5 hours.
Thu Jun 15 04:00:00 BNT [filer02: kern.uptime.filer:info]:   4:00am up 20:12 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 622633 iSCSI ops
Thu Jun 15 04:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 4 hours.
Thu Jun 15 05:00:00 BNT [filer02: kern.uptime.filer:info]:   5:00am up 21:12 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 622633 iSCSI ops
Thu Jun 15 05:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 3 hours.
Thu Jun 15 06:00:00 BNT [filer02: kern.uptime.filer:info]:   6:00am up 22:12 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 622633 iSCSI ops
Thu Jun 15 06:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 2 hours.
Thu Jun 15 07:00:00 BNT [filer02: kern.uptime.filer:info]:   7:00am up 23:13 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 622633 iSCSI ops
Thu Jun 15 07:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 1 hour.
Thu Jun 15 07:00:00 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] is still present in the system and should be removed.
Thu Jun 15 07:00:00 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] is still present in the system and should be removed.
Thu Jun 15 07:00:00 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] is still present in the system and should be removed.
Thu Jun 15 07:31:15 BNT [filer02: fci.link.break:error]: Link break detected on Fibre Channel adapter 0a.
Thu Jun 15 07:31:16 BNT [filer02: fci.adapter.error:warning]: Fibre Channel adapter driver encountered error "Failed to get loop map" on adapter 0a.
Wed Jun 14 23:32:04 GMT [localhost: cf.fsm.takeoverOfPartnerEnabled:notice]: Cluster monitor: takeover of filer01 enabled
Wed Jun 14 23:32:04 GMT [localhost: rc:notice]: The system was down for 11 seconds
Thu Jun 15 07:32:05 BNT [filer02: cf.fsm.takeoverByPartnerEnabled:notice]: Cluster monitor: takeover of filer02 by filer01 enabled
Thu Jun 15 07:31:28 BNT [filer02: cf.nm.nicTransitionUp:info]: replayed event: Interconnect link 0 is UP
Thu Jun 15 07:31:35 BNT [filer02: fci.initialization.failed:error]: replayed event: Initialization failed on Fibre Channel adapter 0b.
Thu Jun 15 07:31:37 BNT [filer02: nvram.battery.turned.on:info]: replayed event: The NVRAM battery is turned ON. It is turned OFF during system shutdown.
Thu Jun 15 07:31:40 BNT [filer02: diskown.isEnabled:info]: replayed event: software ownership has been enabled for this system
Thu Jun 15 07:31:44 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.2 is a local HA mailbox disk.
Thu Jun 15 07:31:44 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.0 is a local HA mailbox disk.
Thu Jun 15 07:31:44 BNT [filer02: fmmb.instStat.change:info]: replayed event: normal mailbox instance on local side.
Thu Jun 15 07:31:44 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.7 is a partner HA mailbox disk.
Thu Jun 15 07:31:44 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.3 is a partner HA mailbox disk.
Thu Jun 15 07:31:44 BNT [filer02: fmmb.instStat.change:info]: replayed event: normal mailbox instance on partner side.
Thu Jun 15 07:31:45 BNT [filer02: cf.fm.partner:info]: replayed event: Cluster monitor: partner 'filer01'
Thu Jun 15 07:31:45 BNT [filer02: cf.fm.timeMasterStatus:info]: replayed event: Acting as cluster time slave
Thu Jun 15 07:31:45 BNT [filer02: raid.assim.disk.brokenPreAssim:error]: replayed event: Broken Disk 0a.40 Shelf ? Bay ? [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] detected prior to assimilation. It should be removed.
Thu Jun 15 07:31:45 BNT [filer02: raid.assim.disk.brokenPreAssim:error]: replayed event: Broken Disk 0a.35 Shelf ? Bay ? [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] detected prior to assimilation. It should be removed.
Thu Jun 15 07:31:45 BNT [filer02: raid.assim.disk.brokenPreAssim:error]: replayed event: Broken Disk 0a.33 Shelf ? Bay ? [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] detected prior to assimilation. It should be removed.
Thu Jun 15 07:31:46 BNT [filer02: coredump.spare.none:info]: replayed event: No sparecore disk was found.
Thu Jun 15 07:31:46 BNT [filer02: raid.vol.replay.nvram:info]: replayed event: Performing raid replay on volume(s)
Thu Jun 15 07:31:46 BNT [filer02: raid.cksum.replay.summary:info]: replayed event: Replayed 0 checksum blocks.
Thu Jun 15 07:31:46 BNT [filer02: raid.stripe.replay.summary:info]: replayed event: Replayed 0 stripes.
Thu Jun 15 07:31:58 BNT [filer02: cf.fm.launch:info]: replayed event: Launching cluster monitor
Thu Jun 15 07:31:58 BNT [filer02: cf.fm.partner:info]: replayed event: Cluster monitor: partner 'filer01'
Thu Jun 15 07:32:02 BNT [filer02: cf.fsm.takeoverOfPartnerDisabled:notice]: replayed event: Cluster monitor: takeover of filer01 disabled (partner booting)
Thu Jun 15 07:32:04 BNT [filer02: fcp.service.adapter:warning]: replayed event: FCP started but no adapter is present!
Thu Jun 15 07:32:04 BNT [filer02: cf.fsm.takeoverOfPartnerEnabled:notice]: replayed event: Cluster monitor: takeover of filer01 enabled
Thu Jun 15 07:32:04 BNT [filer02: kern.syslog.msg:notice]: replayed event: The system was down for 11 seconds
Thu Jun 15 07:32:05 BNT [filer02: cf.fsm.takeoverByPartnerEnabled:notice]: replayed event: Cluster monitor: takeover of filer02 by filer01 enabled
Thu Jun 15 07:32:05 BNT [filer02: disk.failmsg:error]: Disk 0a.40 (N00V0W8L): message received.
Thu Jun 15 07:32:05 BNT [filer02: disk.failmsg:error]: Disk 0a.35 (N00WLAUL): message received.
Thu Jun 15 07:32:05 BNT [filer02: disk.failmsg:error]: Disk 0a.33 (N00W75PL): message received.
Thu Jun 15 07:32:05 BNT [filer02: raid.disk.unload.done:info]: Unload of Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] has completed successfully
Thu Jun 15 07:32:05 BNT [filer02: raid.disk.unload.done:info]: Unload of Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] has completed successfully
Thu Jun 15 07:32:05 BNT [filer02: raid.disk.unload.done:info]: Unload of Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] has completed successfully
Thu Jun 15 07:32:05 BNT [filer02: raid.fdr.failed.ok:info]: Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] successfully deleted from spare pool
Thu Jun 15 07:32:05 BNT [filer02: raid.fdr.failed.ok:info]: Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] successfully deleted from spare pool
Thu Jun 15 07:32:05 BNT [filer02: raid.fdr.failed.ok:info]: Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] successfully deleted from spare pool
Thu Jun 15 07:32:06 BNT [filer02: dfu.firmwareUpToDate:info]: Firmware is up-to-date on all disk drives
Thu Jun 15 07:32:07 BNT [filer02: perf.archive.start:info]: Performance archiver started. Sampling 22 objects and 198 counters.
Thu Jun 15 07:32:09 BNT [filer02: netif.linkUp:info]: Ethernet e0a: Link up.
Thu Jun 15 07:32:09 BNT [filer02: netif.linkUp:info]: Ethernet e0b: Link up.
Thu Jun 15 07:32:10 BNT [filer02: pvif.switchLink:warning]: vif1: switching to e0a
Thu Jun 15 07:32:11 BNT [filer02: rc:info]: relog syslog Thu Jun 15 07:31:31 BNT [filer02: fci.adapter.error:warning]: Fibre Channel adapter driver encountered error "Failed t
Thu Jun 15 07:32:11 BNT [filer02: rc:info]: relog syslog Thu Jun 15 07:31:31 BNT [filer02: fci.adapter.reset:debug]: Resetting Fibre Channel adapter 0a.
Thu Jun 15 07:32:12 BNT [filer02: iscsi.service.startup:info]: iSCSI service startup
Thu Jun 15 07:32:12 BNT [filer02: cmds.sysconf.logErr:error]: sysconfig: Unless directed by NetApp Global Services the cf.takeover.on_panic option should be set to On. .
Thu Jun 15 07:32:12 BNT [filer02: raid.rg.recons.missing:notice]: RAID group /aggr1/plex0/rg0 is missing 2 disk(s).
Thu Jun 15 07:32:12 BNT [filer02: raid.rg.recons.cantStart:warning]: The reconstruction cannot start in RAID group /aggr1/plex0/rg0: No 520 bps disk of required size available in spare pool, targeting any  spare pool
Thu Jun 15 07:32:12 BNT [filer02: mgr.boot.disk_done:info]: NetApp Release 7.3.2 boot complete. Last disk update written at Thu Jun 15 07:31:02 BNT 2017
Thu Jun 15 07:32:12 BNT [filer02: monitor.raid.brokenDisk:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken.
Thu Jun 15 07:32:12 BNT [filer02: monitor.brokenDisk.notice:notice]: When two disks are broken in raid_dp volume,  the system shuts down automatically every 24  hours to encourage you to replace the disk.  If you reboot the system it will run for another  24 hours before shutting down. (The 24 hour timeout  may be increased by altering the "raid.timeout"  value using the "options" command.)
Thu Jun 15 07:32:13 BNT [filer02: diskown.errorReadingOwnership:warning]: error 3 (disk failed) while reading ownership on disk 0a.24 (S/N N00SSSAL)
Thu Jun 15 07:32:13 BNT [filer02: diskown.errorReadingOwnership:warning]: error 3 (disk failed) while reading ownership on disk 0a.26 (S/N N00V0UWL)
Thu Jun 15 07:32:13 BNT [filer02: shelf.config.spha:info]: System is using single path HA attached storage only.
Thu Jun 15 07:32:13 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] is still present in the system and should be removed.
Thu Jun 15 07:32:13 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] is still present in the system and should be removed.
Thu Jun 15 07:32:13 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] is still present in the system and should be removed.
Thu Jun 15 07:32:13 BNT [filer02: mgr.boot.reason_ok:notice]: System rebooted after power-on.
Thu Jun 15 07:32:35 BNT [filer02: raid.rg.spares.low:warning]: /aggr1/plex0/rg0
Thu Jun 15 07:32:38 BNT [filer02: nbt.nbns.registrationComplete:info]: NBT: All CIFS name registrations have completed for the local server.
Thu Jun 15 07:32:40 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (CLUSTER ERROR: DISK/SHELF COUNT MISMATCH)
Thu Jun 15 07:32:47 BNT last message repeated 2 times
Thu Jun 15 07:32:47 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201706090806.0) was not posted for host (0), it was dropped from queue due to lack of space
Thu Jun 15 07:32:53 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu Jun 15 07:32:53 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201706091200.0) was not posted for host (0), it was dropped from queue due to lack of space
Thu Jun 15 07:32:59 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu Jun 15 07:32:59 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201706091600.0) was not posted for host (0), it was dropped from queue due to lack of space
Thu Jun 15 07:33:06 BNT [filer02: monitor.globalStatus.nonCritical:warning]: Disk on adapter 0a, shelf 2, bay 8, failed. Disk on adapter 0a, shelf 2, bay 3, failed. Disk on adapter 0a, shelf 2, bay 1, failed.  There are not enough spare disks. /vol/vol1 is full (using or reserving 100% of space and 0% of inodes, using 92% of reserve).
Thu Jun 15 07:33:21 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu Jun 15 07:33:21 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201706092000.0) was not posted for host (0), it was dropped from queue due to lack of space
Thu Jun 15 07:37:57 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu Jun 15 07:41:53 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu Jun 15 07:45:28 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu Jun 15 07:49:02 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu Jun 15 07:53:30 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu Jun 15 07:56:04 BNT [filer02: iscsi.notice:notice]: ISCSI: New session from initiator iqn.1991-05.com.microsoft:pcb-722-ad1.pcbl722dom.local at IP addr 10.72.4.1
Thu Jun 15 07:57:29 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu Jun 15 08:00:00 BNT [filer02: kern.uptime.filer:info]:   8:00am up 27 mins, 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 10 iSCSI ops
Thu Jun 15 08:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 24 hours.
Thu Jun 15 08:00:30 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Thu Jun 15 08:00:38 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu Jun 15 08:00:39 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201706100000.0) was not posted for host (0), it was dropped from queue due to lack of space
Thu Jun 15 08:05:06 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu Jun 15 08:08:50 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu Jun 15 08:08:50 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201706100800.0) for host (0) was not posted to NetApp. The system will drop the message
Thu Jun 15 09:00:00 BNT [filer02: kern.uptime.filer:info]:   9:00am up  1:27 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 305 iSCSI ops
Thu Jun 15 09:00:01 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 23 hours.
Thu Jun 15 10:00:00 BNT [filer02: kern.uptime.filer:info]:  10:00am up  2:27 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 5601 iSCSI ops
Thu Jun 15 10:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 22 hours.
Thu Jun 15 11:00:00 BNT [filer02: kern.uptime.filer:info]:  11:00am up  3:27 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 5775 iSCSI ops
Thu Jun 15 11:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 21 hours.
Thu Jun 15 12:00:00 BNT [filer02: kern.uptime.filer:info]:  12:00pm up  4:27 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 6095 iSCSI ops
Thu Jun 15 12:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 20 hours.
Thu Jun 15 12:00:15 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Thu Jun 15 12:00:30 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Thu Jun 15 12:04:22 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Thu Jun 15 12:08:33 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Thu Jun 15 12:12:16 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Thu Jun 15 12:16:02 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Thu Jun 15 12:36:16 BNT last message repeated 5 times
Thu Jun 15 12:40:03 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Thu Jun 15 12:55:31 BNT last message repeated 4 times
Thu Jun 15 12:55:31 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201706100930.0) for host (0) was not posted to NetApp. The system will drop the message
Thu Jun 15 13:00:00 BNT [filer02: kern.uptime.filer:info]:   1:00pm up  5:27 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 14948 iSCSI ops
Thu Jun 15 13:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 19 hours.
Thu Jun 15 14:00:00 BNT [filer02: kern.uptime.filer:info]:   2:00pm up  6:27 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 20744 iSCSI ops
Thu Jun 15 14:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 18 hours.
Thu Jun 15 15:00:00 BNT [filer02: kern.uptime.filer:info]:   3:00pm up  7:27 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 20745 iSCSI ops
Thu Jun 15 15:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 17 hours.
Thu Jun 15 16:00:00 BNT [filer02: kern.uptime.filer:info]:   4:00pm up  8:27 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 20745 iSCSI ops
Thu Jun 15 16:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 16 hours.
Thu Jun 15 16:00:16 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Thu Jun 15 16:00:31 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SHUTDOWN PENDING (degraded mode))
Thu Jun 15 16:04:08 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SHUTDOWN PENDING (degraded mode))
Thu Jun 15 16:08:16 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SHUTDOWN PENDING (degraded mode))
Thu Jun 15 16:12:09 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SHUTDOWN PENDING (degraded mode))
Thu Jun 15 16:15:54 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SHUTDOWN PENDING (degraded mode))
Thu Jun 15 16:35:12 BNT last message repeated 5 times
Thu Jun 15 16:39:19 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SHUTDOWN PENDING (degraded mode))
Thu Jun 15 16:55:18 BNT last message repeated 4 times
Thu Jun 15 16:55:18 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201706100930.1) for host (0) was not posted to NetApp. The system will drop the message
Thu Jun 15 17:00:00 BNT [filer02: kern.uptime.filer:info]:   5:00pm up  9:27 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 20745 iSCSI ops
Thu Jun 15 17:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 15 hours.
Thu Jun 15 18:00:00 BNT [filer02: kern.uptime.filer:info]:   6:00pm up 10:28 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 20745 iSCSI ops
Thu Jun 15 18:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 14 hours.
Thu Jun 15 19:00:00 BNT [filer02: kern.uptime.filer:info]:   7:00pm up 11:28 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 20745 iSCSI ops
Thu Jun 15 19:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 13 hours.
Thu Jun 15 20:00:00 BNT [filer02: kern.uptime.filer:info]:   8:00pm up 12:28 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 20745 iSCSI ops
Thu Jun 15 20:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 12 hours.
Thu Jun 15 20:00:01 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Thu Jun 15 20:00:21 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (REBOOT)
Thu Jun 15 20:03:54 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (REBOOT)
Thu Jun 15 20:08:18 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (REBOOT)
Thu Jun 15 20:12:18 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (REBOOT)
Thu Jun 15 20:16:17 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (REBOOT)
Thu Jun 15 20:35:39 BNT last message repeated 5 times
Thu Jun 15 20:39:37 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (REBOOT)
Thu Jun 15 20:55:43 BNT last message repeated 4 times
Thu Jun 15 20:55:43 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201706100931.0) for host (0) was not posted to NetApp. The system will drop the message
Thu Jun 15 21:00:00 BNT [filer02: kern.uptime.filer:info]:   9:00pm up 13:28 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 20745 iSCSI ops
Thu Jun 15 21:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 11 hours.
Thu Jun 15 22:00:00 BNT [filer02: kern.uptime.filer:info]:  10:00pm up 14:28 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 20745 iSCSI ops
Thu Jun 15 22:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 10 hours.
Thu Jun 15 23:00:00 BNT [filer02: kern.uptime.filer:info]:  11:00pm up 15:28 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 20745 iSCSI ops
Thu Jun 15 23:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 9 hours.
Fri Jun 16 00:00:00 BNT [filer02: kern.uptime.filer:info]:  12:00am up 16:28 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 20745 iSCSI ops
Fri Jun 16 00:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 8 hours.
Fri Jun 16 00:00:00 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of nightly.0 snapshot (Snapshot operation failed: No space left on device.)
Fri Jun 16 00:00:11 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SPARES_LOW)
Fri Jun 16 00:03:51 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SPARES_LOW)
Fri Jun 16 00:07:46 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SPARES_LOW)
Fri Jun 16 00:11:34 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SPARES_LOW)
Fri Jun 16 00:15:49 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SPARES_LOW)
Fri Jun 16 00:34:36 BNT last message repeated 5 times
Fri Jun 16 00:38:15 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SPARES_LOW)
Fri Jun 16 00:53:58 BNT last message repeated 4 times
Fri Jun 16 00:53:58 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201706100931.1) for host (0) was not posted to NetApp. The system will drop the message
Fri Jun 16 01:00:00 BNT [filer02: kern.uptime.filer:info]:   1:00am up 17:28 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 20745 iSCSI ops
Fri Jun 16 01:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 7 hours.
Fri Jun 16 02:00:00 BNT [filer02: kern.uptime.filer:info]:   2:00am up 18:28 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 20745 iSCSI ops
Fri Jun 16 02:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 6 hours.
Fri Jun 16 03:00:00 BNT [filer02: kern.uptime.filer:info]:   3:00am up 19:28 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 20745 iSCSI ops
Fri Jun 16 03:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 5 hours.
Fri Jun 16 04:00:00 BNT [filer02: kern.uptime.filer:info]:   4:00am up 20:28 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 20745 iSCSI ops
Fri Jun 16 04:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 4 hours.
Fri Jun 16 05:00:00 BNT [filer02: kern.uptime.filer:info]:   5:00am up 21:28 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 20745 iSCSI ops
Fri Jun 16 05:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 3 hours.
Fri Jun 16 06:00:00 BNT [filer02: kern.uptime.filer:info]:   6:00am up 22:28 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 20745 iSCSI ops
Fri Jun 16 06:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 2 hours.
Fri Jun 16 07:00:00 BNT [filer02: kern.uptime.filer:info]:   7:00am up 23:28 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 20745 iSCSI ops
Fri Jun 16 07:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 1 hour.
Fri Jun 16 07:00:00 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] is still present in the system and should be removed.
Fri Jun 16 07:00:00 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] is still present in the system and should be removed.
Fri Jun 16 07:00:00 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] is still present in the system and should be removed.
Fri Jun 16 07:27:35 BNT [filer02: fci.link.break:error]: Link break detected on Fibre Channel adapter 0a.
Fri Jun 16 07:27:39 BNT [filer02: fci.adapter.error:warning]: Fibre Channel adapter driver encountered error "Failed to get loop map" on adapter 0a.
Thu Jun 15 23:28:18 GMT [localhost: rc:notice]: The system was down for 21 seconds
Thu Jun 15 23:28:18 GMT [localhost: cf.fsm.takeoverOfPartnerEnabled:notice]: Cluster monitor: takeover of filer01 enabled
Fri Jun 16 07:27:42 BNT [filer02: cf.nm.nicTransitionUp:info]: replayed event: Interconnect link 0 is UP
Fri Jun 16 07:27:49 BNT [filer02: fci.initialization.failed:error]: replayed event: Initialization failed on Fibre Channel adapter 0b.
Fri Jun 16 07:28:19 BNT [filer02: cf.fsm.takeoverByPartnerEnabled:notice]: Cluster monitor: takeover of filer02 by filer01 enabled
Fri Jun 16 07:27:51 BNT [filer02: nvram.battery.turned.on:info]: replayed event: The NVRAM battery is turned ON. It is turned OFF during system shutdown.
Fri Jun 16 07:27:54 BNT [filer02: diskown.isEnabled:info]: replayed event: software ownership has been enabled for this system
Fri Jun 16 07:27:58 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.2 is a local HA mailbox disk.
Fri Jun 16 07:27:58 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.0 is a local HA mailbox disk.
Fri Jun 16 07:27:58 BNT [filer02: fmmb.instStat.change:info]: replayed event: normal mailbox instance on local side.
Fri Jun 16 07:27:58 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.7 is a partner HA mailbox disk.
Fri Jun 16 07:27:58 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.3 is a partner HA mailbox disk.
Fri Jun 16 07:27:58 BNT [filer02: fmmb.instStat.change:info]: replayed event: normal mailbox instance on partner side.
Fri Jun 16 07:27:59 BNT [filer02: cf.fm.partner:info]: replayed event: Cluster monitor: partner 'filer01'
Fri Jun 16 07:27:59 BNT [filer02: cf.fm.timeMasterStatus:info]: replayed event: Acting as cluster time slave
Fri Jun 16 07:27:59 BNT [filer02: raid.assim.disk.brokenPreAssim:error]: replayed event: Broken Disk 0a.40 Shelf ? Bay ? [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] detected prior to assimilation. It should be removed.
Fri Jun 16 07:27:59 BNT [filer02: raid.assim.disk.brokenPreAssim:error]: replayed event: Broken Disk 0a.35 Shelf ? Bay ? [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] detected prior to assimilation. It should be removed.
Fri Jun 16 07:27:59 BNT [filer02: raid.assim.disk.brokenPreAssim:error]: replayed event: Broken Disk 0a.33 Shelf ? Bay ? [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] detected prior to assimilation. It should be removed.
Fri Jun 16 07:28:00 BNT [filer02: coredump.spare.none:info]: replayed event: No sparecore disk was found.
Fri Jun 16 07:28:00 BNT [filer02: raid.vol.replay.nvram:info]: replayed event: Performing raid replay on volume(s)
Fri Jun 16 07:28:00 BNT [filer02: raid.cksum.replay.summary:info]: replayed event: Replayed 0 checksum blocks.
Fri Jun 16 07:28:00 BNT [filer02: raid.stripe.replay.summary:info]: replayed event: Replayed 0 stripes.
Fri Jun 16 07:28:03 BNT [filer02: ses.status.psWarning:warning]: replayed event: DS12-ESAS shelf 0 on channel 0c power warning for Power supply 1: non-critical status; DC undervoltage fault. This module is on the rear side of the shelf, at the left.
Fri Jun 16 07:28:03 BNT [filer02: ses.status.psWarning:warning]: replayed event: DS12-ESAS shelf 0 on channel 0c power warning for Power supply 2: non-critical status; DC undervoltage fault. This module is on the rear side of the shelf, at the right.
Fri Jun 16 07:28:12 BNT [filer02: cf.fm.launch:info]: replayed event: Launching cluster monitor
Fri Jun 16 07:28:12 BNT [filer02: cf.fm.partner:info]: replayed event: Cluster monitor: partner 'filer01'
Fri Jun 16 07:28:16 BNT [filer02: cf.fsm.takeoverOfPartnerDisabled:notice]: replayed event: Cluster monitor: takeover of filer01 disabled (partner booting)
Fri Jun 16 07:28:17 BNT [filer02: fcp.service.adapter:warning]: replayed event: FCP started but no adapter is present!
Fri Jun 16 07:28:18 BNT [filer02: kern.syslog.msg:notice]: replayed event: The system was down for 21 seconds
Fri Jun 16 07:28:18 BNT [filer02: cf.fsm.takeoverOfPartnerEnabled:notice]: replayed event: Cluster monitor: takeover of filer01 enabled
Fri Jun 16 07:28:19 BNT [filer02: disk.failmsg:error]: Disk 0a.40 (N00V0W8L): message received.
Fri Jun 16 07:28:19 BNT [filer02: disk.failmsg:error]: Disk 0a.35 (N00WLAUL): message received.
Fri Jun 16 07:28:19 BNT [filer02: disk.failmsg:error]: Disk 0a.33 (N00W75PL): message received.
Fri Jun 16 07:28:19 BNT [filer02: raid.disk.unload.done:info]: Unload of Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] has completed successfully
Fri Jun 16 07:28:19 BNT [filer02: raid.disk.unload.done:info]: Unload of Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] has completed successfully
Fri Jun 16 07:28:19 BNT [filer02: raid.disk.unload.done:info]: Unload of Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] has completed successfully
Fri Jun 16 07:28:19 BNT [filer02: raid.fdr.failed.ok:info]: Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] successfully deleted from spare pool
Fri Jun 16 07:28:19 BNT [filer02: raid.fdr.failed.ok:info]: Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] successfully deleted from spare pool
Fri Jun 16 07:28:19 BNT [filer02: raid.fdr.failed.ok:info]: Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] successfully deleted from spare pool
Fri Jun 16 07:28:20 BNT [filer02: dfu.firmwareUpToDate:info]: Firmware is up-to-date on all disk drives
Fri Jun 16 07:28:21 BNT [filer02: perf.archive.start:info]: Performance archiver started. Sampling 22 objects and 198 counters.
Fri Jun 16 07:28:23 BNT [filer02: netif.linkUp:info]: Ethernet e0b: Link up.
Fri Jun 16 07:28:23 BNT [filer02: netif.linkUp:info]: Ethernet e0a: Link up.
Fri Jun 16 07:28:24 BNT [filer02: pvif.switchLink:warning]: vif1: switching to e0a
Fri Jun 16 07:28:25 BNT [filer02: rc:info]: relog syslog Fri Jun 16 07:27:39 BNT [filer02: fci.adapter.error:warning]: Fibre Channel adapter driver encountered error "Failed t
Fri Jun 16 07:28:25 BNT [filer02: rc:info]: relog syslog Fri Jun 16 07:27:39 BNT [filer02: fci.adapter.reset:debug]: Resetting Fibre Channel adapter 0a.
Fri Jun 16 07:28:25 BNT [filer02: iscsi.service.startup:info]: iSCSI service startup
Fri Jun 16 07:28:25 BNT [filer02: netif.linkDown:info]: Ethernet e0a: Link down, check cable.
Fri Jun 16 07:28:26 BNT [filer02: cmds.sysconf.logErr:error]: sysconfig: Unless directed by NetApp Global Services the cf.takeover.on_panic option should be set to On. .
Fri Jun 16 07:28:26 BNT [filer02: raid.rg.recons.missing:notice]: RAID group /aggr1/plex0/rg0 is missing 2 disk(s).
Fri Jun 16 07:28:26 BNT [filer02: raid.rg.recons.cantStart:warning]: The reconstruction cannot start in RAID group /aggr1/plex0/rg0: No 520 bps disk of required size available in spare pool, targeting any  spare pool
Fri Jun 16 07:28:26 BNT [filer02: mgr.boot.disk_done:info]: NetApp Release 7.3.2 boot complete. Last disk update written at Fri Jun 16 07:27:30 BNT 2017
Fri Jun 16 07:28:26 BNT [filer02: monitor.raid.brokenDisk:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken.
Fri Jun 16 07:28:26 BNT [filer02: monitor.brokenDisk.notice:notice]: When two disks are broken in raid_dp volume,  the system shuts down automatically every 24  hours to encourage you to replace the disk.  If you reboot the system it will run for another  24 hours before shutting down. (The 24 hour timeout  may be increased by altering the "raid.timeout"  value using the "options" command.)
Fri Jun 16 07:28:27 BNT [filer02: diskown.errorReadingOwnership:warning]: error 3 (disk failed) while reading ownership on disk 0a.24 (S/N N00SSSAL)
Fri Jun 16 07:28:27 BNT [filer02: diskown.errorReadingOwnership:warning]: error 3 (disk failed) while reading ownership on disk 0a.26 (S/N N00V0UWL)
Fri Jun 16 07:28:27 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] is still present in the system and should be removed.
Fri Jun 16 07:28:27 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] is still present in the system and should be removed.
Fri Jun 16 07:28:27 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] is still present in the system and should be removed.
Fri Jun 16 07:28:27 BNT [filer02: shelf.config.spha:info]: System is using single path HA attached storage only.
Fri Jun 16 07:28:27 BNT [filer02: mgr.boot.reason_ok:notice]: System rebooted after power-on.
Fri Jun 16 07:28:28 BNT [filer02: netif.linkUp:info]: Ethernet e0a: Link up.
Fri Jun 16 07:28:50 BNT [filer02: raid.rg.spares.low:warning]: /aggr1/plex0/rg0
Fri Jun 16 07:28:52 BNT [filer02: nbt.nbns.registrationComplete:info]: NBT: All CIFS name registrations have completed for the local server.
Fri Jun 16 07:28:56 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (PERFORMANCE SNAPSHOT)
Fri Jun 16 07:29:03 BNT last message repeated 2 times
Fri Jun 16 07:29:03 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201706100931.2) was not posted for host (0), it was dropped from queue due to lack of space
Fri Jun 16 07:29:04 BNT [filer02: monitor.globalStatus.nonCritical:warning]: Disk on adapter 0a, shelf 2, bay 8, failed. Disk on adapter 0a, shelf 2, bay 3, failed. Disk on adapter 0a, shelf 2, bay 1, failed.  There are not enough spare disks. /vol/vol1 is full (using or reserving 100% of space and 0% of inodes, using 92% of reserve).
Fri Jun 16 07:29:09 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (CLUSTER ERROR: DISK/SHELF COUNT MISMATCH)
Fri Jun 16 07:29:09 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201706100936.0) was not posted for host (0), it was dropped from queue due to lack of space
Fri Jun 16 07:29:16 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jun 16 07:29:16 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201706101200.0) was not posted for host (0), it was dropped from queue due to lack of space
Fri Jun 16 07:29:36 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jun 16 07:29:36 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201706101600.0) was not posted for host (0), it was dropped from queue due to lack of space
Fri Jun 16 07:34:07 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jun 16 07:38:56 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jun 16 07:42:49 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jun 16 07:46:36 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jun 16 07:50:30 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jun 16 07:59:11 BNT last message repeated 2 times
Fri Jun 16 07:59:37 BNT [filer02: iscsi.notice:notice]: ISCSI: New session from initiator iqn.1991-05.com.microsoft:pcb-722-ad1.pcbl722dom.local at IP addr 10.72.4.1
Fri Jun 16 08:00:00 BNT [filer02: kern.uptime.filer:info]:   8:00am up 31 mins, 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 10 iSCSI ops
Fri Jun 16 08:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 24 hours.
Fri Jun 16 08:00:45 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Fri Jun 16 08:00:53 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jun 16 08:00:54 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201706102000.0) was not posted for host (0), it was dropped from queue due to lack of space
Fri Jun 16 08:04:29 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (WEEKLY_LOG)
Fri Jun 16 08:04:29 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201706110000.0) for host (0) was not posted to NetApp. The system will drop the message
Fri Jun 16 09:00:00 BNT [filer02: kern.uptime.filer:info]:   9:00am up  1:31 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 113 iSCSI ops
Fri Jun 16 09:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 23 hours.
Fri Jun 16 10:00:00 BNT [filer02: kern.uptime.filer:info]:  10:00am up  2:31 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 30235 iSCSI ops
Fri Jun 16 10:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 22 hours.
Fri Jun 16 11:00:00 BNT [filer02: kern.uptime.filer:info]:  11:00am up  3:31 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 118816 iSCSI ops
Fri Jun 16 11:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 21 hours.
Fri Jun 16 12:00:00 BNT [filer02: kern.uptime.filer:info]:  12:00pm up  4:31 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 227810 iSCSI ops
Fri Jun 16 12:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 20 hours.
Fri Jun 16 12:00:31 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Fri Jun 16 12:00:46 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (PERFORMANCE DATA)
Fri Jun 16 12:04:40 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (PERFORMANCE DATA)
Fri Jun 16 12:08:15 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (PERFORMANCE DATA)
Fri Jun 16 12:12:43 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (PERFORMANCE DATA)
Fri Jun 16 12:16:19 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (PERFORMANCE DATA)
Fri Jun 16 12:32:55 BNT last message repeated 4 times
Fri Jun 16 12:37:06 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (PERFORMANCE DATA)
Fri Jun 16 12:54:10 BNT last message repeated 4 times
Fri Jun 16 12:58:19 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (PERFORMANCE DATA)
Fri Jun 16 12:58:19 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201706110000.1) for host (0) was not posted to NetApp. The system will drop the message
Fri Jun 16 13:00:00 BNT [filer02: kern.uptime.filer:info]:   1:00pm up  5:31 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 247147 iSCSI ops
Fri Jun 16 13:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 19 hours.
Fri Jun 16 14:00:00 BNT [filer02: kern.uptime.filer:info]:   2:00pm up  6:31 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 247845 iSCSI ops
Fri Jun 16 14:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 18 hours.
Fri Jun 16 15:00:00 BNT [filer02: kern.uptime.filer:info]:   3:00pm up  7:31 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 247951 iSCSI ops
Fri Jun 16 15:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 17 hours.
Fri Jun 16 16:00:00 BNT [filer02: kern.uptime.filer:info]:   4:00pm up  8:31 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 249196 iSCSI ops
Fri Jun 16 16:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 16 hours.
Fri Jun 16 16:00:30 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Fri Jun 16 16:00:45 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jun 16 16:04:58 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jun 16 16:09:14 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jun 16 16:13:12 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jun 16 16:16:55 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jun 16 16:33:15 BNT last message repeated 4 times
Fri Jun 16 16:37:35 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jun 16 16:57:41 BNT last message repeated 5 times
Fri Jun 16 16:57:41 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201706110000.2) for host (0) was not posted to NetApp. The system will drop the message
Fri Jun 16 17:00:00 BNT [filer02: kern.uptime.filer:info]:   5:00pm up  9:31 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 250996 iSCSI ops
Fri Jun 16 17:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 15 hours.
Fri Jun 16 18:00:00 BNT [filer02: kern.uptime.filer:info]:   6:00pm up 10:31 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 250996 iSCSI ops
Fri Jun 16 18:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 14 hours.
Fri Jun 16 19:00:00 BNT [filer02: kern.uptime.filer:info]:   7:00pm up 11:31 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 250996 iSCSI ops
Fri Jun 16 19:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 13 hours.
Fri Jun 16 20:00:00 BNT [filer02: kern.uptime.filer:info]:   8:00pm up 12:31 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 250996 iSCSI ops
Fri Jun 16 20:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 12 hours.
Fri Jun 16 20:00:15 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Fri Jun 16 20:00:31 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jun 16 20:04:28 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jun 16 20:08:11 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jun 16 20:11:58 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jun 16 20:16:00 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jun 16 20:33:07 BNT last message repeated 4 times
Fri Jun 16 20:37:23 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jun 16 20:54:17 BNT last message repeated 4 times
Fri Jun 16 20:58:41 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jun 16 20:58:41 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201706110800.0) for host (0) was not posted to NetApp. The system will drop the message
Fri Jun 16 21:00:00 BNT [filer02: kern.uptime.filer:info]:   9:00pm up 13:31 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 250996 iSCSI ops
Fri Jun 16 21:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 11 hours.
Fri Jun 16 22:00:00 BNT [filer02: kern.uptime.filer:info]:  10:00pm up 14:31 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 250996 iSCSI ops
Fri Jun 16 22:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 10 hours.
Fri Jun 16 23:00:00 BNT [filer02: kern.uptime.filer:info]:  11:00pm up 15:31 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 250996 iSCSI ops
Fri Jun 16 23:00:01 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 9 hours.
Sat Jun 17 00:00:00 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of nightly.0 snapshot (Snapshot operation failed: No space left on device.)
Sat Jun 17 00:00:00 BNT [filer02: kern.uptime.filer:info]:  12:00am up 16:32 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 250996 iSCSI ops
Sat Jun 17 00:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 8 hours.
Sat Jun 17 00:00:11 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Sat Jun 17 00:03:54 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Sat Jun 17 00:07:45 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Sat Jun 17 00:12:02 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Sat Jun 17 00:16:26 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Sat Jun 17 00:36:05 BNT last message repeated 5 times
Sat Jun 17 00:39:49 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Sat Jun 17 00:55:10 BNT last message repeated 4 times
Sat Jun 17 00:55:10 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201706130806.0) for host (0) was not posted to NetApp. The system will drop the message
Sat Jun 17 01:00:00 BNT [filer02: kern.uptime.filer:info]:   1:00am up 17:32 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 250996 iSCSI ops
Sat Jun 17 01:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 7 hours.
Sat Jun 17 02:00:00 BNT [filer02: kern.uptime.filer:info]:   2:00am up 18:32 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 250996 iSCSI ops
Sat Jun 17 02:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 6 hours.
Sat Jun 17 03:00:00 BNT [filer02: kern.uptime.filer:info]:   3:00am up 19:32 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 250996 iSCSI ops
Sat Jun 17 03:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 5 hours.
Sat Jun 17 04:00:00 BNT [filer02: kern.uptime.filer:info]:   4:00am up 20:32 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 251007 iSCSI ops
Sat Jun 17 04:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 4 hours.
Sat Jun 17 05:00:00 BNT [filer02: kern.uptime.filer:info]:   5:00am up 21:32 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 251007 iSCSI ops
Sat Jun 17 05:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 3 hours.
Sat Jun 17 06:00:00 BNT [filer02: kern.uptime.filer:info]:   6:00am up 22:32 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 251007 iSCSI ops
Sat Jun 17 06:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 2 hours.
Sat Jun 17 07:00:00 BNT [filer02: kern.uptime.filer:info]:   7:00am up 23:32 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 251007 iSCSI ops
Sat Jun 17 07:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 1 hour.
Sat Jun 17 07:00:01 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] is still present in the system and should be removed.
Sat Jun 17 07:00:01 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] is still present in the system and should be removed.
Sat Jun 17 07:00:01 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] is still present in the system and should be removed.
Sat Jun 17 08:00:00 BNT [filer02: kern.uptime.filer:info]:   8:00am up  1 day, 32 mins, 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 251007 iSCSI ops
Sat Jun 17 08:00:01 BNT [filer02: monitor.brokenDisk.notice:notice]: When two disks are broken in raid_dp volume,  the system shuts down automatically every 24  hours to encourage you to replace the disk.  If you reboot the system it will run for another  24 hours before shutting down. (The 24 hour timeout  may be increased by altering the "raid.timeout"  value using the "options" command.)
Sat Jun 17 08:00:01 BNT [filer02: monitor.shutdown.brokenDisk:EMERGENCY]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system now.
Sat Jun 17 08:00:04 BNT [filer02: cf.fsm.takeoverOfPartnerDisabled:notice]: Cluster monitor: takeover of filer01 disabled (local halt in progress)
Sat Jun 17 08:00:07 BNT [filer02: kern.shutdown:notice]: System shut down because : "BROKEN DISK".
Sun Jun 18 03:44:39 GMT [localhost: rc:notice]: The system was down for 99864 seconds
Sun Jun 18 11:44:13 BNT [filer02: cf.nm.nicTransitionUp:info]: replayed event: Interconnect link 0 is UP
Sun Jun 18 11:44:20 BNT [filer02: fci.initialization.failed:error]: replayed event: Initialization failed on Fibre Channel adapter 0b.
Sun Jun 18 11:44:22 BNT [filer02: nvram.battery.turned.on:info]: replayed event: The NVRAM battery is turned ON. It is turned OFF during system shutdown.
Sun Jun 18 11:44:25 BNT [filer02: diskown.isEnabled:info]: replayed event: software ownership has been enabled for this system
Sun Jun 18 11:44:29 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.2 is a local HA mailbox disk.
Sun Jun 18 11:44:29 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.0 is a local HA mailbox disk.
Sun Jun 18 11:44:29 BNT [filer02: fmmb.instStat.change:info]: replayed event: normal mailbox instance on local side.
Sun Jun 18 11:44:29 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.7 is a partner HA mailbox disk.
Sun Jun 18 11:44:29 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.3 is a partner HA mailbox disk.
Sun Jun 18 11:44:29 BNT [filer02: fmmb.instStat.change:info]: replayed event: normal mailbox instance on partner side.
Sun Jun 18 11:44:29 BNT [filer02: cf.fm.partner:info]: replayed event: Cluster monitor: partner 'filer01'
Sun Jun 18 11:44:29 BNT [filer02: cf.fm.timeMasterStatus:info]: replayed event: Acting as cluster time slave
Sun Jun 18 11:44:30 BNT [filer02: raid.assim.disk.brokenPreAssim:error]: replayed event: Broken Disk 0a.40 Shelf ? Bay ? [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] detected prior to assimilation. It should be removed.
Sun Jun 18 11:44:30 BNT [filer02: raid.assim.disk.brokenPreAssim:error]: replayed event: Broken Disk 0a.35 Shelf ? Bay ? [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] detected prior to assimilation. It should be removed.
Sun Jun 18 11:44:30 BNT [filer02: raid.assim.disk.brokenPreAssim:error]: replayed event: Broken Disk 0a.33 Shelf ? Bay ? [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] detected prior to assimilation. It should be removed.
Sun Jun 18 11:44:30 BNT [filer02: coredump.spare.none:info]: replayed event: No sparecore disk was found.
Sun Jun 18 11:44:31 BNT [filer02: raid.cksum.replay.summary:info]: replayed event: Replayed 0 checksum blocks.
Sun Jun 18 11:44:31 BNT [filer02: raid.stripe.replay.summary:info]: replayed event: Replayed 0 stripes.
Sun Jun 18 11:44:33 BNT [filer02: cf.fm.launch:info]: replayed event: Launching cluster monitor
Sun Jun 18 11:44:33 BNT [filer02: cf.fm.partner:info]: replayed event: Cluster monitor: partner 'filer01'
Sun Jun 18 11:44:33 BNT [filer02: cf.fsm.takeoverOfPartnerDisabled:notice]: replayed event: Cluster monitor: takeover of filer01 disabled (partner booting)
Sun Jun 18 11:44:34 BNT [filer02: ses.status.psWarning:warning]: replayed event: DS12-ESAS shelf 0 on channel 0c power warning for Power supply 1: non-critical status; DC undervoltage fault. This module is on the rear side of the shelf, at the left.
Sun Jun 18 11:44:34 BNT [filer02: ses.status.psWarning:warning]: replayed event: DS12-ESAS shelf 0 on channel 0c power warning for Power supply 2: non-critical status; DC undervoltage fault. This module is on the rear side of the shelf, at the right.
Sun Jun 18 11:44:39 BNT [filer02: fcp.service.adapter:warning]: replayed event: FCP started but no adapter is present!
Sun Jun 18 11:44:39 BNT [filer02: kern.syslog.msg:notice]: replayed event: The system was down for 99864 seconds
Sun Jun 18 11:44:40 BNT [filer02: disk.failmsg:error]: Disk 0a.40 (N00V0W8L): message received.
Sun Jun 18 11:44:40 BNT [filer02: disk.failmsg:error]: Disk 0a.35 (N00WLAUL): message received.
Sun Jun 18 11:44:40 BNT [filer02: disk.failmsg:error]: Disk 0a.33 (N00W75PL): message received.
Sun Jun 18 11:44:40 BNT [filer02: raid.disk.unload.done:info]: Unload of Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] has completed successfully
Sun Jun 18 11:44:40 BNT [filer02: raid.disk.unload.done:info]: Unload of Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] has completed successfully
Sun Jun 18 11:44:40 BNT [filer02: raid.disk.unload.done:info]: Unload of Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] has completed successfully
Sun Jun 18 11:44:40 BNT [filer02: raid.fdr.failed.ok:info]: Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] successfully deleted from spare pool
Sun Jun 18 11:44:40 BNT [filer02: raid.fdr.failed.ok:info]: Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] successfully deleted from spare pool
Sun Jun 18 11:44:40 BNT [filer02: raid.fdr.failed.ok:info]: Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] successfully deleted from spare pool
Sun Jun 18 11:44:41 BNT [filer02: dfu.firmwareUpToDate:info]: Firmware is up-to-date on all disk drives
Sun Jun 18 11:44:41 BNT [filer02: cf.fsm.takeoverOfPartnerEnabled:notice]: Cluster monitor: takeover of filer01 enabled
Sun Jun 18 11:44:41 BNT [filer02: cf.fsm.takeoverByPartnerEnabled:notice]: Cluster monitor: takeover of filer02 by filer01 enabled
Sun Jun 18 11:44:42 BNT [filer02: perf.archive.start:info]: Performance archiver started. Sampling 22 objects and 198 counters.
Sun Jun 18 11:44:44 BNT [filer02: netif.linkUp:info]: Ethernet e0b: Link up.
Sun Jun 18 11:44:45 BNT [filer02: netif.linkUp:info]: Ethernet e0a: Link up.
Sun Jun 18 11:44:45 BNT [filer02: pvif.switchLink:warning]: vif1: switching to e0a
Sun Jun 18 11:44:47 BNT [filer02: iscsi.service.startup:info]: iSCSI service startup
Sun Jun 18 11:44:47 BNT [filer02: iscsi.notice:notice]: ISCSI: New session from initiator iqn.1991-05.com.microsoft:pcb-722-ad1.pcbl722dom.local at IP addr 10.72.4.1
Sun Jun 18 11:44:47 BNT [filer02: cmds.sysconf.logErr:error]: sysconfig: Unless directed by NetApp Global Services the cf.takeover.on_panic option should be set to On. .
Sun Jun 18 11:44:47 BNT [filer02: raid.rg.recons.missing:notice]: RAID group /aggr1/plex0/rg0 is missing 2 disk(s).
Sun Jun 18 11:44:47 BNT [filer02: raid.rg.recons.cantStart:warning]: The reconstruction cannot start in RAID group /aggr1/plex0/rg0: No 520 bps disk of required size available in spare pool, targeting any  spare pool
Sun Jun 18 11:44:48 BNT [filer02: mgr.boot.disk_done:info]: NetApp Release 7.3.2 boot complete. Last disk update written at Sat Jun 17 08:00:09 BNT 2017
Sun Jun 18 11:44:48 BNT [filer02: monitor.raid.brokenDisk:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken.
Sun Jun 18 11:44:48 BNT [filer02: monitor.brokenDisk.notice:notice]: When two disks are broken in raid_dp volume,  the system shuts down automatically every 24  hours to encourage you to replace the disk.  If you reboot the system it will run for another  24 hours before shutting down. (The 24 hour timeout  may be increased by altering the "raid.timeout"  value using the "options" command.)
Sun Jun 18 11:44:48 BNT [filer02: diskown.errorReadingOwnership:warning]: error 3 (disk failed) while reading ownership on disk 0a.24 (S/N N00SSSAL)
Sun Jun 18 11:44:48 BNT [filer02: diskown.errorReadingOwnership:warning]: error 3 (disk failed) while reading ownership on disk 0a.26 (S/N N00V0UWL)
Sun Jun 18 11:44:48 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] is still present in the system and should be removed.
Sun Jun 18 11:44:48 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] is still present in the system and should be removed.
Sun Jun 18 11:44:48 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] is still present in the system and should be removed.
Sun Jun 18 11:44:48 BNT [filer02: shelf.config.spha:info]: System is using single path HA attached storage only.
Sun Jun 18 11:44:48 BNT [filer02: mgr.boot.reason_ok:notice]: System rebooted.
Sun Jun 18 11:45:01 BNT [filer02: raid.rg.spares.low:warning]: /aggr1/plex0/rg0
Sun Jun 18 11:45:02 BNT [filer02: monitor.globalStatus.nonCritical:warning]: Disk on adapter 0a, shelf 2, bay 8, failed. Disk on adapter 0a, shelf 2, bay 3, failed. Disk on adapter 0a, shelf 2, bay 1, failed.  There are not enough spare disks. /vol/vol1 is full (using or reserving 100% of space and 0% of inodes, using 92% of reserve).
Sun Jun 18 11:45:13 BNT [filer02: nbt.nbns.registrationComplete:info]: NBT: All CIFS name registrations have completed for the local server.
Sun Jun 18 11:45:15 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SHUTDOWN PENDING (degraded mode))
Sun Jun 18 11:45:22 BNT last message repeated 2 times
Sun Jun 18 11:45:23 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201706130806.1) was not posted for host (0), it was dropped from queue due to lack of space
Sun Jun 18 11:45:29 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (REBOOT)
Sun Jun 18 11:45:29 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201706130806.2) was not posted for host (0), it was dropped from queue due to lack of space
Sun Jun 18 11:45:36 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SPARES_LOW)
Sun Jun 18 11:45:36 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201706130806.3) was not posted for host (0), it was dropped from queue due to lack of space
Sun Jun 18 11:45:57 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (PERFORMANCE SNAPSHOT)
Sun Jun 18 11:45:57 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201706130807.0) was not posted for host (0), it was dropped from queue due to lack of space
Sun Jun 18 11:50:12 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (CLUSTER ERROR: DISK/SHELF COUNT MISMATCH)
Sun Jun 18 11:50:28 BNT [filer02: cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 0a.33 (016F3AB5:B475F4EC:10505E83:02202755:00000000:00000000:00000000:00000000:00000000:00000000) has recently changed or the node (filer02) is missing the disk.
Sun Jun 18 11:50:28 BNT [filer02: cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 0a.35 (9115483F:9DD43BC4:27172773:99F7D8B4:00000000:00000000:00000000:00000000:00000000:00000000) has recently changed or the node (filer02) is missing the disk.
Sun Jun 18 11:50:28 BNT [filer02: cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 0a.40 (072AEB65:C662A9CF:E58B9917:9BFA787B:00000000:00000000:00000000:00000000:00000000:00000000) has recently changed or the node (filer02) is missing the disk.
Sun Jun 18 11:50:28 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (CLUSTER ERROR: DISK/SHELF COUNT MISMATCH)
Sun Jun 18 11:50:35 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (CLUSTER ERROR: DISK/SHELF COUNT MISMATCH)
Sun Jun 18 11:50:35 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201706130811.0) was not posted for host (0), it was dropped from queue due to lack of space
Sun Jun 18 11:55:10 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Sun Jun 18 11:59:37 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Sun Jun 18 12:00:00 BNT [filer02: kern.uptime.filer:info]:  12:00pm up 14 mins, 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 2734 iSCSI ops
Sun Jun 18 12:00:01 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 24 hours.
Sun Jun 18 12:00:16 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Sun Jun 18 12:00:23 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Sun Jun 18 12:00:23 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201706131200.0) was not posted for host (0), it was dropped from queue due to lack of space
Sun Jun 18 12:04:48 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Sun Jun 18 12:08:22 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Sun Jun 18 12:12:48 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Sun Jun 18 12:12:48 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201706131600.0) for host (0) was not posted to NetApp. The system will drop the message
Sun Jun 18 12:51:49 BNT [filer02: cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 0a.33 (016F3AB5:B475F4EC:10505E83:02202755:00000000:00000000:00000000:00000000:00000000:00000000) has recently changed or the node (filer02) is missing the disk.
Sun Jun 18 12:51:49 BNT [filer02: cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 0a.35 (9115483F:9DD43BC4:27172773:99F7D8B4:00000000:00000000:00000000:00000000:00000000:00000000) has recently changed or the node (filer02) is missing the disk.
Sun Jun 18 12:51:49 BNT [filer02: cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 0a.40 (072AEB65:C662A9CF:E58B9917:9BFA787B:00000000:00000000:00000000:00000000:00000000:00000000) has recently changed or the node (filer02) is missing the disk.
Sun Jun 18 13:00:00 BNT [filer02: kern.uptime.filer:info]:   1:00pm up  1:14 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 3585 iSCSI ops
Sun Jun 18 13:00:01 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 23 hours.
Sun Jun 18 14:00:00 BNT [filer02: kern.uptime.filer:info]:   2:00pm up  2:14 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 4666 iSCSI ops
Sun Jun 18 14:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 22 hours.
Sun Jun 18 15:00:00 BNT [filer02: kern.uptime.filer:info]:   3:00pm up  3:15 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 6210 iSCSI ops
Sun Jun 18 15:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 21 hours.
Sun Jun 18 16:00:00 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Sun Jun 18 16:00:00 BNT [filer02: kern.uptime.filer:info]:   4:00pm up  4:15 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 6412 iSCSI ops
Sun Jun 18 16:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 20 hours.
Sun Jun 18 16:00:19 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Sun Jun 18 16:04:24 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Sun Jun 18 16:08:23 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Sun Jun 18 16:12:14 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Sun Jun 18 16:15:57 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Sun Jun 18 16:32:25 BNT last message repeated 4 times
Sun Jun 18 16:36:52 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Sun Jun 18 16:55:55 BNT last message repeated 5 times
Sun Jun 18 16:55:55 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201706132000.0) for host (0) was not posted to NetApp. The system will drop the message
Sun Jun 18 17:00:00 BNT [filer02: kern.uptime.filer:info]:   5:00pm up  5:15 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 6412 iSCSI ops
Sun Jun 18 17:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 19 hours.
Sun Jun 18 18:00:00 BNT [filer02: kern.uptime.filer:info]:   6:00pm up  6:15 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 6442 iSCSI ops
Sun Jun 18 18:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 18 hours.
Sun Jun 18 19:00:00 BNT [filer02: kern.uptime.filer:info]:   7:00pm up  7:15 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 6442 iSCSI ops
Sun Jun 18 19:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 17 hours.
Sun Jun 18 20:00:00 BNT [filer02: kern.uptime.filer:info]:   8:00pm up  8:15 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 6442 iSCSI ops
Sun Jun 18 20:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 16 hours.
Sun Jun 18 20:00:01 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Sun Jun 18 20:00:16 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Sun Jun 18 20:04:25 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Sun Jun 18 20:08:24 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Sun Jun 18 20:12:15 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Sun Jun 18 20:16:03 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Sun Jun 18 20:35:56 BNT last message repeated 5 times
Sun Jun 18 20:40:02 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Sun Jun 18 20:54:41 BNT last message repeated 4 times
Sun Jun 18 20:54:41 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201706140000.0) for host (0) was not posted to NetApp. The system will drop the message
Sun Jun 18 21:00:00 BNT [filer02: kern.uptime.filer:info]:   9:00pm up  9:15 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 6442 iSCSI ops
Sun Jun 18 21:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 15 hours.
Sun Jun 18 22:00:00 BNT [filer02: kern.uptime.filer:info]:  10:00pm up 10:15 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 6442 iSCSI ops
Sun Jun 18 22:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 14 hours.
Sun Jun 18 23:00:00 BNT [filer02: kern.uptime.filer:info]:  11:00pm up 11:15 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 6442 iSCSI ops
Sun Jun 18 23:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 13 hours.
Mon Jun 19 00:00:00 BNT [filer02: kern.uptime.filer:info]:  12:00am up 12:15 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 6442 iSCSI ops
Mon Jun 19 00:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 12 hours.
Mon Jun 19 00:00:45 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of nightly.0 snapshot (Snapshot operation failed: No space left on device.)
Mon Jun 19 00:00:56 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Mon Jun 19 00:04:29 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Mon Jun 19 00:08:41 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Mon Jun 19 00:12:37 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Mon Jun 19 00:16:18 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Mon Jun 19 00:36:04 BNT last message repeated 5 times
Mon Jun 19 00:39:52 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Mon Jun 19 00:54:54 BNT last message repeated 4 times
Mon Jun 19 00:54:54 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201706140747.0) for host (0) was not posted to NetApp. The system will drop the message
Mon Jun 19 01:00:00 BNT [filer02: kern.uptime.filer:info]:   1:00am up 13:15 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 6442 iSCSI ops
Mon Jun 19 01:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 11 hours.
Mon Jun 19 02:00:00 BNT [filer02: kern.uptime.filer:info]:   2:00am up 14:15 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 6446 iSCSI ops
Mon Jun 19 02:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 10 hours.
Mon Jun 19 03:00:00 BNT [filer02: kern.uptime.filer:info]:   3:00am up 15:15 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 6446 iSCSI ops
Mon Jun 19 03:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 9 hours.
Mon Jun 19 04:00:00 BNT [filer02: kern.uptime.filer:info]:   4:00am up 16:15 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 6446 iSCSI ops
Mon Jun 19 04:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 8 hours.
Mon Jun 19 05:00:00 BNT [filer02: kern.uptime.filer:info]:   5:00am up 17:15 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 6446 iSCSI ops
Mon Jun 19 05:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 7 hours.
Mon Jun 19 06:00:00 BNT [filer02: kern.uptime.filer:info]:   6:00am up 18:15 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 6446 iSCSI ops
Mon Jun 19 06:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 6 hours.
Mon Jun 19 07:00:00 BNT [filer02: kern.uptime.filer:info]:   7:00am up 19:15 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 6446 iSCSI ops
Mon Jun 19 07:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 5 hours.
Mon Jun 19 07:30:59 BNT [filer02: fci.link.break:error]: Link break detected on Fibre Channel adapter 0a.
Mon Jun 19 07:30:59 BNT [filer02: fci.adapter.error:warning]: Fibre Channel adapter driver encountered error "Failed to get loop map" on adapter 0a.
Sun Jun 18 23:31:54 GMT [localhost: rc:notice]: The system was down for 34 seconds
Mon Jun 19 07:31:19 BNT [filer02: cf.nm.nicTransitionUp:info]: replayed event: Interconnect link 0 is UP
Mon Jun 19 07:31:26 BNT [filer02: fci.initialization.failed:error]: replayed event: Initialization failed on Fibre Channel adapter 0b.
Mon Jun 19 07:31:55 BNT [filer02: cf.fsm.takeoverOfPartnerEnabled:notice]: Cluster monitor: takeover of filer01 enabled
Mon Jun 19 07:31:55 BNT [filer02: cf.fsm.takeoverByPartnerEnabled:notice]: Cluster monitor: takeover of filer02 by filer01 enabled
Mon Jun 19 07:31:28 BNT [filer02: nvram.battery.turned.on:info]: replayed event: The NVRAM battery is turned ON. It is turned OFF during system shutdown.
Mon Jun 19 07:31:31 BNT [filer02: diskown.isEnabled:info]: replayed event: software ownership has been enabled for this system
Mon Jun 19 07:31:35 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.2 is a local HA mailbox disk.
Mon Jun 19 07:31:35 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.0 is a local HA mailbox disk.
Mon Jun 19 07:31:35 BNT [filer02: fmmb.instStat.change:info]: replayed event: normal mailbox instance on local side.
Mon Jun 19 07:31:35 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.7 is a partner HA mailbox disk.
Mon Jun 19 07:31:35 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.3 is a partner HA mailbox disk.
Mon Jun 19 07:31:35 BNT [filer02: fmmb.instStat.change:info]: replayed event: normal mailbox instance on partner side.
Mon Jun 19 07:31:36 BNT [filer02: cf.fm.partner:info]: replayed event: Cluster monitor: partner 'filer01'
Mon Jun 19 07:31:36 BNT [filer02: cf.fm.timeMasterStatus:info]: replayed event: Acting as cluster time slave
Mon Jun 19 07:31:36 BNT [filer02: raid.assim.disk.brokenPreAssim:error]: replayed event: Broken Disk 0a.40 Shelf ? Bay ? [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] detected prior to assimilation. It should be removed.
Mon Jun 19 07:31:36 BNT [filer02: raid.assim.disk.brokenPreAssim:error]: replayed event: Broken Disk 0a.35 Shelf ? Bay ? [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] detected prior to assimilation. It should be removed.
Mon Jun 19 07:31:36 BNT [filer02: raid.assim.disk.brokenPreAssim:error]: replayed event: Broken Disk 0a.33 Shelf ? Bay ? [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] detected prior to assimilation. It should be removed.
Mon Jun 19 07:31:37 BNT [filer02: coredump.spare.none:info]: replayed event: No sparecore disk was found.
Mon Jun 19 07:31:37 BNT [filer02: raid.vol.replay.nvram:info]: replayed event: Performing raid replay on volume(s)
Mon Jun 19 07:31:37 BNT [filer02: raid.cksum.replay.summary:info]: replayed event: Replayed 0 checksum blocks.
Mon Jun 19 07:31:37 BNT [filer02: raid.stripe.replay.summary:info]: replayed event: Replayed 0 stripes.
Mon Jun 19 07:31:40 BNT [filer02: ses.status.psWarning:warning]: replayed event: DS12-ESAS shelf 0 on channel 0c power warning for Power supply 1: non-critical status; DC undervoltage fault. This module is on the rear side of the shelf, at the left.
Mon Jun 19 07:31:40 BNT [filer02: ses.status.psWarning:warning]: replayed event: DS12-ESAS shelf 0 on channel 0c power warning for Power supply 2: non-critical status; DC undervoltage fault. This module is on the rear side of the shelf, at the right.
Mon Jun 19 07:31:49 BNT [filer02: cf.fm.launch:info]: replayed event: Launching cluster monitor
Mon Jun 19 07:31:49 BNT [filer02: cf.fm.partner:info]: replayed event: Cluster monitor: partner 'filer01'
Mon Jun 19 07:31:53 BNT [filer02: fcp.service.adapter:warning]: replayed event: FCP started but no adapter is present!
Mon Jun 19 07:31:54 BNT [filer02: kern.syslog.msg:notice]: replayed event: The system was down for 34 seconds
Mon Jun 19 07:31:55 BNT [filer02: disk.failmsg:error]: Disk 0a.40 (N00V0W8L): message received.
Mon Jun 19 07:31:55 BNT [filer02: disk.failmsg:error]: Disk 0a.35 (N00WLAUL): message received.
Mon Jun 19 07:31:55 BNT [filer02: disk.failmsg:error]: Disk 0a.33 (N00W75PL): message received.
Mon Jun 19 07:31:55 BNT [filer02: raid.disk.unload.done:info]: Unload of Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] has completed successfully
Mon Jun 19 07:31:55 BNT [filer02: raid.disk.unload.done:info]: Unload of Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] has completed successfully
Mon Jun 19 07:31:55 BNT [filer02: raid.disk.unload.done:info]: Unload of Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] has completed successfully
Mon Jun 19 07:31:55 BNT [filer02: raid.fdr.failed.ok:info]: Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] successfully deleted from spare pool
Mon Jun 19 07:31:55 BNT [filer02: raid.fdr.failed.ok:info]: Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] successfully deleted from spare pool
Mon Jun 19 07:31:55 BNT [filer02: raid.fdr.failed.ok:info]: Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] successfully deleted from spare pool
Mon Jun 19 07:31:56 BNT [filer02: dfu.firmwareUpToDate:info]: Firmware is up-to-date on all disk drives
Mon Jun 19 07:31:57 BNT [filer02: perf.archive.start:info]: Performance archiver started. Sampling 22 objects and 198 counters.
Mon Jun 19 07:31:59 BNT [filer02: netif.linkUp:info]: Ethernet e0b: Link up.
Mon Jun 19 07:31:59 BNT [filer02: netif.linkUp:info]: Ethernet e0a: Link up.
Mon Jun 19 07:32:00 BNT [filer02: pvif.switchLink:warning]: vif1: switching to e0a
Mon Jun 19 07:32:01 BNT [filer02: rc:info]: relog syslog Mon Jun 19 07:31:04 BNT [filer02: fci.adapter.error:warning]: Fibre Channel adapter driver encountered error "Failed t
Mon Jun 19 07:32:01 BNT [filer02: rc:info]: relog syslog Mon Jun 19 07:31:04 BNT [filer02: fci.adapter.reset:debug]: Resetting Fibre Channel adapter 0a.
Mon Jun 19 07:32:02 BNT [filer02: iscsi.service.startup:info]: iSCSI service startup
Mon Jun 19 07:32:02 BNT [filer02: netif.linkDown:info]: Ethernet e0b: Link down, check cable.
Mon Jun 19 07:32:02 BNT [filer02: netif.linkDown:info]: Ethernet e0a: Link down, check cable.
Mon Jun 19 07:32:02 BNT [filer02: pvif.allLinksDown:CRITICAL]: vif1: all links down
Mon Jun 19 07:32:02 BNT [filer02: cmds.sysconf.logErr:error]: sysconfig: Unless directed by NetApp Global Services the cf.takeover.on_panic option should be set to On. .
Mon Jun 19 07:32:02 BNT [filer02: raid.rg.recons.missing:notice]: RAID group /aggr1/plex0/rg0 is missing 2 disk(s).
Mon Jun 19 07:32:02 BNT [filer02: raid.rg.recons.cantStart:warning]: The reconstruction cannot start in RAID group /aggr1/plex0/rg0: No 520 bps disk of required size available in spare pool, targeting any  spare pool
Mon Jun 19 07:32:02 BNT [filer02: mgr.boot.disk_done:info]: NetApp Release 7.3.2 boot complete. Last disk update written at Mon Jun 19 07:30:44 BNT 2017
Mon Jun 19 07:32:02 BNT [filer02: monitor.raid.brokenDisk:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken.
Mon Jun 19 07:32:02 BNT [filer02: monitor.brokenDisk.notice:notice]: When two disks are broken in raid_dp volume,  the system shuts down automatically every 24  hours to encourage you to replace the disk.  If you reboot the system it will run for another  24 hours before shutting down. (The 24 hour timeout  may be increased by altering the "raid.timeout"  value using the "options" command.)
Mon Jun 19 07:32:03 BNT [filer02: diskown.errorReadingOwnership:warning]: error 3 (disk failed) while reading ownership on disk 0a.24 (S/N N00SSSAL)
Mon Jun 19 07:32:03 BNT [filer02: diskown.errorReadingOwnership:warning]: error 3 (disk failed) while reading ownership on disk 0a.26 (S/N N00V0UWL)
Mon Jun 19 07:32:03 BNT [filer02: shelf.config.spha:info]: System is using single path HA attached storage only.
Mon Jun 19 07:32:03 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] is still present in the system and should be removed.
Mon Jun 19 07:32:03 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] is still present in the system and should be removed.
Mon Jun 19 07:32:03 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] is still present in the system and should be removed.
Mon Jun 19 07:32:03 BNT [filer02: mgr.boot.reason_ok:notice]: System rebooted after power-on.
Mon Jun 19 07:32:05 BNT [filer02: netif.linkUp:info]: Ethernet e0a: Link up.
Mon Jun 19 07:32:05 BNT [filer02: pvif.switchLink:warning]: vif1: switching to e0a
Mon Jun 19 07:32:16 BNT [filer02: netif.linkUp:info]: Ethernet e0b: Link up.
Mon Jun 19 07:32:16 BNT [filer02: pvif.mediaMismatch:warning]: Inconsistent media type observed while adding link e0a under multi-level/multi -vif vif1, This can lead to performance degradation
Mon Jun 19 07:32:26 BNT [filer02: raid.rg.spares.low:warning]: /aggr1/plex0/rg0
Mon Jun 19 07:32:29 BNT [filer02: nbt.nbns.registrationComplete:info]: NBT: All CIFS name registrations have completed for the local server.
Mon Jun 19 07:32:32 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SHUTDOWN PENDING (degraded mode))
Mon Jun 19 07:32:39 BNT last message repeated 2 times
Mon Jun 19 07:32:39 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201706140748.0) was not posted for host (0), it was dropped from queue due to lack of space
Mon Jun 19 07:32:46 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (REBOOT (power on))
Mon Jun 19 07:32:46 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201706140748.1) was not posted for host (0), it was dropped from queue due to lack of space
Mon Jun 19 07:32:53 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SPARES_LOW)
Mon Jun 19 07:32:53 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201706140748.2) was not posted for host (0), it was dropped from queue due to lack of space
Mon Jun 19 07:33:08 BNT [filer02: monitor.globalStatus.nonCritical:warning]: Disk on adapter 0a, shelf 2, bay 8, failed. Disk on adapter 0a, shelf 2, bay 3, failed. Disk on adapter 0a, shelf 2, bay 1, failed.  There are not enough spare disks. /vol/vol1 is full (using or reserving 100% of space and 0% of inodes, using 92% of reserve).
Mon Jun 19 07:33:23 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (PERFORMANCE SNAPSHOT)
Mon Jun 19 07:33:23 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201706140748.3) was not posted for host (0), it was dropped from queue due to lack of space
Mon Jun 19 07:38:08 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Mon Jun 19 07:42:13 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Mon Jun 19 07:45:47 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Mon Jun 19 07:50:02 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Mon Jun 19 07:53:48 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Mon Jun 19 07:58:11 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Mon Jun 19 08:00:00 BNT [filer02: kern.uptime.filer:info]:   8:00am up 27 mins, 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 0 iSCSI ops
Mon Jun 19 08:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 24 hours.
Mon Jun 19 08:00:16 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Mon Jun 19 08:00:23 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Mon Jun 19 08:00:23 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201706140800.0) was not posted for host (0), it was dropped from queue due to lack of space
Mon Jun 19 08:04:45 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Mon Jun 19 08:09:01 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Mon Jun 19 08:09:01 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201706141200.0) for host (0) was not posted to NetApp. The system will drop the message
Mon Jun 19 08:09:43 BNT [filer02: iscsi.notice:notice]: ISCSI: New session from initiator iqn.1991-05.com.microsoft:pcb-722-ad1.pcbl722dom.local at IP addr 10.72.4.1
Mon Jun 19 09:00:00 BNT [filer02: kern.uptime.filer:info]:   9:00am up  1:27 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 364 iSCSI ops
Mon Jun 19 09:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 23 hours.
Mon Jun 19 10:00:00 BNT [filer02: kern.uptime.filer:info]:  10:00am up  2:27 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 101358 iSCSI ops
Mon Jun 19 10:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 22 hours.
Mon Jun 19 11:00:00 BNT [filer02: kern.uptime.filer:info]:  11:00am up  3:27 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 1721821 iSCSI ops
Mon Jun 19 11:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 21 hours.
Mon Jun 19 12:00:00 BNT [filer02: kern.uptime.filer:info]:  12:00pm up  4:27 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 3336556 iSCSI ops
Mon Jun 19 12:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 20 hours.
Mon Jun 19 12:00:16 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Mon Jun 19 12:00:37 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Mon Jun 19 12:04:54 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Mon Jun 19 12:08:43 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Mon Jun 19 12:12:51 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Mon Jun 19 12:16:33 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Mon Jun 19 12:36:23 BNT last message repeated 5 times
Mon Jun 19 12:39:58 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Mon Jun 19 12:56:12 BNT last message repeated 4 times
Mon Jun 19 12:56:12 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201706141600.0) for host (0) was not posted to NetApp. The system will drop the message
Mon Jun 19 13:00:00 BNT [filer02: kern.uptime.filer:info]:   1:00pm up  5:27 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 3361880 iSCSI ops
Mon Jun 19 13:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 19 hours.
Mon Jun 19 14:00:00 BNT [filer02: kern.uptime.filer:info]:   2:00pm up  6:28 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 3412581 iSCSI ops
Mon Jun 19 14:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 18 hours.
Mon Jun 19 15:00:00 BNT [filer02: kern.uptime.filer:info]:   3:00pm up  7:28 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 3727457 iSCSI ops
Mon Jun 19 15:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 17 hours.
Mon Jun 19 16:00:00 BNT [filer02: kern.uptime.filer:info]:   4:00pm up  8:28 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 3803203 iSCSI ops
Mon Jun 19 16:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 16 hours.
Mon Jun 19 16:00:01 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Mon Jun 19 16:00:17 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Mon Jun 19 16:04:20 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Mon Jun 19 16:08:25 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Mon Jun 19 16:12:33 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Mon Jun 19 16:16:47 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Mon Jun 19 16:35:56 BNT last message repeated 5 times
Mon Jun 19 16:40:04 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Mon Jun 19 16:56:51 BNT last message repeated 4 times
Mon Jun 19 16:56:51 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201706142000.0) for host (0) was not posted to NetApp. The system will drop the message
Mon Jun 19 17:00:00 BNT [filer02: kern.uptime.filer:info]:   5:00pm up  9:28 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 3811429 iSCSI ops
Mon Jun 19 17:00:01 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 15 hours.
Mon Jun 19 18:00:00 BNT [filer02: kern.uptime.filer:info]:   6:00pm up 10:28 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 3811429 iSCSI ops
Mon Jun 19 18:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 14 hours.
Mon Jun 19 19:00:00 BNT [filer02: kern.uptime.filer:info]:   7:00pm up 11:28 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 3811429 iSCSI ops
Mon Jun 19 19:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 13 hours.
Mon Jun 19 20:00:00 BNT [filer02: kern.uptime.filer:info]:   8:00pm up 12:28 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 3811429 iSCSI ops
Mon Jun 19 20:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 12 hours.
Mon Jun 19 20:00:00 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Mon Jun 19 20:00:17 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Mon Jun 19 20:04:03 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Mon Jun 19 20:08:02 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Mon Jun 19 20:11:47 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Mon Jun 19 20:15:38 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Mon Jun 19 20:35:50 BNT last message repeated 5 times
Mon Jun 19 20:40:02 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Mon Jun 19 20:55:03 BNT last message repeated 4 times
Mon Jun 19 20:55:03 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201706150000.0) for host (0) was not posted to NetApp. The system will drop the message
Mon Jun 19 21:00:00 BNT [filer02: kern.uptime.filer:info]:   9:00pm up 13:28 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 3811429 iSCSI ops
Mon Jun 19 21:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 11 hours.
Mon Jun 19 22:00:00 BNT [filer02: kern.uptime.filer:info]:  10:00pm up 14:28 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 3811429 iSCSI ops
Mon Jun 19 22:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 10 hours.
Mon Jun 19 23:00:00 BNT [filer02: kern.uptime.filer:info]:  11:00pm up 15:28 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 3811429 iSCSI ops
Mon Jun 19 23:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 9 hours.
Tue Jun 20 00:00:00 BNT [filer02: kern.uptime.filer:info]:  12:00am up 16:28 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 3811429 iSCSI ops
Tue Jun 20 00:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 8 hours.
Tue Jun 20 00:00:45 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of nightly.0 snapshot (Snapshot operation failed: No space left on device.)
Tue Jun 20 00:32:23 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Tue Jun 20 00:56:23 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201706150732.0) for host (0) was not posted to NetApp. The system will drop the message
Tue Jun 20 01:00:00 BNT [filer02: kern.uptime.filer:info]:   1:00am up 17:28 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 3811429 iSCSI ops
Tue Jun 20 01:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 7 hours.
Tue Jun 20 02:00:00 BNT [filer02: kern.uptime.filer:info]:   2:00am up 18:28 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 3811429 iSCSI ops
Tue Jun 20 02:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 6 hours.
Tue Jun 20 03:00:00 BNT [filer02: kern.uptime.filer:info]:   3:00am up 19:28 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 3811429 iSCSI ops
Tue Jun 20 03:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 5 hours.
Tue Jun 20 04:00:00 BNT [filer02: kern.uptime.filer:info]:   4:00am up 20:28 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 3811429 iSCSI ops
Tue Jun 20 04:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 4 hours.
Tue Jun 20 05:00:00 BNT [filer02: kern.uptime.filer:info]:   5:00am up 21:28 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 3811644 iSCSI ops
Tue Jun 20 05:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 3 hours.
Tue Jun 20 06:00:00 BNT [filer02: kern.uptime.filer:info]:   6:00am up 22:28 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 3811644 iSCSI ops
Tue Jun 20 06:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 2 hours.
Tue Jun 20 07:00:00 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] is still present in the system and should be removed.
Tue Jun 20 07:00:00 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] is still present in the system and should be removed.
Tue Jun 20 07:00:00 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] is still present in the system and should be removed.
Tue Jun 20 07:23:47 BNT [filer02: fci.link.break:error]: Link break detected on Fibre Channel adapter 0a.
Tue Jun 20 07:23:50 BNT [filer02: fci.adapter.error:warning]: Fibre Channel adapter driver encountered error "Failed to get loop map" on adapter 0a.
Mon Jun 19 23:24:34 GMT [localhost: rc:notice]: The system was down for 16 seconds
Mon Jun 19 23:24:34 GMT [filer02: cf.fsm.takeoverOfPartnerEnabled:notice]: Cluster monitor: takeover of filer01 enabled
Tue Jun 20 07:23:58 BNT [filer02: cf.nm.nicTransitionUp:info]: replayed event: Interconnect link 0 is UP
Tue Jun 20 07:24:05 BNT [filer02: fci.initialization.failed:error]: replayed event: Initialization failed on Fibre Channel adapter 0b.
Tue Jun 20 07:24:07 BNT [filer02: nvram.battery.turned.on:info]: replayed event: The NVRAM battery is turned ON. It is turned OFF during system shutdown.
Tue Jun 20 07:24:35 BNT [filer02: cf.fsm.takeoverByPartnerEnabled:notice]: Cluster monitor: takeover of filer02 by filer01 enabled
Tue Jun 20 07:24:10 BNT [filer02: diskown.isEnabled:info]: replayed event: software ownership has been enabled for this system
Tue Jun 20 07:24:14 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.2 is a local HA mailbox disk.
Tue Jun 20 07:24:14 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.0 is a local HA mailbox disk.
Tue Jun 20 07:24:14 BNT [filer02: fmmb.instStat.change:info]: replayed event: normal mailbox instance on local side.
Tue Jun 20 07:24:14 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.7 is a partner HA mailbox disk.
Tue Jun 20 07:24:14 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.3 is a partner HA mailbox disk.
Tue Jun 20 07:24:14 BNT [filer02: fmmb.instStat.change:info]: replayed event: normal mailbox instance on partner side.
Tue Jun 20 07:24:15 BNT [filer02: cf.fm.partner:info]: replayed event: Cluster monitor: partner 'filer01'
Tue Jun 20 07:24:15 BNT [filer02: cf.fm.timeMasterStatus:info]: replayed event: Acting as cluster time slave
Tue Jun 20 07:24:15 BNT [filer02: raid.assim.disk.brokenPreAssim:error]: replayed event: Broken Disk 0a.40 Shelf ? Bay ? [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] detected prior to assimilation. It should be removed.
Tue Jun 20 07:24:15 BNT [filer02: raid.assim.disk.brokenPreAssim:error]: replayed event: Broken Disk 0a.35 Shelf ? Bay ? [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] detected prior to assimilation. It should be removed.
Tue Jun 20 07:24:15 BNT [filer02: raid.assim.disk.brokenPreAssim:error]: replayed event: Broken Disk 0a.33 Shelf ? Bay ? [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] detected prior to assimilation. It should be removed.
Tue Jun 20 07:24:16 BNT [filer02: coredump.spare.none:info]: replayed event: No sparecore disk was found.
Tue Jun 20 07:24:16 BNT [filer02: raid.vol.replay.nvram:info]: replayed event: Performing raid replay on volume(s)
Tue Jun 20 07:24:16 BNT [filer02: raid.cksum.replay.summary:info]: replayed event: Replayed 0 checksum blocks.
Tue Jun 20 07:24:16 BNT [filer02: raid.stripe.replay.summary:info]: replayed event: Replayed 0 stripes.
Tue Jun 20 07:24:28 BNT [filer02: cf.fm.launch:info]: replayed event: Launching cluster monitor
Tue Jun 20 07:24:28 BNT [filer02: cf.fm.partner:info]: replayed event: Cluster monitor: partner 'filer01'
Tue Jun 20 07:24:33 BNT [filer02: cf.fsm.takeoverByPartnerDisabled:notice]: replayed event: Cluster monitor: takeover of  by filer01 disabled (partner booting)
Tue Jun 20 07:24:33 BNT [filer02: fcp.service.adapter:warning]: replayed event: FCP started but no adapter is present!
Tue Jun 20 07:24:34 BNT [filer02: kern.syslog.msg:notice]: replayed event: The system was down for 16 seconds
Tue Jun 20 07:24:34 BNT [filer02: cf.fsm.takeoverOfPartnerEnabled:notice]: replayed event: Cluster monitor: takeover of filer01 enabled
Tue Jun 20 07:24:35 BNT [filer02: disk.failmsg:error]: Disk 0a.40 (N00V0W8L): message received.
Tue Jun 20 07:24:35 BNT [filer02: disk.failmsg:error]: Disk 0a.35 (N00WLAUL): message received.
Tue Jun 20 07:24:35 BNT [filer02: disk.failmsg:error]: Disk 0a.33 (N00W75PL): message received.
Tue Jun 20 07:24:35 BNT [filer02: raid.disk.unload.done:info]: Unload of Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] has completed successfully
Tue Jun 20 07:24:35 BNT [filer02: raid.disk.unload.done:info]: Unload of Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] has completed successfully
Tue Jun 20 07:24:35 BNT [filer02: raid.disk.unload.done:info]: Unload of Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] has completed successfully
Tue Jun 20 07:24:35 BNT [filer02: raid.fdr.failed.ok:info]: Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] successfully deleted from spare pool
Tue Jun 20 07:24:35 BNT [filer02: raid.fdr.failed.ok:info]: Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] successfully deleted from spare pool
Tue Jun 20 07:24:35 BNT [filer02: raid.fdr.failed.ok:info]: Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] successfully deleted from spare pool
Tue Jun 20 07:24:36 BNT [filer02: dfu.firmwareUpToDate:info]: Firmware is up-to-date on all disk drives
Tue Jun 20 07:24:37 BNT [filer02: perf.archive.start:info]: Performance archiver started. Sampling 22 objects and 198 counters.
Tue Jun 20 07:24:39 BNT [filer02: netif.linkUp:info]: Ethernet e0b: Link up.
Tue Jun 20 07:24:39 BNT [filer02: netif.linkUp:info]: Ethernet e0a: Link up.
Tue Jun 20 07:24:40 BNT [filer02: pvif.switchLink:warning]: vif1: switching to e0a
Tue Jun 20 07:24:41 BNT [filer02: rc:info]: relog syslog Tue Jun 20 07:23:56 BNT [filer02: fci.adapter.error:warning]: Fibre Channel adapter driver encountered error "Failed t
Tue Jun 20 07:24:41 BNT [filer02: rc:info]: relog syslog Tue Jun 20 07:23:56 BNT [filer02: fci.adapter.reset:debug]: Resetting Fibre Channel adapter 0a.
Tue Jun 20 07:24:41 BNT [filer02: iscsi.service.startup:info]: iSCSI service startup
Tue Jun 20 07:24:42 BNT [filer02: cmds.sysconf.logErr:error]: sysconfig: Unless directed by NetApp Global Services the cf.takeover.on_panic option should be set to On. .
Tue Jun 20 07:24:42 BNT [filer02: raid.rg.recons.missing:notice]: RAID group /aggr1/plex0/rg0 is missing 2 disk(s).
Tue Jun 20 07:24:42 BNT [filer02: raid.rg.recons.cantStart:warning]: The reconstruction cannot start in RAID group /aggr1/plex0/rg0: No 520 bps disk of required size available in spare pool, targeting any  spare pool
Tue Jun 20 07:24:42 BNT [filer02: mgr.boot.disk_done:info]: NetApp Release 7.3.2 boot complete. Last disk update written at Tue Jun 20 07:23:37 BNT 2017
Tue Jun 20 07:24:42 BNT [filer02: monitor.raid.brokenDisk:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken.
Tue Jun 20 07:24:42 BNT [filer02: monitor.brokenDisk.notice:notice]: When two disks are broken in raid_dp volume,  the system shuts down automatically every 24  hours to encourage you to replace the disk.  If you reboot the system it will run for another  24 hours before shutting down. (The 24 hour timeout  may be increased by altering the "raid.timeout"  value using the "options" command.)
Tue Jun 20 07:24:43 BNT [filer02: shelf.config.spha:info]: System is using single path HA attached storage only.
Tue Jun 20 07:24:43 BNT [filer02: diskown.errorReadingOwnership:warning]: error 3 (disk failed) while reading ownership on disk 0a.24 (S/N N00SSSAL)
Tue Jun 20 07:24:43 BNT [filer02: diskown.errorReadingOwnership:warning]: error 3 (disk failed) while reading ownership on disk 0a.26 (S/N N00V0UWL)
Tue Jun 20 07:24:43 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] is still present in the system and should be removed.
Tue Jun 20 07:24:43 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] is still present in the system and should be removed.
Tue Jun 20 07:24:43 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] is still present in the system and should be removed.
Tue Jun 20 07:24:43 BNT [filer02: mgr.boot.reason_ok:notice]: System rebooted after power-on.
Tue Jun 20 07:25:03 BNT [filer02: raid.rg.spares.low:warning]: /aggr1/plex0/rg0
Tue Jun 20 07:25:03 BNT [filer02: monitor.globalStatus.nonCritical:warning]: Disk on adapter 0a, shelf 2, bay 8, failed. Disk on adapter 0a, shelf 2, bay 3, failed. Disk on adapter 0a, shelf 2, bay 1, failed.  There are not enough spare disks. /vol/vol1 is full (using or reserving 100% of space and 0% of inodes, using 92% of reserve).
Tue Jun 20 07:25:09 BNT [filer02: nbt.nbns.registrationComplete:info]: NBT: All CIFS name registrations have completed for the local server.
Tue Jun 20 07:25:13 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SHUTDOWN PENDING (degraded mode))
Tue Jun 20 07:25:20 BNT last message repeated 2 times
Tue Jun 20 07:25:20 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201706150732.1) was not posted for host (0), it was dropped from queue due to lack of space
Tue Jun 20 07:25:27 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (REBOOT (power on))
Tue Jun 20 07:25:27 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201706150732.2) was not posted for host (0), it was dropped from queue due to lack of space
Tue Jun 20 07:25:34 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SPARES_LOW)
Tue Jun 20 07:25:34 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201706150732.3) was not posted for host (0), it was dropped from queue due to lack of space
Tue Jun 20 07:25:53 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (PERFORMANCE SNAPSHOT)
Tue Jun 20 07:25:53 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201706150733.0) was not posted for host (0), it was dropped from queue due to lack of space
Tue Jun 20 07:30:31 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Tue Jun 20 07:34:32 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Tue Jun 20 07:38:44 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Tue Jun 20 07:43:09 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Tue Jun 20 07:47:22 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Tue Jun 20 07:59:41 BNT last message repeated 3 times
Tue Jun 20 08:00:00 BNT [filer02: kern.uptime.filer:info]:   8:00am up 35 mins, 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 0 iSCSI ops
Tue Jun 20 08:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 24 hours.
Tue Jun 20 08:00:01 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Tue Jun 20 08:00:10 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Tue Jun 20 08:00:10 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201706150800.0) for host (0) was not posted to NetApp. The system will drop the message
Tue Jun 20 08:03:27 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Tue Jun 20 08:07:33 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Tue Jun 20 08:11:39 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Tue Jun 20 08:15:39 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Tue Jun 20 08:19:11 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Tue Jun 20 08:35:32 BNT last message repeated 4 times
Tue Jun 20 08:39:34 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Tue Jun 20 08:39:44 BNT [filer02: iscsi.notice:notice]: ISCSI: New session from initiator iqn.1991-05.com.microsoft:pcb-722-ad1.pcbl722dom.local at IP addr 10.72.4.1
Tue Jun 20 08:43:16 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Tue Jun 20 08:47:28 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Tue Jun 20 08:51:56 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Tue Jun 20 08:55:45 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Tue Jun 20 08:59:27 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Tue Jun 20 08:59:27 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201706151200.0) for host (0) was not posted to NetApp. The system will drop the message
Tue Jun 20 09:00:00 BNT [filer02: kern.uptime.filer:info]:   9:00am up  1:35 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 241 iSCSI ops
Tue Jun 20 09:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 23 hours.
Tue Jun 20 10:00:00 BNT [filer02: kern.uptime.filer:info]:  10:00am up  2:35 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 4722 iSCSI ops
Tue Jun 20 10:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 22 hours.
Tue Jun 20 11:00:00 BNT [filer02: kern.uptime.filer:info]:  11:00am up  3:35 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 19846 iSCSI ops
Tue Jun 20 11:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 21 hours.
Tue Jun 20 12:00:00 BNT [filer02: kern.uptime.filer:info]:  12:00pm up  4:35 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 46072 iSCSI ops
Tue Jun 20 12:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 20 hours.
Tue Jun 20 12:00:45 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Tue Jun 20 12:01:00 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Tue Jun 20 12:04:51 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Tue Jun 20 12:09:04 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Tue Jun 20 12:13:22 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Tue Jun 20 12:16:57 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Tue Jun 20 12:33:12 BNT last message repeated 4 times
Tue Jun 20 12:37:40 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Tue Jun 20 12:57:33 BNT last message repeated 5 times
Tue Jun 20 12:57:33 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201706151600.0) for host (0) was not posted to NetApp. The system will drop the message
Tue Jun 20 13:00:00 BNT [filer02: kern.uptime.filer:info]:   1:00pm up  5:35 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 47704 iSCSI ops
Tue Jun 20 13:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 19 hours.
Tue Jun 20 14:00:00 BNT [filer02: kern.uptime.filer:info]:   2:00pm up  6:35 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 48011 iSCSI ops
Tue Jun 20 14:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 18 hours.
Tue Jun 20 15:00:00 BNT [filer02: kern.uptime.filer:info]:   3:00pm up  7:35 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 64257 iSCSI ops
Tue Jun 20 15:00:01 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 17 hours.
Tue Jun 20 16:00:00 BNT [filer02: kern.uptime.filer:info]:   4:00pm up  8:35 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 75948 iSCSI ops
Tue Jun 20 16:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 16 hours.
Tue Jun 20 16:00:46 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Tue Jun 20 16:12:51 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Tue Jun 20 16:21:00 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Tue Jun 20 16:55:41 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201706152000.0) for host (0) was not posted to NetApp. The system will drop the message
Tue Jun 20 17:00:00 BNT [filer02: kern.uptime.filer:info]:   5:00pm up  9:35 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 77448 iSCSI ops
Tue Jun 20 17:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 15 hours.
Tue Jun 20 18:00:00 BNT [filer02: kern.uptime.filer:info]:   6:00pm up 10:35 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 77448 iSCSI ops
Tue Jun 20 18:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 14 hours.
Tue Jun 20 19:00:00 BNT [filer02: kern.uptime.filer:info]:   7:00pm up 11:35 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 77448 iSCSI ops
Tue Jun 20 19:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 13 hours.
Tue Jun 20 20:00:00 BNT [filer02: kern.uptime.filer:info]:   8:00pm up 12:35 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 77448 iSCSI ops
Tue Jun 20 20:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 12 hours.
Tue Jun 20 20:00:31 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Tue Jun 20 20:21:01 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Tue Jun 20 20:57:54 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201706160000.0) for host (0) was not posted to NetApp. The system will drop the message
Tue Jun 20 21:00:00 BNT [filer02: kern.uptime.filer:info]:   9:00pm up 13:35 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 77448 iSCSI ops
Tue Jun 20 21:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 11 hours.
Tue Jun 20 22:00:00 BNT [filer02: kern.uptime.filer:info]:  10:00pm up 14:35 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 77448 iSCSI ops
Tue Jun 20 22:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 10 hours.
Tue Jun 20 23:00:00 BNT [filer02: kern.uptime.filer:info]:  11:00pm up 15:35 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 77448 iSCSI ops
Tue Jun 20 23:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 9 hours.
Wed Jun 21 00:00:00 BNT [filer02: kern.uptime.filer:info]:  12:00am up 16:35 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 77448 iSCSI ops
Wed Jun 21 00:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 8 hours.
Wed Jun 21 00:00:30 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of nightly.0 snapshot (Snapshot operation failed: No space left on device.)
Wed Jun 21 00:54:08 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201706160728.0) for host (0) was not posted to NetApp. The system will drop the message
Wed Jun 21 01:00:00 BNT [filer02: kern.uptime.filer:info]:   1:00am up 17:35 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 77448 iSCSI ops
Wed Jun 21 01:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 7 hours.
Wed Jun 21 02:00:00 BNT [filer02: kern.uptime.filer:info]:   2:00am up 18:35 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 77448 iSCSI ops
Wed Jun 21 02:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 6 hours.
Wed Jun 21 03:00:00 BNT [filer02: kern.uptime.filer:info]:   3:00am up 19:35 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 77448 iSCSI ops
Wed Jun 21 03:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 5 hours.
Wed Jun 21 04:00:00 BNT [filer02: kern.uptime.filer:info]:   4:00am up 20:35 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 77496 iSCSI ops
Wed Jun 21 04:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 4 hours.
Wed Jun 21 05:00:00 BNT [filer02: kern.uptime.filer:info]:   5:00am up 21:35 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 77496 iSCSI ops
Wed Jun 21 05:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 3 hours.
Wed Jun 21 06:00:00 BNT [filer02: kern.uptime.filer:info]:   6:00am up 22:35 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 77496 iSCSI ops
Wed Jun 21 06:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 2 hours.
Wed Jun 21 07:00:00 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] is still present in the system and should be removed.
Wed Jun 21 07:00:00 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] is still present in the system and should be removed.
Wed Jun 21 07:00:00 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] is still present in the system and should be removed.
Wed Jun 21 07:52:34 BNT [filer02: fci.link.break:error]: Link break detected on Fibre Channel adapter 0a.
Tue Jun 20 23:53:11 GMT [localhost: rc:notice]: The system was down for 15 seconds
Wed Jun 21 07:52:35 BNT [filer02: cf.nm.nicTransitionUp:info]: replayed event: Interconnect link 0 is UP
Wed Jun 21 07:52:42 BNT [filer02: fci.initialization.failed:error]: replayed event: Initialization failed on Fibre Channel adapter 0b.
Wed Jun 21 07:52:44 BNT [filer02: nvram.battery.turned.on:info]: replayed event: The NVRAM battery is turned ON. It is turned OFF during system shutdown.
Wed Jun 21 07:52:47 BNT [filer02: diskown.isEnabled:info]: replayed event: software ownership has been enabled for this system
Wed Jun 21 07:52:51 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.2 is a local HA mailbox disk.
Wed Jun 21 07:52:51 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.0 is a local HA mailbox disk.
Wed Jun 21 07:52:51 BNT [filer02: fmmb.instStat.change:info]: replayed event: normal mailbox instance on local side.
Wed Jun 21 07:52:51 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.7 is a partner HA mailbox disk.
Wed Jun 21 07:52:51 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.3 is a partner HA mailbox disk.
Wed Jun 21 07:52:51 BNT [filer02: fmmb.instStat.change:info]: replayed event: normal mailbox instance on partner side.
Wed Jun 21 07:52:52 BNT [filer02: cf.fm.partner:info]: replayed event: Cluster monitor: partner 'filer01'
Wed Jun 21 07:52:52 BNT [filer02: cf.fm.timeMasterStatus:info]: replayed event: Acting as cluster time slave
Wed Jun 21 07:52:52 BNT [filer02: raid.assim.disk.brokenPreAssim:error]: replayed event: Broken Disk 0a.40 Shelf ? Bay ? [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] detected prior to assimilation. It should be removed.
Wed Jun 21 07:52:52 BNT [filer02: raid.assim.disk.brokenPreAssim:error]: replayed event: Broken Disk 0a.35 Shelf ? Bay ? [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] detected prior to assimilation. It should be removed.
Wed Jun 21 07:52:52 BNT [filer02: raid.assim.disk.brokenPreAssim:error]: replayed event: Broken Disk 0a.33 Shelf ? Bay ? [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] detected prior to assimilation. It should be removed.
Wed Jun 21 07:52:53 BNT [filer02: coredump.spare.none:info]: replayed event: No sparecore disk was found.
Wed Jun 21 07:52:53 BNT [filer02: raid.vol.replay.nvram:info]: replayed event: Performing raid replay on volume(s)
Wed Jun 21 07:52:53 BNT [filer02: raid.cksum.replay.summary:info]: replayed event: Replayed 0 checksum blocks.
Wed Jun 21 07:52:53 BNT [filer02: raid.stripe.replay.summary:info]: replayed event: Replayed 0 stripes.
Wed Jun 21 07:52:56 BNT [filer02: ses.status.psWarning:warning]: replayed event: DS12-ESAS shelf 0 on channel 0c power warning for Power supply 1: non-critical status; DC undervoltage fault. This module is on the rear side of the shelf, at the left.
Wed Jun 21 07:52:56 BNT [filer02: ses.status.psWarning:warning]: replayed event: DS12-ESAS shelf 0 on channel 0c power warning for Power supply 2: non-critical status; DC undervoltage fault. This module is on the rear side of the shelf, at the right.
Wed Jun 21 07:53:05 BNT [filer02: cf.fm.launch:info]: replayed event: Launching cluster monitor
Wed Jun 21 07:53:05 BNT [filer02: cf.fm.partner:info]: replayed event: Cluster monitor: partner 'filer01'
Wed Jun 21 07:53:09 BNT [filer02: cf.fsm.takeoverOfPartnerDisabled:notice]: replayed event: Cluster monitor: takeover of filer01 disabled (partner booting)
Wed Jun 21 07:53:10 BNT [filer02: fcp.service.adapter:warning]: replayed event: FCP started but no adapter is present!
Wed Jun 21 07:53:10 BNT [filer02: cf.fsm.takeoverOfPartnerEnabled:notice]: replayed event: Cluster monitor: takeover of filer01 enabled
Wed Jun 21 07:53:10 BNT [filer02: cf.fsm.takeoverByPartnerDisabled:notice]: replayed event: Cluster monitor: takeover of  by filer01 disabled (partner booting)
Wed Jun 21 07:53:11 BNT [filer02: kern.syslog.msg:notice]: replayed event: The system was down for 15 seconds
Wed Jun 21 07:53:12 BNT [filer02: disk.failmsg:error]: Disk 0a.40 (N00V0W8L): message received.
Wed Jun 21 07:53:12 BNT [filer02: disk.failmsg:error]: Disk 0a.35 (N00WLAUL): message received.
Wed Jun 21 07:53:12 BNT [filer02: disk.failmsg:error]: Disk 0a.33 (N00W75PL): message received.
Wed Jun 21 07:53:12 BNT [filer02: raid.disk.unload.done:info]: Unload of Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] has completed successfully
Wed Jun 21 07:53:12 BNT [filer02: raid.disk.unload.done:info]: Unload of Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] has completed successfully
Wed Jun 21 07:53:12 BNT [filer02: raid.disk.unload.done:info]: Unload of Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] has completed successfully
Wed Jun 21 07:53:12 BNT [filer02: raid.fdr.failed.ok:info]: Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] successfully deleted from spare pool
Wed Jun 21 07:53:12 BNT [filer02: raid.fdr.failed.ok:info]: Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] successfully deleted from spare pool
Wed Jun 21 07:53:12 BNT [filer02: raid.fdr.failed.ok:info]: Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] successfully deleted from spare pool
Wed Jun 21 07:53:12 BNT [filer02: cf.fsm.takeoverByPartnerEnabled:notice]: Cluster monitor: takeover of filer02 by filer01 enabled
Wed Jun 21 07:53:13 BNT [filer02: dfu.firmwareUpToDate:info]: Firmware is up-to-date on all disk drives
Wed Jun 21 07:53:14 BNT [filer02: perf.archive.start:info]: Performance archiver started. Sampling 22 objects and 198 counters.
Wed Jun 21 07:53:16 BNT [filer02: netif.linkUp:info]: Ethernet e0b: Link up.
Wed Jun 21 07:53:16 BNT [filer02: netif.linkUp:info]: Ethernet e0a: Link up.
Wed Jun 21 07:53:17 BNT [filer02: pvif.switchLink:warning]: vif1: switching to e0a
Wed Jun 21 07:53:18 BNT [filer02: rc:info]: relog syslog Wed Jun 21 07:25:53 BNT [filer02: mem_scrub_admin:debug]: mem_scrub: completed system memory scrub (count = 1)
Wed Jun 21 07:53:18 BNT [filer02: rc:info]: relog syslog Wed Jun 21 07:52:34 BNT [filer02: fci.link.break:error]: Link break detected on Fibre Channel adapter 0a.
Wed Jun 21 07:53:18 BNT [filer02: iscsi.service.startup:info]: iSCSI service startup
Wed Jun 21 07:53:19 BNT [filer02: cmds.sysconf.logErr:error]: sysconfig: Unless directed by NetApp Global Services the cf.takeover.on_panic option should be set to On. .
Wed Jun 21 07:53:19 BNT [filer02: raid.rg.recons.missing:notice]: RAID group /aggr1/plex0/rg0 is missing 2 disk(s).
Wed Jun 21 07:53:19 BNT [filer02: raid.rg.recons.cantStart:warning]: The reconstruction cannot start in RAID group /aggr1/plex0/rg0: No 520 bps disk of required size available in spare pool, targeting any  spare pool
Wed Jun 21 07:53:19 BNT [filer02: mgr.boot.disk_done:info]: NetApp Release 7.3.2 boot complete. Last disk update written at Wed Jun 21 07:52:27 BNT 2017
Wed Jun 21 07:53:19 BNT [filer02: monitor.raid.brokenDisk:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken.
Wed Jun 21 07:53:19 BNT [filer02: monitor.brokenDisk.notice:notice]: When two disks are broken in raid_dp volume,  the system shuts down automatically every 24  hours to encourage you to replace the disk.  If you reboot the system it will run for another  24 hours before shutting down. (The 24 hour timeout  may be increased by altering the "raid.timeout"  value using the "options" command.)
Wed Jun 21 07:53:19 BNT [filer02: diskown.errorReadingOwnership:warning]: error 3 (disk failed) while reading ownership on disk 0a.24 (S/N N00SSSAL)
Wed Jun 21 07:53:19 BNT [filer02: diskown.errorReadingOwnership:warning]: error 3 (disk failed) while reading ownership on disk 0a.26 (S/N N00V0UWL)
Wed Jun 21 07:53:19 BNT [filer02: shelf.config.spha:info]: System is using single path HA attached storage only.
Wed Jun 21 07:53:19 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] is still present in the system and should be removed.
Wed Jun 21 07:53:19 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] is still present in the system and should be removed.
Wed Jun 21 07:53:19 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] is still present in the system and should be removed.
Wed Jun 21 07:53:20 BNT [filer02: mgr.boot.reason_ok:notice]: System rebooted after power-on.
Wed Jun 21 07:53:43 BNT [filer02: raid.rg.spares.low:warning]: /aggr1/plex0/rg0
Wed Jun 21 07:53:47 BNT [filer02: nbt.nbns.registrationComplete:info]: NBT: All CIFS name registrations have completed for the local server.
Wed Jun 21 07:53:49 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SHUTDOWN PENDING (degraded mode))
Wed Jun 21 07:53:57 BNT last message repeated 2 times
Wed Jun 21 07:53:57 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201706160728.1) was not posted for host (0), it was dropped from queue due to lack of space
Wed Jun 21 07:54:03 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (REBOOT (power on))
Wed Jun 21 07:54:03 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201706160729.0) was not posted for host (0), it was dropped from queue due to lack of space
Wed Jun 21 07:54:06 BNT [filer02: monitor.globalStatus.nonCritical:warning]: Disk on adapter 0a, shelf 2, bay 8, failed. Disk on adapter 0a, shelf 2, bay 3, failed. Disk on adapter 0a, shelf 2, bay 1, failed.  There are not enough spare disks. /vol/vol1 is full (using or reserving 100% of space and 0% of inodes, using 92% of reserve).
Wed Jun 21 07:54:10 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SPARES_LOW)
Wed Jun 21 07:54:10 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201706160729.1) was not posted for host (0), it was dropped from queue due to lack of space
Wed Jun 21 07:54:30 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (PERFORMANCE SNAPSHOT)
Wed Jun 21 07:54:30 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201706160729.2) was not posted for host (0), it was dropped from queue due to lack of space
Wed Jun 21 07:59:32 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Wed Jun 21 08:00:07 BNT [filer02: kern.uptime.filer:info]:   8:00am up 6 mins, 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 0 iSCSI ops
Wed Jun 21 08:00:07 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 24 hours.
Wed Jun 21 08:00:18 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Wed Jun 21 08:00:26 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Wed Jun 21 08:00:26 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201706160800.0) was not posted for host (0), it was dropped from queue due to lack of space
Wed Jun 21 08:04:43 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Wed Jun 21 08:08:59 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Wed Jun 21 08:13:11 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Wed Jun 21 08:16:49 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Wed Jun 21 08:20:32 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Wed Jun 21 08:28:41 BNT last message repeated 2 times
Wed Jun 21 08:28:41 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201706161200.0) for host (0) was not posted to NetApp. The system will drop the message
Wed Jun 21 08:30:10 BNT [filer02: iscsi.notice:notice]: ISCSI: New session from initiator iqn.1991-05.com.microsoft:pcb-722-ad1.pcbl722dom.local at IP addr 10.72.4.1
Wed Jun 21 09:00:00 BNT [filer02: kern.uptime.filer:info]:   9:00am up  1:06 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 153 iSCSI ops
Wed Jun 21 09:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 23 hours.
Wed Jun 21 10:00:00 BNT [filer02: kern.uptime.filer:info]:  10:00am up  2:06 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 7519 iSCSI ops
Wed Jun 21 10:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 22 hours.
Wed Jun 21 11:00:00 BNT [filer02: kern.uptime.filer:info]:  11:00am up  3:06 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 220435 iSCSI ops
Wed Jun 21 11:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 21 hours.
Wed Jun 21 12:00:00 BNT [filer02: kern.uptime.filer:info]:  12:00pm up  4:06 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 569284 iSCSI ops
Wed Jun 21 12:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 20 hours.
Wed Jun 21 12:00:30 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Wed Jun 21 12:00:45 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Wed Jun 21 12:04:49 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Wed Jun 21 12:08:25 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Wed Jun 21 12:12:18 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Wed Jun 21 12:16:09 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Wed Jun 21 12:33:03 BNT last message repeated 4 times
Wed Jun 21 12:37:30 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Wed Jun 21 12:56:27 BNT last message repeated 5 times
Wed Jun 21 12:56:27 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201706161600.0) for host (0) was not posted to NetApp. The system will drop the message
Wed Jun 21 13:00:00 BNT [filer02: kern.uptime.filer:info]:   1:00pm up  5:06 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 581849 iSCSI ops
Wed Jun 21 13:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 19 hours.
Wed Jun 21 14:00:00 BNT [filer02: kern.uptime.filer:info]:   2:00pm up  6:06 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 897173 iSCSI ops
Wed Jun 21 14:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 18 hours.
Wed Jun 21 15:00:00 BNT [filer02: kern.uptime.filer:info]:   3:00pm up  7:06 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 1244630 iSCSI ops
Wed Jun 21 15:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 17 hours.
Wed Jun 21 16:00:00 BNT [filer02: kern.uptime.filer:info]:   4:00pm up  8:06 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 1252238 iSCSI ops
Wed Jun 21 16:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 16 hours.
Wed Jun 21 16:00:15 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Wed Jun 21 16:00:32 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Wed Jun 21 16:04:36 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Wed Jun 21 16:08:59 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Wed Jun 21 16:12:38 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Wed Jun 21 16:16:37 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Wed Jun 21 16:36:22 BNT last message repeated 5 times
Wed Jun 21 16:40:02 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Wed Jun 21 16:55:53 BNT last message repeated 4 times
Wed Jun 21 16:55:53 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201706162000.0) for host (0) was not posted to NetApp. The system will drop the message
Wed Jun 21 17:00:00 BNT [filer02: kern.uptime.filer:info]:   5:00pm up  9:06 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 1252238 iSCSI ops
Wed Jun 21 17:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 15 hours.
Wed Jun 21 18:00:00 BNT [filer02: kern.uptime.filer:info]:   6:00pm up 10:06 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 1252238 iSCSI ops
Wed Jun 21 18:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 14 hours.
Wed Jun 21 19:00:00 BNT [filer02: kern.uptime.filer:info]:   7:00pm up 11:06 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 1252238 iSCSI ops
Wed Jun 21 19:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 13 hours.
Wed Jun 21 20:00:00 BNT [filer02: kern.uptime.filer:info]:   8:00pm up 12:06 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 1252238 iSCSI ops
Wed Jun 21 20:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 12 hours.
Wed Jun 21 20:00:16 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Wed Jun 21 20:16:22 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Wed Jun 21 20:56:33 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201706170000.0) for host (0) was not posted to NetApp. The system will drop the message
Wed Jun 21 21:00:00 BNT [filer02: kern.uptime.filer:info]:   9:00pm up 13:06 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 1252238 iSCSI ops
Wed Jun 21 21:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 11 hours.
Wed Jun 21 22:00:00 BNT [filer02: kern.uptime.filer:info]:  10:00pm up 14:07 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 1252238 iSCSI ops
Wed Jun 21 22:00:01 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 10 hours.
Wed Jun 21 23:00:00 BNT [filer02: kern.uptime.filer:info]:  11:00pm up 15:07 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 1252238 iSCSI ops
Wed Jun 21 23:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 9 hours.
Thu Jun 22 00:00:00 BNT [filer02: kern.uptime.filer:info]:  12:00am up 16:07 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 1252238 iSCSI ops
Thu Jun 22 00:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 8 hours.
Thu Jun 22 00:00:01 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of nightly.0 snapshot (Snapshot operation failed: No space left on device.)
Thu Jun 22 00:38:54 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Thu Jun 22 00:54:46 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Thu Jun 22 00:54:46 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201706181144.0) for host (0) was not posted to NetApp. The system will drop the message
Thu Jun 22 01:00:00 BNT [filer02: kern.uptime.filer:info]:   1:00am up 17:07 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 1252238 iSCSI ops
Thu Jun 22 01:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 7 hours.
Thu Jun 22 02:00:00 BNT [filer02: kern.uptime.filer:info]:   2:00am up 18:07 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 1252238 iSCSI ops
Thu Jun 22 02:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 6 hours.
Thu Jun 22 03:00:00 BNT [filer02: kern.uptime.filer:info]:   3:00am up 19:07 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 1252238 iSCSI ops
Thu Jun 22 03:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 5 hours.
Thu Jun 22 04:00:00 BNT [filer02: kern.uptime.filer:info]:   4:00am up 20:07 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 1252238 iSCSI ops
Thu Jun 22 04:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 4 hours.
Thu Jun 22 05:00:00 BNT [filer02: kern.uptime.filer:info]:   5:00am up 21:07 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 1252238 iSCSI ops
Thu Jun 22 05:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 3 hours.
Thu Jun 22 06:00:00 BNT [filer02: kern.uptime.filer:info]:   6:00am up 22:07 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 1252238 iSCSI ops
Thu Jun 22 06:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 2 hours.
Thu Jun 22 07:00:00 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] is still present in the system and should be removed.
Thu Jun 22 07:00:00 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] is still present in the system and should be removed.
Thu Jun 22 07:00:00 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] is still present in the system and should be removed.
Wed Jun 21 23:43:46 GMT [localhost: cf.fsm.takeoverOfPartnerEnabled:notice]: Cluster monitor: takeover of filer01 enabled
Wed Jun 21 23:43:46 GMT [localhost: rc:notice]: The system was down for 31 seconds
Thu Jun 22 07:43:47 BNT [filer02: cf.fsm.takeoverByPartnerEnabled:notice]: Cluster monitor: takeover of filer02 by filer01 enabled
Thu Jun 22 07:43:10 BNT [filer02: cf.nm.nicTransitionUp:info]: replayed event: Interconnect link 0 is UP
Thu Jun 22 07:43:17 BNT [filer02: fci.initialization.failed:error]: replayed event: Initialization failed on Fibre Channel adapter 0b.
Thu Jun 22 07:43:19 BNT [filer02: nvram.battery.turned.on:info]: replayed event: The NVRAM battery is turned ON. It is turned OFF during system shutdown.
Thu Jun 22 07:43:22 BNT [filer02: diskown.isEnabled:info]: replayed event: software ownership has been enabled for this system
Thu Jun 22 07:43:26 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.2 is a local HA mailbox disk.
Thu Jun 22 07:43:26 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.0 is a local HA mailbox disk.
Thu Jun 22 07:43:26 BNT [filer02: fmmb.instStat.change:info]: replayed event: normal mailbox instance on local side.
Thu Jun 22 07:43:26 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.7 is a partner HA mailbox disk.
Thu Jun 22 07:43:26 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.3 is a partner HA mailbox disk.
Thu Jun 22 07:43:26 BNT [filer02: fmmb.instStat.change:info]: replayed event: normal mailbox instance on partner side.
Thu Jun 22 07:43:27 BNT [filer02: cf.fm.partner:info]: replayed event: Cluster monitor: partner 'filer01'
Thu Jun 22 07:43:27 BNT [filer02: cf.fm.timeMasterStatus:info]: replayed event: Acting as cluster time slave
Thu Jun 22 07:43:27 BNT [filer02: raid.assim.disk.brokenPreAssim:error]: replayed event: Broken Disk 0a.40 Shelf ? Bay ? [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] detected prior to assimilation. It should be removed.
Thu Jun 22 07:43:27 BNT [filer02: raid.assim.disk.brokenPreAssim:error]: replayed event: Broken Disk 0a.35 Shelf ? Bay ? [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] detected prior to assimilation. It should be removed.
Thu Jun 22 07:43:27 BNT [filer02: raid.assim.disk.brokenPreAssim:error]: replayed event: Broken Disk 0a.33 Shelf ? Bay ? [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] detected prior to assimilation. It should be removed.
Thu Jun 22 07:43:28 BNT [filer02: coredump.spare.none:info]: replayed event: No sparecore disk was found.
Thu Jun 22 07:43:29 BNT [filer02: raid.vol.replay.nvram:info]: replayed event: Performing raid replay on volume(s)
Thu Jun 22 07:43:29 BNT [filer02: raid.cksum.replay.summary:info]: replayed event: Replayed 0 checksum blocks.
Thu Jun 22 07:43:29 BNT [filer02: raid.stripe.replay.summary:info]: replayed event: Replayed 0 stripes.
Thu Jun 22 07:43:31 BNT [filer02: ses.status.psWarning:warning]: replayed event: DS12-ESAS shelf 0 on channel 0c power warning for Power supply 1: non-critical status; DC undervoltage fault. This module is on the rear side of the shelf, at the left.
Thu Jun 22 07:43:31 BNT [filer02: ses.status.psWarning:warning]: replayed event: DS12-ESAS shelf 0 on channel 0c power warning for Power supply 2: non-critical status; DC undervoltage fault. This module is on the rear side of the shelf, at the right.
Thu Jun 22 07:43:40 BNT [filer02: cf.fm.launch:info]: replayed event: Launching cluster monitor
Thu Jun 22 07:43:41 BNT [filer02: cf.fm.partner:info]: replayed event: Cluster monitor: partner 'filer01'
Thu Jun 22 07:43:46 BNT [filer02: fcp.service.adapter:warning]: replayed event: FCP started but no adapter is present!
Thu Jun 22 07:43:46 BNT [filer02: cf.fsm.takeoverOfPartnerEnabled:notice]: replayed event: Cluster monitor: takeover of filer01 enabled
Thu Jun 22 07:43:46 BNT [filer02: kern.syslog.msg:notice]: replayed event: The system was down for 31 seconds
Thu Jun 22 07:43:47 BNT [filer02: cf.fsm.takeoverByPartnerEnabled:notice]: replayed event: Cluster monitor: takeover of filer02 by filer01 enabled
Thu Jun 22 07:43:48 BNT [filer02: disk.failmsg:error]: Disk 0a.40 (N00V0W8L): message received.
Thu Jun 22 07:43:48 BNT [filer02: disk.failmsg:error]: Disk 0a.35 (N00WLAUL): message received.
Thu Jun 22 07:43:48 BNT [filer02: disk.failmsg:error]: Disk 0a.33 (N00W75PL): message received.
Thu Jun 22 07:43:48 BNT [filer02: raid.disk.unload.done:info]: Unload of Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] has completed successfully
Thu Jun 22 07:43:48 BNT [filer02: raid.disk.unload.done:info]: Unload of Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] has completed successfully
Thu Jun 22 07:43:48 BNT [filer02: raid.disk.unload.done:info]: Unload of Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] has completed successfully
Thu Jun 22 07:43:48 BNT [filer02: raid.fdr.failed.ok:info]: Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] successfully deleted from spare pool
Thu Jun 22 07:43:48 BNT [filer02: raid.fdr.failed.ok:info]: Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] successfully deleted from spare pool
Thu Jun 22 07:43:48 BNT [filer02: raid.fdr.failed.ok:info]: Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] successfully deleted from spare pool
Thu Jun 22 07:43:48 BNT [filer02: dfu.firmwareUpToDate:info]: Firmware is up-to-date on all disk drives
Thu Jun 22 07:43:50 BNT [filer02: perf.archive.start:info]: Performance archiver started. Sampling 22 objects and 198 counters.
Thu Jun 22 07:43:52 BNT [filer02: netif.linkUp:info]: Ethernet e0a: Link up.
Thu Jun 22 07:43:52 BNT [filer02: netif.linkUp:info]: Ethernet e0b: Link up.
Thu Jun 22 07:43:53 BNT [filer02: pvif.switchLink:warning]: vif1: switching to e0a
Thu Jun 22 07:43:54 BNT [filer02: rc:info]: relog syslog Thu Jun 22 07:00:00 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TS
Thu Jun 22 07:43:54 BNT [filer02: rc:info]: relog syslog Thu Jun 22 07:00:00 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TS
Thu Jun 22 07:43:54 BNT [filer02: iscsi.service.startup:info]: iSCSI service startup
Thu Jun 22 07:43:55 BNT [filer02: cmds.sysconf.logErr:error]: sysconfig: Unless directed by NetApp Global Services the cf.takeover.on_panic option should be set to On. .
Thu Jun 22 07:43:55 BNT [filer02: raid.rg.recons.missing:notice]: RAID group /aggr1/plex0/rg0 is missing 2 disk(s).
Thu Jun 22 07:43:55 BNT [filer02: raid.rg.recons.cantStart:warning]: The reconstruction cannot start in RAID group /aggr1/plex0/rg0: No 520 bps disk of required size available in spare pool, targeting any  spare pool
Thu Jun 22 07:43:55 BNT [filer02: mgr.boot.disk_done:info]: NetApp Release 7.3.2 boot complete. Last disk update written at Thu Jun 22 07:42:53 BNT 2017
Thu Jun 22 07:43:55 BNT [filer02: monitor.raid.brokenDisk:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken.
Thu Jun 22 07:43:55 BNT [filer02: monitor.brokenDisk.notice:notice]: When two disks are broken in raid_dp volume,  the system shuts down automatically every 24  hours to encourage you to replace the disk.  If you reboot the system it will run for another  24 hours before shutting down. (The 24 hour timeout  may be increased by altering the "raid.timeout"  value using the "options" command.)
Thu Jun 22 07:43:55 BNT [filer02: diskown.errorReadingOwnership:warning]: error 3 (disk failed) while reading ownership on disk 0a.24 (S/N N00SSSAL)
Thu Jun 22 07:43:55 BNT [filer02: diskown.errorReadingOwnership:warning]: error 3 (disk failed) while reading ownership on disk 0a.26 (S/N N00V0UWL)
Thu Jun 22 07:43:56 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] is still present in the system and should be removed.
Thu Jun 22 07:43:56 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] is still present in the system and should be removed.
Thu Jun 22 07:43:56 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] is still present in the system and should be removed.
Thu Jun 22 07:43:56 BNT [filer02: shelf.config.spha:info]: System is using single path HA attached storage only.
Thu Jun 22 07:43:56 BNT [filer02: mgr.boot.reason_ok:notice]: System rebooted after power-on.
Thu Jun 22 07:44:00 BNT [filer02: raid.rg.spares.low:warning]: /aggr1/plex0/rg0
Thu Jun 22 07:44:11 BNT [filer02: monitor.globalStatus.nonCritical:warning]: Disk on adapter 0a, shelf 2, bay 8, failed. Disk on adapter 0a, shelf 2, bay 3, failed. Disk on adapter 0a, shelf 2, bay 1, failed.  There are not enough spare disks. /vol/vol1 is full (using or reserving 100% of space and 0% of inodes, using 92% of reserve).
Thu Jun 22 07:44:22 BNT [filer02: nbt.nbns.registrationComplete:info]: NBT: All CIFS name registrations have completed for the local server.
Thu Jun 22 07:44:24 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SHUTDOWN PENDING (degraded mode))
Thu Jun 22 07:44:31 BNT last message repeated 2 times
Thu Jun 22 07:44:31 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201706181145.0) was not posted for host (0), it was dropped from queue due to lack of space
Thu Jun 22 07:44:38 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (REBOOT)
Thu Jun 22 07:44:38 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201706181145.1) was not posted for host (0), it was dropped from queue due to lack of space
Thu Jun 22 07:44:44 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SPARES_LOW)
Thu Jun 22 07:44:44 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201706181145.2) was not posted for host (0), it was dropped from queue due to lack of space
Thu Jun 22 07:45:06 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (PERFORMANCE SNAPSHOT)
Thu Jun 22 07:45:06 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201706181145.3) was not posted for host (0), it was dropped from queue due to lack of space
Thu Jun 22 07:50:00 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (CLUSTER ERROR: DISK/SHELF COUNT MISMATCH)
Thu Jun 22 07:53:56 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (CLUSTER ERROR: DISK/SHELF COUNT MISMATCH)
Thu Jun 22 07:57:29 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (CLUSTER ERROR: DISK/SHELF COUNT MISMATCH)
Thu Jun 22 08:00:00 BNT [filer02: kern.uptime.filer:info]:   8:00am up 15 mins, 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 0 iSCSI ops
Thu Jun 22 08:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 24 hours.
Thu Jun 22 08:00:15 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Thu Jun 22 08:00:24 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (CLUSTER ERROR: DISK/SHELF COUNT MISMATCH)
Thu Jun 22 08:00:24 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201706181150.0) was not posted for host (0), it was dropped from queue due to lack of space
Thu Jun 22 08:04:44 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu Jun 22 08:09:02 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu Jun 22 08:12:57 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu Jun 22 08:17:26 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu Jun 22 08:21:22 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu Jun 22 08:21:22 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201706181200.0) for host (0) was not posted to NetApp. The system will drop the message
Thu Jun 22 08:52:31 BNT [filer02: iscsi.notice:notice]: ISCSI: New session from initiator iqn.1991-05.com.microsoft:pcb-722-ad1.pcbl722dom.local at IP addr 10.72.4.1
Thu Jun 22 09:00:00 BNT [filer02: kern.uptime.filer:info]:   9:00am up  1:15 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 135 iSCSI ops
Thu Jun 22 09:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 23 hours.
Thu Jun 22 10:00:00 BNT [filer02: kern.uptime.filer:info]:  10:00am up  2:15 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 4482 iSCSI ops
Thu Jun 22 10:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 22 hours.
Thu Jun 22 11:00:00 BNT [filer02: kern.uptime.filer:info]:  11:00am up  3:16 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 92856 iSCSI ops
Thu Jun 22 11:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 21 hours.
Thu Jun 22 12:00:00 BNT [filer02: kern.uptime.filer:info]:  12:00pm up  4:16 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 431574 iSCSI ops
Thu Jun 22 12:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 20 hours.
Thu Jun 22 12:00:01 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Thu Jun 22 12:00:21 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu Jun 22 12:04:44 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu Jun 22 12:09:07 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu Jun 22 12:13:32 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu Jun 22 12:17:32 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu Jun 22 12:37:34 BNT last message repeated 5 times
Thu Jun 22 12:41:16 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu Jun 22 12:58:07 BNT last message repeated 4 times
Thu Jun 22 12:58:07 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201706181600.0) for host (0) was not posted to NetApp. The system will drop the message
Thu Jun 22 13:00:00 BNT [filer02: kern.uptime.filer:info]:   1:00pm up  5:16 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 646503 iSCSI ops
Thu Jun 22 13:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 19 hours.
Thu Jun 22 14:00:00 BNT [filer02: kern.uptime.filer:info]:   2:00pm up  6:16 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 710507 iSCSI ops
Thu Jun 22 14:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 18 hours.
Thu Jun 22 15:00:00 BNT [filer02: kern.uptime.filer:info]:   3:00pm up  7:16 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 710587 iSCSI ops
Thu Jun 22 15:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 17 hours.
Thu Jun 22 16:00:00 BNT [filer02: kern.uptime.filer:info]:   4:00pm up  8:16 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 710587 iSCSI ops
Thu Jun 22 16:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 16 hours.
Thu Jun 22 16:00:00 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Thu Jun 22 16:00:22 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu Jun 22 16:04:26 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu Jun 22 16:08:37 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu Jun 22 16:12:48 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu Jun 22 16:17:07 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu Jun 22 16:33:55 BNT last message repeated 4 times
Thu Jun 22 16:37:44 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu Jun 22 16:49:06 BNT last message repeated 3 times
Thu Jun 22 16:56:51 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201706182000.0) for host (0) was not posted to NetApp. The system will drop the message
Thu Jun 22 17:00:00 BNT [filer02: kern.uptime.filer:info]:   5:00pm up  9:16 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 710587 iSCSI ops
Thu Jun 22 17:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 15 hours.
Thu Jun 22 18:00:00 BNT [filer02: kern.uptime.filer:info]:   6:00pm up 10:16 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 710620 iSCSI ops
Thu Jun 22 18:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 14 hours.
Thu Jun 22 19:00:00 BNT [filer02: kern.uptime.filer:info]:   7:00pm up 11:16 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 710620 iSCSI ops
Thu Jun 22 19:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 13 hours.
Thu Jun 22 20:00:00 BNT [filer02: kern.uptime.filer:info]:   8:00pm up 12:16 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 710620 iSCSI ops
Thu Jun 22 20:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 12 hours.
Thu Jun 22 20:00:45 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Thu Jun 22 20:52:45 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu Jun 22 20:56:19 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201706190000.0) for host (0) was not posted to NetApp. The system will drop the message
Thu Jun 22 21:00:00 BNT [filer02: kern.uptime.filer:info]:   9:00pm up 13:16 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 710620 iSCSI ops
Thu Jun 22 21:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 11 hours.
Thu Jun 22 22:00:00 BNT [filer02: kern.uptime.filer:info]:  10:00pm up 14:16 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 710620 iSCSI ops
Thu Jun 22 22:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 10 hours.
Thu Jun 22 23:00:00 BNT [filer02: kern.uptime.filer:info]:  11:00pm up 15:16 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 710620 iSCSI ops
Thu Jun 22 23:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 9 hours.
Fri Jun 23 00:00:00 BNT [filer02: kern.uptime.filer:info]:  12:00am up 16:16 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 710620 iSCSI ops
Fri Jun 23 00:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 8 hours.
Fri Jun 23 00:00:30 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of nightly.0 snapshot (Snapshot operation failed: No space left on device.)
Fri Jun 23 00:31:32 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Fri Jun 23 00:47:09 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Fri Jun 23 00:55:21 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Fri Jun 23 00:55:21 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201706190732.0) for host (0) was not posted to NetApp. The system will drop the message
Fri Jun 23 01:00:00 BNT [filer02: kern.uptime.filer:info]:   1:00am up 17:16 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 710620 iSCSI ops
Fri Jun 23 01:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 7 hours.
Fri Jun 23 02:00:00 BNT [filer02: kern.uptime.filer:info]:   2:00am up 18:16 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 710620 iSCSI ops
Fri Jun 23 02:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 6 hours.
Fri Jun 23 03:00:00 BNT [filer02: kern.uptime.filer:info]:   3:00am up 19:16 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 710620 iSCSI ops
Fri Jun 23 03:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 5 hours.
Fri Jun 23 04:00:00 BNT [filer02: kern.uptime.filer:info]:   4:00am up 20:16 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 710620 iSCSI ops
Fri Jun 23 04:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 4 hours.
Fri Jun 23 05:00:00 BNT [filer02: kern.uptime.filer:info]:   5:00am up 21:16 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 710620 iSCSI ops
Fri Jun 23 05:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 3 hours.
Fri Jun 23 06:00:00 BNT [filer02: kern.uptime.filer:info]:   6:00am up 22:16 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 710620 iSCSI ops
Fri Jun 23 06:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 2 hours.
Fri Jun 23 07:00:00 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] is still present in the system and should be removed.
Fri Jun 23 07:00:00 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] is still present in the system and should be removed.
Fri Jun 23 07:00:00 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] is still present in the system and should be removed.
Thu Jun 22 23:51:04 GMT [localhost: rc:notice]: The system was down for 33 seconds
Fri Jun 23 07:50:28 BNT [filer02: cf.nm.nicTransitionUp:info]: replayed event: Interconnect link 0 is UP
Fri Jun 23 07:50:35 BNT [filer02: fci.initialization.failed:error]: replayed event: Initialization failed on Fibre Channel adapter 0b.
Fri Jun 23 07:50:37 BNT [filer02: nvram.battery.turned.on:info]: replayed event: The NVRAM battery is turned ON. It is turned OFF during system shutdown.
Fri Jun 23 07:50:40 BNT [filer02: diskown.isEnabled:info]: replayed event: software ownership has been enabled for this system
Fri Jun 23 07:50:44 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.2 is a local HA mailbox disk.
Fri Jun 23 07:50:44 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.0 is a local HA mailbox disk.
Fri Jun 23 07:50:44 BNT [filer02: fmmb.instStat.change:info]: replayed event: normal mailbox instance on local side.
Fri Jun 23 07:50:44 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.7 is a partner HA mailbox disk.
Fri Jun 23 07:50:44 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.3 is a partner HA mailbox disk.
Fri Jun 23 07:50:44 BNT [filer02: fmmb.instStat.change:info]: replayed event: normal mailbox instance on partner side.
Fri Jun 23 07:50:45 BNT [filer02: cf.fm.partner:info]: replayed event: Cluster monitor: partner 'filer01'
Fri Jun 23 07:50:45 BNT [filer02: cf.fm.timeMasterStatus:info]: replayed event: Acting as cluster time slave
Fri Jun 23 07:50:45 BNT [filer02: raid.assim.disk.brokenPreAssim:error]: replayed event: Broken Disk 0a.40 Shelf ? Bay ? [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] detected prior to assimilation. It should be removed.
Fri Jun 23 07:50:45 BNT [filer02: raid.assim.disk.brokenPreAssim:error]: replayed event: Broken Disk 0a.35 Shelf ? Bay ? [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] detected prior to assimilation. It should be removed.
Fri Jun 23 07:50:45 BNT [filer02: raid.assim.disk.brokenPreAssim:error]: replayed event: Broken Disk 0a.33 Shelf ? Bay ? [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] detected prior to assimilation. It should be removed.
Fri Jun 23 07:50:46 BNT [filer02: coredump.spare.none:info]: replayed event: No sparecore disk was found.
Fri Jun 23 07:50:46 BNT [filer02: raid.vol.replay.nvram:info]: replayed event: Performing raid replay on volume(s)
Fri Jun 23 07:50:46 BNT [filer02: raid.cksum.replay.summary:info]: replayed event: Replayed 0 checksum blocks.
Fri Jun 23 07:50:46 BNT [filer02: raid.stripe.replay.summary:info]: replayed event: Replayed 0 stripes.
Fri Jun 23 07:50:58 BNT [filer02: cf.fm.launch:info]: replayed event: Launching cluster monitor
Fri Jun 23 07:50:58 BNT [filer02: cf.fm.partner:info]: replayed event: Cluster monitor: partner 'filer01'
Fri Jun 23 07:51:02 BNT [filer02: cf.fsm.takeoverOfPartnerDisabled:notice]: replayed event: Cluster monitor: takeover of filer01 disabled (partner booting)
Fri Jun 23 07:51:04 BNT [filer02: fcp.service.adapter:warning]: replayed event: FCP started but no adapter is present!
Fri Jun 23 07:51:04 BNT [filer02: cf.fsm.takeoverOfPartnerEnabled:notice]: replayed event: Cluster monitor: takeover of filer01 enabled
Fri Jun 23 07:51:04 BNT [filer02: cf.fsm.takeoverByPartnerDisabled:notice]: replayed event: Cluster monitor: takeover of  by filer01 disabled (partner booting)
Fri Jun 23 07:51:04 BNT [filer02: kern.syslog.msg:notice]: replayed event: The system was down for 33 seconds
Fri Jun 23 07:51:06 BNT [filer02: disk.failmsg:error]: Disk 0a.40 (N00V0W8L): message received.
Fri Jun 23 07:51:06 BNT [filer02: disk.failmsg:error]: Disk 0a.35 (N00WLAUL): message received.
Fri Jun 23 07:51:06 BNT [filer02: disk.failmsg:error]: Disk 0a.33 (N00W75PL): message received.
Fri Jun 23 07:51:06 BNT [filer02: raid.disk.unload.done:info]: Unload of Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] has completed successfully
Fri Jun 23 07:51:06 BNT [filer02: raid.disk.unload.done:info]: Unload of Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] has completed successfully
Fri Jun 23 07:51:06 BNT [filer02: raid.disk.unload.done:info]: Unload of Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] has completed successfully
Fri Jun 23 07:51:06 BNT [filer02: raid.fdr.failed.ok:info]: Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] successfully deleted from spare pool
Fri Jun 23 07:51:06 BNT [filer02: raid.fdr.failed.ok:info]: Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] successfully deleted from spare pool
Fri Jun 23 07:51:06 BNT [filer02: raid.fdr.failed.ok:info]: Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] successfully deleted from spare pool
Fri Jun 23 07:51:06 BNT [filer02: cf.fsm.takeoverByPartnerEnabled:notice]: Cluster monitor: takeover of filer02 by filer01 enabled
Fri Jun 23 07:51:06 BNT [filer02: dfu.firmwareUpToDate:info]: Firmware is up-to-date on all disk drives
Fri Jun 23 07:51:08 BNT [filer02: perf.archive.start:info]: Performance archiver started. Sampling 22 objects and 198 counters.
Fri Jun 23 07:51:10 BNT [filer02: netif.linkUp:info]: Ethernet e0b: Link up.
Fri Jun 23 07:51:10 BNT [filer02: netif.linkUp:info]: Ethernet e0a: Link up.
Fri Jun 23 07:51:11 BNT [filer02: pvif.switchLink:warning]: vif1: switching to e0a
Fri Jun 23 07:51:12 BNT [filer02: rc:info]: relog syslog Fri Jun 23 07:00:00 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TS
Fri Jun 23 07:51:12 BNT [filer02: rc:info]: relog syslog Fri Jun 23 07:45:07 BNT [filer02: mem_scrub_admin:debug]: mem_scrub: completed system memory scrub (count = 1)
Fri Jun 23 07:51:12 BNT [filer02: iscsi.service.startup:info]: iSCSI service startup
Fri Jun 23 07:51:13 BNT [filer02: netif.linkDown:info]: Ethernet e0a: Link down, check cable.
Fri Jun 23 07:51:13 BNT [filer02: cmds.sysconf.logErr:error]: sysconfig: Unless directed by NetApp Global Services the cf.takeover.on_panic option should be set to On. .
Fri Jun 23 07:51:13 BNT [filer02: raid.rg.recons.missing:notice]: RAID group /aggr1/plex0/rg0 is missing 2 disk(s).
Fri Jun 23 07:51:13 BNT [filer02: raid.rg.recons.cantStart:warning]: The reconstruction cannot start in RAID group /aggr1/plex0/rg0: No 520 bps disk of required size available in spare pool, targeting any  spare pool
Fri Jun 23 07:51:13 BNT [filer02: mgr.boot.disk_done:info]: NetApp Release 7.3.2 boot complete. Last disk update written at Fri Jun 23 07:50:11 BNT 2017
Fri Jun 23 07:51:13 BNT [filer02: monitor.raid.brokenDisk:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken.
Fri Jun 23 07:51:13 BNT [filer02: monitor.brokenDisk.notice:notice]: When two disks are broken in raid_dp volume,  the system shuts down automatically every 24  hours to encourage you to replace the disk.  If you reboot the system it will run for another  24 hours before shutting down. (The 24 hour timeout  may be increased by altering the "raid.timeout"  value using the "options" command.)
Fri Jun 23 07:51:14 BNT [filer02: shelf.config.spha:info]: System is using single path HA attached storage only.
Fri Jun 23 07:51:14 BNT [filer02: diskown.errorReadingOwnership:warning]: error 3 (disk failed) while reading ownership on disk 0a.24 (S/N N00SSSAL)
Fri Jun 23 07:51:14 BNT [filer02: diskown.errorReadingOwnership:warning]: error 3 (disk failed) while reading ownership on disk 0a.26 (S/N N00V0UWL)
Fri Jun 23 07:51:14 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] is still present in the system and should be removed.
Fri Jun 23 07:51:14 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] is still present in the system and should be removed.
Fri Jun 23 07:51:14 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] is still present in the system and should be removed.
Fri Jun 23 07:51:14 BNT [filer02: mgr.boot.reason_ok:notice]: System rebooted after power-on.
Fri Jun 23 07:51:15 BNT [filer02: netif.linkUp:info]: Ethernet e0a: Link up.
Fri Jun 23 07:51:37 BNT [filer02: raid.rg.spares.low:warning]: /aggr1/plex0/rg0
Fri Jun 23 07:51:40 BNT [filer02: nbt.nbns.registrationComplete:info]: NBT: All CIFS name registrations have completed for the local server.
Fri Jun 23 07:51:46 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SHUTDOWN PENDING (degraded mode))
Fri Jun 23 07:51:53 BNT last message repeated 2 times
Fri Jun 23 07:51:53 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201706190732.1) was not posted for host (0), it was dropped from queue due to lack of space
Fri Jun 23 07:52:00 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (REBOOT (power on))
Fri Jun 23 07:52:00 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201706190732.2) was not posted for host (0), it was dropped from queue due to lack of space
Fri Jun 23 07:52:07 BNT [filer02: monitor.globalStatus.nonCritical:warning]: Disk on adapter 0a, shelf 2, bay 8, failed. Disk on adapter 0a, shelf 2, bay 3, failed. Disk on adapter 0a, shelf 2, bay 1, failed.  There are not enough spare disks. /vol/vol1 is full (using or reserving 100% of space and 0% of inodes, using 92% of reserve).
Fri Jun 23 07:52:07 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SPARES_LOW)
Fri Jun 23 07:52:07 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201706190732.3) was not posted for host (0), it was dropped from queue due to lack of space
Fri Jun 23 07:52:24 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (PERFORMANCE SNAPSHOT)
Fri Jun 23 07:52:24 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201706190733.0) was not posted for host (0), it was dropped from queue due to lack of space
Fri Jun 23 07:57:10 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jun 23 08:00:07 BNT [filer02: kern.uptime.filer:info]:   8:00am up 8 mins, 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 0 iSCSI ops
Fri Jun 23 08:00:07 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 24 hours.
Fri Jun 23 08:00:31 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Fri Jun 23 08:00:38 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jun 23 08:00:38 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201706190800.0) was not posted for host (0), it was dropped from queue due to lack of space
Fri Jun 23 08:04:50 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jun 23 08:09:12 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jun 23 08:13:23 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jun 23 08:17:34 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jun 23 08:21:27 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jun 23 08:28:53 BNT last message repeated 2 times
Fri Jun 23 08:28:53 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201706191200.0) for host (0) was not posted to NetApp. The system will drop the message
Fri Jun 23 09:00:00 BNT [filer02: kern.uptime.filer:info]:   9:00am up  1:08 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 0 iSCSI ops
Fri Jun 23 09:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 23 hours.
Fri Jun 23 09:24:34 BNT [filer02: iscsi.notice:notice]: ISCSI: New session from initiator iqn.1991-05.com.microsoft:pcb-722-ad1.pcbl722dom.local at IP addr 10.72.4.1
Fri Jun 23 10:00:00 BNT [filer02: kern.uptime.filer:info]:  10:00am up  2:08 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 237 iSCSI ops
Fri Jun 23 10:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 22 hours.
Fri Jun 23 11:00:00 BNT [filer02: kern.uptime.filer:info]:  11:00am up  3:08 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 237 iSCSI ops
Fri Jun 23 11:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 21 hours.
Fri Jun 23 12:00:00 BNT [filer02: kern.uptime.filer:info]:  12:00pm up  4:08 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 237 iSCSI ops
Fri Jun 23 12:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 20 hours.
Fri Jun 23 12:00:16 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Fri Jun 23 12:00:34 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jun 23 12:04:49 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jun 23 12:08:40 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jun 23 12:12:17 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jun 23 12:16:26 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jun 23 12:32:34 BNT last message repeated 4 times
Fri Jun 23 12:36:34 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jun 23 12:56:43 BNT last message repeated 5 times
Fri Jun 23 12:56:43 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201706191600.0) for host (0) was not posted to NetApp. The system will drop the message
Fri Jun 23 13:00:00 BNT [filer02: kern.uptime.filer:info]:   1:00pm up  5:08 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 237 iSCSI ops
Fri Jun 23 13:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 19 hours.
Fri Jun 23 14:00:00 BNT [filer02: kern.uptime.filer:info]:   2:00pm up  6:08 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 237 iSCSI ops
Fri Jun 23 14:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 18 hours.
Fri Jun 23 15:00:00 BNT [filer02: kern.uptime.filer:info]:   3:00pm up  7:08 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 237 iSCSI ops
Fri Jun 23 15:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 17 hours.
Fri Jun 23 16:00:00 BNT [filer02: kern.uptime.filer:info]:   4:00pm up  8:08 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 237 iSCSI ops
Fri Jun 23 16:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 16 hours.
Fri Jun 23 16:00:15 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Fri Jun 23 16:00:34 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jun 23 16:04:20 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jun 23 16:08:34 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jun 23 16:12:57 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jun 23 16:17:03 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jun 23 16:35:51 BNT last message repeated 5 times
Fri Jun 23 16:40:07 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jun 23 16:56:16 BNT last message repeated 4 times
Fri Jun 23 16:56:16 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201706192000.0) for host (0) was not posted to NetApp. The system will drop the message
Fri Jun 23 17:00:00 BNT [filer02: kern.uptime.filer:info]:   5:00pm up  9:08 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 237 iSCSI ops
Fri Jun 23 17:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 15 hours.
Fri Jun 23 18:00:00 BNT [filer02: kern.uptime.filer:info]:   6:00pm up 10:08 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 237 iSCSI ops
Fri Jun 23 18:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 14 hours.
Fri Jun 23 19:00:00 BNT [filer02: kern.uptime.filer:info]:   7:00pm up 11:09 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 237 iSCSI ops
Fri Jun 23 19:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 13 hours.
Fri Jun 23 20:00:00 BNT [filer02: kern.uptime.filer:info]:   8:00pm up 12:09 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 237 iSCSI ops
Fri Jun 23 20:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 12 hours.
Fri Jun 23 20:00:00 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Fri Jun 23 20:08:25 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jun 23 20:28:15 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jun 23 20:56:56 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201706200000.0) for host (0) was not posted to NetApp. The system will drop the message
Fri Jun 23 21:00:00 BNT [filer02: kern.uptime.filer:info]:   9:00pm up 13:09 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 237 iSCSI ops
Fri Jun 23 21:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 11 hours.
Fri Jun 23 22:00:00 BNT [filer02: kern.uptime.filer:info]:  10:00pm up 14:09 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 237 iSCSI ops
Fri Jun 23 22:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 10 hours.
Fri Jun 23 23:00:00 BNT [filer02: kern.uptime.filer:info]:  11:00pm up 15:09 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 237 iSCSI ops
Fri Jun 23 23:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 9 hours.
Sat Jun 24 00:00:01 BNT [filer02: kern.uptime.filer:info]:  12:00am up 16:09 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 237 iSCSI ops
Sat Jun 24 00:00:01 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 8 hours.
Sat Jun 24 00:00:01 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of nightly.0 snapshot (Snapshot operation failed: No space left on device.)
Sat Jun 24 00:40:35 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Sat Jun 24 00:57:08 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201706200724.0) for host (0) was not posted to NetApp. The system will drop the message
Sat Jun 24 01:00:00 BNT [filer02: kern.uptime.filer:info]:   1:00am up 17:09 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 237 iSCSI ops
Sat Jun 24 01:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 7 hours.
Sat Jun 24 02:00:00 BNT [filer02: kern.uptime.filer:info]:   2:00am up 18:09 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 237 iSCSI ops
Sat Jun 24 02:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 6 hours.
Sat Jun 24 03:00:00 BNT [filer02: kern.uptime.filer:info]:   3:00am up 19:09 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 237 iSCSI ops
Sat Jun 24 03:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 5 hours.
Sat Jun 24 04:00:00 BNT [filer02: kern.uptime.filer:info]:   4:00am up 20:09 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 237 iSCSI ops
Sat Jun 24 04:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 4 hours.
Sat Jun 24 05:00:00 BNT [filer02: kern.uptime.filer:info]:   5:00am up 21:09 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 237 iSCSI ops
Sat Jun 24 05:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 3 hours.
Sat Jun 24 06:00:00 BNT [filer02: kern.uptime.filer:info]:   6:00am up 22:09 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 237 iSCSI ops
Sat Jun 24 06:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 2 hours.
Sat Jun 24 07:00:00 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] is still present in the system and should be removed.
Sat Jun 24 07:00:00 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] is still present in the system and should be removed.
Sat Jun 24 07:00:00 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] is still present in the system and should be removed.
Sat Jun 24 08:00:00 BNT [filer02: monitor.brokenDisk.notice:notice]: When two disks are broken in raid_dp volume,  the system shuts down automatically every 24  hours to encourage you to replace the disk.  If you reboot the system it will run for another  24 hours before shutting down. (The 24 hour timeout  may be increased by altering the "raid.timeout"  value using the "options" command.)
Sat Jun 24 08:00:00 BNT [filer02: monitor.shutdown.brokenDisk:EMERGENCY]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system now.
Sat Jun 24 08:00:04 BNT [filer02: cf.fsm.takeoverOfPartnerDisabled:notice]: Cluster monitor: takeover of filer01 disabled (local halt in progress)
Sat Jun 24 08:00:06 BNT [filer02: kern.shutdown:notice]: System shut down because : "BROKEN DISK".
Wed Jun 28 23:52:51 GMT [localhost: rc:notice]: The system was down for 431559 seconds
Thu Jun 29 07:52:26 BNT [filer02: cf.nm.nicTransitionUp:info]: replayed event: Interconnect link 0 is UP
Thu Jun 29 07:52:33 BNT [filer02: fci.initialization.failed:error]: replayed event: Initialization failed on Fibre Channel adapter 0b.
Thu Jun 29 07:52:35 BNT [filer02: nvram.battery.turned.on:info]: replayed event: The NVRAM battery is turned ON. It is turned OFF during system shutdown.
Thu Jun 29 07:52:38 BNT [filer02: diskown.isEnabled:info]: replayed event: software ownership has been enabled for this system
Thu Jun 29 07:52:42 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.2 is a local HA mailbox disk.
Thu Jun 29 07:52:42 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.0 is a local HA mailbox disk.
Thu Jun 29 07:52:42 BNT [filer02: fmmb.instStat.change:info]: replayed event: normal mailbox instance on local side.
Thu Jun 29 07:52:42 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.7 is a partner HA mailbox disk.
Thu Jun 29 07:52:42 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.3 is a partner HA mailbox disk.
Thu Jun 29 07:52:42 BNT [filer02: fmmb.instStat.change:info]: replayed event: normal mailbox instance on partner side.
Thu Jun 29 07:52:42 BNT [filer02: cf.fm.partner:info]: replayed event: Cluster monitor: partner 'filer01'
Thu Jun 29 07:52:42 BNT [filer02: cf.fm.timeMasterStatus:info]: replayed event: Acting as cluster time slave
Thu Jun 29 07:52:43 BNT [filer02: raid.assim.disk.brokenPreAssim:error]: replayed event: Broken Disk 0a.40 Shelf ? Bay ? [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] detected prior to assimilation. It should be removed.
Thu Jun 29 07:52:43 BNT [filer02: raid.assim.disk.brokenPreAssim:error]: replayed event: Broken Disk 0a.35 Shelf ? Bay ? [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] detected prior to assimilation. It should be removed.
Thu Jun 29 07:52:43 BNT [filer02: raid.assim.disk.brokenPreAssim:error]: replayed event: Broken Disk 0a.33 Shelf ? Bay ? [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] detected prior to assimilation. It should be removed.
Thu Jun 29 07:52:43 BNT [filer02: coredump.spare.none:info]: replayed event: No sparecore disk was found.
Thu Jun 29 07:52:44 BNT [filer02: raid.cksum.replay.summary:info]: replayed event: Replayed 0 checksum blocks.
Thu Jun 29 07:52:44 BNT [filer02: raid.stripe.replay.summary:info]: replayed event: Replayed 0 stripes.
Thu Jun 29 07:52:46 BNT [filer02: cf.fm.launch:info]: replayed event: Launching cluster monitor
Thu Jun 29 07:52:46 BNT [filer02: cf.fm.partner:info]: replayed event: Cluster monitor: partner 'filer01'
Thu Jun 29 07:52:46 BNT [filer02: cf.fsm.takeoverOfPartnerDisabled:notice]: replayed event: Cluster monitor: takeover of filer01 disabled (partner booting)
Thu Jun 29 07:52:51 BNT [filer02: fcp.service.adapter:warning]: replayed event: FCP started but no adapter is present!
Thu Jun 29 07:52:51 BNT [filer02: kern.syslog.msg:notice]: replayed event: The system was down for 431559 seconds
Thu Jun 29 07:52:53 BNT [filer02: disk.failmsg:error]: Disk 0a.40 (N00V0W8L): message received.
Thu Jun 29 07:52:53 BNT [filer02: disk.failmsg:error]: Disk 0a.35 (N00WLAUL): message received.
Thu Jun 29 07:52:53 BNT [filer02: disk.failmsg:error]: Disk 0a.33 (N00W75PL): message received.
Thu Jun 29 07:52:53 BNT [filer02: raid.disk.unload.done:info]: Unload of Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] has completed successfully
Thu Jun 29 07:52:53 BNT [filer02: raid.disk.unload.done:info]: Unload of Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] has completed successfully
Thu Jun 29 07:52:53 BNT [filer02: raid.disk.unload.done:info]: Unload of Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] has completed successfully
Thu Jun 29 07:52:53 BNT [filer02: raid.fdr.failed.ok:info]: Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] successfully deleted from spare pool
Thu Jun 29 07:52:53 BNT [filer02: raid.fdr.failed.ok:info]: Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] successfully deleted from spare pool
Thu Jun 29 07:52:53 BNT [filer02: raid.fdr.failed.ok:info]: Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] successfully deleted from spare pool
Thu Jun 29 07:52:53 BNT [filer02: dfu.firmwareUpToDate:info]: Firmware is up-to-date on all disk drives
Thu Jun 29 07:52:54 BNT [filer02: cf.fsm.takeoverOfPartnerEnabled:notice]: Cluster monitor: takeover of filer01 enabled
Thu Jun 29 07:52:54 BNT [filer02: cf.fsm.takeoverByPartnerEnabled:notice]: Cluster monitor: takeover of filer02 by filer01 enabled
Thu Jun 29 07:52:55 BNT [filer02: perf.archive.start:info]: Performance archiver started. Sampling 22 objects and 198 counters.
Thu Jun 29 07:52:57 BNT [filer02: netif.linkUp:info]: Ethernet e0a: Link up.
Thu Jun 29 07:52:58 BNT [filer02: pvif.switchLink:warning]: vif1: switching to e0a
Thu Jun 29 07:52:58 BNT [filer02: netif.linkDown:info]: Ethernet e0b: Link down, check cable.
Thu Jun 29 07:53:02 BNT [filer02: netif.linkUp:info]: Ethernet e0b: Link up.
Thu Jun 29 07:53:03 BNT [filer02: iscsi.service.startup:info]: iSCSI service startup
Thu Jun 29 07:53:04 BNT [filer02: cmds.sysconf.logErr:error]: sysconfig: Unless directed by NetApp Global Services the cf.takeover.on_panic option should be set to On. .
Thu Jun 29 07:53:04 BNT [filer02: raid.rg.recons.missing:notice]: RAID group /aggr1/plex0/rg0 is missing 2 disk(s).
Thu Jun 29 07:53:04 BNT [filer02: raid.rg.recons.cantStart:warning]: The reconstruction cannot start in RAID group /aggr1/plex0/rg0: No 520 bps disk of required size available in spare pool, targeting any  spare pool
Thu Jun 29 07:53:04 BNT [filer02: mgr.boot.disk_done:info]: NetApp Release 7.3.2 boot complete. Last disk update written at Sat Jun 24 08:00:08 BNT 2017
Thu Jun 29 07:53:04 BNT [filer02: monitor.raid.brokenDisk:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken.
Thu Jun 29 07:53:04 BNT [filer02: monitor.brokenDisk.notice:notice]: When two disks are broken in raid_dp volume,  the system shuts down automatically every 24  hours to encourage you to replace the disk.  If you reboot the system it will run for another  24 hours before shutting down. (The 24 hour timeout  may be increased by altering the "raid.timeout"  value using the "options" command.)
Thu Jun 29 07:53:04 BNT [filer02: iscsi.notice:notice]: ISCSI: New session from initiator iqn.1991-05.com.microsoft:pcb-722-ad1.pcbl722dom.local at IP addr 10.72.4.1
Thu Jun 29 07:53:05 BNT [filer02: diskown.errorReadingOwnership:warning]: error 3 (disk failed) while reading ownership on disk 0a.26 (S/N N00V0UWL)
Thu Jun 29 07:53:05 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] is still present in the system and should be removed.
Thu Jun 29 07:53:05 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] is still present in the system and should be removed.
Thu Jun 29 07:53:05 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] is still present in the system and should be removed.
Thu Jun 29 07:53:05 BNT [filer02: shelf.config.spha:info]: System is using single path HA attached storage only.
Thu Jun 29 07:53:05 BNT [filer02: mgr.boot.reason_ok:notice]: System rebooted.
Thu Jun 29 07:53:29 BNT [filer02: raid.rg.spares.low:warning]: /aggr1/plex0/rg0
Thu Jun 29 07:53:33 BNT [filer02: nbt.nbns.registrationComplete:info]: NBT: All CIFS name registrations have completed for the local server.
Thu Jun 29 07:53:36 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SHUTDOWN PENDING (degraded mode))
Thu Jun 29 07:53:44 BNT last message repeated 2 times
Thu Jun 29 07:53:44 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201706200725.0) was not posted for host (0), it was dropped from queue due to lack of space
Thu Jun 29 07:53:51 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (REBOOT (power on))
Thu Jun 29 07:53:51 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201706200725.1) was not posted for host (0), it was dropped from queue due to lack of space
Thu Jun 29 07:53:58 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SPARES_LOW)
Thu Jun 29 07:53:58 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201706200725.2) was not posted for host (0), it was dropped from queue due to lack of space
Thu Jun 29 07:54:10 BNT [filer02: monitor.globalStatus.nonCritical:warning]: Disk on adapter 0a, shelf 2, bay 8, failed. Disk on adapter 0a, shelf 2, bay 3, failed. Disk on adapter 0a, shelf 2, bay 1, failed.  There are not enough spare disks. /vol/vol1 is full (using or reserving 100% of space and 0% of inodes, using 92% of reserve).
Thu Jun 29 07:54:17 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (PERFORMANCE SNAPSHOT)
Thu Jun 29 07:54:17 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201706200725.3) was not posted for host (0), it was dropped from queue due to lack of space
Thu Jun 29 07:58:55 BNT [filer02: cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 0a.24 (37AF1465:B13F6B87:04EAEFC0:98AAFB7E:00000000:00000000:00000000:00000000:00000000:00000000) has recently changed or the node (filer01) is missing the disk.
Thu Jun 29 07:58:55 BNT [filer02: cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 0a.33 (016F3AB5:B475F4EC:10505E83:02202755:00000000:00000000:00000000:00000000:00000000:00000000) has recently changed or the node (filer02) is missing the disk.
Thu Jun 29 07:58:55 BNT [filer02: cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 0a.35 (9115483F:9DD43BC4:27172773:99F7D8B4:00000000:00000000:00000000:00000000:00000000:00000000) has recently changed or the node (filer02) is missing the disk.
Thu Jun 29 07:58:55 BNT [filer02: cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 0a.40 (072AEB65:C662A9CF:E58B9917:9BFA787B:00000000:00000000:00000000:00000000:00000000:00000000) has recently changed or the node (filer02) is missing the disk.
Thu Jun 29 07:58:55 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu Jun 29 07:59:02 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu Jun 29 07:59:02 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201706200800.0) was not posted for host (0), it was dropped from queue due to lack of space
Thu Jun 29 08:00:08 BNT [filer02: kern.uptime.filer:info]:   8:00am up 6 mins, 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 2697 iSCSI ops
Thu Jun 29 08:00:08 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 24 hours.
Thu Jun 29 08:00:18 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Thu Jun 29 08:00:27 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu Jun 29 08:00:27 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201706201200.0) was not posted for host (0), it was dropped from queue due to lack of space
Thu Jun 29 08:05:08 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu Jun 29 08:08:47 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu Jun 29 08:13:04 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu Jun 29 08:17:21 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu Jun 29 08:21:29 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu Jun 29 08:25:59 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu Jun 29 08:25:59 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201706201600.0) for host (0) was not posted to NetApp. The system will drop the message
Thu Jun 29 09:00:00 BNT [filer02: kern.uptime.filer:info]:   9:00am up  1:06 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 2739 iSCSI ops
Thu Jun 29 09:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 23 hours.
Thu Jun 29 09:00:15 BNT [filer02: cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 0a.24 (37AF1465:B13F6B87:04EAEFC0:98AAFB7E:00000000:00000000:00000000:00000000:00000000:00000000) has recently changed or the node (filer01) is missing the disk.
Thu Jun 29 09:00:15 BNT [filer02: cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 0a.33 (016F3AB5:B475F4EC:10505E83:02202755:00000000:00000000:00000000:00000000:00000000:00000000) has recently changed or the node (filer02) is missing the disk.
Thu Jun 29 09:00:15 BNT [filer02: cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 0a.35 (9115483F:9DD43BC4:27172773:99F7D8B4:00000000:00000000:00000000:00000000:00000000:00000000) has recently changed or the node (filer02) is missing the disk.
Thu Jun 29 09:00:15 BNT [filer02: cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 0a.40 (072AEB65:C662A9CF:E58B9917:9BFA787B:00000000:00000000:00000000:00000000:00000000:00000000) has recently changed or the node (filer02) is missing the disk.
Thu Jun 29 10:00:00 BNT [filer02: kern.uptime.filer:info]:  10:00am up  2:06 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 11127 iSCSI ops
Thu Jun 29 10:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 22 hours.
Thu Jun 29 11:00:00 BNT [filer02: kern.uptime.filer:info]:  11:00am up  3:06 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 13108 iSCSI ops
Thu Jun 29 11:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 21 hours.
Thu Jun 29 12:00:00 BNT [filer02: kern.uptime.filer:info]:  12:00pm up  4:06 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 21449 iSCSI ops
Thu Jun 29 12:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 20 hours.
Thu Jun 29 12:00:16 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Thu Jun 29 12:00:35 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu Jun 29 12:04:19 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu Jun 29 12:08:30 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu Jun 29 12:12:55 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu Jun 29 12:17:11 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu Jun 29 12:36:47 BNT last message repeated 5 times
Thu Jun 29 12:40:50 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu Jun 29 12:57:31 BNT last message repeated 4 times
Thu Jun 29 12:57:31 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201706202000.0) for host (0) was not posted to NetApp. The system will drop the message
Thu Jun 29 13:00:00 BNT [filer02: kern.uptime.filer:info]:   1:00pm up  5:06 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 21553 iSCSI ops
Thu Jun 29 13:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 19 hours.
Thu Jun 29 14:00:00 BNT [filer02: kern.uptime.filer:info]:   2:00pm up  6:06 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 21553 iSCSI ops
Thu Jun 29 14:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 18 hours.
Thu Jun 29 15:00:00 BNT [filer02: kern.uptime.filer:info]:   3:00pm up  7:06 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 23175 iSCSI ops
Thu Jun 29 15:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 17 hours.
Thu Jun 29 16:00:00 BNT [filer02: kern.uptime.filer:info]:   4:00pm up  8:06 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 23208 iSCSI ops
Thu Jun 29 16:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 16 hours.
Thu Jun 29 16:00:16 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Thu Jun 29 16:00:40 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu Jun 29 16:49:14 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu Jun 29 16:57:25 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201706210000.0) for host (0) was not posted to NetApp. The system will drop the message
Thu Jun 29 17:00:00 BNT [filer02: kern.uptime.filer:info]:   5:00pm up  9:06 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 23208 iSCSI ops
Thu Jun 29 17:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 15 hours.
Thu Jun 29 18:00:00 BNT [filer02: kern.uptime.filer:info]:   6:00pm up 10:06 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 23208 iSCSI ops
Thu Jun 29 18:00:01 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 14 hours.
Thu Jun 29 19:00:00 BNT [filer02: kern.uptime.filer:info]:   7:00pm up 11:07 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 23208 iSCSI ops
Thu Jun 29 19:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 13 hours.
Thu Jun 29 20:00:00 BNT [filer02: kern.uptime.filer:info]:   8:00pm up 12:07 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 23208 iSCSI ops
Thu Jun 29 20:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 12 hours.
Thu Jun 29 20:00:00 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Thu Jun 29 20:25:52 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Thu Jun 29 20:50:17 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Thu Jun 29 20:57:59 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201706210753.0) for host (0) was not posted to NetApp. The system will drop the message
Thu Jun 29 21:00:00 BNT [filer02: kern.uptime.filer:info]:   9:00pm up 13:07 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 23208 iSCSI ops
Thu Jun 29 21:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 11 hours.
Thu Jun 29 22:00:00 BNT [filer02: kern.uptime.filer:info]:  10:00pm up 14:07 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 23208 iSCSI ops
Thu Jun 29 22:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 10 hours.
Thu Jun 29 23:00:00 BNT [filer02: kern.uptime.filer:info]:  11:00pm up 15:07 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 23208 iSCSI ops
Thu Jun 29 23:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 9 hours.
Fri Jun 30 00:00:00 BNT [filer02: kern.uptime.filer:info]:  12:00am up 16:07 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 23208 iSCSI ops
Fri Jun 30 00:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 8 hours.
Fri Jun 30 00:00:01 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of nightly.0 snapshot (Snapshot operation failed: No space left on device.)
Fri Jun 30 00:12:49 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SHUTDOWN PENDING (degraded mode))
Fri Jun 30 00:40:26 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SHUTDOWN PENDING (degraded mode))
Fri Jun 30 00:56:37 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201706210753.1) for host (0) was not posted to NetApp. The system will drop the message
Fri Jun 30 01:00:00 BNT [filer02: kern.uptime.filer:info]:   1:00am up 17:07 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 23208 iSCSI ops
Fri Jun 30 01:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 7 hours.
Fri Jun 30 02:00:00 BNT [filer02: kern.uptime.filer:info]:   2:00am up 18:07 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 23208 iSCSI ops
Fri Jun 30 02:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 6 hours.
Fri Jun 30 03:00:00 BNT [filer02: kern.uptime.filer:info]:   3:00am up 19:07 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 23208 iSCSI ops
Fri Jun 30 03:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 5 hours.
Fri Jun 30 04:00:00 BNT [filer02: kern.uptime.filer:info]:   4:00am up 20:07 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 23208 iSCSI ops
Fri Jun 30 04:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 4 hours.
Fri Jun 30 05:00:00 BNT [filer02: kern.uptime.filer:info]:   5:00am up 21:07 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 23208 iSCSI ops
Fri Jun 30 05:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 3 hours.
Fri Jun 30 06:00:00 BNT [filer02: kern.uptime.filer:info]:   6:00am up 22:07 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 23208 iSCSI ops
Fri Jun 30 06:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 2 hours.
Fri Jun 30 07:00:00 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] is still present in the system and should be removed.
Fri Jun 30 07:00:00 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] is still present in the system and should be removed.
Fri Jun 30 07:00:00 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] is still present in the system and should be removed.
Fri Jun 30 07:40:24 BNT [filer02: fci.link.break:error]: Link break detected on Fibre Channel adapter 0a.
Thu Jun 29 23:41:00 GMT [localhost: rc:notice]: The system was down for 11 seconds
Fri Jun 30 07:41:01 BNT [filer02: cf.fsm.takeoverByPartnerEnabled:notice]: Cluster monitor: takeover of filer02 by filer01 enabled
Fri Jun 30 07:40:23 BNT [filer02: cf.nm.nicTransitionUp:info]: replayed event: Interconnect link 0 is UP
Fri Jun 30 07:40:30 BNT [filer02: fci.initialization.failed:error]: replayed event: Initialization failed on Fibre Channel adapter 0b.
Fri Jun 30 07:40:32 BNT [filer02: nvram.battery.turned.on:info]: replayed event: The NVRAM battery is turned ON. It is turned OFF during system shutdown.
Fri Jun 30 07:40:35 BNT [filer02: diskown.isEnabled:info]: replayed event: software ownership has been enabled for this system
Fri Jun 30 07:40:39 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.2 is a local HA mailbox disk.
Fri Jun 30 07:40:39 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.0 is a local HA mailbox disk.
Fri Jun 30 07:40:39 BNT [filer02: fmmb.instStat.change:info]: replayed event: normal mailbox instance on local side.
Fri Jun 30 07:40:39 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.7 is a partner HA mailbox disk.
Fri Jun 30 07:40:39 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.3 is a partner HA mailbox disk.
Fri Jun 30 07:40:39 BNT [filer02: fmmb.instStat.change:info]: replayed event: normal mailbox instance on partner side.
Fri Jun 30 07:40:40 BNT [filer02: cf.fm.partner:info]: replayed event: Cluster monitor: partner 'filer01'
Fri Jun 30 07:40:40 BNT [filer02: cf.fm.timeMasterStatus:info]: replayed event: Acting as cluster time slave
Fri Jun 30 07:40:40 BNT [filer02: raid.assim.disk.brokenPreAssim:error]: replayed event: Broken Disk 0a.40 Shelf ? Bay ? [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] detected prior to assimilation. It should be removed.
Fri Jun 30 07:40:40 BNT [filer02: raid.assim.disk.brokenPreAssim:error]: replayed event: Broken Disk 0a.35 Shelf ? Bay ? [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] detected prior to assimilation. It should be removed.
Fri Jun 30 07:40:40 BNT [filer02: raid.assim.disk.brokenPreAssim:error]: replayed event: Broken Disk 0a.33 Shelf ? Bay ? [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] detected prior to assimilation. It should be removed.
Fri Jun 30 07:40:41 BNT [filer02: coredump.spare.none:info]: replayed event: No sparecore disk was found.
Fri Jun 30 07:40:41 BNT [filer02: raid.vol.replay.nvram:info]: replayed event: Performing raid replay on volume(s)
Fri Jun 30 07:40:41 BNT [filer02: raid.cksum.replay.summary:info]: replayed event: Replayed 0 checksum blocks.
Fri Jun 30 07:40:41 BNT [filer02: raid.stripe.replay.summary:info]: replayed event: Replayed 0 stripes.
Fri Jun 30 07:40:53 BNT [filer02: cf.fm.launch:info]: replayed event: Launching cluster monitor
Fri Jun 30 07:40:53 BNT [filer02: cf.fm.partner:info]: replayed event: Cluster monitor: partner 'filer01'
Fri Jun 30 07:40:57 BNT [filer02: cf.fsm.takeoverOfPartnerDisabled:notice]: replayed event: Cluster monitor: takeover of filer01 disabled (partner booting)
Fri Jun 30 07:40:59 BNT [filer02: cf.fsm.takeoverOfPartnerEnabled:notice]: replayed event: Cluster monitor: takeover of filer01 enabled
Fri Jun 30 07:40:59 BNT [filer02: fcp.service.adapter:warning]: replayed event: FCP started but no adapter is present!
Fri Jun 30 07:41:00 BNT [filer02: kern.syslog.msg:notice]: replayed event: The system was down for 11 seconds
Fri Jun 30 07:41:01 BNT [filer02: cf.fsm.takeoverByPartnerEnabled:notice]: replayed event: Cluster monitor: takeover of filer02 by filer01 enabled
Fri Jun 30 07:41:01 BNT [filer02: disk.failmsg:error]: Disk 0a.40 (N00V0W8L): message received.
Fri Jun 30 07:41:01 BNT [filer02: disk.failmsg:error]: Disk 0a.35 (N00WLAUL): message received.
Fri Jun 30 07:41:01 BNT [filer02: disk.failmsg:error]: Disk 0a.33 (N00W75PL): message received.
Fri Jun 30 07:41:01 BNT [filer02: raid.disk.unload.done:info]: Unload of Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] has completed successfully
Fri Jun 30 07:41:01 BNT [filer02: raid.disk.unload.done:info]: Unload of Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] has completed successfully
Fri Jun 30 07:41:01 BNT [filer02: raid.disk.unload.done:info]: Unload of Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] has completed successfully
Fri Jun 30 07:41:01 BNT [filer02: raid.fdr.failed.ok:info]: Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] successfully deleted from spare pool
Fri Jun 30 07:41:01 BNT [filer02: raid.fdr.failed.ok:info]: Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] successfully deleted from spare pool
Fri Jun 30 07:41:01 BNT [filer02: raid.fdr.failed.ok:info]: Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] successfully deleted from spare pool
Fri Jun 30 07:41:02 BNT [filer02: dfu.firmwareUpToDate:info]: Firmware is up-to-date on all disk drives
Fri Jun 30 07:41:03 BNT [filer02: perf.archive.start:info]: Performance archiver started. Sampling 22 objects and 198 counters.
Fri Jun 30 07:41:05 BNT [filer02: netif.linkUp:info]: Ethernet e0a: Link up.
Fri Jun 30 07:41:05 BNT [filer02: netif.linkUp:info]: Ethernet e0b: Link up.
Fri Jun 30 07:41:06 BNT [filer02: pvif.switchLink:warning]: vif1: switching to e0a
Fri Jun 30 07:41:07 BNT [filer02: rc:info]: relog syslog Fri Jun 30 07:00:00 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TS
Fri Jun 30 07:41:07 BNT [filer02: rc:info]: relog syslog Fri Jun 30 07:40:24 BNT [filer02: fci.link.break:error]: Link break detected on Fibre Channel adapter 0a.
Fri Jun 30 07:41:07 BNT [filer02: iscsi.service.startup:info]: iSCSI service startup
Fri Jun 30 07:41:08 BNT [filer02: cmds.sysconf.logErr:error]: sysconfig: Unless directed by NetApp Global Services the cf.takeover.on_panic option should be set to On. .
Fri Jun 30 07:41:08 BNT [filer02: raid.rg.recons.missing:notice]: RAID group /aggr1/plex0/rg0 is missing 2 disk(s).
Fri Jun 30 07:41:08 BNT [filer02: raid.rg.recons.cantStart:warning]: The reconstruction cannot start in RAID group /aggr1/plex0/rg0: No 520 bps disk of required size available in spare pool, targeting any  spare pool
Fri Jun 30 07:41:08 BNT [filer02: mgr.boot.disk_done:info]: NetApp Release 7.3.2 boot complete. Last disk update written at Fri Jun 30 07:40:13 BNT 2017
Fri Jun 30 07:41:08 BNT [filer02: monitor.raid.brokenDisk:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken.
Fri Jun 30 07:41:08 BNT [filer02: monitor.brokenDisk.notice:notice]: When two disks are broken in raid_dp volume,  the system shuts down automatically every 24  hours to encourage you to replace the disk.  If you reboot the system it will run for another  24 hours before shutting down. (The 24 hour timeout  may be increased by altering the "raid.timeout"  value using the "options" command.)
Fri Jun 30 07:41:08 BNT [filer02: diskown.errorReadingOwnership:warning]: error 3 (disk failed) while reading ownership on disk 0a.24 (S/N N00SSSAL)
Fri Jun 30 07:41:08 BNT [filer02: diskown.errorReadingOwnership:warning]: error 3 (disk failed) while reading ownership on disk 0a.26 (S/N N00V0UWL)
Fri Jun 30 07:41:08 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] is still present in the system and should be removed.
Fri Jun 30 07:41:08 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] is still present in the system and should be removed.
Fri Jun 30 07:41:08 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] is still present in the system and should be removed.
Fri Jun 30 07:41:08 BNT [filer02: shelf.config.spha:info]: System is using single path HA attached storage only.
Fri Jun 30 07:41:09 BNT [filer02: mgr.boot.reason_ok:notice]: System rebooted after power-on.
Fri Jun 30 07:41:31 BNT [filer02: raid.rg.spares.low:warning]: /aggr1/plex0/rg0
Fri Jun 30 07:41:37 BNT [filer02: nbt.nbns.registrationComplete:info]: NBT: All CIFS name registrations have completed for the local server.
Fri Jun 30 07:41:38 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (REBOOT (power on))
Fri Jun 30 07:41:46 BNT last message repeated 2 times
Fri Jun 30 07:41:46 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201706210753.2) was not posted for host (0), it was dropped from queue due to lack of space
Fri Jun 30 07:41:53 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SPARES_LOW)
Fri Jun 30 07:41:53 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201706210754.0) was not posted for host (0), it was dropped from queue due to lack of space
Fri Jun 30 07:41:59 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (PERFORMANCE SNAPSHOT)
Fri Jun 30 07:42:00 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201706210754.1) was not posted for host (0), it was dropped from queue due to lack of space
Fri Jun 30 07:42:09 BNT [filer02: monitor.globalStatus.nonCritical:warning]: Disk on adapter 0a, shelf 2, bay 8, failed. Disk on adapter 0a, shelf 2, bay 3, failed. Disk on adapter 0a, shelf 2, bay 1, failed.  There are not enough spare disks. /vol/vol1 is full (using or reserving 100% of space and 0% of inodes, using 92% of reserve).
Fri Jun 30 07:42:20 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jun 30 07:42:20 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201706210800.0) was not posted for host (0), it was dropped from queue due to lack of space
Fri Jun 30 07:47:01 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jun 30 07:51:30 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jun 30 07:55:43 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jun 30 08:00:00 BNT [filer02: kern.uptime.filer:info]:   8:00am up 18 mins, 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 0 iSCSI ops
Fri Jun 30 08:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 24 hours.
Fri Jun 30 08:00:09 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jun 30 08:00:31 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Fri Jun 30 08:00:38 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jun 30 08:00:39 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201706211200.0) was not posted for host (0), it was dropped from queue due to lack of space
Fri Jun 30 08:05:00 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jun 30 08:09:18 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jun 30 08:13:03 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jun 30 08:16:51 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jun 30 08:16:51 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201706211600.0) for host (0) was not posted to NetApp. The system will drop the message
Fri Jun 30 09:00:00 BNT [filer02: kern.uptime.filer:info]:   9:00am up  1:18 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 0 iSCSI ops
Fri Jun 30 09:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 23 hours.
Fri Jun 30 10:00:00 BNT [filer02: kern.uptime.filer:info]:  10:00am up  2:18 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 0 iSCSI ops
Fri Jun 30 10:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 22 hours.
Fri Jun 30 10:14:21 BNT [filer02: iscsi.notice:notice]: ISCSI: New session from initiator iqn.1991-05.com.microsoft:pcb-722-ad1.pcbl722dom.local at IP addr 10.72.4.1
Fri Jun 30 11:00:00 BNT [filer02: kern.uptime.filer:info]:  11:00am up  3:18 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 2930 iSCSI ops
Fri Jun 30 11:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 21 hours.
Fri Jun 30 12:00:00 BNT [filer02: kern.uptime.filer:info]:  12:00pm up  4:18 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 3496 iSCSI ops
Fri Jun 30 12:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 20 hours.
Fri Jun 30 12:00:16 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Fri Jun 30 12:00:38 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jun 30 12:04:22 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jun 30 12:07:56 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jun 30 12:11:41 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jun 30 12:16:08 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jun 30 12:36:04 BNT last message repeated 5 times
Fri Jun 30 12:39:51 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jun 30 12:54:57 BNT last message repeated 4 times
Fri Jun 30 12:54:57 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201706212000.0) for host (0) was not posted to NetApp. The system will drop the message
Fri Jun 30 13:00:00 BNT [filer02: kern.uptime.filer:info]:   1:00pm up  5:18 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 3496 iSCSI ops
Fri Jun 30 13:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 19 hours.
Fri Jun 30 14:00:00 BNT [filer02: kern.uptime.filer:info]:   2:00pm up  6:18 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 3496 iSCSI ops
Fri Jun 30 14:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 18 hours.
Fri Jun 30 15:00:00 BNT [filer02: kern.uptime.filer:info]:   3:00pm up  7:18 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 3496 iSCSI ops
Fri Jun 30 15:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 17 hours.
Fri Jun 30 16:00:00 BNT [filer02: kern.uptime.filer:info]:   4:00pm up  8:18 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 5397 iSCSI ops
Fri Jun 30 16:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 16 hours.
Fri Jun 30 16:00:15 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Fri Jun 30 16:00:34 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jun 30 16:04:19 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jun 30 16:07:51 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jun 30 16:12:01 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jun 30 16:16:06 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jun 30 16:31:45 BNT last message repeated 4 times
Fri Jun 30 16:36:15 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jun 30 16:48:10 BNT last message repeated 3 times
Fri Jun 30 16:56:09 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201706220000.0) for host (0) was not posted to NetApp. The system will drop the message
Fri Jun 30 17:00:00 BNT [filer02: kern.uptime.filer:info]:   5:00pm up  9:19 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 5420 iSCSI ops
Fri Jun 30 17:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 15 hours.
Fri Jun 30 18:00:00 BNT [filer02: kern.uptime.filer:info]:   6:00pm up 10:19 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 5437 iSCSI ops
Fri Jun 30 18:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 14 hours.
Fri Jun 30 19:00:00 BNT [filer02: kern.uptime.filer:info]:   7:00pm up 11:19 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 5437 iSCSI ops
Fri Jun 30 19:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 13 hours.
Fri Jun 30 20:00:00 BNT [filer02: kern.uptime.filer:info]:   8:00pm up 12:19 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 5437 iSCSI ops
Fri Jun 30 20:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 12 hours.
Fri Jun 30 20:00:00 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Fri Jun 30 20:36:00 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Fri Jun 30 20:56:45 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201706220743.0) for host (0) was not posted to NetApp. The system will drop the message
Fri Jun 30 21:00:00 BNT [filer02: kern.uptime.filer:info]:   9:00pm up 13:19 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 5437 iSCSI ops
Fri Jun 30 21:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 11 hours.
Fri Jun 30 22:00:00 BNT [filer02: kern.uptime.filer:info]:  10:00pm up 14:19 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 5437 iSCSI ops
Fri Jun 30 22:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 10 hours.
Fri Jun 30 23:00:00 BNT [filer02: kern.uptime.filer:info]:  11:00pm up 15:19 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 5437 iSCSI ops
Fri Jun 30 23:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 9 hours.
Sat Jul  1 00:00:00 BNT [filer02: kern.uptime.filer:info]:  12:00am up 16:19 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 5437 iSCSI ops
Sat Jul  1 00:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 8 hours.
Sat Jul  1 00:00:45 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of nightly.0 snapshot (Snapshot operation failed: No space left on device.)
Sat Jul  1 00:15:48 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SHUTDOWN PENDING (degraded mode))
Sat Jul  1 00:54:04 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201706220744.0) for host (0) was not posted to NetApp. The system will drop the message
Sat Jul  1 01:00:00 BNT [filer02: kern.uptime.filer:info]:   1:00am up 17:19 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 5437 iSCSI ops
Sat Jul  1 01:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 7 hours.
Sat Jul  1 02:00:00 BNT [filer02: kern.uptime.filer:info]:   2:00am up 18:19 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 5437 iSCSI ops
Sat Jul  1 02:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 6 hours.
Sat Jul  1 03:00:00 BNT [filer02: kern.uptime.filer:info]:   3:00am up 19:19 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 5437 iSCSI ops
Sat Jul  1 03:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 5 hours.
Sat Jul  1 04:00:00 BNT [filer02: kern.uptime.filer:info]:   4:00am up 20:19 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 5437 iSCSI ops
Sat Jul  1 04:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 4 hours.
Sat Jul  1 05:00:00 BNT [filer02: kern.uptime.filer:info]:   5:00am up 21:19 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 5437 iSCSI ops
Sat Jul  1 05:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 3 hours.
Sat Jul  1 06:00:00 BNT [filer02: kern.uptime.filer:info]:   6:00am up 22:19 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 5437 iSCSI ops
Sat Jul  1 06:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 2 hours.
Sat Jul  1 07:00:00 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] is still present in the system and should be removed.
Sat Jul  1 07:00:00 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] is still present in the system and should be removed.
Sat Jul  1 07:00:00 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] is still present in the system and should be removed.
Sat Jul  1 08:00:00 BNT [filer02: monitor.brokenDisk.notice:notice]: When two disks are broken in raid_dp volume,  the system shuts down automatically every 24  hours to encourage you to replace the disk.  If you reboot the system it will run for another  24 hours before shutting down. (The 24 hour timeout  may be increased by altering the "raid.timeout"  value using the "options" command.)
Sat Jul  1 08:00:00 BNT [filer02: monitor.shutdown.brokenDisk:EMERGENCY]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system now.
Sat Jul  1 08:00:03 BNT [filer02: cf.fsm.takeoverOfPartnerDisabled:notice]: Cluster monitor: takeover of filer01 disabled (local halt in progress)
Sat Jul  1 08:00:06 BNT [filer02: kern.shutdown:notice]: System shut down because : "BROKEN DISK".
Sun Jul  2 23:47:44 GMT [localhost: rc:notice]: The system was down for 172054 seconds
Mon Jul  3 07:47:19 BNT [filer02: cf.nm.nicTransitionUp:info]: replayed event: Interconnect link 0 is UP
Mon Jul  3 07:47:26 BNT [filer02: fci.initialization.failed:error]: replayed event: Initialization failed on Fibre Channel adapter 0b.
Mon Jul  3 07:47:28 BNT [filer02: nvram.battery.turned.on:info]: replayed event: The NVRAM battery is turned ON. It is turned OFF during system shutdown.
Mon Jul  3 07:47:31 BNT [filer02: diskown.isEnabled:info]: replayed event: software ownership has been enabled for this system
Mon Jul  3 07:47:35 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.2 is a local HA mailbox disk.
Mon Jul  3 07:47:35 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.0 is a local HA mailbox disk.
Mon Jul  3 07:47:35 BNT [filer02: fmmb.instStat.change:info]: replayed event: normal mailbox instance on local side.
Mon Jul  3 07:47:35 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.7 is a partner HA mailbox disk.
Mon Jul  3 07:47:35 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.3 is a partner HA mailbox disk.
Mon Jul  3 07:47:35 BNT [filer02: fmmb.instStat.change:info]: replayed event: normal mailbox instance on partner side.
Mon Jul  3 07:47:35 BNT [filer02: cf.fm.partner:info]: replayed event: Cluster monitor: partner 'filer01'
Mon Jul  3 07:47:35 BNT [filer02: cf.fm.timeMasterStatus:info]: replayed event: Acting as cluster time slave
Mon Jul  3 07:47:36 BNT [filer02: raid.assim.disk.brokenPreAssim:error]: replayed event: Broken Disk 0a.40 Shelf ? Bay ? [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] detected prior to assimilation. It should be removed.
Mon Jul  3 07:47:36 BNT [filer02: raid.assim.disk.brokenPreAssim:error]: replayed event: Broken Disk 0a.35 Shelf ? Bay ? [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] detected prior to assimilation. It should be removed.
Mon Jul  3 07:47:36 BNT [filer02: raid.assim.disk.brokenPreAssim:error]: replayed event: Broken Disk 0a.33 Shelf ? Bay ? [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] detected prior to assimilation. It should be removed.
Mon Jul  3 07:47:36 BNT [filer02: coredump.spare.none:info]: replayed event: No sparecore disk was found.
Mon Jul  3 07:47:37 BNT [filer02: raid.cksum.replay.summary:info]: replayed event: Replayed 0 checksum blocks.
Mon Jul  3 07:47:37 BNT [filer02: raid.stripe.replay.summary:info]: replayed event: Replayed 0 stripes.
Mon Jul  3 07:47:39 BNT [filer02: cf.fm.launch:info]: replayed event: Launching cluster monitor
Mon Jul  3 07:47:39 BNT [filer02: cf.fm.partner:info]: replayed event: Cluster monitor: partner 'filer01'
Mon Jul  3 07:47:39 BNT [filer02: cf.fsm.takeoverOfPartnerDisabled:notice]: replayed event: Cluster monitor: takeover of filer01 disabled (partner booting)
Mon Jul  3 07:47:44 BNT [filer02: fcp.service.adapter:warning]: replayed event: FCP started but no adapter is present!
Mon Jul  3 07:47:44 BNT [filer02: kern.syslog.msg:notice]: replayed event: The system was down for 172054 seconds
Mon Jul  3 07:47:46 BNT [filer02: disk.failmsg:error]: Disk 0a.40 (N00V0W8L): message received.
Mon Jul  3 07:47:46 BNT [filer02: disk.failmsg:error]: Disk 0a.35 (N00WLAUL): message received.
Mon Jul  3 07:47:46 BNT [filer02: disk.failmsg:error]: Disk 0a.33 (N00W75PL): message received.
Mon Jul  3 07:47:46 BNT [filer02: raid.disk.unload.done:info]: Unload of Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] has completed successfully
Mon Jul  3 07:47:46 BNT [filer02: raid.disk.unload.done:info]: Unload of Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] has completed successfully
Mon Jul  3 07:47:46 BNT [filer02: raid.disk.unload.done:info]: Unload of Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] has completed successfully
Mon Jul  3 07:47:46 BNT [filer02: raid.fdr.failed.ok:info]: Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] successfully deleted from spare pool
Mon Jul  3 07:47:46 BNT [filer02: raid.fdr.failed.ok:info]: Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] successfully deleted from spare pool
Mon Jul  3 07:47:46 BNT [filer02: raid.fdr.failed.ok:info]: Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] successfully deleted from spare pool
Mon Jul  3 07:47:47 BNT [filer02: dfu.firmwareUpToDate:info]: Firmware is up-to-date on all disk drives
Mon Jul  3 07:47:47 BNT [filer02: cf.fsm.takeoverOfPartnerEnabled:notice]: Cluster monitor: takeover of filer01 enabled
Mon Jul  3 07:47:47 BNT [filer02: cf.fsm.takeoverByPartnerEnabled:notice]: Cluster monitor: takeover of filer02 by filer01 enabled
Mon Jul  3 07:47:48 BNT [filer02: perf.archive.start:info]: Performance archiver started. Sampling 22 objects and 198 counters.
Mon Jul  3 07:47:50 BNT [filer02: netif.linkUp:info]: Ethernet e0a: Link up.
Mon Jul  3 07:47:50 BNT [filer02: netif.linkUp:info]: Ethernet e0b: Link up.
Mon Jul  3 07:47:51 BNT [filer02: pvif.switchLink:warning]: vif1: switching to e0a
Mon Jul  3 07:47:52 BNT [filer02: iscsi.service.startup:info]: iSCSI service startup
Mon Jul  3 07:47:53 BNT [filer02: cmds.sysconf.logErr:error]: sysconfig: Unless directed by NetApp Global Services the cf.takeover.on_panic option should be set to On. .
Mon Jul  3 07:47:53 BNT [filer02: raid.rg.recons.missing:notice]: RAID group /aggr1/plex0/rg0 is missing 2 disk(s).
Mon Jul  3 07:47:53 BNT [filer02: raid.rg.recons.cantStart:warning]: The reconstruction cannot start in RAID group /aggr1/plex0/rg0: No 520 bps disk of required size available in spare pool, targeting any  spare pool
Mon Jul  3 07:47:53 BNT [filer02: mgr.boot.disk_done:info]: NetApp Release 7.3.2 boot complete. Last disk update written at Sat Jul  1 08:00:08 BNT 2017
Mon Jul  3 07:47:53 BNT [filer02: monitor.raid.brokenDisk:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken.
Mon Jul  3 07:47:53 BNT [filer02: monitor.brokenDisk.notice:notice]: When two disks are broken in raid_dp volume,  the system shuts down automatically every 24  hours to encourage you to replace the disk.  If you reboot the system it will run for another  24 hours before shutting down. (The 24 hour timeout  may be increased by altering the "raid.timeout"  value using the "options" command.)
Mon Jul  3 07:47:53 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] is still present in the system and should be removed.
Mon Jul  3 07:47:53 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] is still present in the system and should be removed.
Mon Jul  3 07:47:53 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] is still present in the system and should be removed.
Mon Jul  3 07:47:53 BNT [filer02: shelf.config.spha:info]: System is using single path HA attached storage only.
Mon Jul  3 07:47:54 BNT [filer02: mgr.boot.reason_ok:notice]: System rebooted.
Mon Jul  3 07:47:56 BNT [filer02: iscsi.notice:notice]: ISCSI: New session from initiator iqn.1991-05.com.microsoft:pcb-722-ad1.pcbl722dom.local at IP addr 10.72.4.1
Mon Jul  3 07:48:02 BNT [filer02: raid.rg.spares.low:warning]: /aggr1/plex0/rg0
Mon Jul  3 07:48:10 BNT [filer02: monitor.globalStatus.nonCritical:warning]: Disk on adapter 0a, shelf 2, bay 8, failed. Disk on adapter 0a, shelf 2, bay 3, failed. Disk on adapter 0a, shelf 2, bay 1, failed.  There are not enough spare disks. /vol/vol1 is full (using or reserving 100% of space and 0% of inodes, using 92% of reserve).
Mon Jul  3 07:48:22 BNT [filer02: nbt.nbns.registrationComplete:info]: NBT: All CIFS name registrations have completed for the local server.
Mon Jul  3 07:48:27 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (REBOOT (power on))
Mon Jul  3 07:48:35 BNT last message repeated 2 times
Mon Jul  3 07:48:35 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201706220744.1) was not posted for host (0), it was dropped from queue due to lack of space
Mon Jul  3 07:48:42 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SPARES_LOW)
Mon Jul  3 07:48:42 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201706220744.2) was not posted for host (0), it was dropped from queue due to lack of space
Mon Jul  3 07:48:49 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (PERFORMANCE SNAPSHOT)
Mon Jul  3 07:48:49 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201706220745.0) was not posted for host (0), it was dropped from queue due to lack of space
Mon Jul  3 07:49:05 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Mon Jul  3 07:49:05 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201706220800.0) was not posted for host (0), it was dropped from queue due to lack of space
Mon Jul  3 07:53:45 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Mon Jul  3 07:53:47 BNT [filer02: cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 0a.24 (37AF1465:B13F6B87:04EAEFC0:98AAFB7E:00000000:00000000:00000000:00000000:00000000:00000000) has recently changed or the node (filer01) is missing the disk.
Mon Jul  3 07:53:47 BNT [filer02: cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 0a.26 (DB493CEF:613B738F:C6775410:52ABD6E0:00000000:00000000:00000000:00000000:00000000:00000000) has recently changed or the node (filer01) is missing the disk.
Mon Jul  3 07:53:47 BNT [filer02: cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 0a.33 (016F3AB5:B475F4EC:10505E83:02202755:00000000:00000000:00000000:00000000:00000000:00000000) has recently changed or the node (filer02) is missing the disk.
Mon Jul  3 07:53:47 BNT [filer02: cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 0a.35 (9115483F:9DD43BC4:27172773:99F7D8B4:00000000:00000000:00000000:00000000:00000000:00000000) has recently changed or the node (filer02) is missing the disk.
Mon Jul  3 07:53:47 BNT [filer02: cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 0a.40 (072AEB65:C662A9CF:E58B9917:9BFA787B:00000000:00000000:00000000:00000000:00000000:00000000) has recently changed or the node (filer02) is missing the disk.
Mon Jul  3 07:53:48 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Mon Jul  3 07:53:55 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Mon Jul  3 07:53:55 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201706221200.0) was not posted for host (0), it was dropped from queue due to lack of space
Mon Jul  3 07:57:48 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Mon Jul  3 08:00:00 BNT [filer02: kern.uptime.filer:info]:   8:00am up 11 mins, 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 2691 iSCSI ops
Mon Jul  3 08:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 24 hours.
Mon Jul  3 08:00:15 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Mon Jul  3 08:00:23 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Mon Jul  3 08:00:24 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201706221600.0) was not posted for host (0), it was dropped from queue due to lack of space
Mon Jul  3 08:04:26 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Mon Jul  3 08:08:53 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Mon Jul  3 08:13:10 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Mon Jul  3 08:16:49 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Mon Jul  3 08:16:49 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201706222000.0) for host (0) was not posted to NetApp. The system will drop the message
Mon Jul  3 08:55:07 BNT [filer02: cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 0a.24 (37AF1465:B13F6B87:04EAEFC0:98AAFB7E:00000000:00000000:00000000:00000000:00000000:00000000) has recently changed or the node (filer01) is missing the disk.
Mon Jul  3 08:55:07 BNT [filer02: cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 0a.26 (DB493CEF:613B738F:C6775410:52ABD6E0:00000000:00000000:00000000:00000000:00000000:00000000) has recently changed or the node (filer01) is missing the disk.
Mon Jul  3 08:55:07 BNT [filer02: cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 0a.33 (016F3AB5:B475F4EC:10505E83:02202755:00000000:00000000:00000000:00000000:00000000:00000000) has recently changed or the node (filer02) is missing the disk.
Mon Jul  3 08:55:07 BNT [filer02: cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 0a.35 (9115483F:9DD43BC4:27172773:99F7D8B4:00000000:00000000:00000000:00000000:00000000:00000000) has recently changed or the node (filer02) is missing the disk.
Mon Jul  3 08:55:07 BNT [filer02: cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 0a.40 (072AEB65:C662A9CF:E58B9917:9BFA787B:00000000:00000000:00000000:00000000:00000000:00000000) has recently changed or the node (filer02) is missing the disk.
Mon Jul  3 09:00:00 BNT [filer02: kern.uptime.filer:info]:   9:00am up  1:11 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 2733 iSCSI ops
Mon Jul  3 09:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 23 hours.
Mon Jul  3 10:00:00 BNT [filer02: kern.uptime.filer:info]:  10:00am up  2:11 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 35757 iSCSI ops
Mon Jul  3 10:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 22 hours.
Mon Jul  3 11:00:00 BNT [filer02: kern.uptime.filer:info]:  11:00am up  3:11 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 49460 iSCSI ops
Mon Jul  3 11:00:01 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 21 hours.
Mon Jul  3 12:00:00 BNT [filer02: kern.uptime.filer:info]:  12:00pm up  4:11 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 49769 iSCSI ops
Mon Jul  3 12:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 20 hours.
Mon Jul  3 12:00:15 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Mon Jul  3 12:00:39 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Mon Jul  3 12:05:04 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Mon Jul  3 12:09:33 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Mon Jul  3 12:13:22 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Mon Jul  3 12:17:34 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Mon Jul  3 12:37:15 BNT last message repeated 5 times
Mon Jul  3 12:41:35 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Mon Jul  3 12:57:31 BNT last message repeated 4 times
Mon Jul  3 12:57:31 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201706230000.0) for host (0) was not posted to NetApp. The system will drop the message
Mon Jul  3 13:00:00 BNT [filer02: kern.uptime.filer:info]:   1:00pm up  5:11 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 49769 iSCSI ops
Mon Jul  3 13:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 19 hours.
Mon Jul  3 14:00:00 BNT [filer02: kern.uptime.filer:info]:   2:00pm up  6:11 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 76030 iSCSI ops
Mon Jul  3 14:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 18 hours.
Mon Jul  3 15:00:00 BNT [filer02: kern.uptime.filer:info]:   3:00pm up  7:11 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 96505 iSCSI ops
Mon Jul  3 15:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 17 hours.
Mon Jul  3 16:00:00 BNT [filer02: kern.uptime.filer:info]:   4:00pm up  8:12 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 107551 iSCSI ops
Mon Jul  3 16:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 16 hours.
Mon Jul  3 16:00:00 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Mon Jul  3 16:12:16 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Mon Jul  3 16:16:34 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Mon Jul  3 16:24:28 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Mon Jul  3 16:55:49 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201706230751.0) for host (0) was not posted to NetApp. The system will drop the message
Mon Jul  3 17:00:00 BNT [filer02: kern.uptime.filer:info]:   5:00pm up  9:12 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 108960 iSCSI ops
Mon Jul  3 17:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 15 hours.
Mon Jul  3 18:00:00 BNT [filer02: kern.uptime.filer:info]:   6:00pm up 10:12 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 192573 iSCSI ops
Mon Jul  3 18:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 14 hours.
Mon Jul  3 19:00:00 BNT [filer02: kern.uptime.filer:info]:   7:00pm up 11:12 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 192573 iSCSI ops
Mon Jul  3 19:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 13 hours.
Mon Jul  3 20:00:00 BNT [filer02: kern.uptime.filer:info]:   8:00pm up 12:12 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 192573 iSCSI ops
Mon Jul  3 20:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 12 hours.
Mon Jul  3 20:00:01 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Mon Jul  3 20:20:08 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SHUTDOWN PENDING (degraded mode))
Mon Jul  3 20:55:25 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201706230751.1) for host (0) was not posted to NetApp. The system will drop the message
Mon Jul  3 21:00:00 BNT [filer02: kern.uptime.filer:info]:   9:00pm up 13:12 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 192573 iSCSI ops
Mon Jul  3 21:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 11 hours.
Mon Jul  3 22:00:00 BNT [filer02: kern.uptime.filer:info]:  10:00pm up 14:12 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 192573 iSCSI ops
Mon Jul  3 22:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 10 hours.
Mon Jul  3 23:00:00 BNT [filer02: kern.uptime.filer:info]:  11:00pm up 15:12 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 192573 iSCSI ops
Mon Jul  3 23:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 9 hours.
Tue Jul  4 00:00:00 BNT [filer02: kern.uptime.filer:info]:  12:00am up 16:12 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 192573 iSCSI ops
Tue Jul  4 00:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 8 hours.
Tue Jul  4 00:00:46 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of nightly.0 snapshot (Snapshot operation failed: No space left on device.)
Tue Jul  4 00:12:54 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (REBOOT (power on))
Tue Jul  4 00:16:30 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (REBOOT (power on))
Tue Jul  4 00:55:55 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201706230751.2) for host (0) was not posted to NetApp. The system will drop the message
Tue Jul  4 01:00:00 BNT [filer02: kern.uptime.filer:info]:   1:00am up 17:12 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 192573 iSCSI ops
Tue Jul  4 01:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 7 hours.
Tue Jul  4 02:00:00 BNT [filer02: kern.uptime.filer:info]:   2:00am up 18:12 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 192573 iSCSI ops
Tue Jul  4 02:00:01 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 6 hours.
Tue Jul  4 03:00:00 BNT [filer02: kern.uptime.filer:info]:   3:00am up 19:12 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 192573 iSCSI ops
Tue Jul  4 03:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 5 hours.
Tue Jul  4 04:00:00 BNT [filer02: kern.uptime.filer:info]:   4:00am up 20:12 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 192573 iSCSI ops
Tue Jul  4 04:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 4 hours.
Tue Jul  4 05:00:00 BNT [filer02: kern.uptime.filer:info]:   5:00am up 21:12 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 192573 iSCSI ops
Tue Jul  4 05:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 3 hours.
Tue Jul  4 06:00:00 BNT [filer02: kern.uptime.filer:info]:   6:00am up 22:12 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 192573 iSCSI ops
Tue Jul  4 06:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 2 hours.
Tue Jul  4 07:00:00 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] is still present in the system and should be removed.
Tue Jul  4 07:00:00 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] is still present in the system and should be removed.
Tue Jul  4 07:00:00 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] is still present in the system and should be removed.
Tue Jul  4 07:44:12 BNT [filer02: fci.link.break:error]: Link break detected on Fibre Channel adapter 0a.
Mon Jul  3 23:44:46 GMT [localhost: rc:notice]: The system was down for 4 seconds
Tue Jul  4 07:44:09 BNT [filer02: cf.nm.nicTransitionUp:info]: replayed event: Interconnect link 0 is UP
Tue Jul  4 07:44:16 BNT [filer02: fci.initialization.failed:error]: replayed event: Initialization failed on Fibre Channel adapter 0b.
Tue Jul  4 07:44:18 BNT [filer02: nvram.battery.turned.on:info]: replayed event: The NVRAM battery is turned ON. It is turned OFF during system shutdown.
Tue Jul  4 07:44:47 BNT [filer02: cf.fsm.takeoverByPartnerEnabled:notice]: Cluster monitor: takeover of filer02 by filer01 enabled
Tue Jul  4 07:44:21 BNT [filer02: diskown.isEnabled:info]: replayed event: software ownership has been enabled for this system
Tue Jul  4 07:44:25 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.2 is a local HA mailbox disk.
Tue Jul  4 07:44:25 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.0 is a local HA mailbox disk.
Tue Jul  4 07:44:25 BNT [filer02: fmmb.instStat.change:info]: replayed event: normal mailbox instance on local side.
Tue Jul  4 07:44:25 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.7 is a partner HA mailbox disk.
Tue Jul  4 07:44:25 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.3 is a partner HA mailbox disk.
Tue Jul  4 07:44:25 BNT [filer02: fmmb.instStat.change:info]: replayed event: normal mailbox instance on partner side.
Tue Jul  4 07:44:26 BNT [filer02: cf.fm.partner:info]: replayed event: Cluster monitor: partner 'filer01'
Tue Jul  4 07:44:26 BNT [filer02: cf.fm.timeMasterStatus:info]: replayed event: Acting as cluster time slave
Tue Jul  4 07:44:26 BNT [filer02: raid.assim.disk.brokenPreAssim:error]: replayed event: Broken Disk 0a.40 Shelf ? Bay ? [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] detected prior to assimilation. It should be removed.
Tue Jul  4 07:44:26 BNT [filer02: raid.assim.disk.brokenPreAssim:error]: replayed event: Broken Disk 0a.35 Shelf ? Bay ? [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] detected prior to assimilation. It should be removed.
Tue Jul  4 07:44:26 BNT [filer02: raid.assim.disk.brokenPreAssim:error]: replayed event: Broken Disk 0a.33 Shelf ? Bay ? [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] detected prior to assimilation. It should be removed.
Tue Jul  4 07:44:27 BNT [filer02: coredump.spare.none:info]: replayed event: No sparecore disk was found.
Tue Jul  4 07:44:27 BNT [filer02: raid.vol.replay.nvram:info]: replayed event: Performing raid replay on volume(s)
Tue Jul  4 07:44:27 BNT [filer02: raid.cksum.replay.summary:info]: replayed event: Replayed 0 checksum blocks.
Tue Jul  4 07:44:27 BNT [filer02: raid.stripe.replay.summary:info]: replayed event: Replayed 0 stripes.
Tue Jul  4 07:44:30 BNT [filer02: ses.status.psWarning:warning]: replayed event: DS12-ESAS shelf 0 on channel 0c power warning for Power supply 1: non-critical status; DC undervoltage fault. This module is on the rear side of the shelf, at the left.
Tue Jul  4 07:44:30 BNT [filer02: ses.status.psWarning:warning]: replayed event: DS12-ESAS shelf 0 on channel 0c power warning for Power supply 2: non-critical status; DC undervoltage fault. This module is on the rear side of the shelf, at the right.
Tue Jul  4 07:44:39 BNT [filer02: cf.fm.launch:info]: replayed event: Launching cluster monitor
Tue Jul  4 07:44:39 BNT [filer02: cf.fm.partner:info]: replayed event: Cluster monitor: partner 'filer01'
Tue Jul  4 07:44:44 BNT [filer02: cf.fsm.takeoverOfPartnerDisabled:notice]: replayed event: Cluster monitor: takeover of filer01 disabled (partner booting)
Tue Jul  4 07:44:45 BNT [filer02: cf.fsm.takeoverOfPartnerEnabled:notice]: replayed event: Cluster monitor: takeover of filer01 enabled
Tue Jul  4 07:44:45 BNT [filer02: cf.fsm.takeoverByPartnerDisabled:notice]: replayed event: Cluster monitor: takeover of  by filer01 disabled (partner booting)
Tue Jul  4 07:44:45 BNT [filer02: fcp.service.adapter:warning]: replayed event: FCP started but no adapter is present!
Tue Jul  4 07:44:46 BNT [filer02: kern.syslog.msg:notice]: replayed event: The system was down for 4 seconds
Tue Jul  4 07:44:47 BNT [filer02: disk.failmsg:error]: Disk 0a.40 (N00V0W8L): message received.
Tue Jul  4 07:44:47 BNT [filer02: disk.failmsg:error]: Disk 0a.35 (N00WLAUL): message received.
Tue Jul  4 07:44:47 BNT [filer02: disk.failmsg:error]: Disk 0a.33 (N00W75PL): message received.
Tue Jul  4 07:44:47 BNT [filer02: raid.disk.unload.done:info]: Unload of Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] has completed successfully
Tue Jul  4 07:44:47 BNT [filer02: raid.disk.unload.done:info]: Unload of Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] has completed successfully
Tue Jul  4 07:44:47 BNT [filer02: raid.disk.unload.done:info]: Unload of Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] has completed successfully
Tue Jul  4 07:44:47 BNT [filer02: raid.fdr.failed.ok:info]: Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] successfully deleted from spare pool
Tue Jul  4 07:44:47 BNT [filer02: raid.fdr.failed.ok:info]: Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] successfully deleted from spare pool
Tue Jul  4 07:44:47 BNT [filer02: raid.fdr.failed.ok:info]: Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] successfully deleted from spare pool
Tue Jul  4 07:44:48 BNT [filer02: dfu.firmwareUpToDate:info]: Firmware is up-to-date on all disk drives
Tue Jul  4 07:44:49 BNT [filer02: perf.archive.start:info]: Performance archiver started. Sampling 22 objects and 198 counters.
Tue Jul  4 07:44:51 BNT [filer02: netif.linkUp:info]: Ethernet e0b: Link up.
Tue Jul  4 07:44:51 BNT [filer02: netif.linkUp:info]: Ethernet e0a: Link up.
Tue Jul  4 07:44:52 BNT [filer02: pvif.switchLink:warning]: vif1: switching to e0a
Tue Jul  4 07:44:53 BNT [filer02: rc:info]: relog syslog Tue Jul  4 07:00:00 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TS
Tue Jul  4 07:44:53 BNT [filer02: rc:info]: relog syslog Tue Jul  4 07:44:12 BNT [filer02: fci.link.break:error]: Link break detected on Fibre Channel adapter 0a.
Tue Jul  4 07:44:53 BNT [filer02: iscsi.service.startup:info]: iSCSI service startup
Tue Jul  4 07:44:54 BNT [filer02: cmds.sysconf.logErr:error]: sysconfig: Unless directed by NetApp Global Services the cf.takeover.on_panic option should be set to On. .
Tue Jul  4 07:44:54 BNT [filer02: raid.rg.recons.missing:notice]: RAID group /aggr1/plex0/rg0 is missing 2 disk(s).
Tue Jul  4 07:44:54 BNT [filer02: raid.rg.recons.cantStart:warning]: The reconstruction cannot start in RAID group /aggr1/plex0/rg0: No 520 bps disk of required size available in spare pool, targeting any  spare pool
Tue Jul  4 07:44:54 BNT [filer02: mgr.boot.disk_done:info]: NetApp Release 7.3.2 boot complete. Last disk update written at Tue Jul  4 07:44:00 BNT 2017
Tue Jul  4 07:44:54 BNT [filer02: monitor.raid.brokenDisk:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken.
Tue Jul  4 07:44:54 BNT [filer02: monitor.brokenDisk.notice:notice]: When two disks are broken in raid_dp volume,  the system shuts down automatically every 24  hours to encourage you to replace the disk.  If you reboot the system it will run for another  24 hours before shutting down. (The 24 hour timeout  may be increased by altering the "raid.timeout"  value using the "options" command.)
Tue Jul  4 07:44:54 BNT [filer02: diskown.errorReadingOwnership:warning]: error 3 (disk failed) while reading ownership on disk 0a.24 (S/N N00SSSAL)
Tue Jul  4 07:44:54 BNT [filer02: diskown.errorReadingOwnership:warning]: error 3 (disk failed) while reading ownership on disk 0a.26 (S/N N00V0UWL)
Tue Jul  4 07:44:54 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] is still present in the system and should be removed.
Tue Jul  4 07:44:54 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] is still present in the system and should be removed.
Tue Jul  4 07:44:54 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] is still present in the system and should be removed.
Tue Jul  4 07:44:54 BNT [filer02: shelf.config.spha:info]: System is using single path HA attached storage only.
Tue Jul  4 07:44:55 BNT [filer02: mgr.boot.reason_ok:notice]: System rebooted after power-on.
Tue Jul  4 07:45:01 BNT [filer02: raid.rg.spares.low:warning]: /aggr1/plex0/rg0
Tue Jul  4 07:45:12 BNT [filer02: monitor.globalStatus.nonCritical:warning]: Disk on adapter 0a, shelf 2, bay 8, failed. Disk on adapter 0a, shelf 2, bay 3, failed. Disk on adapter 0a, shelf 2, bay 1, failed.  There are not enough spare disks. /vol/vol1 is full (using or reserving 100% of space and 0% of inodes, using 92% of reserve).
Tue Jul  4 07:45:23 BNT [filer02: nbt.nbns.registrationComplete:info]: NBT: All CIFS name registrations have completed for the local server.
Tue Jul  4 07:45:26 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SPARES_LOW)
Tue Jul  4 07:45:34 BNT last message repeated 2 times
Tue Jul  4 07:45:34 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201706230752.0) was not posted for host (0), it was dropped from queue due to lack of space
Tue Jul  4 07:45:41 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (PERFORMANCE SNAPSHOT)
Tue Jul  4 07:45:41 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201706230752.1) was not posted for host (0), it was dropped from queue due to lack of space
Tue Jul  4 07:45:48 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Tue Jul  4 07:45:48 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201706230800.0) was not posted for host (0), it was dropped from queue due to lack of space
Tue Jul  4 07:46:06 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Tue Jul  4 07:46:06 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201706231200.0) was not posted for host (0), it was dropped from queue due to lack of space
Tue Jul  4 07:50:56 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Tue Jul  4 07:55:16 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Tue Jul  4 07:58:58 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Tue Jul  4 08:00:00 BNT [filer02: kern.uptime.filer:info]:   8:00am up 14 mins, 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 0 iSCSI ops
Tue Jul  4 08:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 24 hours.
Tue Jul  4 08:00:16 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Tue Jul  4 08:00:24 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Tue Jul  4 08:00:24 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201706231600.0) was not posted for host (0), it was dropped from queue due to lack of space
Tue Jul  4 08:04:05 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Tue Jul  4 08:08:15 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Tue Jul  4 08:12:26 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Tue Jul  4 08:16:50 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Tue Jul  4 08:21:00 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Tue Jul  4 08:21:00 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201706232000.0) for host (0) was not posted to NetApp. The system will drop the message
Tue Jul  4 08:52:19 BNT [filer02: iscsi.notice:notice]: ISCSI: New session from initiator iqn.1991-05.com.microsoft:pcb-722-ad1.pcbl722dom.local at IP addr 10.72.4.1
Tue Jul  4 09:00:01 BNT [filer02: kern.uptime.filer:info]:   9:00am up  1:14 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 91 iSCSI ops
Tue Jul  4 09:00:01 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 23 hours.
Tue Jul  4 10:00:00 BNT [filer02: kern.uptime.filer:info]:  10:00am up  2:14 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 5679 iSCSI ops
Tue Jul  4 10:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 22 hours.
Tue Jul  4 11:00:00 BNT [filer02: kern.uptime.filer:info]:  11:00am up  3:15 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 16847 iSCSI ops
Tue Jul  4 11:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 21 hours.
Tue Jul  4 12:00:00 BNT [filer02: kern.uptime.filer:info]:  12:00pm up  4:15 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 16847 iSCSI ops
Tue Jul  4 12:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 20 hours.
Tue Jul  4 12:00:01 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Tue Jul  4 12:00:24 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Tue Jul  4 12:04:06 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Tue Jul  4 12:07:44 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Tue Jul  4 12:11:55 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Tue Jul  4 12:15:26 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Tue Jul  4 12:31:32 BNT last message repeated 4 times
Tue Jul  4 12:35:58 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Tue Jul  4 12:55:40 BNT last message repeated 5 times
Tue Jul  4 12:55:40 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201706240000.0) for host (0) was not posted to NetApp. The system will drop the message
Tue Jul  4 13:00:00 BNT [filer02: kern.uptime.filer:info]:   1:00pm up  5:15 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 16864 iSCSI ops
Tue Jul  4 13:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 19 hours.
Tue Jul  4 14:00:00 BNT [filer02: kern.uptime.filer:info]:   2:00pm up  6:15 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 18753 iSCSI ops
Tue Jul  4 14:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 18 hours.
Tue Jul  4 15:00:00 BNT [filer02: kern.uptime.filer:info]:   3:00pm up  7:15 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 20598 iSCSI ops
Tue Jul  4 15:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 17 hours.
Tue Jul  4 16:00:00 BNT [filer02: kern.uptime.filer:info]:   4:00pm up  8:15 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 20698 iSCSI ops
Tue Jul  4 16:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 16 hours.
Tue Jul  4 16:00:00 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Tue Jul  4 16:00:25 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Tue Jul  4 16:04:43 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Tue Jul  4 16:08:33 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Tue Jul  4 16:12:15 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Tue Jul  4 16:15:47 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Tue Jul  4 16:35:49 BNT last message repeated 5 times
Tue Jul  4 16:40:06 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Tue Jul  4 16:51:34 BNT last message repeated 3 times
Tue Jul  4 16:55:14 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201706290753.0) for host (0) was not posted to NetApp. The system will drop the message
Tue Jul  4 17:00:00 BNT [filer02: kern.uptime.filer:info]:   5:00pm up  9:15 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 69397 iSCSI ops
Tue Jul  4 17:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 15 hours.
Tue Jul  4 18:00:00 BNT [filer02: kern.uptime.filer:info]:   6:00pm up 10:15 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 71839 iSCSI ops
Tue Jul  4 18:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 14 hours.
Tue Jul  4 19:00:00 BNT [filer02: kern.uptime.filer:info]:   7:00pm up 11:15 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 71839 iSCSI ops
Tue Jul  4 19:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 13 hours.
Tue Jul  4 20:00:00 BNT [filer02: kern.uptime.filer:info]:   8:00pm up 12:15 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 71839 iSCSI ops
Tue Jul  4 20:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 12 hours.
Tue Jul  4 20:00:45 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Tue Jul  4 20:04:47 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SHUTDOWN PENDING (degraded mode))
Tue Jul  4 20:32:37 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SHUTDOWN PENDING (degraded mode))
Tue Jul  4 20:53:52 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SHUTDOWN PENDING (degraded mode))
Tue Jul  4 20:58:12 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SHUTDOWN PENDING (degraded mode))
Tue Jul  4 20:58:12 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201706290753.1) for host (0) was not posted to NetApp. The system will drop the message
Tue Jul  4 21:00:00 BNT [filer02: kern.uptime.filer:info]:   9:00pm up 13:15 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 71839 iSCSI ops
Tue Jul  4 21:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 11 hours.
Tue Jul  4 22:00:00 BNT [filer02: kern.uptime.filer:info]:  10:00pm up 14:15 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 71839 iSCSI ops
Tue Jul  4 22:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 10 hours.
Tue Jul  4 23:00:00 BNT [filer02: kern.uptime.filer:info]:  11:00pm up 15:15 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 71839 iSCSI ops
Tue Jul  4 23:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 9 hours.
Wed Jul  5 00:00:00 BNT [filer02: kern.uptime.filer:info]:  12:00am up 16:15 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 71839 iSCSI ops
Wed Jul  5 00:00:01 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 8 hours.
Wed Jul  5 00:00:46 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of nightly.0 snapshot (Snapshot operation failed: No space left on device.)
Wed Jul  5 00:12:48 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (REBOOT)
Wed Jul  5 00:58:45 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201706290753.2) for host (0) was not posted to NetApp. The system will drop the message
Wed Jul  5 01:00:00 BNT [filer02: kern.uptime.filer:info]:   1:00am up 17:15 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 71839 iSCSI ops
Wed Jul  5 01:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 7 hours.
Wed Jul  5 02:00:00 BNT [filer02: kern.uptime.filer:info]:   2:00am up 18:15 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 71839 iSCSI ops
Wed Jul  5 02:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 6 hours.
Wed Jul  5 03:00:00 BNT [filer02: kern.uptime.filer:info]:   3:00am up 19:15 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 71839 iSCSI ops
Wed Jul  5 03:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 5 hours.
Wed Jul  5 04:00:00 BNT [filer02: kern.uptime.filer:info]:   4:00am up 20:15 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 71839 iSCSI ops
Wed Jul  5 04:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 4 hours.
Wed Jul  5 05:00:00 BNT [filer02: kern.uptime.filer:info]:   5:00am up 21:15 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 71839 iSCSI ops
Wed Jul  5 05:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 3 hours.
Wed Jul  5 06:00:00 BNT [filer02: kern.uptime.filer:info]:   6:00am up 22:15 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 71839 iSCSI ops
Wed Jul  5 06:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 2 hours.
Wed Jul  5 07:00:01 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] is still present in the system and should be removed.
Wed Jul  5 07:00:01 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] is still present in the system and should be removed.
Wed Jul  5 07:00:01 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] is still present in the system and should be removed.
Wed Jul  5 07:54:19 BNT [filer02: ses.status.psWarning:warning]: DS14-Mk2-AT shelf 1 on channel 0a power warning for Power supply 2: non-critical status; DC undervoltage fault. This module is on the rear side of the shelf, at the right.
Wed Jul  5 07:54:21 BNT [filer02: fci.link.break:error]: Link break detected on Fibre Channel adapter 0a.
Wed Jul  5 07:54:26 BNT [filer02: fci.adapter.error:warning]: Fibre Channel adapter driver encountered error "Failed to get loop map" on adapter 0a.
Tue Jul  4 23:54:53 GMT [localhost: rc:notice]: The system was down for 10 seconds
Tue Jul  4 23:54:54 GMT [filer02: cf.fsm.takeoverOfPartnerEnabled:notice]: Cluster monitor: takeover of filer01 enabled
Tue Jul  4 23:54:54 GMT [filer02: cf.fsm.takeoverByPartnerEnabled:notice]: Cluster monitor: takeover of filer02 by filer01 enabled
Wed Jul  5 07:54:18 BNT [filer02: cf.nm.nicTransitionUp:info]: replayed event: Interconnect link 0 is UP
Wed Jul  5 07:54:25 BNT [filer02: fci.initialization.failed:error]: replayed event: Initialization failed on Fibre Channel adapter 0b.
Wed Jul  5 07:54:27 BNT [filer02: nvram.battery.turned.on:info]: replayed event: The NVRAM battery is turned ON. It is turned OFF during system shutdown.
Wed Jul  5 07:54:30 BNT [filer02: diskown.isEnabled:info]: replayed event: software ownership has been enabled for this system
Wed Jul  5 07:54:34 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.2 is a local HA mailbox disk.
Wed Jul  5 07:54:34 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.0 is a local HA mailbox disk.
Wed Jul  5 07:54:34 BNT [filer02: fmmb.instStat.change:info]: replayed event: normal mailbox instance on local side.
Wed Jul  5 07:54:34 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.7 is a partner HA mailbox disk.
Wed Jul  5 07:54:34 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.3 is a partner HA mailbox disk.
Wed Jul  5 07:54:34 BNT [filer02: fmmb.instStat.change:info]: replayed event: normal mailbox instance on partner side.
Wed Jul  5 07:54:34 BNT [filer02: cf.fm.partner:info]: replayed event: Cluster monitor: partner 'filer01'
Wed Jul  5 07:54:34 BNT [filer02: cf.fm.timeMasterStatus:info]: replayed event: Acting as cluster time slave
Wed Jul  5 07:54:35 BNT [filer02: raid.assim.disk.brokenPreAssim:error]: replayed event: Broken Disk 0a.35 Shelf ? Bay ? [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] detected prior to assimilation. It should be removed.
Wed Jul  5 07:54:35 BNT [filer02: raid.assim.disk.brokenPreAssim:error]: replayed event: Broken Disk 0a.40 Shelf ? Bay ? [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] detected prior to assimilation. It should be removed.
Wed Jul  5 07:54:35 BNT [filer02: raid.assim.disk.brokenPreAssim:error]: replayed event: Broken Disk 0a.33 Shelf ? Bay ? [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] detected prior to assimilation. It should be removed.
Wed Jul  5 07:54:35 BNT [filer02: coredump.spare.none:info]: replayed event: No sparecore disk was found.
Wed Jul  5 07:54:36 BNT [filer02: raid.vol.replay.nvram:info]: replayed event: Performing raid replay on volume(s)
Wed Jul  5 07:54:36 BNT [filer02: raid.cksum.replay.summary:info]: replayed event: Replayed 0 checksum blocks.
Wed Jul  5 07:54:36 BNT [filer02: raid.stripe.replay.summary:info]: replayed event: Replayed 0 stripes.
Wed Jul  5 07:54:39 BNT [filer02: ses.status.psWarning:warning]: replayed event: DS12-ESAS shelf 0 on channel 0c power warning for Power supply 1: non-critical status; DC undervoltage fault. This module is on the rear side of the shelf, at the left.
Wed Jul  5 07:54:39 BNT [filer02: ses.status.psWarning:warning]: replayed event: DS12-ESAS shelf 0 on channel 0c power warning for Power supply 2: non-critical status; DC undervoltage fault. This module is on the rear side of the shelf, at the right.
Wed Jul  5 07:54:48 BNT [filer02: cf.fm.launch:info]: replayed event: Launching cluster monitor
Wed Jul  5 07:54:48 BNT [filer02: cf.fm.partner:info]: replayed event: Cluster monitor: partner 'filer01'
Wed Jul  5 07:54:53 BNT [filer02: fcp.service.adapter:warning]: replayed event: FCP started but no adapter is present!
Wed Jul  5 07:54:53 BNT [filer02: cf.fsm.takeoverByPartnerDisabled:notice]: replayed event: Cluster monitor: takeover of  by filer01 disabled (partner booting)
Wed Jul  5 07:54:53 BNT [filer02: kern.syslog.msg:notice]: replayed event: The system was down for 10 seconds
Wed Jul  5 07:54:54 BNT [filer02: cf.fsm.takeoverOfPartnerEnabled:notice]: replayed event: Cluster monitor: takeover of filer01 enabled
Wed Jul  5 07:54:54 BNT [filer02: cf.fsm.takeoverByPartnerEnabled:notice]: replayed event: Cluster monitor: takeover of filer02 by filer01 enabled
Wed Jul  5 07:54:55 BNT [filer02: disk.failmsg:error]: Disk 0a.35 (N00WLAUL): message received.
Wed Jul  5 07:54:55 BNT [filer02: disk.failmsg:error]: Disk 0a.40 (N00V0W8L): message received.
Wed Jul  5 07:54:55 BNT [filer02: disk.failmsg:error]: Disk 0a.33 (N00W75PL): message received.
Wed Jul  5 07:54:55 BNT [filer02: raid.disk.unload.done:info]: Unload of Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] has completed successfully
Wed Jul  5 07:54:55 BNT [filer02: raid.disk.unload.done:info]: Unload of Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] has completed successfully
Wed Jul  5 07:54:55 BNT [filer02: raid.disk.unload.done:info]: Unload of Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] has completed successfully
Wed Jul  5 07:54:55 BNT [filer02: raid.fdr.failed.ok:info]: Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] successfully deleted from spare pool
Wed Jul  5 07:54:55 BNT [filer02: raid.fdr.failed.ok:info]: Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] successfully deleted from spare pool
Wed Jul  5 07:54:55 BNT [filer02: raid.fdr.failed.ok:info]: Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] successfully deleted from spare pool
Wed Jul  5 07:54:55 BNT [filer02: dfu.firmwareUpToDate:info]: Firmware is up-to-date on all disk drives
Wed Jul  5 07:54:57 BNT [filer02: perf.archive.start:info]: Performance archiver started. Sampling 22 objects and 198 counters.
Wed Jul  5 07:54:58 BNT [filer02: netif.linkUp:info]: Ethernet e0b: Link up.
Wed Jul  5 07:54:59 BNT [filer02: netif.linkUp:info]: Ethernet e0a: Link up.
Wed Jul  5 07:54:59 BNT [filer02: pvif.switchLink:warning]: vif1: switching to e0a
Wed Jul  5 07:55:01 BNT [filer02: rc:info]: relog syslog Wed Jul  5 07:54:28 BNT [filer02: fci.adapter.error:warning]: Fibre Channel adapter driver encountered error "Failed t
Wed Jul  5 07:55:01 BNT [filer02: rc:info]: relog syslog Wed Jul  5 07:54:28 BNT [filer02: fci.adapter.reset:debug]: Resetting Fibre Channel adapter 0a.
Wed Jul  5 07:55:01 BNT [filer02: iscsi.service.startup:info]: iSCSI service startup
Wed Jul  5 07:55:02 BNT [filer02: cmds.sysconf.logErr:error]: sysconfig: Unless directed by NetApp Global Services the cf.takeover.on_panic option should be set to On. .
Wed Jul  5 07:55:02 BNT [filer02: raid.rg.recons.missing:notice]: RAID group /aggr1/plex0/rg0 is missing 2 disk(s).
Wed Jul  5 07:55:02 BNT [filer02: raid.rg.recons.cantStart:warning]: The reconstruction cannot start in RAID group /aggr1/plex0/rg0: No 520 bps disk of required size available in spare pool, targeting any  spare pool
Wed Jul  5 07:55:02 BNT [filer02: mgr.boot.disk_done:info]: NetApp Release 7.3.2 boot complete. Last disk update written at Wed Jul  5 07:54:17 BNT 2017
Wed Jul  5 07:55:02 BNT [filer02: monitor.raid.brokenDisk:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken.
Wed Jul  5 07:55:02 BNT [filer02: monitor.brokenDisk.notice:notice]: When two disks are broken in raid_dp volume,  the system shuts down automatically every 24  hours to encourage you to replace the disk.  If you reboot the system it will run for another  24 hours before shutting down. (The 24 hour timeout  may be increased by altering the "raid.timeout"  value using the "options" command.)
Wed Jul  5 07:55:02 BNT [filer02: diskown.errorReadingOwnership:warning]: error 3 (disk failed) while reading ownership on disk 0a.24 (S/N N00SSSAL)
Wed Jul  5 07:55:02 BNT [filer02: diskown.errorReadingOwnership:warning]: error 3 (disk failed) while reading ownership on disk 0a.26 (S/N N00V0UWL)
Wed Jul  5 07:55:02 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] is still present in the system and should be removed.
Wed Jul  5 07:55:02 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] is still present in the system and should be removed.
Wed Jul  5 07:55:02 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] is still present in the system and should be removed.
Wed Jul  5 07:55:02 BNT [filer02: shelf.config.spha:info]: System is using single path HA attached storage only.
Wed Jul  5 07:55:02 BNT [filer02: mgr.boot.reason_ok:notice]: System rebooted after power-on.
Wed Jul  5 07:55:28 BNT [filer02: raid.rg.spares.low:warning]: /aggr1/plex0/rg0
Wed Jul  5 07:55:31 BNT [filer02: nbt.nbns.registrationComplete:info]: NBT: All CIFS name registrations have completed for the local server.
Wed Jul  5 07:55:36 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SPARES_LOW)
Wed Jul  5 07:55:44 BNT last message repeated 2 times
Wed Jul  5 07:55:44 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201706290753.3) was not posted for host (0), it was dropped from queue due to lack of space
Wed Jul  5 07:55:51 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (PERFORMANCE SNAPSHOT)
Wed Jul  5 07:55:51 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201706290754.0) was not posted for host (0), it was dropped from queue due to lack of space
Wed Jul  5 07:55:59 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (CLUSTER ERROR: DISK/SHELF COUNT MISMATCH)
Wed Jul  5 07:55:59 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201706290758.0) was not posted for host (0), it was dropped from queue due to lack of space
Wed Jul  5 07:56:10 BNT [filer02: monitor.globalStatus.nonCritical:warning]: Disk on adapter 0a, shelf 2, bay 3, failed. Disk on adapter 0a, shelf 2, bay 8, failed. Disk on adapter 0a, shelf 2, bay 1, failed.  There are not enough spare disks. /vol/vol1 is full (using or reserving 100% of space and 0% of inodes, using 92% of reserve).
Wed Jul  5 07:56:14 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Wed Jul  5 07:56:14 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201706290800.0) was not posted for host (0), it was dropped from queue due to lack of space
Wed Jul  5 08:00:07 BNT [filer02: kern.uptime.filer:info]:   8:00am up 5 mins, 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 0 iSCSI ops
Wed Jul  5 08:00:07 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 24 hours.
Wed Jul  5 08:00:41 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Wed Jul  5 08:04:41 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Wed Jul  5 08:08:48 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Wed Jul  5 08:13:14 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Wed Jul  5 08:17:27 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Wed Jul  5 08:34:12 BNT last message repeated 4 times
Wed Jul  5 08:34:12 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201706291200.0) for host (0) was not posted to NetApp. The system will drop the message
Wed Jul  5 08:57:13 BNT [filer02: iscsi.notice:notice]: ISCSI: New session from initiator iqn.1991-05.com.microsoft:pcb-722-ad1.pcbl722dom.local at IP addr 10.72.4.1
Wed Jul  5 09:00:00 BNT [filer02: kern.uptime.filer:info]:   9:00am up  1:04 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 3140 iSCSI ops
Wed Jul  5 09:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 23 hours.
Wed Jul  5 10:00:00 BNT [filer02: kern.uptime.filer:info]:  10:00am up  2:04 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 9486 iSCSI ops
Wed Jul  5 10:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 22 hours.
Wed Jul  5 11:00:00 BNT [filer02: kern.uptime.filer:info]:  11:00am up  3:04 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 55654 iSCSI ops
Wed Jul  5 11:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 21 hours.
Wed Jul  5 12:00:00 BNT [filer02: kern.uptime.filer:info]:  12:00pm up  4:04 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 332395 iSCSI ops
Wed Jul  5 12:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 20 hours.
Wed Jul  5 12:00:15 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Wed Jul  5 12:00:39 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Wed Jul  5 12:04:26 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Wed Jul  5 12:08:24 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Wed Jul  5 12:12:03 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Wed Jul  5 12:16:20 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Wed Jul  5 12:36:19 BNT last message repeated 5 times
Wed Jul  5 12:40:24 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Wed Jul  5 12:56:30 BNT last message repeated 4 times
Wed Jul  5 12:56:30 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201706291600.0) for host (0) was not posted to NetApp. The system will drop the message
Wed Jul  5 13:00:00 BNT [filer02: kern.uptime.filer:info]:   1:00pm up  5:04 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 349577 iSCSI ops
Wed Jul  5 13:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 19 hours.
Wed Jul  5 14:00:00 BNT [filer02: kern.uptime.filer:info]:   2:00pm up  6:04 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 350643 iSCSI ops
Wed Jul  5 14:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 18 hours.
Wed Jul  5 15:00:00 BNT [filer02: kern.uptime.filer:info]:   3:00pm up  7:05 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 382354 iSCSI ops
Wed Jul  5 15:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 17 hours.
Wed Jul  5 16:00:00 BNT [filer02: kern.uptime.filer:info]:   4:00pm up  8:05 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 453992 iSCSI ops
Wed Jul  5 16:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 16 hours.
Wed Jul  5 16:00:00 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Wed Jul  5 16:00:24 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Wed Jul  5 16:04:40 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Wed Jul  5 16:08:31 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Wed Jul  5 16:12:27 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Wed Jul  5 16:16:05 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Wed Jul  5 16:32:02 BNT last message repeated 4 times
Wed Jul  5 16:36:30 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Wed Jul  5 16:52:53 BNT last message repeated 4 times
Wed Jul  5 16:56:44 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Wed Jul  5 16:56:44 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201706292000.0) for host (0) was not posted to NetApp. The system will drop the message
Wed Jul  5 17:00:00 BNT [filer02: kern.uptime.filer:info]:   5:00pm up  9:05 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 455324 iSCSI ops
Wed Jul  5 17:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 15 hours.
Wed Jul  5 18:00:00 BNT [filer02: kern.uptime.filer:info]:   6:00pm up 10:05 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 458314 iSCSI ops
Wed Jul  5 18:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 14 hours.
Wed Jul  5 19:00:00 BNT [filer02: kern.uptime.filer:info]:   7:00pm up 11:05 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 459089 iSCSI ops
Wed Jul  5 19:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 13 hours.
Wed Jul  5 20:00:00 BNT [filer02: kern.uptime.filer:info]:   8:00pm up 12:05 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 459154 iSCSI ops
Wed Jul  5 20:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 12 hours.
Wed Jul  5 20:00:01 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Wed Jul  5 20:37:12 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Wed Jul  5 20:56:19 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201706300000.0) for host (0) was not posted to NetApp. The system will drop the message
Wed Jul  5 21:00:00 BNT [filer02: kern.uptime.filer:info]:   9:00pm up 13:05 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 459154 iSCSI ops
Wed Jul  5 21:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 11 hours.
Wed Jul  5 22:00:00 BNT [filer02: kern.uptime.filer:info]:  10:00pm up 14:05 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 459154 iSCSI ops
Wed Jul  5 22:00:01 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 10 hours.
Wed Jul  5 23:00:00 BNT [filer02: kern.uptime.filer:info]:  11:00pm up 15:05 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 459154 iSCSI ops
Wed Jul  5 23:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 9 hours.
Thu Jul  6 00:00:00 BNT [filer02: kern.uptime.filer:info]:  12:00am up 16:05 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 459154 iSCSI ops
Thu Jul  6 00:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 8 hours.
Thu Jul  6 00:00:46 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of nightly.0 snapshot (Snapshot operation failed: No space left on device.)
Thu Jul  6 00:23:33 BNT [filer02: asup.post.host:info]: Autosupport (SYSTEM CONFIGURATION WARNING) cannot connect to url support.netapp.com/asupprod/post/1.0/postAsup (can not connect to IP address 216.240.21.18 using port 443: Connection timed out)
Thu Jul  6 00:59:12 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201706300741.0) for host (0) was not posted to NetApp. The system will drop the message
Thu Jul  6 01:00:00 BNT [filer02: kern.uptime.filer:info]:   1:00am up 17:05 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 459154 iSCSI ops
Thu Jul  6 01:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 7 hours.
Thu Jul  6 02:00:00 BNT [filer02: kern.uptime.filer:info]:   2:00am up 18:05 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 459154 iSCSI ops
Thu Jul  6 02:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 6 hours.
Thu Jul  6 03:00:00 BNT [filer02: kern.uptime.filer:info]:   3:00am up 19:05 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 459154 iSCSI ops
Thu Jul  6 03:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 5 hours.
Thu Jul  6 04:00:00 BNT [filer02: kern.uptime.filer:info]:   4:00am up 20:05 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 459154 iSCSI ops
Thu Jul  6 04:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 4 hours.
Thu Jul  6 05:00:00 BNT [filer02: kern.uptime.filer:info]:   5:00am up 21:05 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 459154 iSCSI ops
Thu Jul  6 05:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 3 hours.
Thu Jul  6 06:00:00 BNT [filer02: kern.uptime.filer:info]:   6:00am up 22:05 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 459154 iSCSI ops
Thu Jul  6 06:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 2 hours.
Thu Jul  6 07:00:00 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] is still present in the system and should be removed.
Thu Jul  6 07:00:00 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] is still present in the system and should be removed.
Thu Jul  6 07:00:00 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] is still present in the system and should be removed.
Wed Jul  5 23:55:04 GMT [localhost: rc:notice]: The system was down for 36 seconds
Wed Jul  5 23:55:04 GMT [filer02: cf.fsm.takeoverOfPartnerEnabled:notice]: Cluster monitor: takeover of filer01 enabled
Thu Jul  6 07:54:29 BNT [filer02: cf.nm.nicTransitionUp:info]: replayed event: Interconnect link 0 is UP
Thu Jul  6 07:54:36 BNT [filer02: fci.initialization.failed:error]: replayed event: Initialization failed on Fibre Channel adapter 0b.
Thu Jul  6 07:54:38 BNT [filer02: nvram.battery.turned.on:info]: replayed event: The NVRAM battery is turned ON. It is turned OFF during system shutdown.
Thu Jul  6 07:55:05 BNT [filer02: cf.fsm.takeoverByPartnerEnabled:notice]: Cluster monitor: takeover of filer02 by filer01 enabled
Thu Jul  6 07:54:41 BNT [filer02: diskown.isEnabled:info]: replayed event: software ownership has been enabled for this system
Thu Jul  6 07:54:45 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.2 is a local HA mailbox disk.
Thu Jul  6 07:54:45 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.0 is a local HA mailbox disk.
Thu Jul  6 07:54:45 BNT [filer02: fmmb.instStat.change:info]: replayed event: normal mailbox instance on local side.
Thu Jul  6 07:54:46 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.7 is a partner HA mailbox disk.
Thu Jul  6 07:54:46 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.3 is a partner HA mailbox disk.
Thu Jul  6 07:54:46 BNT [filer02: fmmb.instStat.change:info]: replayed event: normal mailbox instance on partner side.
Thu Jul  6 07:54:46 BNT [filer02: cf.fm.partner:info]: replayed event: Cluster monitor: partner 'filer01'
Thu Jul  6 07:54:46 BNT [filer02: cf.fm.timeMasterStatus:info]: replayed event: Acting as cluster time slave
Thu Jul  6 07:54:46 BNT [filer02: raid.assim.disk.brokenPreAssim:error]: replayed event: Broken Disk 0a.40 Shelf ? Bay ? [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] detected prior to assimilation. It should be removed.
Thu Jul  6 07:54:46 BNT [filer02: raid.assim.disk.brokenPreAssim:error]: replayed event: Broken Disk 0a.35 Shelf ? Bay ? [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] detected prior to assimilation. It should be removed.
Thu Jul  6 07:54:46 BNT [filer02: raid.assim.disk.brokenPreAssim:error]: replayed event: Broken Disk 0a.33 Shelf ? Bay ? [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] detected prior to assimilation. It should be removed.
Thu Jul  6 07:54:47 BNT [filer02: coredump.spare.none:info]: replayed event: No sparecore disk was found.
Thu Jul  6 07:54:47 BNT [filer02: raid.vol.replay.nvram:info]: replayed event: Performing raid replay on volume(s)
Thu Jul  6 07:54:47 BNT [filer02: raid.cksum.replay.summary:info]: replayed event: Replayed 0 checksum blocks.
Thu Jul  6 07:54:47 BNT [filer02: raid.stripe.replay.summary:info]: replayed event: Replayed 0 stripes.
Thu Jul  6 07:54:50 BNT [filer02: ses.status.psWarning:warning]: replayed event: DS12-ESAS shelf 0 on channel 0c power warning for Power supply 1: non-critical status; DC undervoltage fault. This module is on the rear side of the shelf, at the left.
Thu Jul  6 07:54:50 BNT [filer02: ses.status.psWarning:warning]: replayed event: DS12-ESAS shelf 0 on channel 0c power warning for Power supply 2: non-critical status; DC undervoltage fault. This module is on the rear side of the shelf, at the right.
Thu Jul  6 07:54:59 BNT [filer02: cf.fm.launch:info]: replayed event: Launching cluster monitor
Thu Jul  6 07:54:59 BNT [filer02: cf.fm.partner:info]: replayed event: Cluster monitor: partner 'filer01'
Thu Jul  6 07:55:03 BNT [filer02: cf.fsm.takeoverOfPartnerDisabled:notice]: replayed event: Cluster monitor: takeover of filer01 disabled (partner booting)
Thu Jul  6 07:55:04 BNT [filer02: fcp.service.adapter:warning]: replayed event: FCP started but no adapter is present!
Thu Jul  6 07:55:04 BNT [filer02: kern.syslog.msg:notice]: replayed event: The system was down for 36 seconds
Thu Jul  6 07:55:04 BNT [filer02: cf.fsm.takeoverOfPartnerEnabled:notice]: replayed event: Cluster monitor: takeover of filer01 enabled
Thu Jul  6 07:55:06 BNT [filer02: disk.failmsg:error]: Disk 0a.40 (N00V0W8L): message received.
Thu Jul  6 07:55:06 BNT [filer02: disk.failmsg:error]: Disk 0a.35 (N00WLAUL): message received.
Thu Jul  6 07:55:06 BNT [filer02: disk.failmsg:error]: Disk 0a.33 (N00W75PL): message received.
Thu Jul  6 07:55:06 BNT [filer02: raid.disk.unload.done:info]: Unload of Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] has completed successfully
Thu Jul  6 07:55:06 BNT [filer02: raid.disk.unload.done:info]: Unload of Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] has completed successfully
Thu Jul  6 07:55:06 BNT [filer02: raid.disk.unload.done:info]: Unload of Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] has completed successfully
Thu Jul  6 07:55:06 BNT [filer02: raid.fdr.failed.ok:info]: Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] successfully deleted from spare pool
Thu Jul  6 07:55:06 BNT [filer02: raid.fdr.failed.ok:info]: Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] successfully deleted from spare pool
Thu Jul  6 07:55:06 BNT [filer02: raid.fdr.failed.ok:info]: Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] successfully deleted from spare pool
Thu Jul  6 07:55:06 BNT [filer02: dfu.firmwareUpToDate:info]: Firmware is up-to-date on all disk drives
Thu Jul  6 07:55:08 BNT [filer02: perf.archive.start:info]: Performance archiver started. Sampling 22 objects and 198 counters.
Thu Jul  6 07:55:10 BNT [filer02: netif.linkUp:info]: Ethernet e0b: Link up.
Thu Jul  6 07:55:10 BNT [filer02: netif.linkUp:info]: Ethernet e0a: Link up.
Thu Jul  6 07:55:10 BNT [filer02: pvif.switchLink:warning]: vif1: switching to e0a
Thu Jul  6 07:55:12 BNT [filer02: rc:info]: relog syslog Thu Jul  6 07:00:00 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TS
Thu Jul  6 07:55:12 BNT [filer02: rc:info]: relog syslog Thu Jul  6 07:00:00 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TS
Thu Jul  6 07:55:12 BNT [filer02: iscsi.service.startup:info]: iSCSI service startup
Thu Jul  6 07:55:13 BNT [filer02: cmds.sysconf.logErr:error]: sysconfig: Unless directed by NetApp Global Services the cf.takeover.on_panic option should be set to On. .
Thu Jul  6 07:55:13 BNT [filer02: raid.rg.recons.missing:notice]: RAID group /aggr1/plex0/rg0 is missing 2 disk(s).
Thu Jul  6 07:55:13 BNT [filer02: raid.rg.recons.cantStart:warning]: The reconstruction cannot start in RAID group /aggr1/plex0/rg0: No 520 bps disk of required size available in spare pool, targeting any  spare pool
Thu Jul  6 07:55:13 BNT [filer02: mgr.boot.disk_done:info]: NetApp Release 7.3.2 boot complete. Last disk update written at Thu Jul  6 07:54:12 BNT 2017
Thu Jul  6 07:55:13 BNT [filer02: monitor.raid.brokenDisk:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken.
Thu Jul  6 07:55:13 BNT [filer02: monitor.brokenDisk.notice:notice]: When two disks are broken in raid_dp volume,  the system shuts down automatically every 24  hours to encourage you to replace the disk.  If you reboot the system it will run for another  24 hours before shutting down. (The 24 hour timeout  may be increased by altering the "raid.timeout"  value using the "options" command.)
Thu Jul  6 07:55:14 BNT [filer02: diskown.errorReadingOwnership:warning]: error 3 (disk failed) while reading ownership on disk 0a.24 (S/N N00SSSAL)
Thu Jul  6 07:55:14 BNT [filer02: diskown.errorReadingOwnership:warning]: error 3 (disk failed) while reading ownership on disk 0a.26 (S/N N00V0UWL)
Thu Jul  6 07:55:14 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] is still present in the system and should be removed.
Thu Jul  6 07:55:14 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] is still present in the system and should be removed.
Thu Jul  6 07:55:14 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] is still present in the system and should be removed.
Thu Jul  6 07:55:14 BNT [filer02: shelf.config.spha:info]: System is using single path HA attached storage only.
Thu Jul  6 07:55:14 BNT [filer02: mgr.boot.reason_ok:notice]: System rebooted after power-on.
Thu Jul  6 07:55:41 BNT [filer02: raid.rg.spares.low:warning]: /aggr1/plex0/rg0
Thu Jul  6 07:55:43 BNT [filer02: nbt.nbns.registrationComplete:info]: NBT: All CIFS name registrations have completed for the local server.
Thu Jul  6 07:55:47 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SHUTDOWN PENDING (degraded mode))
Thu Jul  6 07:55:55 BNT last message repeated 2 times
Thu Jul  6 07:55:55 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201706300741.1) was not posted for host (0), it was dropped from queue due to lack of space
Thu Jul  6 07:56:02 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (REBOOT (power on))
Thu Jul  6 07:56:02 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201706300741.2) was not posted for host (0), it was dropped from queue due to lack of space
Thu Jul  6 07:56:10 BNT [filer02: monitor.globalStatus.nonCritical:warning]: Disk on adapter 0a, shelf 2, bay 8, failed. Disk on adapter 0a, shelf 2, bay 3, failed. Disk on adapter 0a, shelf 2, bay 1, failed.  There are not enough spare disks. /vol/vol1 is full (using or reserving 100% of space and 0% of inodes, using 92% of reserve).
Thu Jul  6 07:56:10 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SPARES_LOW)
Thu Jul  6 07:56:10 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201706300741.3) was not posted for host (0), it was dropped from queue due to lack of space
Thu Jul  6 07:56:26 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (PERFORMANCE SNAPSHOT)
Thu Jul  6 07:56:26 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201706300742.0) was not posted for host (0), it was dropped from queue due to lack of space
Thu Jul  6 08:00:02 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Thu Jul  6 08:00:07 BNT [filer02: kern.uptime.filer:info]:   8:00am up 5 mins, 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 0 iSCSI ops
Thu Jul  6 08:00:07 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 24 hours.
Thu Jul  6 08:00:12 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu Jul  6 08:00:12 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201706300800.0) was not posted for host (0), it was dropped from queue due to lack of space
Thu Jul  6 08:04:33 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu Jul  6 08:08:59 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu Jul  6 08:12:58 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu Jul  6 08:17:06 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu Jul  6 08:21:34 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu Jul  6 08:34:02 BNT last message repeated 3 times
Thu Jul  6 08:34:02 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201706301200.0) for host (0) was not posted to NetApp. The system will drop the message
Thu Jul  6 08:39:34 BNT [filer02: iscsi.notice:notice]: ISCSI: New session from initiator iqn.1991-05.com.microsoft:pcb-722-ad1.pcbl722dom.local at IP addr 10.72.4.1
Thu Jul  6 09:00:00 BNT [filer02: kern.uptime.filer:info]:   9:00am up  1:04 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 994 iSCSI ops
Thu Jul  6 09:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 23 hours.
Thu Jul  6 10:00:00 BNT [filer02: kern.uptime.filer:info]:  10:00am up  2:04 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 162669 iSCSI ops
Thu Jul  6 10:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 22 hours.
Thu Jul  6 11:00:00 BNT [filer02: kern.uptime.filer:info]:  11:00am up  3:04 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 681619 iSCSI ops
Thu Jul  6 11:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 21 hours.
Thu Jul  6 12:00:00 BNT [filer02: kern.uptime.filer:info]:  12:00pm up  4:04 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 933960 iSCSI ops
Thu Jul  6 12:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 20 hours.
Thu Jul  6 12:00:30 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Thu Jul  6 12:00:56 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu Jul  6 12:05:22 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu Jul  6 12:09:23 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu Jul  6 12:13:51 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu Jul  6 12:17:50 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu Jul  6 12:36:42 BNT last message repeated 5 times
Thu Jul  6 12:40:26 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu Jul  6 12:55:35 BNT last message repeated 4 times
Thu Jul  6 12:55:35 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201706301600.0) for host (0) was not posted to NetApp. The system will drop the message
Thu Jul  6 13:00:00 BNT [filer02: kern.uptime.filer:info]:   1:00pm up  5:04 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 1032762 iSCSI ops
Thu Jul  6 13:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 19 hours.
Thu Jul  6 14:00:00 BNT [filer02: kern.uptime.filer:info]:   2:00pm up  6:04 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 1104535 iSCSI ops
Thu Jul  6 14:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 18 hours.
Thu Jul  6 15:00:00 BNT [filer02: kern.uptime.filer:info]:   3:00pm up  7:04 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 1112815 iSCSI ops
Thu Jul  6 15:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 17 hours.
Thu Jul  6 16:00:00 BNT [filer02: kern.uptime.filer:info]:   4:00pm up  8:04 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 2216933 iSCSI ops
Thu Jul  6 16:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 16 hours.
Thu Jul  6 16:00:15 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Thu Jul  6 16:00:41 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu Jul  6 16:04:52 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu Jul  6 16:09:11 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu Jul  6 16:13:19 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu Jul  6 16:17:15 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu Jul  6 16:37:14 BNT last message repeated 5 times
Thu Jul  6 16:40:46 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu Jul  6 16:56:56 BNT last message repeated 4 times
Thu Jul  6 16:56:56 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201706302000.0) for host (0) was not posted to NetApp. The system will drop the message
Thu Jul  6 17:00:00 BNT [filer02: kern.uptime.filer:info]:   5:00pm up  9:04 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 2227046 iSCSI ops
Thu Jul  6 17:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 15 hours.
Thu Jul  6 18:00:00 BNT [filer02: kern.uptime.filer:info]:   6:00pm up 10:04 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 2356921 iSCSI ops
Thu Jul  6 18:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 14 hours.
Thu Jul  6 19:00:00 BNT [filer02: kern.uptime.filer:info]:   7:00pm up 11:04 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 2471843 iSCSI ops
Thu Jul  6 19:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 13 hours.
Thu Jul  6 20:00:00 BNT [filer02: kern.uptime.filer:info]:   8:00pm up 12:04 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 2471843 iSCSI ops
Thu Jul  6 20:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 12 hours.
Thu Jul  6 20:00:16 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Thu Jul  6 20:00:40 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu Jul  6 20:20:40 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu Jul  6 20:32:45 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu Jul  6 20:48:39 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Thu Jul  6 20:56:47 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201707010000.0) for host (0) was not posted to NetApp. The system will drop the message
Thu Jul  6 21:00:00 BNT [filer02: kern.uptime.filer:info]:   9:00pm up 13:05 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 2471843 iSCSI ops
Thu Jul  6 21:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 11 hours.
Thu Jul  6 22:00:00 BNT [filer02: kern.uptime.filer:info]:  10:00pm up 14:05 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 2471843 iSCSI ops
Thu Jul  6 22:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 10 hours.
Thu Jul  6 23:00:00 BNT [filer02: kern.uptime.filer:info]:  11:00pm up 15:05 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 2471843 iSCSI ops
Thu Jul  6 23:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 9 hours.
Fri Jul  7 00:00:00 BNT [filer02: kern.uptime.filer:info]:  12:00am up 16:05 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 2471843 iSCSI ops
Fri Jul  7 00:00:01 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 8 hours.
Fri Jul  7 00:00:01 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of nightly.0 snapshot (Snapshot operation failed: No space left on device.)
Fri Jul  7 00:23:49 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SYSTEM CONFIGURATION WARNING)
Fri Jul  7 00:56:30 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201707030747.0) for host (0) was not posted to NetApp. The system will drop the message
Fri Jul  7 01:00:00 BNT [filer02: kern.uptime.filer:info]:   1:00am up 17:05 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 2471843 iSCSI ops
Fri Jul  7 01:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 7 hours.
Fri Jul  7 02:00:00 BNT [filer02: kern.uptime.filer:info]:   2:00am up 18:05 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 2471843 iSCSI ops
Fri Jul  7 02:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 6 hours.
Fri Jul  7 03:00:00 BNT [filer02: kern.uptime.filer:info]:   3:00am up 19:05 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 2471843 iSCSI ops
Fri Jul  7 03:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 5 hours.
Fri Jul  7 04:00:00 BNT [filer02: kern.uptime.filer:info]:   4:00am up 20:05 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 2471843 iSCSI ops
Fri Jul  7 04:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 4 hours.
Fri Jul  7 05:00:00 BNT [filer02: kern.uptime.filer:info]:   5:00am up 21:05 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 2471843 iSCSI ops
Fri Jul  7 05:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 3 hours.
Fri Jul  7 06:00:00 BNT [filer02: kern.uptime.filer:info]:   6:00am up 22:05 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 2471843 iSCSI ops
Fri Jul  7 06:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 2 hours.
Fri Jul  7 07:00:00 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] is still present in the system and should be removed.
Fri Jul  7 07:00:00 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] is still present in the system and should be removed.
Fri Jul  7 07:00:00 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] is still present in the system and should be removed.
Fri Jul  7 08:00:00 BNT [filer02: monitor.brokenDisk.notice:notice]: When two disks are broken in raid_dp volume,  the system shuts down automatically every 24  hours to encourage you to replace the disk.  If you reboot the system it will run for another  24 hours before shutting down. (The 24 hour timeout  may be increased by altering the "raid.timeout"  value using the "options" command.)
Fri Jul  7 08:00:00 BNT [filer02: monitor.shutdown.brokenDisk:EMERGENCY]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system now.
Fri Jul  7 08:00:03 BNT [filer02: cf.fsm.takeoverOfPartnerDisabled:notice]: Cluster monitor: takeover of filer01 disabled (local halt in progress)
Fri Jul  7 08:00:07 BNT [filer02: kern.shutdown:notice]: System shut down because : "BROKEN DISK".
Fri Jul  7 00:25:55 GMT [localhost: rc:notice]: The system was down for 1544 seconds
Fri Jul  7 08:25:29 BNT [filer02: cf.nm.nicTransitionUp:info]: replayed event: Interconnect link 0 is UP
Fri Jul  7 08:25:36 BNT [filer02: fci.initialization.failed:error]: replayed event: Initialization failed on Fibre Channel adapter 0b.
Fri Jul  7 08:25:38 BNT [filer02: nvram.battery.turned.on:info]: replayed event: The NVRAM battery is turned ON. It is turned OFF during system shutdown.
Fri Jul  7 08:25:41 BNT [filer02: diskown.isEnabled:info]: replayed event: software ownership has been enabled for this system
Fri Jul  7 08:25:45 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.2 is a local HA mailbox disk.
Fri Jul  7 08:25:45 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.0 is a local HA mailbox disk.
Fri Jul  7 08:25:45 BNT [filer02: fmmb.instStat.change:info]: replayed event: normal mailbox instance on local side.
Fri Jul  7 08:25:45 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.7 is a partner HA mailbox disk.
Fri Jul  7 08:25:45 BNT [filer02: fmmb.current.lock.disk:info]: replayed event: Disk 0c.00.3 is a partner HA mailbox disk.
Fri Jul  7 08:25:45 BNT [filer02: fmmb.instStat.change:info]: replayed event: normal mailbox instance on partner side.
Fri Jul  7 08:25:45 BNT [filer02: cf.fm.partner:info]: replayed event: Cluster monitor: partner 'filer01'
Fri Jul  7 08:25:45 BNT [filer02: cf.fm.timeMasterStatus:info]: replayed event: Acting as cluster time slave
Fri Jul  7 08:25:46 BNT [filer02: raid.assim.disk.brokenPreAssim:error]: replayed event: Broken Disk 0a.40 Shelf ? Bay ? [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] detected prior to assimilation. It should be removed.
Fri Jul  7 08:25:46 BNT [filer02: raid.assim.disk.brokenPreAssim:error]: replayed event: Broken Disk 0a.35 Shelf ? Bay ? [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] detected prior to assimilation. It should be removed.
Fri Jul  7 08:25:46 BNT [filer02: raid.assim.disk.brokenPreAssim:error]: replayed event: Broken Disk 0a.33 Shelf ? Bay ? [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] detected prior to assimilation. It should be removed.
Fri Jul  7 08:25:46 BNT [filer02: coredump.spare.none:info]: replayed event: No sparecore disk was found.
Fri Jul  7 08:25:47 BNT [filer02: raid.cksum.replay.summary:info]: replayed event: Replayed 0 checksum blocks.
Fri Jul  7 08:25:47 BNT [filer02: raid.stripe.replay.summary:info]: replayed event: Replayed 0 stripes.
Fri Jul  7 08:25:48 BNT [filer02: cf.fm.launch:info]: replayed event: Launching cluster monitor
Fri Jul  7 08:25:49 BNT [filer02: cf.fm.partner:info]: replayed event: Cluster monitor: partner 'filer01'
Fri Jul  7 08:25:49 BNT [filer02: cf.fsm.takeoverOfPartnerDisabled:notice]: replayed event: Cluster monitor: takeover of filer01 disabled (partner booting)
Fri Jul  7 08:25:54 BNT [filer02: fcp.service.adapter:warning]: replayed event: FCP started but no adapter is present!
Fri Jul  7 08:25:55 BNT [filer02: kern.syslog.msg:notice]: replayed event: The system was down for 1544 seconds
Fri Jul  7 08:25:56 BNT [filer02: disk.failmsg:error]: Disk 0a.40 (N00V0W8L): message received.
Fri Jul  7 08:25:56 BNT [filer02: disk.failmsg:error]: Disk 0a.35 (N00WLAUL): message received.
Fri Jul  7 08:25:56 BNT [filer02: disk.failmsg:error]: Disk 0a.33 (N00W75PL): message received.
Fri Jul  7 08:25:56 BNT [filer02: raid.disk.unload.done:info]: Unload of Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] has completed successfully
Fri Jul  7 08:25:56 BNT [filer02: raid.disk.unload.done:info]: Unload of Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] has completed successfully
Fri Jul  7 08:25:56 BNT [filer02: raid.disk.unload.done:info]: Unload of Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] has completed successfully
Fri Jul  7 08:25:56 BNT [filer02: raid.fdr.failed.ok:info]: Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] successfully deleted from spare pool
Fri Jul  7 08:25:56 BNT [filer02: raid.fdr.failed.ok:info]: Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] successfully deleted from spare pool
Fri Jul  7 08:25:56 BNT [filer02: raid.fdr.failed.ok:info]: Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] successfully deleted from spare pool
Fri Jul  7 08:25:57 BNT [filer02: dfu.firmwareUpToDate:info]: Firmware is up-to-date on all disk drives
Fri Jul  7 08:25:57 BNT [filer02: cf.fsm.takeoverOfPartnerEnabled:notice]: Cluster monitor: takeover of filer01 enabled
Fri Jul  7 08:25:57 BNT [filer02: cf.fsm.takeoverByPartnerEnabled:notice]: Cluster monitor: takeover of filer02 by filer01 enabled
Fri Jul  7 08:25:58 BNT [filer02: perf.archive.start:info]: Performance archiver started. Sampling 22 objects and 198 counters.
Fri Jul  7 08:26:00 BNT [filer02: netif.linkUp:info]: Ethernet e0b: Link up.
Fri Jul  7 08:26:00 BNT [filer02: netif.linkUp:info]: Ethernet e0a: Link up.
Fri Jul  7 08:26:01 BNT [filer02: pvif.switchLink:warning]: vif1: switching to e0a
Fri Jul  7 08:26:02 BNT [filer02: iscsi.service.startup:info]: iSCSI service startup
Fri Jul  7 08:26:03 BNT [filer02: cmds.sysconf.logErr:error]: sysconfig: Unless directed by NetApp Global Services the cf.takeover.on_panic option should be set to On. .
Fri Jul  7 08:26:03 BNT [filer02: raid.rg.recons.missing:notice]: RAID group /aggr1/plex0/rg0 is missing 2 disk(s).
Fri Jul  7 08:26:03 BNT [filer02: raid.rg.recons.cantStart:warning]: The reconstruction cannot start in RAID group /aggr1/plex0/rg0: No 520 bps disk of required size available in spare pool, targeting any  spare pool
Fri Jul  7 08:26:03 BNT [filer02: mgr.boot.disk_done:info]: NetApp Release 7.3.2 boot complete. Last disk update written at Fri Jul  7 08:00:08 BNT 2017
Fri Jul  7 08:26:03 BNT [filer02: monitor.raid.brokenDisk:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken.
Fri Jul  7 08:26:03 BNT [filer02: monitor.brokenDisk.notice:notice]: When two disks are broken in raid_dp volume,  the system shuts down automatically every 24  hours to encourage you to replace the disk.  If you reboot the system it will run for another  24 hours before shutting down. (The 24 hour timeout  may be increased by altering the "raid.timeout"  value using the "options" command.)
Fri Jul  7 08:26:03 BNT [filer02: diskown.errorReadingOwnership:warning]: error 3 (disk failed) while reading ownership on disk 0a.24 (S/N N00SSSAL)
Fri Jul  7 08:26:03 BNT [filer02: diskown.errorReadingOwnership:warning]: error 3 (disk failed) while reading ownership on disk 0a.26 (S/N N00V0UWL)
Fri Jul  7 08:26:03 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] is still present in the system and should be removed.
Fri Jul  7 08:26:03 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] is still present in the system and should be removed.
Fri Jul  7 08:26:03 BNT [filer02: raid.fdr.reminder:warning]: Failed Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] is still present in the system and should be removed.
Fri Jul  7 08:26:03 BNT [filer02: shelf.config.spha:info]: System is using single path HA attached storage only.
Fri Jul  7 08:26:03 BNT [filer02: mgr.boot.reason_ok:notice]: System rebooted.
Fri Jul  7 08:26:04 BNT [filer02: iscsi.notice:notice]: ISCSI: New session from initiator iqn.1991-05.com.microsoft:pcb-722-ad1.pcbl722dom.local at IP addr 10.72.4.1
Fri Jul  7 08:26:30 BNT [filer02: raid.rg.spares.low:warning]: /aggr1/plex0/rg0
Fri Jul  7 08:26:33 BNT [filer02: nbt.nbns.registrationComplete:info]: NBT: All CIFS name registrations have completed for the local server.
Fri Jul  7 08:26:39 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SHUTDOWN PENDING (degraded mode))
Fri Jul  7 08:26:47 BNT last message repeated 2 times
Fri Jul  7 08:26:47 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201707030748.0) was not posted for host (0), it was dropped from queue due to lack of space
Fri Jul  7 08:26:55 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (REBOOT)
Fri Jul  7 08:26:55 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201707030748.1) was not posted for host (0), it was dropped from queue due to lack of space
Fri Jul  7 08:27:03 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SPARES_LOW)
Fri Jul  7 08:27:03 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201707030748.2) was not posted for host (0), it was dropped from queue due to lack of space
Fri Jul  7 08:27:11 BNT [filer02: monitor.globalStatus.nonCritical:warning]: Disk on adapter 0a, shelf 2, bay 8, failed. Disk on adapter 0a, shelf 2, bay 3, failed. Disk on adapter 0a, shelf 2, bay 1, failed.  There are not enough spare disks. /vol/vol1 is full (using or reserving 100% of space and 0% of inodes, using 92% of reserve).
Fri Jul  7 08:27:16 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (PERFORMANCE SNAPSHOT)
Fri Jul  7 08:27:16 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201707030749.0) was not posted for host (0), it was dropped from queue due to lack of space
Fri Jul  7 08:31:40 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (CLUSTER ERROR: DISK/SHELF COUNT MISMATCH)
Fri Jul  7 08:31:48 BNT [filer02: cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 0a.33 (016F3AB5:B475F4EC:10505E83:02202755:00000000:00000000:00000000:00000000:00000000:00000000) has recently changed or the node (filer02) is missing the disk.
Fri Jul  7 08:31:48 BNT [filer02: cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 0a.35 (9115483F:9DD43BC4:27172773:99F7D8B4:00000000:00000000:00000000:00000000:00000000:00000000) has recently changed or the node (filer02) is missing the disk.
Fri Jul  7 08:31:48 BNT [filer02: cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 0a.40 (072AEB65:C662A9CF:E58B9917:9BFA787B:00000000:00000000:00000000:00000000:00000000:00000000) has recently changed or the node (filer02) is missing the disk.
Fri Jul  7 08:31:48 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (CLUSTER ERROR: DISK/SHELF COUNT MISMATCH)
Fri Jul  7 08:31:56 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (CLUSTER ERROR: DISK/SHELF COUNT MISMATCH)
Fri Jul  7 08:31:56 BNT [filer02: asup.post.drop.enqueue:info]: Autosupport message (201707030753.0) was not posted for host (0), it was dropped from queue due to lack of space
Fri Jul  7 08:35:56 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jul  7 08:39:54 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jul  7 08:44:07 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jul  7 08:48:00 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jul  7 08:52:06 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jul  7 08:55:44 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jul  7 08:55:44 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201707030800.0) for host (0) was not posted to NetApp. The system will drop the message
Fri Jul  7 09:00:00 BNT [filer02: kern.uptime.filer:info]:   9:00am up 33 mins, 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 67305 iSCSI ops
Fri Jul  7 09:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 24 hours.
Fri Jul  7 09:32:09 BNT [filer02: cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 0a.33 (016F3AB5:B475F4EC:10505E83:02202755:00000000:00000000:00000000:00000000:00000000:00000000) has recently changed or the node (filer02) is missing the disk.
Fri Jul  7 09:32:09 BNT [filer02: cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 0a.35 (9115483F:9DD43BC4:27172773:99F7D8B4:00000000:00000000:00000000:00000000:00000000:00000000) has recently changed or the node (filer02) is missing the disk.
Fri Jul  7 09:32:09 BNT [filer02: cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 0a.40 (072AEB65:C662A9CF:E58B9917:9BFA787B:00000000:00000000:00000000:00000000:00000000:00000000) has recently changed or the node (filer02) is missing the disk.
Fri Jul  7 10:00:00 BNT [filer02: kern.uptime.filer:info]:  10:00am up  1:33 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 790538 iSCSI ops
Fri Jul  7 10:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 23 hours.
Fri Jul  7 11:00:00 BNT [filer02: kern.uptime.filer:info]:  11:00am up  2:33 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 1056205 iSCSI ops
Fri Jul  7 11:00:01 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 22 hours.
Fri Jul  7 12:00:00 BNT [filer02: kern.uptime.filer:info]:  12:00pm up  3:33 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 1057228 iSCSI ops
Fri Jul  7 12:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 21 hours.
Fri Jul  7 12:00:31 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Fri Jul  7 12:00:57 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jul  7 12:04:59 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jul  7 12:09:06 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jul  7 12:12:59 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jul  7 12:16:37 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jul  7 12:33:04 BNT last message repeated 4 times
Fri Jul  7 12:37:12 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jul  7 12:54:07 BNT last message repeated 4 times
Fri Jul  7 12:58:28 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jul  7 12:58:28 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201707031200.0) for host (0) was not posted to NetApp. The system will drop the message
Fri Jul  7 13:00:00 BNT [filer02: kern.uptime.filer:info]:   1:00pm up  4:33 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 1911162 iSCSI ops
Fri Jul  7 13:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 20 hours.
Fri Jul  7 14:00:00 BNT [filer02: kern.uptime.filer:info]:   2:00pm up  5:33 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 3509330 iSCSI ops
Fri Jul  7 14:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 19 hours.
Fri Jul  7 15:00:00 BNT [filer02: kern.uptime.filer:info]:   3:00pm up  6:33 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 5093943 iSCSI ops
Fri Jul  7 15:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 18 hours.
Fri Jul  7 16:00:00 BNT [filer02: kern.uptime.filer:info]:   4:00pm up  7:33 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 7184081 iSCSI ops
Fri Jul  7 16:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 17 hours.
Fri Jul  7 16:00:16 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Fri Jul  7 16:00:38 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jul  7 16:04:58 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jul  7 16:09:03 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jul  7 16:12:35 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jul  7 16:16:20 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jul  7 16:31:36 BNT last message repeated 4 times
Fri Jul  7 16:39:37 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jul  7 16:56:02 BNT [filer02: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201707031600.0) for host (0) was not posted to NetApp. The system will drop the message
Fri Jul  7 17:00:00 BNT [filer02: kern.uptime.filer:info]:   5:00pm up  8:33 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 8830999 iSCSI ops
Fri Jul  7 17:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 16 hours.
Fri Jul  7 18:00:00 BNT [filer02: kern.uptime.filer:info]:   6:00pm up  9:33 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 10596085 iSCSI ops
Fri Jul  7 18:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 15 hours.
Fri Jul  7 19:00:00 BNT [filer02: kern.uptime.filer:info]:   7:00pm up 10:33 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 12396693 iSCSI ops
Fri Jul  7 19:00:00 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 14 hours.
Fri Jul  7 20:00:00 BNT [filer02: kern.uptime.filer:info]:   8:00pm up 11:33 0 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 FCP ops, 14137442 iSCSI ops
Fri Jul  7 20:00:01 BNT [filer02: monitor.shutdown.brokenDisk.pending:warning]: the parity disk and a data disk in RAID group /aggr1/plex0/rg0 are broken. Halting system in 13 hours.
Fri Jul  7 20:00:15 BNT [filer02: wafl.snap.create.skip.reason:notice]: vol1: skipping creation of hourly.0 snapshot (Snapshot operation failed: No space left on device.)
Fri Jul  7 20:09:09 BNT [filer02: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jul  7 20:42:36 BNT [filer02: disk.senseError:error]: Disk 0a.32: op 0x28:01b99bd0:0090 sector 0 SCSI:aborted command -  (b 88 0 0)
Fri Jul  7 20:42:36 BNT [filer02: raid.config.filesystem.disk.failed:error]: File system Disk /aggr1/plex0/rg0/0a.32 Shelf 2 Bay 0 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V103L] failed.
Fri Jul  7 20:42:36 BNT [filer02: raid.vol.failed:CRITICAL]: Aggregate aggr1: Failed due to multi-disk error
Fri Jul  7 20:42:36 BNT [filer02: disk.failmsg:error]: Disk 0a.32 (N00V103L): message received, maintenance center recommended.
Fri Jul  7 20:42:36 BNT [filer02: raid.disk.unload.done:info]: Unload of Disk 0a.32 Shelf 2 Bay 0 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V103L] has completed successfully
Fri Jul  7 20:42:46 BNT [filer02/filer01: iscsi.service.startup:info]: iSCSI service startup
Fri Jul  7 20:42:47 BNT [filer02/filer01: raid.fdr.failed.ok:info]: Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] successfully deleted from spare pool
Fri Jul  7 20:42:47 BNT [filer02/filer01: raid.fdr.failed.ok:info]: Disk 0a.32 Shelf 2 Bay 0 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V103L] successfully deleted from spare pool
Fri Jul  7 20:42:47 BNT [filer02/filer01: raid.fdr.failed.ok:info]: Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] successfully deleted from spare pool
Fri Jul  7 20:42:47 BNT [filer02/filer01: raid.fdr.failed.ok:info]: Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] successfully deleted from spare pool
Fri Jul  7 20:42:47 BNT [filer02/filer01: cf_takeover:info]: relog syslog Fri Jul  7 20:42:36 BNT [filer02: raid.disk.unload.done:info]: Unload of Disk 0a.32 Shelf 2 Bay 0 [NETAPP   X269_HJUPI
Fri Jul  7 20:42:47 BNT [filer02/filer01: cf_takeover:info]: relog syslog Fri Jul  7 20:42:36 BNT [filer02: disk.partner.msgStatus:debug]: The partner/CDO has returned status 0 for msg sent fo
Fri Jul  7 20:42:47 BNT [filer02/filer01: iscsi.notice:notice]: ISCSI: New session from initiator iqn.1991-05.com.microsoft:pcb-722-ad1.pcbl722dom.local at IP addr 10.72.4.1
Fri Jul  7 20:43:00 BNT [filer02/filer01: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jul  7 20:43:00 BNT [filer02/filer01: asup.post.drop.enqueue:info]: Autosupport message (201707032000.0) was not posted for host (1), it was dropped from queue due to lack of space
Fri Jul  7 20:43:00 BNT [filer02/filer01: monitor.globalStatus.critical:CRITICAL]: filer01 has taken over this node. Disk on adapter 0a, shelf 2, bay 3, failed. Disk on adapter 0a, shelf 2, bay 0, failed. Disk on adapter 0a, shelf 2, bay 1, failed. Disk on adapter 0a, shelf 2, bay 8, failed.  There are not enough spare disks. 
Fri Jul  7 20:43:11 BNT [filer02/filer01: nbt.nbns.registrationComplete:info]: NBT: All CIFS name registrations have completed for the partner server.
Fri Jul  7 20:46:57 BNT [filer02/filer01: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jul  7 20:51:06 BNT [filer02/filer01: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jul  7 20:54:46 BNT [filer02/filer01: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jul  7 20:58:35 BNT [filer02/filer01: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jul  7 21:02:47 BNT [filer02/filer01: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jul  7 21:21:49 BNT last message repeated 5 times
Fri Jul  7 21:25:39 BNT [filer02/filer01: asup.post.disconnected:info]: Autosupport was disconnected during transmission (SCHEDULED SNAPSHOT CREATE FAILED)
Fri Jul  7 21:37:53 BNT last message repeated 3 times
Fri Jul  7 21:37:53 BNT [filer02/filer01: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201707040000.0) for host (1) was not posted to NetApp. The system will drop the message
Sat Jul  8 08:00:00 BNT [filer02/filer01: raid.fdr.reminder:warning]: Failed Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] is still present in the system and should be removed.
Sat Jul  8 08:00:00 BNT [filer02/filer01: raid.fdr.reminder:warning]: Failed Disk 0a.32 Shelf 2 Bay 0 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V103L] is still present in the system and should be removed.
Sat Jul  8 08:00:00 BNT [filer02/filer01: raid.fdr.reminder:warning]: Failed Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] is still present in the system and should be removed.
Sat Jul  8 08:00:00 BNT [filer02/filer01: raid.fdr.reminder:warning]: Failed Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] is still present in the system and should be removed.
Sat Jul  8 10:02:59 BNT [filer02/filer01: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201707052000.0) for host (1) was not posted to NetApp. The system will drop the message
Sat Jul  8 10:02:59 BNT [filer02/filer01: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201707060000.0) for host (1) was not posted to NetApp. The system will drop the message
Sat Jul  8 10:02:59 BNT [filer02/filer01: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201707060755.0) for host (1) was not posted to NetApp. The system will drop the message
Sat Jul  8 10:02:59 BNT [filer02/filer01: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201707060755.1) for host (1) was not posted to NetApp. The system will drop the message
Sat Jul  8 10:02:59 BNT [filer02/filer01: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201707060755.2) for host (1) was not posted to NetApp. The system will drop the message
Sat Jul  8 10:02:59 BNT [filer02/filer01: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201707060756.0) for host (1) was not posted to NetApp. The system will drop the message
Sat Jul  8 10:02:59 BNT [filer02/filer01: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201707060756.1) for host (1) was not posted to NetApp. The system will drop the message
Sat Jul  8 10:02:59 BNT [filer02/filer01: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201707060800.0) for host (1) was not posted to NetApp. The system will drop the message
Sat Jul  8 10:02:59 BNT [filer02/filer01: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201707061200.0) for host (1) was not posted to NetApp. The system will drop the message
Sat Jul  8 10:02:59 BNT [filer02/filer01: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201707061600.0) for host (1) was not posted to NetApp. The system will drop the message
Sat Jul  8 10:02:59 BNT [filer02/filer01: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201707062000.0) for host (1) was not posted to NetApp. The system will drop the message
Sat Jul  8 10:02:59 BNT [filer02/filer01: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201707070000.0) for host (1) was not posted to NetApp. The system will drop the message
Sat Jul  8 10:02:59 BNT [filer02/filer01: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201707070826.0) for host (1) was not posted to NetApp. The system will drop the message
Sat Jul  8 10:02:59 BNT [filer02/filer01: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201707070826.1) for host (1) was not posted to NetApp. The system will drop the message
Sat Jul  8 10:02:59 BNT [filer02/filer01: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201707070826.2) for host (1) was not posted to NetApp. The system will drop the message
Sat Jul  8 10:02:59 BNT [filer02/filer01: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201707070826.3) for host (1) was not posted to NetApp. The system will drop the message
Sat Jul  8 10:02:59 BNT [filer02/filer01: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201707070827.0) for host (1) was not posted to NetApp. The system will drop the message
Sat Jul  8 10:02:59 BNT [filer02/filer01: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201707070831.0) for host (1) was not posted to NetApp. The system will drop the message
Sat Jul  8 10:02:59 BNT [filer02/filer01: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201707071200.0) for host (1) was not posted to NetApp. The system will drop the message
Sat Jul  8 10:02:59 BNT [filer02/filer01: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201707071600.0) for host (1) was not posted to NetApp. The system will drop the message
Sat Jul  8 10:02:59 BNT [filer02/filer01: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201707072000.0) for host (1) was not posted to NetApp. The system will drop the message
Sat Jul  8 10:02:59 BNT [filer02/filer01: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201707072042.0) for host (1) was not posted to NetApp. The system will drop the message
Sat Jul  8 10:03:00 BNT [filer02/filer01: iscsi.service.startup:info]: iSCSI service startup
Sat Jul  8 10:03:00 BNT [filer02/filer01: raid.fdr.failed.ok:info]: Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] successfully deleted from spare pool
Sat Jul  8 10:03:00 BNT [filer02/filer01: raid.fdr.failed.ok:info]: Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] successfully deleted from spare pool
Sat Jul  8 10:03:00 BNT [filer02/filer01: raid.fdr.failed.ok:info]: Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] successfully deleted from spare pool
Sat Jul  8 10:03:00 BNT [filer02/filer01: raid.fdr.failed.ok:info]: Disk 0a.32 Shelf 2 Bay 0 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V103L] successfully deleted from spare pool
Sat Jul  8 10:03:23 BNT [filer02/filer01: asup.post.disconnected:info]: Autosupport was disconnected during transmission (REBOOT (CLUSTER TAKEOVER))
Sat Jul  8 10:03:34 BNT [filer02/filer01: nbt.nbns.registrationComplete:info]: NBT: All CIFS name registrations have completed for the partner server.
Sat Jul  8 10:04:00 BNT [filer02/filer01: monitor.globalStatus.critical:CRITICAL]: filer01 has taken over this node. Disk on adapter 0a, shelf 2, bay 8, failed. Disk on adapter 0a, shelf 2, bay 3, failed. Disk on adapter 0a, shelf 2, bay 1, failed. Disk on adapter 0a, shelf 2, bay 0, failed.  There are not enough spare disks. 
Sat Jul  8 10:08:11 BNT [filer02/filer01: iscsi.service.startup:info]: iSCSI service startup
Sat Jul  8 10:08:11 BNT [filer02/filer01: raid.fdr.failed.ok:info]: Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] successfully deleted from spare pool
Sat Jul  8 10:08:11 BNT [filer02/filer01: raid.fdr.failed.ok:info]: Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] successfully deleted from spare pool
Sat Jul  8 10:08:11 BNT [filer02/filer01: raid.fdr.failed.ok:info]: Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] successfully deleted from spare pool
Sat Jul  8 10:08:11 BNT [filer02/filer01: raid.fdr.failed.ok:info]: Disk 0a.32 Shelf 2 Bay 0 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V103L] successfully deleted from spare pool
Sat Jul  8 10:08:32 BNT [filer02/filer01: asup.post.disconnected:info]: Autosupport was disconnected during transmission (REBOOT (CLUSTER TAKEOVER))
Sat Jul  8 10:08:44 BNT [filer02/filer01: nbt.nbns.registrationComplete:info]: NBT: All CIFS name registrations have completed for the partner server.
Sat Jul  8 10:09:00 BNT [filer02/filer01: monitor.globalStatus.critical:CRITICAL]: filer01 has taken over this node. Disk on adapter 0a, shelf 2, bay 8, failed. Disk on adapter 0a, shelf 2, bay 3, failed. Disk on adapter 0a, shelf 2, bay 1, failed. Disk on adapter 0a, shelf 2, bay 0, failed.  There are not enough spare disks. 
Sat Jul  8 10:12:45 BNT [filer02/filer01: asup.post.disconnected:info]: Autosupport was disconnected during transmission (REBOOT (CLUSTER TAKEOVER))
Sat Jul  8 10:16:23 BNT [filer02/filer01: asup.post.disconnected:info]: Autosupport was disconnected during transmission (REBOOT (CLUSTER TAKEOVER))
Sat Jul  8 10:20:47 BNT [filer02/filer01: asup.post.disconnected:info]: Autosupport was disconnected during transmission (REBOOT (CLUSTER TAKEOVER))
Sat Jul  8 10:24:41 BNT [filer02/filer01: asup.post.disconnected:info]: Autosupport was disconnected during transmission (REBOOT (CLUSTER TAKEOVER))
Sat Jul  8 10:28:35 BNT [filer02/filer01: asup.post.disconnected:info]: Autosupport was disconnected during transmission (REBOOT (CLUSTER TAKEOVER))
Sat Jul  8 10:36:55 BNT last message repeated 2 times
Sat Jul  8 10:41:47 BNT [filer02/filer01: asup.post.host:info]: Autosupport (REBOOT (CLUSTER TAKEOVER)) cannot connect to url support.netapp.com/asupprod/post/1.0/postAsup (Could not find hostname 'support.netapp.com', hostname lookup resolution error: Host name lookup failure)
Sat Jul  8 10:41:52 BNT [filer02/filer01: iscsi.notice:notice]: ISCSI: New session from initiator iqn.1991-05.com.microsoft:pcb-722-ad1.pcbl722dom.local at IP addr 10.72.4.1
Sat Jul  8 10:41:55 BNT last message repeated 2 times
Sat Jul  8 10:45:59 BNT [filer02/filer01: asup.post.disconnected:info]: Autosupport was disconnected during transmission (REBOOT (CLUSTER TAKEOVER))
Sat Jul  8 10:50:05 BNT [filer02/filer01: asup.post.disconnected:info]: Autosupport was disconnected during transmission (REBOOT (CLUSTER TAKEOVER))
Sat Jul  8 10:54:40 BNT [filer02/filer01: asup.post.host:info]: Autosupport (REBOOT (CLUSTER TAKEOVER)) cannot connect to url support.netapp.com/asupprod/post/1.0/postAsup (Could not find hostname 'support.netapp.com', hostname lookup resolution error: Host name lookup failure)
Sat Jul  8 10:54:46 BNT [filer02/filer01: iscsi.notice:notice]: ISCSI: New session from initiator iqn.1991-05.com.microsoft:pcb-722-ad1.pcbl722dom.local at IP addr 10.72.4.1
Sat Jul  8 10:54:47 BNT [filer02/filer01: iscsi.notice:notice]: ISCSI: New session from initiator iqn.1991-05.com.microsoft:pcb-722-ad1.pcbl722dom.local at IP addr 10.72.4.1
Sat Jul  8 10:58:25 BNT [filer02/filer01: asup.post.disconnected:info]: Autosupport was disconnected during transmission (REBOOT (CLUSTER TAKEOVER))
Sat Jul  8 11:02:23 BNT [filer02/filer01: asup.post.disconnected:info]: Autosupport was disconnected during transmission (REBOOT (CLUSTER TAKEOVER))
Sat Jul  8 11:06:24 BNT [filer02/filer01: iscsi.service.startup:info]: iSCSI service startup
Sat Jul  8 11:06:24 BNT [filer02/filer01: raid.fdr.failed.ok:info]: Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] successfully deleted from spare pool
Sat Jul  8 11:06:24 BNT [filer02/filer01: raid.fdr.failed.ok:info]: Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] successfully deleted from spare pool
Sat Jul  8 11:06:24 BNT [filer02/filer01: raid.fdr.failed.ok:info]: Disk 0a.32 Shelf 2 Bay 0 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V103L] successfully deleted from spare pool
Sat Jul  8 11:06:24 BNT [filer02/filer01: raid.fdr.failed.ok:info]: Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] successfully deleted from spare pool
Sat Jul  8 11:06:47 BNT [filer02/filer01: asup.post.disconnected:info]: Autosupport was disconnected during transmission (REBOOT (CLUSTER TAKEOVER))
Sat Jul  8 11:06:56 BNT [filer02/filer01: nbt.nbns.registrationComplete:info]: NBT: All CIFS name registrations have completed for the partner server.
Sat Jul  8 11:07:00 BNT [filer02/filer01: monitor.globalStatus.critical:CRITICAL]: filer01 has taken over this node. Disk on adapter 0a, shelf 2, bay 8, failed. Disk on adapter 0a, shelf 2, bay 3, failed. Disk on adapter 0a, shelf 2, bay 0, failed. Disk on adapter 0a, shelf 2, bay 1, failed.  There are not enough spare disks. 
Sat Jul  8 11:11:58 BNT [filer02/filer01: asup.post.host:info]: Autosupport (REBOOT (CLUSTER TAKEOVER)) cannot connect to url support.netapp.com/asupprod/post/1.0/postAsup (Could not find hostname 'support.netapp.com', hostname lookup resolution error: Host name lookup failure)
Sat Jul  8 11:13:02 BNT [filer02/filer01: iscsi.notice:notice]: ISCSI: New session from initiator iqn.1991-05.com.microsoft:pcb-722-ad1.pcbl722dom.local at IP addr 10.72.4.1
Sat Jul  8 11:16:12 BNT [filer02/filer01: asup.post.disconnected:info]: Autosupport was disconnected during transmission (REBOOT (CLUSTER TAKEOVER))
Sat Jul  8 11:18:21 BNT [filer02/filer01: iscsi.notice:notice]: ISCSI: New session from initiator iqn.1991-05.com.microsoft:pcb-722-ad1.pcbl722dom.local at IP addr 10.72.4.1
Sat Jul  8 11:18:28 BNT [filer02/filer01: iscsi.notice:notice]: ISCSI: New session from initiator iqn.1991-05.com.microsoft:pcb-722-ad1.pcbl722dom.local at IP addr 10.72.4.1
Sat Jul  8 11:20:17 BNT [filer02/filer01: asup.post.disconnected:info]: Autosupport was disconnected during transmission (REBOOT (CLUSTER TAKEOVER))
Sat Jul  8 11:24:38 BNT [filer02/filer01: asup.post.disconnected:info]: Autosupport was disconnected during transmission (REBOOT (CLUSTER TAKEOVER))
Sat Jul  8 11:29:05 BNT [filer02/filer01: asup.post.disconnected:info]: Autosupport was disconnected during transmission (REBOOT (CLUSTER TAKEOVER))
Sat Jul  8 13:01:43 BNT [filer02/filer01: iscsi.service.startup:info]: iSCSI service startup
Sat Jul  8 13:01:43 BNT [filer02/filer01: raid.fdr.failed.ok:info]: Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] successfully deleted from spare pool
Sat Jul  8 13:01:43 BNT [filer02/filer01: raid.fdr.failed.ok:info]: Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] successfully deleted from spare pool
Sat Jul  8 13:01:43 BNT [filer02/filer01: raid.fdr.failed.ok:info]: Disk 0a.32 Shelf 2 Bay 0 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V103L] successfully deleted from spare pool
Sat Jul  8 13:01:43 BNT [filer02/filer01: raid.fdr.failed.ok:info]: Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] successfully deleted from spare pool
Sat Jul  8 13:01:48 BNT [filer02/filer01: iscsi.notice:notice]: ISCSI: New session from initiator iqn.1991-05.com.microsoft:pcb-722-ad1.pcbl722dom.local at IP addr 10.72.4.1
Sat Jul  8 13:02:00 BNT [filer02/filer01: monitor.globalStatus.critical:CRITICAL]: filer01 has taken over this node. Disk on adapter 0a, shelf 2, bay 8, failed. Disk on adapter 0a, shelf 2, bay 1, failed. Disk on adapter 0a, shelf 2, bay 0, failed. Disk on adapter 0a, shelf 2, bay 3, failed.  There are not enough spare disks. 
Sat Jul  8 13:02:04 BNT [filer02/filer01: asup.post.disconnected:info]: Autosupport was disconnected during transmission (REBOOT (CLUSTER TAKEOVER))
Sat Jul  8 13:02:12 BNT [filer02/filer01: nbt.nbns.registrationComplete:info]: NBT: All CIFS name registrations have completed for the partner server.
Sat Jul  8 13:06:23 BNT [filer02/filer01: asup.post.disconnected:info]: Autosupport was disconnected during transmission (REBOOT (CLUSTER TAKEOVER))
Sat Jul  8 13:10:10 BNT [filer02/filer01: asup.post.disconnected:info]: Autosupport was disconnected during transmission (REBOOT (CLUSTER TAKEOVER))
Sat Jul  8 13:13:41 BNT [filer02/filer01: asup.post.disconnected:info]: Autosupport was disconnected during transmission (REBOOT (CLUSTER TAKEOVER))
Sat Jul  8 13:18:05 BNT [filer02/filer01: asup.post.disconnected:info]: Autosupport was disconnected during transmission (REBOOT (CLUSTER TAKEOVER))
Sat Jul  8 13:22:19 BNT [filer02/filer01: asup.post.disconnected:info]: Autosupport was disconnected during transmission (REBOOT (CLUSTER TAKEOVER))
Sat Jul  8 13:42:18 BNT last message repeated 5 times
Sat Jul  8 13:46:37 BNT [filer02/filer01: asup.post.disconnected:info]: Autosupport was disconnected during transmission (REBOOT (CLUSTER TAKEOVER))
Sat Jul  8 13:59:24 BNT last message repeated 3 times
Sat Jul  8 13:59:24 BNT [filer02/filer01: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201707081301.0) for host (1) was not posted to NetApp. The system will drop the message
Sun Jul  9 01:00:01 BNT [filer02/filer01: raid.rg.scrub.start:notice]: partner:/aggr0/plex0/rg0: starting scrub
Sun Jul  9 01:26:08 BNT [filer02/filer01: asup.post.disconnected:info]: Autosupport was disconnected during transmission (WEEKLY_LOG)
Sun Jul  9 01:29:40 BNT [filer02/filer01: asup.post.disconnected:info]: Autosupport was disconnected during transmission (WEEKLY_LOG)
Sun Jul  9 01:34:07 BNT [filer02/filer01: asup.post.disconnected:info]: Autosupport was disconnected during transmission (WEEKLY_LOG)
Sun Jul  9 01:38:07 BNT [filer02/filer01: asup.post.disconnected:info]: Autosupport was disconnected during transmission (WEEKLY_LOG)
Sun Jul  9 01:42:30 BNT [filer02/filer01: asup.post.disconnected:info]: Autosupport was disconnected during transmission (WEEKLY_LOG)
Sun Jul  9 01:59:17 BNT last message repeated 4 times
Sun Jul  9 02:03:07 BNT [filer02/filer01: asup.post.disconnected:info]: Autosupport was disconnected during transmission (WEEKLY_LOG)
Sun Jul  9 02:23:13 BNT last message repeated 5 times
Sun Jul  9 02:23:13 BNT [filer02/filer01: asup.post.drop:error]: Autosupport message (/etc/log/autosupport/201707090000.0) for host (1) was not posted to NetApp. The system will drop the message
Sun Jul  9 13:00:01 BNT [filer02/filer01: raid.fdr.reminder:warning]: Failed Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] is still present in the system and should be removed.
Sun Jul  9 13:00:01 BNT [filer02/filer01: raid.fdr.reminder:warning]: Failed Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] is still present in the system and should be removed.
Sun Jul  9 13:00:01 BNT [filer02/filer01: raid.fdr.reminder:warning]: Failed Disk 0a.32 Shelf 2 Bay 0 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V103L] is still present in the system and should be removed.
Sun Jul  9 13:00:01 BNT [filer02/filer01: raid.fdr.reminder:warning]: Failed Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] is still present in the system and should be removed.
Mon Jul 10 13:00:01 BNT [filer02/filer01: raid.fdr.reminder:warning]: Failed Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] is still present in the system and should be removed.
Mon Jul 10 13:00:01 BNT [filer02/filer01: raid.fdr.reminder:warning]: Failed Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] is still present in the system and should be removed.
Mon Jul 10 13:00:01 BNT [filer02/filer01: raid.fdr.reminder:warning]: Failed Disk 0a.32 Shelf 2 Bay 0 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V103L] is still present in the system and should be removed.
Mon Jul 10 13:00:01 BNT [filer02/filer01: raid.fdr.reminder:warning]: Failed Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] is still present in the system and should be removed.
Tue Jul 11 12:00:30 BNT [filer02/filer01: HTTPPool01:warning]: HTTP Authentication from 10.72.6.30 using NTLM failed
Tue Jul 11 13:00:01 BNT [filer02/filer01: raid.fdr.reminder:warning]: Failed Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] is still present in the system and should be removed.
Tue Jul 11 13:00:01 BNT [filer02/filer01: raid.fdr.reminder:warning]: Failed Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] is still present in the system and should be removed.
Tue Jul 11 13:00:01 BNT [filer02/filer01: raid.fdr.reminder:warning]: Failed Disk 0a.32 Shelf 2 Bay 0 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V103L] is still present in the system and should be removed.
Tue Jul 11 13:00:01 BNT [filer02/filer01: raid.fdr.reminder:warning]: Failed Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] is still present in the system and should be removed.
Wed Jul 12 07:50:08 BNT [filer02/filer01: HTTPPool00:warning]: HTTP Authentication from 10.72.5.52 using NTLM failed
Wed Jul 12 08:02:40 BNT [filer02/filer01: HTTPPool02:warning]: HTTP Authentication from 10.72.6.2 using NTLM failed
Wed Jul 12 13:00:00 BNT [filer02/filer01: raid.fdr.reminder:warning]: Failed Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] is still present in the system and should be removed.
Wed Jul 12 13:00:00 BNT [filer02/filer01: raid.fdr.reminder:warning]: Failed Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] is still present in the system and should be removed.
Wed Jul 12 13:00:00 BNT [filer02/filer01: raid.fdr.reminder:warning]: Failed Disk 0a.32 Shelf 2 Bay 0 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V103L] is still present in the system and should be removed.
Wed Jul 12 13:00:00 BNT [filer02/filer01: raid.fdr.reminder:warning]: Failed Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] is still present in the system and should be removed.
Thu Jul 13 09:00:35 BNT [filer02/filer01: HTTPPool03:warning]: HTTP Authentication from 10.72.5.52 using NTLM failed
Thu Jul 13 13:00:00 BNT [filer02/filer01: raid.fdr.reminder:warning]: Failed Disk 0a.40 Shelf 2 Bay 8 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V0W8L] is still present in the system and should be removed.
Thu Jul 13 13:00:00 BNT [filer02/filer01: raid.fdr.reminder:warning]: Failed Disk 0a.33 Shelf 2 Bay 1 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00W75PL] is still present in the system and should be removed.
Thu Jul 13 13:00:00 BNT [filer02/filer01: raid.fdr.reminder:warning]: Failed Disk 0a.32 Shelf 2 Bay 0 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00V103L] is still present in the system and should be removed.
Thu Jul 13 13:00:00 BNT [filer02/filer01: raid.fdr.reminder:warning]: Failed Disk 0a.35 Shelf 2 Bay 3 [NETAPP   X269_HJUPI01TSSX NA01] S/N [N00WLAUL] is still present in the system and should be removed.
Thu Jul 13 15:44:22 BNT [filer02/filer01: HTTPPool04:warning]: HTTP Authentication from 10.72.4.1 using NTLM failed
Thu Jul 13 17:19:19 BNT [filer02/filer01: Java_Thread:error]: netapp.fv.volumes.manage.DiskReport: FilerView : The system could not fetch the disk scrub information 
Thu Jul 13 17:24:19 BNT [filer02/filer01: Java_Thread:error]: netapp.fv.volumes.manage.DiskReport: FilerView : The system could not fetch the disk scrub information 
Thu Jul 13 17:24:57 BNT [filer02/filer01: Java_Thread:error]: netapp.fv.volumes.manage.DiskReport: FilerView : The system could not fetch the disk scrub information 
Thu Jul 13 17:31:54 BNT [filer02/filer01: Java_Thread:error]: netapp.fv.volumes.manage.DiskReport: FilerView : The system could not fetch the disk scrub information 
Thu Jul 13 18:32:45 BNT [filer02/filer01: HTTPPool02:warning]: HTTP Authentication from 10.72.5.22 using NTLM failed