Thank you @SpindleNinja ... Our support ended last month and management doesn't want to pay for it even though I presented a case for them to -- I hope they would listen because i'm already loosing it and letting them know today how important it is to our architecture.
Anyways pardon my frustratration! pwwwwwwwwwww... breath , breath ... thanks, I needed to vent!
I got logs from the sp: I highlighted the most recent ones
Record 2414: Wed Jun 24 02:56:34 2020 [Heartbeat.notice]: Heartbeat start: Set SP time. Old time: Wed Jun 24 02:56:34 2020. New time: Wed Jun 24 02:56:35 2020.
Record 2415: Fri Jun 26 00:47:34 2020 [Heartbeat.notice]: Heartbeat time adjusted: Set SP time. Old time: Fri Jun 26 00:47:32 2020. New time: Fri Jun 26 00:47:34 2020.
Record 2416: Sat Jun 27 19:30:34 2020 [Heartbeat.notice]: Heartbeat time adjusted: Set SP time. Old time: Sat Jun 27 19:30:32 2020. New time: Sat Jun 27 19:30:34 2020.
Record 2417: Mon Jun 29 13:57:34 2020 [Heartbeat.notice]: Heartbeat time adjusted: Set SP time. Old time: Mon Jun 29 13:57:32 2020. New time: Mon Jun 29 13:57:34 2020.
Record 2418: Wed Jul 1 07:47:34 2020 [Heartbeat.notice]: Heartbeat time adjusted: Set SP time. Old time: Wed Jul 1 07:47:32 2020. New time: Wed Jul 1 07:47:34 2020.
Record 2419: Thu Jul 2 23:53:35 2020 [Heartbeat.notice]: Heartbeat time adjusted: Set SP time. Old time: Thu Jul 2 23:53:33 2020. New time: Thu Jul 2 23:53:35 2020.
Record 2420: Sat Jul 4 17:59:35 2020 [Heartbeat.notice]: Heartbeat time adjusted: Set SP time. Old time: Sat Jul 4 17:59:33 2020. New time: Sat Jul 4 17:59:35 2020.
Record 2421: Mon Jul 6 10:59:35 2020 [Heartbeat.notice]: Heartbeat time adjusted: Set SP time. Old time: Mon Jul 6 10:59:33 2020. New time: Mon Jul 6 10:59:35 2020.
Record 2422: Wed Jul 8 04:38:35 2020 [Heartbeat.notice]: Heartbeat time adjusted: Set SP time. Old time: Wed Jul 8 04:38:33 2020. New time: Wed Jul 8 04:38:35 2020.
Record 2423: Thu Jul 9 21:56:35 2020 [Heartbeat.notice]: Heartbeat time adjusted: Set SP time. Old time: Thu Jul 9 21:56:33 2020. New time: Thu Jul 9 21:56:35 2020.
Record 2424: Sat Jul 11 15:04:35 2020 [Heartbeat.notice]: Heartbeat time adjusted: Set SP time. Old time: Sat Jul 11 15:04:33 2020. New time: Sat Jul 11 15:04:35 2020.
Record 2425: Mon Jul 13 07:37:35 2020 [Heartbeat.notice]: Heartbeat time adjusted: Set SP time. Old time: Mon Jul 13 07:37:33 2020. New time: Mon Jul 13 07:37:35 2020.
Record 2426: Wed Jul 15 00:52:35 2020 [Heartbeat.notice]: Heartbeat time adjusted: Set SP time. Old time: Wed Jul 15 00:52:33 2020. New time: Wed Jul 15 00:52:35 2020.
Record 2427: Thu Jul 16 17:10:35 2020 [Heartbeat.notice]: Heartbeat time adjusted: Set SP time. Old time: Thu Jul 16 17:10:33 2020. New time: Thu Jul 16 17:10:35 2020.
Record 2428: Sat Jul 18 09:51:35 2020 [Heartbeat.notice]: Heartbeat time adjusted: Set SP time. Old time: Sat Jul 18 09:51:33 2020. New time: Sat Jul 18 09:51:35 2020.
Record 2429: Mon Jul 20 01:50:35 2020 [Heartbeat.notice]: Heartbeat time adjusted: Set SP time. Old time: Mon Jul 20 01:50:33 2020. New time: Mon Jul 20 01:50:35 2020.
Record 2430: Tue Jul 21 18:39:35 2020 [Heartbeat.notice]: Heartbeat time adjusted: Set SP time. Old time: Tue Jul 21 18:39:33 2020. New time: Tue Jul 21 18:39:35 2020.
Record 2431: Thu Jul 23 12:06:35 2020 [Heartbeat.notice]: Heartbeat time adjusted: Set SP time. Old time: Thu Jul 23 12:06:33 2020. New time: Thu Jul 23 12:06:35 2020.
Record 2432: Sat Jul 25 06:23:35 2020 [Heartbeat.notice]: Heartbeat time adjusted: Set SP time. Old time: Sat Jul 25 06:23:33 2020. New time: Sat Jul 25 06:23:35 2020.
Record 2433: Thu Jan 1 00:00:36 1970 [IPMI.notice]: b404 | c0 | OEM: ffff7000ff00 | ManufId: 150300 | SP Power Reset
Record 2434: Thu Jan 1 00:00:36 1970 [IPMI.notice]: b504 | c0 | OEM: fcff70560000 | ManufId: 150300 | POS Register: Power on Reset(Normal Power Cycle)
Record 2435: Thu Jan 1 00:00:41 1970 [IPMI.notice]: b604 | 02 | EVT: 0157ff88 | SASS_1.2V | Assertion Event, "Upper Non-critical going high"
Record 2436: Thu Jan 1 00:00:41 1970 [IPMI.notice]: b704 | 02 | EVT: 0159ff8e | SASS_1.2V | Assertion Event, "Upper Critical going high"
Record 2437: Thu Jan 1 00:00:43 1970 [IPMI.notice]: b804 | 02 | EVT: 0301ffff | Power_Good | Assertion Event, "State Asserted"
Record 2438: Thu Jan 1 00:00:43 1970 [IPMI.notice]: b904 | 02 | EVT: 0301ffff | Power_Proc_OK | Assertion Event, "State Asserted"
Record 2439: Thu Jan 1 00:00:43 1970 [IPMI.notice]: ba04 | 02 | EVT: 0301ffff | Controller_Fault | Assertion Event, "State Asserted"
Record 2440: Thu Jan 1 00:00:43 1970 [IPMI.notice]: bb04 | 02 | EVT: 0900ffff | Wrench_Port_Up | Assertion Event, "Device Disabled"
Record 2441: Thu Jan 1 00:00:46 1970 [IPMI.notice]: bc04 | 02 | EVT: 81597d8e | SASS_1.2V | Deassertion Event, "Upper Critical going high"
Record 2442: Thu Jan 1 00:00:46 1970 [IPMI.notice]: bd04 | 02 | EVT: 81577d88 | SASS_1.2V | Deassertion Event, "Upper Non-critical going high"
Record 2443: Thu Jan 1 00:01:33 1970 [IPMI.notice]: be04 | 02 | EVT: 0901ffff | Wrench_Port_Up | Assertion Event, "Device Enabled"
Record 2444: Thu Jan 1 00:02:21 1970 [SP.notice]: Running primary version 2.10
Record 2445: Thu Jan 1 00:02:33 1970 [IPMI.warning]: FRUID 1 Access error
Record 2446: Thu Jan 1 00:02:45 1970 [IPMI.notice]: bf04 | 02 | EVT: 6fc100ff | System_FW_Status | Assertion Event, "Unspecified"
Record 2447: Thu Jan 1 00:02:49 1970 [IPMI.warning]: FRUID 1 Access error
Record 2448: Thu Jan 1 00:03:06 1970 [IPMI.warning]: FRUID 1 Access error
Record 2449: Thu Jan 1 00:03:22 1970 [IPMI.warning]: FRUID 1 Access error
Record 2450: Thu Jan 1 00:03:38 1970 [IPMI.warning]: FRUID 1 Access error
Record 2451: Thu Jan 1 00:03:55 1970 [IPMI.warning]: FRUID 1 Access error
Record 2452: Thu Jan 1 00:04:09 1970 [IPMI.warning]: FRUID 1 Access error
Record 2453: Sun Jul 26 11:39:44 2020 [BIOS.warning]: POST error 0x00a5: Definition not available Additional data: 0x00000000 0x00000000
Record 2454: Thu Jan 1 00:04:37 1970 [IPMI.notice]: c004 | 02 | EVT: 6fc204ff | System_FW_Status | Assertion Event, "Restoring MCH Values"
Record 2455: Sun Jul 26 11:39:49 2020 [CFE.notice]: Loader time adjust: Set SP time. Old time: Thu Jan 1 00:04:40 1970. New time: Sun Jul 26 11:39:49 2020.
Record 2456: Sun Jul 26 11:39:49 2020 [Boot Loader.notice]: Received time sync
Record 2457: Sun Jul 26 11:39:49 2020 [IPMI.notice]: c104 | 02 | EVT: 6fc000ff | System_FW_Status | Assertion Event, "Unspecified fatal firmware error"
Record 2458: Sun Jul 26 11:39:49 2020 [Boot Loader.critical]: Abort Autoboot due to BIOS POST failure.
Record 2459: Sun Jul 26 11:39:49 2020 [Trap Event.critical]: hwassist post_error (26)
Record 2460: Sun Jul 26 11:39:50 2020 [IPMI.warning]: FRUID 1 Access error
Record 2461: Sun Jul 26 11:39:52 2020 [IPMI.notice]: c204 | 02 | EVT: 6fc213ff | System_FW_Status | Assertion Event, "System boot initiated"
Record 2462: Sun Jul 26 11:39:57 2020 [IPMI.notice]: c304 | 02 | EVT: 6fc220ff | System_FW_Status | Assertion Event, "Bootloader is running"
Record 2463: Sun Jul 26 11:40:25 2020 [IPMI.warning]: FRUID 1 Access error
Record 2464: Sun Jul 26 11:40:42 2020 [IPMI.warning]: FRUID 1 Access error
Record 2465: Sun Jul 26 11:40:58 2020 [IPMI.warning]: FRUID 1 Access error
Record 2466: Sun Jul 26 11:41:14 2020 [IPMI.warning]: FRUID 1 Access error
Record 2467: Sun Jul 26 11:41:31 2020 [IPMI.warning]: FRUID 1 Access error
Record 2468: Sun Jul 26 11:41:47 2020 [IPMI.warning]: FRUID 1 Access error
Record 2469: Sun Jul 26 11:42:04 2020 [IPMI.warning]: FRUID 1 Access error
Record 2470: Sun Jul 26 11:42:20 2020 [IPMI.warning]: FRUID 1 Access error
Record 2471: Sun Jul 26 11:42:37 2020 [IPMI.warning]: FRUID 1 Access error
Record 2472: Sun Jul 26 11:42:53 2020 [IPMI.warning]: FRUID 1 Access error
Record 2473: Sun Jul 26 11:43:09 2020 [IPMI.warning]: FRUID 1 Access error
Record 2474: Sun Jul 26 11:43:26 2020 [IPMI.warning]: FRUID 1 Access error
Record 2475: Sun Jul 26 11:43:42 2020 [IPMI.warning]: FRUID 1 Access error
Record 2476: Sun Jul 26 11:43:58 2020 [IPMI.warning]: FRUID 1 Access error
Record 2477: Sun Jul 26 11:44:15 2020 [IPMI.warning]: FRUID 1 Access error
Record 2478: Sun Jul 26 11:44:31 2020 [IPMI.warning]: FRUID 1 Access error
Record 2479: Sun Jul 26 11:44:48 2020 [IPMI.warning]: FRUID 1 Access error
Record 2480: Sun Jul 26 11:45:04 2020 [IPMI.warning]: FRUID 1 Access error
Record 2481: Sun Jul 26 11:45:20 2020 [IPMI.warning]: FRUID 1 Access error
Record 2482: Sun Jul 26 11:45:42 2020 [IPMI.warning]: FRUID 1 Access error
Record 2483: Sun Jul 26 11:45:58 2020 [IPMI.warning]: FRUID 1 Access error
Record 2484: Sun Jul 26 11:46:14 2020 [IPMI.warning]: FRUID 1 Access error
Record 2485: Sun Jul 26 11:46:39 2020 [IPMI.warning]: FRUID 1 Access error
Record 2486: Sun Jul 26 11:46:43 2020 [ASUP.notice]: First notification email | (SYSTEM_BOOT_FAILED (POST failed)) CRITICAL | Send failed
Record 2487: Sun Jul 26 11:46:56 2020 [IPMI.warning]: FRUID 1 Access error
Record 2488: Sun Jul 26 11:47:12 2020 [IPMI.warning]: FRUID 1 Access error
Record 2489: Sun Jul 26 11:47:29 2020 [IPMI.warning]: FRUID 1 Access error
Record 2490: Sun Jul 26 11:47:45 2020 [IPMI.warning]: FRUID 1 Access error
Record 2491: Sun Jul 26 11:48:02 2020 [IPMI.warning]: FRUID 1 Access error
Record 2492: Sun Jul 26 11:48:18 2020 [IPMI.warning]: FRUID 1 Access error
Record 2493: Sun Jul 26 11:48:40 2020 [IPMI.warning]: FRUID 1 Access error
Record 2494: Sun Jul 26 11:48:40 2020 [IPMI.critical]: Rebooting SP due to task restarts
Record 2495: Sun Jul 26 11:48:40 2020 [IPMI.critical]: df: 98304 35160 63144 36%
Record 2496: Sun Jul 26 11:48:40 2020 [IPMI.critical]: fp: 795 0 12590
Record 2497: Sun Jul 26 11:48:40 2020 [IPMI.critical]: uptime: 811.890015 672.890015
Record 2498: Sun Jul 26 11:48:40 2020 [IPMI.critical]: ldavg: 1.480000 1.340000 0.860000 4/122 2237
Record 2499: Thu Jan 1 00:00:35 1970 [IPMI.notice]: c404 | c0 | OEM: ffff70005100 | ManufId: 150300 | SP Reset Internally
Record 2500: Thu Jan 1 00:00:41 1970 [IPMI.notice]: c504 | 02 | EVT: 0301ffff | Power_Good | Assertion Event, "State Asserted"
Record 2501: Thu Jan 1 00:00:42 1970 [IPMI.notice]: c604 | 02 | EVT: 0301ffff | Power_Proc_OK | Assertion Event, "State Asserted"
Record 2502: Thu Jan 1 00:00:42 1970 [IPMI.notice]: c704 | 02 | EVT: 6fc220ff | System_FW_Status | Assertion Event, "Bootloader is running"
Record 2503: Thu Jan 1 00:00:43 1970 [IPMI.notice]: c804 | 02 | EVT: 0301ffff | Controller_Fault | Assertion Event, "State Asserted"
Record 2504: Thu Jan 1 00:01:03 1970 [SP.notice]: Running primary version 2.10
Record 2505: Sun Jul 26 11:50:56 2020 [CFE.notice]: Loader time adjust: Set SP time. Old time: Thu Jan 1 00:01:34 1970. New time: Sun Jul 26 11:50:56 2020.
Record 2506: Sun Jul 26 11:50:56 2020 [Boot Loader.notice]: Received time sync
Record 2507: Sun Jul 26 12:02:30 2020 [SP.critical]: Heartbeat stopped
Record 2508: Sun Jul 26 12:02:30 2020 [Trap Event.warning]: hwassist loss_of_heartbeat (30)
Record 2509: Sun Jul 26 12:02:47 2020 [ASUP.notice]: First notification email | (HEARTBEAT_LOSS) WARNING | Send failed
Record 2510: Sun Jul 26 12:17:31 2020 [ASUP.notice]: Reminder email | (HEARTBEAT_LOSS) WARNING | Send failed
Record 2511: Tue Jul 28 20:46:15 2020 [IPMI.notice]: c904 | 02 | EVT: 0900ffff | Wrench_Port_Up | Assertion Event, "Device Disabled"
Record 2512: Tue Jul 28 20:48:23 2020 [IPMI.notice]: ca04 | 02 | EVT: 0901ffff | Wrench_Port_Up | Assertion Event, "Device Enabled"
Record 2513: Tue Jul 28 21:39:38 2020 [SP CLI.notice]: "log in from Serial Console"
SP Morgan-01>
Thanks for your help