OCUM 5.2 CORE (7-Mode) on Linux - SQL Startup Error

arsalankhan · ‎2017-09-07

We had abnormal shutdown of the OCUM core Linux based server couple of times.

Right after that DFM service couldnt start successfully.

Basically customer did the kernel upgrade and rebooted the server without stopping DFM services properly. Second time they rebooted again without stopping the services.

Now we are getting this error in logs:

I. 09/07 14:16:12. Disconnecting shared memory client, process id not found

I. 09/07 14:16:12. Disconnected SharedMemory client's AppInfo: IP=10.8.88.100;HOST=dfm-na-p.enp.a.com.sa;OSUSER=root;OS='Linux 2.6.32-696.1.1.el6.x86_64 #1 SMP T

ue Mar 21 12:19:18 EDT 2017 x';EXE=/opt/NTAPdfm/sbin/dfmmonitor;PID=0x2443;THREAD=0x7f8fb96e6700;VERSION=16.0.0.2087;API=ODBC;TIMEZONEADJUSTMENT=180;APPINFO='EXE=dfmm

onitor'

I. 09/07 14:16:12. Disconnecting shared memory client, process id not found

I. 09/07 14:16:12. Disconnected SharedMemory client's AppInfo: IP=10.8.88.100;HOST=dfm-na-p.enp.a.com.sa;OSUSER=root;OS='Linux 2.6.32-696.1.1.el6.x86_64 #1 SMP T

ue Mar 21 12:19:18 EDT 2017 x';EXE=/opt/NTAPdfm/sbin/dfmmonitor;PID=0x2443;THREAD=0x7f8fb95e5700;VERSION=16.0.0.2087;API=ODBC;TIMEZONEADJUSTMENT=180;APPINFO='EXE=dfmm

onitor'

I. 09/07 14:16:19. Database server shutdown requested by DBSTOP

I. 09/07 14:16:19. Starting checkpoint of "monitordb" (monitordb.db) at Thu Sep 07 2017 14:16

I. 09/07 14:16:20. TCPIP listener on IP address 0.0.0.0:2638 is exiting

I. 09/07 14:16:20. Finished checkpoint of "monitordb" (monitordb.db) at Thu Sep 07 2017 14:16

I. 09/07 14:16:20. TCPIP listener on IP address (::):2638 is exiting

I. 09/07 14:16:22. Database server stopped at Thu Sep 07 2017 14:16

I. 09/07 14:18:51. SQL Anywhere Network Server Version 16.0.0.2087

I. 09/07 14:18:51. OEM Authenticated Edition, licensed only for use with authenticated OEM applications.

I. 09/07 14:18:51.

I. 09/07 14:18:51. Use of this software is governed by the Sybase License Agreement.

I. 09/07 14:18:51. Refer to http://www.sybase.com/softwarelicenses.

I. 09/07 14:18:51.

I. 09/07 14:18:51. Connection limit (licensed seats): 1000

I. 09/07 14:18:51. Processors detected: 1 (containing 4 logical processors)

I. 09/07 14:18:51. Maximum number of processors the server will use: 1 physical processor(s), 4 core(s)

I. 09/07 14:18:51. This server is licensed to:

I. 09/07 14:18:51. DFM User

I. 09/07 14:18:51. NetApp

I. 09/07 14:18:51. Running Linux 2.6.32-696.6.3.el6.x86_64.debug #1 SMP Fri Jun 30 13:28:42 EDT 2017 on X86_64

I. 09/07 14:18:51. Server built for X86_64 processor architecture

I. 09/07 14:18:51. 4051432K of memory used for caching

I. 09/07 14:18:51. Minimum cache size: 16752K, maximum cache size: 16370656K

I. 09/07 14:18:51. Using a maximum page size of 8192 bytes

I. 09/07 14:18:51. Multiprogramming level: minimum:4, current:128, maximum:500

I. 09/07 14:18:51. Automatic tuning of multiprogramming level is enabled

I. 09/07 14:18:51. Starting database "monitordb" (/opt/NetAppDFM/data/monitordb.db) at Thu Sep 07 2017 14:18

I. 09/07 14:18:54. Transaction log: monitordb.log

I. 09/07 14:18:54. Starting checkpoint of "monitordb" (monitordb.db) at Thu Sep 07 2017 14:18

I. 09/07 14:18:54. Finished checkpoint of "monitordb" (monitordb.db) at Thu Sep 07 2017 14:18

I. 09/07 14:18:54. Database "monitordb" (monitordb.db) started at Thu Sep 07 2017 14:18

I. 09/07 14:18:54. This database is licensed for use with:

I. 09/07 14:18:54. Application: DataFabric Manager

I. 09/07 14:18:54. Company: NetApp

I. 09/07 14:18:54. Database server started at Thu Sep 07 2017 14:18

I. 09/07 14:18:54. Trying to start SharedMemory link ...

I. 09/07 14:18:54. SharedMemory link started successfully

I. 09/07 14:18:54. Trying to start TCPIP link ...

I. 09/07 14:18:55. Starting on port 2638

I. 09/07 14:19:00. TCPIP link started successfully

I. 09/07 14:19:00. Now accepting requests

I. 09/07 14:19:05. Cache size adjusted to 3926560K

I. 09/07 14:19:05. Cache size adjusted to 3895536K

I. 09/07 14:19:05. Cache size adjusted to 3887880K

I. 09/07 14:19:05. Cache size adjusted to 3886016K

I. 09/07 14:19:05. Cache size adjusted to 3885776K

I. 09/07 14:30:47. Database server shutdown due to TERM signal

I. 09/07 14:30:47. Starting checkpoint of "monitordb" (monitordb.db) at Thu Sep 07 2017 14:30

I. 09/07 14:30:47. TCPIP listener on IP address (::):2638 is exiting

I. 09/07 14:30:47. Finished checkpoint of "monitordb" (monitordb.db) at Thu Sep 07 2017 14:30

I. 09/07 14:30:47. TCPIP listener on IP address 0.0.0.0:2638 is exiting

I. 09/07 14:30:48. Database server stopped at Thu Sep 07 2017 14:30

I. 09/07 14:32:10. SQL Anywhere Network Server Version 16.0.0.2087

I. 09/07 14:32:10. OEM Authenticated Edition, licensed only for use with authenticated OEM applications.

I. 09/07 14:32:10.

I. 09/07 14:32:10. Use of this software is governed by the Sybase License Agreement.

I. 09/07 14:32:10. Refer to http://www.sybase.com/softwarelicenses.

I. 09/07 14:32:10.

I. 09/07 14:32:10. Connection limit (licensed seats): 1000

I. 09/07 14:32:10. Processors detected: 1 (containing 4 logical processors)

I. 09/07 14:32:10. Maximum number of processors the server will use: 1 physical processor(s), 4 core(s)

I. 09/07 14:32:10. This server is licensed to:

I. 09/07 14:32:10. DFM User

I. 09/07 14:32:10. NetApp

I. 09/07 14:32:10. Running Linux 2.6.32-696.6.3.el6.x86_64 #1 SMP Fri Jun 30 13:24:18 EDT 2017 on X86_64

I. 09/07 14:32:10. Server built for X86_64 processor architecture

I. 09/07 14:32:10. 4068480K of memory used for caching

I. 09/07 14:32:10. Minimum cache size: 16784K, maximum cache size: 16436104K

I. 09/07 14:32:10. Using a maximum page size of 8192 bytes

I. 09/07 14:32:10. Multiprogramming level: minimum:4, current:128, maximum:500

I. 09/07 14:32:10. Automatic tuning of multiprogramming level is enabled

I. 09/07 14:32:10. Starting database "monitordb" (/opt/NetAppDFM/data/monitordb.db) at Thu Sep 07 2017 14:32

E. 09/07 14:32:10. *** ERROR *** Assertion failed: 201865 (16.0.0.2087)

Checkpoint log: Invalid pages at 0x74bb9 and 0x74c63

Seems like This is the actual error

"Assertion failed: 201865 (16.0.0.2087)

Checkpoint log: Invalid pages at 0x74bb9 and 0x74c63"

Could someone please help us into this ?

marz · ‎2017-09-12

If you have a backup for the DFM server (.ndb file) captured before the host upgrade, safeguard it - you may need it as a restore option.

Open a case with Support - the database itself (monitordb.db) will need to be investigated to determine the details behind the page faults.