Solved: Re: Not enough memory while trying to install OC 5.0.1 on 64bit Linux

marcinl · ‎2012-06-15

Hello,

I was trying to install OC Core 5.0.1 on 64bit SLES 11.2 with 4GB of RAM (hosted on ESX).

The installation quits with following messages:

Setting up sql ...

Problem initializing the database.

error: %post(NTAPdfm-5.0-7864.x86_64) scriptlet failed, exit status 1

In /var/log/messages I can see this:

Jun 14 13:56:52 monitoring SQLAnywhere(dbinit_engine): SQL Anywhere Personal Server Version 10.0.1.4213

Jun 14 13:56:52 monitoring SQLAnywhere(dbinit_engine): 2 physical processor(s) detected.

Jun 14 13:56:52 monitoring SQLAnywhere(dbinit_engine): This server is licensed to:

Jun 14 13:56:52 monitoring SQLAnywhere(dbinit_engine): DFM User

Jun 14 13:56:52 monitoring SQLAnywhere(dbinit_engine): NetApp

Jun 14 13:56:52 monitoring SQLAnywhere(dbinit_engine): Running Linux 3.0.13-0.27-default #1 SMP Wed Feb 15 13:33:49 UTC 2012 (d73692b) on X86_64

Jun 14 13:56:52 monitoring SQLAnywhere(dbinit_engine): Server built for X86_64 processor architecture

Jun 14 13:56:52 monitoring SQLAnywhere(dbinit_engine): Not enough memory

I have increased RAM to 6GB and got the same error. There is enough memory during the installation (checked via top on 2nd console while installing, was never below 2GB free RAM).

There is enough free space on disk:

monitoring:~ # df -h

Filesystem Size Used Avail Use% Mounted on

/dev/sda6 9.9G 2.9G 6.5G 31% /

devtmpfs 1.9G 144K 1.9G 1% /dev

tmpfs 1.9G 0 1.9G 0% /dev/shm

/dev/sda7 36G 1.4G 33G 5% /opt

/dev/sda5 2.0G 162M 1.8G 9% /var

I have increased number of semaphores and open files, /etc/sysctrl.conf contains following lines now:

kernel.sem=250 32000 32 1024

fs.file-max = 65535

I have also changed dfm option for database cache (while it was partially installed) but I don't think this is preserved between the installations (before installing it again I need to remove the NTAPdfm package).

Has anyone experienced such behavior ? What could cause it? 6GB RAM for OnCommand 5 without any objects in the database should be more than enough.

salekseev · ‎2012-06-19

I was having the same problem. Launched it in Linux 2.6 compatibility mode and it worked:

cbnapsysman:~ # uname26

cbnapsysman:~> ./occore-setup-5-0-1-linux-x64.sh -l blah

Preparing to install OnCommand Core Package

Unpacking files needed for the installation ...

By default, the DataFabric Manager server sends AutoSupport event messages

and weekly reports to NetApp Technical Support. Messages are sent

over secure HTTPS, HTTP or SMTP(HTTPS by default). This information is used to

assist you with troubleshooting and problem resolution. AutoSupport will

significantly speed problem determination and resolution should a problem occur

on the system.

Status AutoSupport event messages will be sent to NetApp once per

week and will start 24 hours after installation. If you do not want

AutoSupport event messages sent to NetApp, you can disable

AutoSupport by setting the AutoSupport Enabled global option to "No".

This option will initially be set to "Unknown" unless manually set, and

will automatically become "Yes" after 24 hours if not set manually to "No".

If you do not want to include private data such as IP addresses, host names,

and user names, set the AutoSupport Content global option to "minimal".

Please respond with "Yes" if you have read and agree with the use of the

AutoSupport feature on this DataFabric Manager server or "No" to abort the

installation.[y,n,?]:y

Please choose the edition of OnCommand Core Package installation.

[1] Express edition

- Simplifies deployment with automated configuration.

- Provides the same level of data protection capability as the

Standard edition, but with reduced monitoring capabilities.

- Runs OnCommand Core and OnCommand Host on the same server.

- Recommended for use with up to four storage systems and one virtualized server.

[2] Standard edition

- Recommended for full storage monitoring capabilities in all deployments.

- Must be installed on a dedicated system.

Please enter your choice of OnCommand Core Package installation.[1,2](default 2):1

Beginning the OnCommand Core Package installation ...

Preparing... ########################################### [100%]

1:NTAPdfm ########################################### [100%]

Installing scripts in /etc/init.d directory.

insserv: Service network is missed in the runlevels 4 to use service NTAPdfm

NTAPdfm 0:off 1:off 2:off 3:on 4:on 5:on 6:off

Creating the DataFabric Manager server keystore.

Configuring DataFabric Manager server services.

Setting up sql ...

Starting SQL ...

Setting up DBA user ...

Defining SQL login procedure ...

Setting up transaction log management...

Setting up deadlock reporting infrastructure...

Disabling deadlock logging...

Defining SQL schema ...

Defining SQL Views ...

Defining SQL debug procedures ...

Defining metadata accessors ...

Setting up DFM user ...

Updating global options and monitoring intervals.

Setting up default schedule for database backup.

Configured Express edition.

Creating or updating sample backup schedules and throttles.

Successfully created VMware sample application policy.

Successfully created Hyperv sample application policy.

Creating the canned provisioning policies.

Canned provisioning policies have been created successfully.

Created resource pool Default Storage Pool (96).

Canned storage services have been created successfully.

Creating or updating sample DP policies.

Creating or updating sample DR policies.

Stopping SQL ...

SQL Anywhere Stop Server Utility Version 10.0.1.4213

Enabled DataFabric Manager server license.

Installing the online help.

Generating DataFabric Manager server certificate files.

Service: sql started.

Starting DataFabric Manager server services.

Service: sql started.

Not attempting to start sql because service is started already.

Setting default reportDesignPath because global option reportDesignPath is empty

Service: webui started.

Service: http started.

Service: eventd started.

Service: monitor started.

Service: scheduler started.

Service: server started.

Service: watchdog started.

Installing Filer Plugins.

New plugin files installed to /opt/NTAPdfm/plugins/filer/6.5.1.

New plugin files installed to /opt/NTAPdfm/plugins/filer/6.5.2.

New plugin files installed to /opt/NTAPdfm/plugins/filer/6.5.3.

New plugin files installed to /opt/NTAPdfm/plugins/filer/6.5.4.

New plugin files installed to /opt/NTAPdfm/plugins/filer/6.5.5.

New plugin files installed to /opt/NTAPdfm/plugins/filer/6.5.6.

New plugin files installed to /opt/NTAPdfm/plugins/filer/6.5.7.

New plugin files installed to /opt/NTAPdfm/plugins/filer/7.0.

New plugin files installed to /opt/NTAPdfm/plugins/filer/7.0.1.

New plugin files installed to /opt/NTAPdfm/plugins/filer/7.0.2.

New plugin files installed to /opt/NTAPdfm/plugins/filer/7.0.3.

New plugin files installed to /opt/NTAPdfm/plugins/filer/7.0.4.

New plugin files installed to /opt/NTAPdfm/plugins/filer/7.0.5.

New plugin files installed to /opt/NTAPdfm/plugins/filer/7.0.6.

New plugin files installed to /opt/NTAPdfm/plugins/filer/7.0.7.

New plugin files installed to /opt/NTAPdfm/plugins/filer/7.1.

New plugin files installed to /opt/NTAPdfm/plugins/filer/7.1.1.

New plugin files installed to /opt/NTAPdfm/plugins/filer/7.1.2.

New plugin files installed to /opt/NTAPdfm/plugins/filer/7.1.3.

New plugin files installed to /opt/NTAPdfm/plugins/filer/7.2.

New plugin files installed to /opt/NTAPdfm/plugins/filer/7.2.1.

New plugin files installed to /opt/NTAPdfm/plugins/filer/7.2.2.

New plugin files installed to /opt/NTAPdfm/plugins/filer/7.2.3.

New plugin files installed to /opt/NTAPdfm/plugins/filer/7.2.4.

New plugin files installed to /opt/NTAPdfm/plugins/filer/7.2.5.

New plugin files installed to /opt/NTAPdfm/plugins/filer/7.2.6.

New plugin files installed to /opt/NTAPdfm/plugins/filer/7.2.7.

New plugin files installed to /opt/NTAPdfm/plugins/filer/7.3.

New plugin files installed to /opt/NTAPdfm/plugins/filer/7.3.1.

New plugin files installed to /opt/NTAPdfm/plugins/filer/7.3.2.

New plugin files installed to /opt/NTAPdfm/plugins/filer/7.3.3.

New plugin files installed to /opt/NTAPdfm/plugins/filer/7.3.4.

New plugin files installed to /opt/NTAPdfm/plugins/filer/7.3.5.

New plugin files installed to /opt/NTAPdfm/plugins/filer/8.0.

New plugin files installed to /opt/NTAPdfm/plugins/filer/8.0.1.

The Express edition of the OnCommand Core Package, including the DataFabric Manager server, has

been installed on your computer.

Please do not change any setup options without consulting NetApp support.

If you are managing a virtual server environment, you must now install the OnCommand Host Package,

if you have not already installed it.

You can now access OnCommand console at

http://cbnapsysman.dfci.harvard.edu:8080/start.html

to configure and monitor hosts and storage with DataFabric Manager server.

OnCommand console must be run from a Windows-based browser.

For help on the dfm command, enter

dfm help

at the command prompt.

View solution in original post

kryan · ‎2012-06-15

https://kb.netapp.com/support/index?page=content&id=2016762

Try the following:

dfm prompt> dfm database set dbcachesize=""
dfm prompt> dfm service start sql

This will set the option back to the default value. I see that you have adjusted it, but not sure which values you have used.

marcinl · ‎2012-06-15

Hi Kryan,

The problem is that this occurs during installation process. Database is not initialized after installation.

monitoring:~ # dfm database set dbcachesize=""

Left option settings unchanged.

monitoring:~ # dfm service start sql

Error: Unable to determine if the database needs to be upgraded to the latest file format.

Error: could not start 'sql' service.

monitoring:~ #

kryan · ‎2012-06-15

What other applications, if any, are installed on this server?

Also, is SELinux enabled? If so, set SELinux to permissive mode.

kryan · ‎2012-06-15

Another item to check - is the memory for the VM reserved or allocated? It would be better to have lower amount (4Gb) reserved as opposed to a larger amount set as allocated.

marcinl · ‎2012-06-15

Memory was just reserved but it's allocated now - 6GB.

Still the same error while installing OC.

SELinux not installed. Nothing else on the server - it's fresh installation.

mahala · ‎2012-06-15

Hi Marcin,

As I remember you had installed OC earlier. So was that on SELinux or was that lower version ?

marcinl · ‎2012-06-18

That was older version of SLES and OC 5.0.0.
I will try with OC 5.0.0 now also.

marz · ‎2012-06-18

Marcin

You need to adjust

kernel.sem=250 32000 32 1024

to read as so in /etc/sysctl.conf

kernel.sem=250 256000 32 1024

SEMMNI is correct at 1024 for OnCommand (and Operations Manager) running on Linux. SEMMNI * SEMMSL = SEMMNS, or 1024 * 250 = 256000

I also noticed in your screenshot that CPU was 0 MHz where Memory was 6144Mb. This summary value looks odd.

marcinl · ‎2012-06-18

Hi Alex,
I have changed the kernel parameters:

monitoring:~ # sysctl -a | grep sem

kernel.sem = 250 32000 32 1024

monitoring:~ # sysctl -p

net.ipv4.icmp_echo_ignore_broadcasts = 1

net.ipv4.conf.all.rp_filter = 1

fs.inotify.max_user_watches = 65536

net.ipv4.conf.default.promote_secondaries = 1

net.ipv4.conf.all.promote_secondaries = 1

kernel.sem = 250 256000 32 1024

fs.file-max = 65535

monitoring:~ # sysctl -a | grep sem

kernel.sem = 250 256000 32 1024

I'm not an VMware expert but I think the CPU displayed as "0" means just that nothing is strictly reserved to my virtual image.

Same error when installing OC5:

Setting up sql ...

Problem initializing the database.

error: %post(NTAPdfm-5.0-7864.x86_64) scriptlet failed, exit status 1

monitoring:~ # tail /var/log/messages

Jun 18 15:53:13 monitoring SQLAnywhere(dbinit_engine):

Jun 18 15:53:13 monitoring SQLAnywhere(dbinit_engine): iAnywhere Solutions, Inc., One Sybase Drive, Dublin, CA 94568, USA

Jun 18 15:53:13 monitoring SQLAnywhere(dbinit_engine):

Jun 18 15:53:13 monitoring SQLAnywhere(dbinit_engine): 2 physical processor(s) detected.

Jun 18 15:53:13 monitoring SQLAnywhere(dbinit_engine): This server is licensed to:

Jun 18 15:53:13 monitoring SQLAnywhere(dbinit_engine): DFM User

Jun 18 15:53:13 monitoring SQLAnywhere(dbinit_engine): NetApp

Jun 18 15:53:13 monitoring SQLAnywhere(dbinit_engine): Running Linux 3.0.13-0.27-default #1 SMP Wed Feb 15 13:33:49 UTC 2012 (d73692b) on X86_64

Jun 18 15:53:13 monitoring SQLAnywhere(dbinit_engine): Server built for X86_64 processor architecture

Jun 18 15:53:13 monitoring SQLAnywhere(dbinit_engine): Not enough memory

marcinl · ‎2012-06-19

The same with OC 5.0:

Creating the DataFabric Manager server keystore.

Configuring DataFabric Manager server services.

Setting up sql ...

Problem initializing the database.

error: %post(NTAPdfm-5.0-7636.x86_64) scriptlet failed, exit status 1

monitoring:~ # tail /var/log/messages

Jun 19 11:06:00 monitoring SQLAnywhere(dbinit_engine):

Jun 19 11:06:00 monitoring SQLAnywhere(dbinit_engine): iAnywhere Solutions, Inc., One Sybase Drive, Dublin, CA 94568, USA

Jun 19 11:06:00 monitoring SQLAnywhere(dbinit_engine):

Jun 19 11:06:00 monitoring SQLAnywhere(dbinit_engine): 2 physical processor(s) detected.

Jun 19 11:06:00 monitoring SQLAnywhere(dbinit_engine): This server is licensed to:

Jun 19 11:06:00 monitoring SQLAnywhere(dbinit_engine): DFM User

Jun 19 11:06:00 monitoring SQLAnywhere(dbinit_engine): NetApp

Jun 19 11:06:00 monitoring SQLAnywhere(dbinit_engine): Running Linux 3.0.13-0.27-default #1 SMP Wed Feb 15 13:33:49 UTC 2012 (d73692b) on X86_64

Jun 19 11:06:00 monitoring SQLAnywhere(dbinit_engine): Server built for X86_64 processor architecture

Jun 19 11:06:00 monitoring SQLAnywhere(dbinit_engine): Not enough memory

marcinl · ‎2012-06-19

OK. I have manually copied content of the occore-setup-5-0-1-linux-x64.sh file.

Then started each part manually.

The problem occures when trying to run command :

dfm service setup -l DFMKEY

Setting up sql ...

Problem initializing the database.

monitoring:/opt/NTAPdfm # echo $?

1

Then in /var/log/messages the same error message as previously about not enough memory.

In strace I can see that it's trying to read f.e. /proc/meminfo

This is how it looks currently:

cat /proc/meminfo

MemTotal: 5994916 kB

MemFree: 1333220 kB

Buffers: 64572 kB

Cached: 4357500 kB

SwapCached: 4796 kB

Active: 2987144 kB

Inactive: 1451528 kB

Active(anon): 10516 kB

Inactive(anon): 6088 kB

Active(file): 2976628 kB

Inactive(file): 1445440 kB

Unevictable: 0 kB

Mlocked: 0 kB

SwapTotal: 2095100 kB

SwapFree: 2078116 kB

Dirty: 64 kB

Writeback: 0 kB

AnonPages: 13300 kB

Mapped: 56536 kB

Shmem: 4 kB

Slab: 136132 kB

SReclaimable: 113072 kB

SUnreclaim: 23060 kB

KernelStack: 1376 kB

PageTables: 3272 kB

NFS_Unstable: 0 kB

Bounce: 0 kB

WritebackTmp: 0 kB

CommitLimit: 5092556 kB

Committed_AS: 102252 kB

VmallocTotal: 34359738367 kB

VmallocUsed: 292420 kB

VmallocChunk: 34359443244 kB

HardwareCorrupted: 0 kB

AnonHugePages: 0 kB

HugePages_Total: 0

HugePages_Free: 0

HugePages_Rsvd: 0

HugePages_Surp: 0

Hugepagesize: 2048 kB

DirectMap4k: 43008 kB

DirectMap2M: 6248448 kB

Attached strace output.

salekseev · ‎2012-06-19