9.7 simulator root vol0 keeps filling up snap and auto support turned off

[mjgratke]

Something like logging is filling root vol0 up you can see the creep in used over just a few seconds.   What is running to fill it up?   snap and auto support turned off      Thanks 

cluster97-01*> df
Filesystem kbytes used avail capacity Mounted on
/vol/vol0/ 10957620 956916 10000704 9% /vol/vol0/
/vol/vol0/.snapshot 576716 0 576716 0% /vol/vol0/.snapshot
cluster97-01*> df
Filesystem kbytes used avail capacity Mounted on
/vol/vol0/ 10957620 957004 10000616 9% /vol/vol0/
/vol/vol0/.snapshot 576716 0 576716 0% /vol/vol0/.snapshot
cluster97-01*> df
Filesystem kbytes used avail capacity Mounted on
/vol/vol0/ 10957620 957632 9999988 9% /vol/vol0/
/vol/vol0/.snapshot 576716 0 576716 0% /vol/vol0/.snapshot
cluster97-01*>

[JohnChampion · Accepted solution]

# Expand root volume size
# Add disk(s) to root aggregate
storage aggregate add-disks -aggregate aggr0 -diskcount 1

# Set root volume to new size
volume size -vserver <vserver_name> -volume vol0 -new-size 1.5GB

# Verify
volume show -volume vol0

# Disable root snapshots

run local
snap delete -a -f vol0
snap sched vol0 0 0 0
snap autodelete vol0 on
snap autodelete vol0 target_free_space 35
snap autodelete vol0

...and have a look at this post - it's a bit old but much still applies and works - especially making more and larger disks.

https://community.netapp.com/t5/Developer-Network-Articles-and-Resources/Building-and-Customizing-NetApp-Simulators/ta-p/87157/highlight/true

[mjgratke]

Thanks for the reply.   I have increased vol0 size and it's going to take longer fill up but something is still constantly consuming free space.  I assume logging but i can not track it down.    Will the simulator manage the growth so vol0 will not run out of space?    

[ploiesti]

I`ve recovered space from diag mode --> systemshell node run...bla bla...

right there on the low level BSD(linux stuff) you can delete log files ( A LOT of log files).

cluster2-01% ll
total 40
-rw-r--r-- 1 diag nogroup 1145 Dec 12 15:05 .cshrc
-rw------- 1 diag nogroup 545 Jan 9 09:16 .history
-rw-r--r-- 1 diag nogroup 323 Dec 12 15:05 .login
-rw-r--r-- 1 diag nogroup 91 Dec 12 15:05 .login_conf
-rw------- 1 diag nogroup 301 Dec 12 15:05 .mail_aliases
-rw-r--r-- 1 diag nogroup 267 Dec 12 15:05 .mailrc
-rw-r--r-- 1 diag nogroup 881 Dec 12 15:05 .profile
-rw------- 1 diag nogroup 212 Dec 12 15:05 .rhosts
-rw-r--r-- 1 diag nogroup 782 Dec 12 15:05 .shrc
drwxr-xr-x 2 root nogroup 512 Jan 9 09:21 .ssh-systemshell/
cluster2-01% cd /
cluster2-01% ll
total 38545
-r--r--r-- 1 root wheel 398 Jan 9 2020 .valgrindrc
-rw-r--r-- 1 root wheel 168 Jan 9 2020 BUILD
-r--r--r-- 1 root wheel 63 Jan 9 2020 COMPAT.TXT
-r--r--r-- 1 root wheel 6142 Jan 9 2020 COPYRIGHT
-r-xr-xr-x 1 root wheel 52779 Jan 9 2020 INSTALL*
-r--r--r-- 1 root wheel 1867 Jan 9 2020 README.TXT
-r--r--r-- 1 2087 30 4 Jan 17 2020 VERSION
drwxr-xr-x 2 root wheel 1024 Jan 9 2020 bin/
drwxr-xr-x 6 root wheel 512 Jan 9 2020 boot/
-r--r--r-- 1 root wheel 172318 Jan 9 2020 cap.xml
drwxr-xr-x 1 root wheel 32768 Jan 1 1980 cfcard/
dr-xr-xr-x 2 root wheel 512 Jan 9 09:18 clus/
dr-xr-xr-x 6 root wheel 512 Jan 9 09:18 dev/
drwxr-xr-x 32 root wheel 6656 Jan 17 2020 etc/
drwxr-xr-x 2 root wheel 1024 Jan 9 2020 fw/
drwxr-xr-x 3 root wheel 512 Jan 9 2020 lib/
drwxr-xr-x 3 root wheel 512 Jan 9 2020 libexec/
drwxr-xr-x 6 root wheel 512 Jan 9 2020 mnt/
drwxr-xr-x 8 root wheel 4096 Jan 9 09:26 mroot/
-rw-r--r-- 1 root wheel 4720265 Jan 9 2020 mroot.tgz
drwxr-xr-x 3 root wheel 512 Jan 9 2020 mroot_late/
-rw-r--r-- 1 root wheel 34196628 Jan 9 2020 mroot_late.tgz
-r-xr-xr-x 1 root wheel 868 Jan 9 2020 netapp_nodar_build_check*
drwxr-xr-x 2 root wheel 512 Jan 9 2020 nfsroot/
drwxr-xr-x 3 root wheel 512 Jan 9 2020 ontap/
drwxr-xr-x 2 root wheel 512 Jan 9 2020 ovl/
drwxr-xr-x 2 root wheel 512 Jan 9 2020 partner/
drwxr-xr-x 5 root wheel 512 Jan 9 2020 platform/
dr-xr-xr-x 1 root wheel 0 Jan 9 09:26 proc/
drwxr-xr-x 3 root wheel 512 Jan 9 2020 root/
drwxr-xr-x 2 root wheel 3584 Jan 17 2020 sbin/
drwxr-xr-x 8 root wheel 512 Dec 12 15:03 sim/
drwxr-xr-x 19 root wheel 512 Jan 9 2020 sldiag/
drwxrwxrwt 7 root wheel 1024 Jan 9 09:26 tmp/
drwxr-xr-x 12 root wheel 512 Jan 9 2020 usr/
drwxr-xr-x 24 root wheel 1024 Jan 9 09:26 var/
-rw-r--r-- 1 root wheel 40178 Jan 17 2020 varfs.tgz
-r--r--r-- 1 root wheel 37495 Jan 9 2020 vs_conf_files.tgz
drwxr-xr-x 3 root wheel 512 Jan 9 2020 webjail/
cluster2-01% cd /v
var/ varfs.tgz vs_conf_files.tgz
cluster2-01% cd /var/log
cluster2-01% ls
apache_logs netapp_fpolicy.log
audit.log.00000000000000000028 netapp_frulog.log
audit.log.00000000000000000029 netapp_mdns.log
bcomd.log.00000000000000000135 netapp_mes.log
bcomd.log.00000000000000000136 netapp_mgmt.log
cm-daemon.log.00000000000000000149 netsetup.log
cm-daemon.log.00000000000000000150 notifyd.log.00000000000000002259
cro_proxy.log.00000000000000000024 notifyd.log.00000000000000002260
cro_proxy.log.00000000000000000025 perfstatd.log.00000000000000000050
crs.log.00000000000000000218 perfstatd.log.00000000000000000051
crs.log.00000000000000000219 php-fpm.access.log.00000000000000001119
debug.log.00000000000000000043 php-fpm.access.log.00000000000000001120
debug.log.00000000000000000044 php-fpm.error.log.00000000000000000001
fpolicy.log.00000000000000000029 php-fpm.error.log.00000000000000000002
fpolicy.log.00000000000000000030 php.log.00000000000000000001
hashd.log.00000000000000000005 secd.log.00000000000000000124
hashd.log.00000000000000000006 secd.log.00000000000000000125
kmip2_client.log.00000000000000000080 servprocd.log.00000000000000000248
kmip2_client.log.00000000000000000081 servprocd.log.00000000000000000249
licensed.log.00000000000000000027 spmd.log.00000000000000000077
licensed.log.00000000000000000028 spmd.log.00000000000000000078
memsnap.log syslog_CA_info
messages.log.00000000000000000063 syslog_dest_info
messages.log.00000000000000000064 sysmgr.log.00000000000000012746
mgwd-start.log sysmgr.log.00000000000000012747
mgwd-start.log.1 utx.lastlogin
mgwd.log.00000000000000000556 utx.log
mgwd.log.00000000000000000557 vifmgr.log.00000000000000000240
mlogd-start.log vifmgr.log.00000000000000000241
mlogd-start.log.1 vldb.log.00000000000000000171
mlogd_stats.0.log vldb.log.00000000000000000172
mlogd_stats.1.log vserverdr.log.00000000000000000742
mlogd_stats.2.log vserverdr.log.00000000000000000743
mount_spin.log ypbind.log.00000000000000000001
ndmpd.log.00000000000000000003 ypbind.log.00000000000000000002
ndmpd.log.00000000000000000004

[mikesmit]

Deleting files is a good temporary solution. I dug a little deeper and found out that the performance archiver was filling up vol0 on my simulator also. I found this kb article which describes symptoms similar to what I was seeing (KB article 162421 . The following commands dramatically reduced the space consumed in vol0 on my simulator and the added side benefit was that I didn't have to continually increase the size of vol0. 
Reduce percentage of vol0 disk space that is used for perf archive from 5% (Default) to 2% (Absolute minimum):
statistics archive config modify -datastore-max-percent 2
Reduce the number of days to retain the perf archive data for to 5 from 15 (default):
statistics archive config modify -datastore-max-retention 5

This is what the output looks like after the changes are made:

windy::*> statistics archive config show

Is the Performance Archive Enabled?: true
Base Path of Performance Archive Datastore: /mroot/etc/log/stats/ccma
Days to Retain Performance Archive Data: 5
Maximum Percentage of Root Volume Used for Performance Archive Data: 2%
Maximum Size of Performance Archive Data on Disk: 36.62MB
Minutes Between Performance Archive Data File Rotations: 2
Minutes Between Perfstat Collections: 0
Date Perfstat Collection Will Be Turned off: -
Minutes Between OPM Data File Rotations: 5
Minutes Between ASUP Data File Rotations: 30
Days to Retain ASUP Archive Data: 28
Data ONTAP Version of Installed Configuration: 9.7.0
Enabled Performance Preset Names: asup-hourly, asup_cfg,
copy_offload, css, default,
mcc_performance, nas, opm,
ps, san, statit, sysstat_all,
sysstat_mp, waffinity_stats,
wafl_bufstats, wafl_cpm,
wafl_spinhi, wafl_susp

[Richinydc]

Hi all, 

I am having similar issues with /vol/vol0 filling up. snap list shows no snapshots (I have disabled them for vol0). I am running 9.7 for this simulator and a lot of the commands list above do not show up as available commands ( statistics archive config show and systemshell do not show up). I can't seem to find what logs or files are filling up vol0. I am new to the simulator so, I am clearly missing something. 

Thanks in advance for any advice