Volume is full and LUN cannot be accessed

[PANTELIS_KOMATAS]

Dear all

We are facing a serious problem with our Netapp filer FAS2050 with version 7.3.2. A volume is hsown as being full and the size of the volume is 2 TB. The volume contains a LUN that it is 1.5 TB bif but when we run df -h on the filer it shows the volume to be absoletely full with no space left at all. There are no snapshots on the volume to delete.

Since the volume has A-SIS on it can not grow beyond 2 TB. When we bring the LUN online the vmware hosts with ESX 4.0 which access the LUN seem to hung and the only way to remedy this is to offline the volume. So despite that the data is there we do not have any way to get the data out of the lun. Any ides?  We have tries to perform an ndmpcopy to another volume on the ha pair filer but it fails with No spece left on device message. Also we have tried the commnad sis undo volume name but we get an error : Volume is in transition state.

Please help!!!!

[radek_kubka]

blocks that had been processed by deduplication scan are frozen until next scan. This means that if you have "1TB LUN, de-dupe enabled (& run) with no savings" - 1TB of your space is frozen for unknown amount of time

'Frozen' as locked & unavailable for modification? This changes things a little bit. Are you sure about this? I honestly admit that I wasn't aware about this ASIS 'feature', which may be a prefect explanation for fractional reserve behaviour.

[radek_kubka]

I've decided to do my (late) homework, so looked into the TR-3505 (http://media.netapp.com/documents/tr-3505.pdf)

Page 45 onwards:

Configuration A: Default LUN Configuration

1. LUN space reservation value = on

2. Volume fractional reserve value = 100

3. Volume guarantee = volume

[...]

When a LUN containing default values is deduplicated, no apparent savings are observed by the storage administrator because the LUN by default was space reserved when it was created and the fractional reserve was set to 100% in the volume. Any blocks freed through deduplication are allocated to the fractional reserve area. This configuration means that overwrite to the LUN should never fail, even if it is overwritten entirely.

Configuration E: LUN Configuration for Maximum Aggregate Space Savings

1. LUN space reservation value = off

2. Volume fractional reserve value = any value from 0–100

3. Volume guarantee = none

[...]

This configuration "forces" the free blocks out of the volume and into the aggregate free pool, where the blocks can be reprovisioned for any other volumes in the aggregate.

From this thread though, it looks like things doesn't always work like described in Config E above.

[aborzenkov]

From this thread though, it looks like things doesn't always work like described in Config E above.

Bur OP did not have Config E; lun space reservation was enabled as can be seen in FR kicking in. With LUN space reservation off no FR is ever allocated and its value is irrelevant.

[radek_kubka]

Well, it's a long thread, but this is what we could read a handful of posts above (https://communities.netapp.com/message/80531#80531😞

Lun guarante was always off.

This is what makes things really weird...

[aborzenkov]

Are you sure about this?

Now no more It does require more temporary space during extended period of time though ...

simsim*> df tv
Filesystem              kbytes       used      avail capacity  Mounted on
/vol/tv/                 20480      15080       5400      74%  /vol/tv/
bor@opensuse:~> sudo dd if=/dev/urandom of=/dev/sdb1 bs=1M  oflag=direct count=4
4+0 records in
4+0 records out
4194304 bytes (4.2 MB) copied, 3.46915 s, 1.2 MB/s
simsim*> df tv
Filesystem              kbytes       used      avail capacity  Mounted on
/vol/tv/                 20480      16692       3788      82%  /vol/tv/
simsim*> df tv
Filesystem              kbytes       used      avail capacity  Mounted on
/vol/tv/                 20480      19200       1280      94%  /vol/tv/
simsim*> df tv
Filesystem              kbytes       used      avail capacity  Mounted on
/vol/tv/                 20480      15080       5400      74%  /vol/tv/

And if you are unlucky enough ...

bor@opensuse:~> sudo dd if=/dev/urandom of=/dev/sdb1 bs=1M  oflag=direct count=6
dd: writing `/dev/sdb1': Input/output error
6+0 records in
5+0 records out
5242880 bytes (5.2 MB) copied, 6.95034 s, 754 kB/s
simsim*> Tue May  1 13:00:54 MSK [simsim:wafl.vol.full:notice]: file system on volume tv is full
Tue May  1 13:00:54 MSK [simsim:scsitarget.lun.noSpace:error]: LUN '/vol/tv/lv' has run out of space.
Tue May  1 13:00:55 MSK [simsim:lun.offline:warning]: LUN /vol/tv/lv has been taken offline
simsim*> df tv
Filesystem              kbytes       used      avail capacity  Mounted on
/vol/tv/                 20480      15144       5336      74%  /vol/tv/
simsim*> lun show /vol/tv/lv
        /vol/tv/lv                    15m (15728640)      (r/w, offline, mapped)

I do not know how long it takes to release overwritten blocks, but it seems to take significantly more time than in normal case.