Active IQ Unified Manager Discussions

DFM (OPs manager) volume-autosized email alert.

waynehapu
17,376 Views

Hi All

I am trying to get an DFM email alert sent when volumes auto-grow. This is because my Production SQL LUNs are in volumes that are set to grow automatically if they reach their pre-defined threshold. If the SQL volumes needs to autogrow then i would like to be alerted, investigate (with the DBAs) why and if required increase the size of the volumes (or decrease the size if the increase is not warranted).


However my attempt to get this alert through has failed so far. If I "test" the alarm i do get a test email, however the SNMP trap does not seem to be sent from the filers to the admin host where DFM resides (which should trigger the email alert). I receive other email alerts for other events just fine.


I have created a test volume and added it to an Alarm in DFM. I have then enabled the "vol autosize" function on this volume and then filled the volume up with data files. When the volume reaches its capacity threshold I can see a message on the filer console that the volumes does indeed autogrow but am not receiving the email alert about it.

STEPS FOLLOWED:
1)Add the DFM server as traphost to filer using snmp traphost add <dfm-ip-address> and set init to "1".

2)Create a volume of size say 2GB on ontap and let dfm discover it.

3)Enable autosize for newly created volume by vol autosize <vol> -m <size> on eg: vol autosize vol_autosize -m 3g -i 100g on

4)Created an e-mail alert(alarm) for the event " volume-autosized" on group of volumes which my test volume is part of.

5)I have written data on this volume to fill it up and trigger autogrow function (successfully).

6)Now on filer console you can see meesages like "JUNE 28 17:06:02 EST [NASHC01: wafl.vol.autoSize.done:info]: Automatic increase size of volume 'test_vol_autogrow' by 102400 kbytes done.."

However as stated i do not receive the email alert from DFM warning me that the volume has grown in size.


Can anyone advise how to get this email alert working? Thanks in advance.

Regards
WH

20 REPLIES 20

pradeepl
17,293 Views

Hi,

The problem could be DFM is not receiving the SNMP trap for this event.

Can you paste the output of dfm report view snmp-traps-all <host-ip/host-id>?

Regards

Pradeep L

waynehapu
17,293 Views

Thanks Pradeep,

Output to "dfm report view snmp-traps-all" listed below....

C:\>dfm report view snmp-traps-all 192.168.22.91

There are no trap generated events.

pradeepl
17,293 Views

Hi,

The problem is clear. DFM is not receiving any traps from that particular storage system. Once DFM receives traps, it will generate the event

and an e-mail alert will be sent for the same.

Regards

Pradeep L

waynehapu
17,294 Views

Hi - Yes Pradeep,

I had guessed that the traps were not getting through. Can you help to understand why this is? and how to fix it?

Thanks

WH

pradeepl
16,239 Views

Hi,

Can you check that port no 162 which iss used by dfm by default for getting traps is not used by any other application/process?

# dfm options list |grep -i snmptrap

snmpTrapListenerEnabled Yes

snmpTrapListenerPort 162

snmpTrapRcvdMaxPerWindow 250

snmpTrapRcvdWindowSize 5 minutes

Regards

Pradeep L

waynehapu
16,242 Views

Hi Pradeep

The Options output is the same as yours…

Kind Regards

Wayne Hapuku

Enterprise Storage Administrator

Information Technology Directorate

Email: Wayne.Hapuku@det.nsw.edu.au<mailto:Wayne.Hapuku@det.nsw.edu.au>

Deskphone: 02 9302 7556

Level 8 (West), 8 Central Avenue

Australian Technology Park

EVELEIGH NSW 2015

www.det.nsw.edu.au<http://www.det.nsw.edu.au/

RichardSopp
17,292 Views

Is the trap listener enabled within DFM?

dfm option list snmpTrapListenerEnabled

adaikkap
17,292 Views

Also check the following command in the filer.

sim1> snmp

contact:

location:

        Lab-DC1

authtrap:

        0

init:

        1

traphosts:

        192.168.98.10 (192.168.98.10) <192.168.98.10>

community:

        ro public

sim1>

Make sure your DFM server is under traphosts and init is set to 1.

waynehapu
17,292 Views

Hi Adai,

As stipulated earlier I have added the DFM server as traphost and init is set to 1 (but here is the output anyway).

Filer1> snmp

contact:

       

location:

        St Leonards

authtrap:

        0

init:

        1

traphosts:

        192.168.86.39 (uppstxxxxxx.xxx.mgmt.det) <192.168.86.39>

community:

        ro ingarlic

thanks for your reply though.

adaikkap
17,292 Views

Hi WH,

             This is what I did.

My snmp setting on the filer.

sim1> snmp

contact:

location:

        lab-DC

authtrap:

        0

init:

        1

traphosts:

        192.168.98.10 (192.168.98.10) <192.168.98.10>

community:

        ro public

sim1>

My DFM settings

C:\>dfm options list snmpTrapListenerEnabled

Option                  Value

----------------------- ------------------------------

snmpTrapListenerEnabled Yes

C:\>

Then created a volume in the filer.

sim1> vol create auto_grow_test -s none aggr1 25m

Creation of volume 'auto_grow_test' with size 25m on containing aggregate

'aggr1' has completed.

sim1>

Turned on autosize and below are its details.

sim1> vol autosize auto_grow_test

Volume autosize is currently ON for volume 'auto_grow_test'.

The volume is set to grow to a maximum of 40 MB, in increments of 5 MB.

sim1>

Discovered that volume in dfm, forcefully instead of waiting by issuing the below cli.

dfm host discover <filerid/name>

Then kept writing to the volume such that it will autogrow.

Below are the messages from the filer.

sim1*> df -h auto_grow_test

Filesystem               total       used      avail capacity  Mounted on

/vol/auto_grow_test/       26MB       21MB     4768KB      82%  /vol/auto_grow_test/

/vol/auto_grow_test/.snapshot     6756KB        0KB     6756KB       0%  /vol/auto_grow_test/.snapshot

sim1*>

sim1*> Tue Jul 26 13:23:07 GMT [wafl.vol.autoSize.done:info]: Automatic increase size of volume 'auto_grow_test' by 5120 kbytes done.

sim1*> Tue Jul 26 13:23:17 GMT [wafl.vol.autoSize.done:info]: Automatic increase size of volume 'auto_grow_test' by 2048 kbytes done.

sim1*> Tue Jul 26 13:23:30 GMT [wafl.vol.autoSize.fail:info]: Unable to grow volume 'auto_grow_test' to recover space: Volume cannot be grown beyond maximum growth limit

Tue Jul 26 13:24:00 GMT [monitor.globalStatus.nonCritical:warning]: /vol/auto_grow_test is full (using or reserving 100% of space and 9% of inodes, using 100% of reserve).

Below is the report in DFM where I see the auto-size event.

C:\>dfm report view events-history 228

Severity    Event ID Event                                                          Triggered    Ack'ed By Ack'ed       Del

----------- -------- -------------------------------------------------------------- ------------ --------- ------------ ---

Error       325      Volume Full                                                    26 Jul 13:20

Information 324      Volume Autosized                                               26 Jul 13:20

Information 322      Volume Autosized                                               26 Jul 13:20

Information 319      Volume Autosized                                               26 Jul 13:18

Warning     317      Volume Almost Full                                             26 Jul 13:18

Information 315      Volume Autosized                                               26 Jul 13:18

Normal      313      No Schedule Conflict between snapshot and snapvault schedules  26 Jul 13:18

Normal      312      No Schedule Conflict between snapshot and SnapMirror schedules 26 Jul 13:18

Normal      311      Snapshots Age: Normal                                          26 Jul 13:18

Normal      310      Snapshots Count: Normal                                        26 Jul 13:18

Information 309      Volume Autosized                                               26 Jul 13:18

Information 307      Volume Autosized                                               26 Jul 13:18

Normal      304      Volume Space Reserve OK                                        26 Jul 13:17

Normal      303      Volume Next Snapshot Possible                                  26 Jul 13:17

Normal      302      Volume First Snapshot OK                                       26 Jul 13:17

Normal      301      Inodes Utilization Normal                                      26 Jul 13:17

Normal      300      Volume Space Normal                                            26 Jul 13:17

Normal      299      Scheduled Snapshots Enabled                                    26 Jul 13:17

Normal      298      Volume Online                                                  26 Jul 13:17

C:\>

Where 228 is the volume id in dfm.

Hope this helps.

Regards

adai

waynehapu
16,238 Views

HI Adai,

Here is the output to your queries….

Also the SNMP trap settings on DFM server..

Kind Regards

Wayne Hapuku

Enterprise Storage Administrator

Information Technology Directorate

Email: Wayne.Hapuku@det.nsw.edu.au<mailto:Wayne.Hapuku@det.nsw.edu.au>

Deskphone: 02 9302 7556

Level 8 (West), 8 Central Avenue

Australian Technology Park

EVELEIGH NSW 2015

www.det.nsw.edu.au<http://www.det.nsw.edu.au/

waynehapu
17,292 Views

Thanks richardsopp

Output to "dfm option list snmpTrapListenerEnabled"

C:\>dfm option list snmpTrapListenerEnabled
Option                  Value
----------------------- ------------------------------
snmpTrapListenerEnabled Yes

Regards WH

waynehapu
16,237 Views

Thanks guys,

Today is my last day at my present employer so will not be contactable at this email addy. The problem is unresolved but is now someone elses issue.

Thanks for your help so far.

WH

jason_lempka
16,237 Views

I'm actually having the same problem as Wayne, so I've been watching and following this thread.

I originally did not have the snmp configuration set appropriately on my filers.  I did not have the traphost set to send traps to the DFM server as I did not realize this was a requirement.  So I've updated one of my filers with:

snmp init 1

snmp traphost add <dfmserveraddress>

I have an alarm configured to send me email upon a Volume Autosize event and a Host ColdStart Event.  On that filer I have filled a test volume such that an autogrow process happened.  I can see that in the messages file on the filer.  But I never received the event on the DFM server.

I have installed Wireshark on the DFM server (it's running on Windows) and I sniffed for traffic on port 162.  On the filer I issued the command:

snmp init 1

which is supposed to send a coldStart trap to the traphost.  In wireshark I see the trap come across the wire.  But the DFM Server does not show a trap received:

C:\Windows\system32>dfm options list snmpTrapListenerEnabled

Option                  Value

----------------------- ------------------------------

snmpTrapListenerEnabled Yes

C:\Windows\system32>dfm report view snmp-traps-all

There are no trap generated events.

Can anyone provide some pointers on what else I should be looking for?

Thanks!

Jason

pradeepl
16,237 Views

Hi Jason,

1)Can you confirm that snmp service is running on the windows machine in services.msc?

2)Also, Can you confirm that port 162 is not used by any other processes?

Type netstat –ano, and ensure no process is using port 162.

Regards

Pradeep L

jason_lempka
11,959 Views

Pradeep,

Thank you for your response.

The Windows SNMP service is configured and running on our DFM server.  Here is the output of "netstat -anop udp":

C:\Windows\system32>netstat -anop udp

Active Connections

  Proto  Local Address          Foreign Address        State           PID

  UDP    0.0.0.0:123            *:*                                    820

  UDP    0.0.0.0:161            *:*                                    2456

  UDP    0.0.0.0:162            *:*                                    2152

  UDP    0.0.0.0:500            *:*                                    764

  UDP    0.0.0.0:4500           *:*                                    764

  UDP    0.0.0.0:5355           *:*                                    912

  UDP    10.1.25.148:137        *:*                                    4

  UDP    10.1.25.148:138        *:*                                    4

  UDP    127.0.0.1:51649        *:*                                    912

  UDP    127.0.0.1:57373        *:*                                    456

  UDP    127.0.0.1:61736        *:*                                    764

  UDP    127.0.0.1:62537        *:*                                    1760

where PID 2456 is snmp.exe as provided by Windows and PID 2152 is dfmserver.exe *32

Here is a netstat output showing one and only one service listening on port 162:

C:\Windows\system32>netstat -ano | findstr 162

  UDP    0.0.0.0:162            *:*                                    2152

My traffic sniffing indicates that the SNMP Trap from my filer is version 1.  The Network Credentials for my filers on the DFM server were configured for SNMPv3.  I've added a specific entry for the filer's IP address (as indicated in the network capture) for SNMPv1 with the proper community string hoping that would help but so far it has not:

C:\Windows\system32>dfm report view snmp-traps-all

There are no trap generated events.

Thanks!

Jason

pradeepl
11,959 Views

Hi Jason,

Do a dfm service stop server and dfm service start server.

In dfmserver.log ,can you see messages like this?

Aug 01 15:23:17 [dfmserver: INFO]: [18406:0xb7f9d6c0]: Starting Event Poller thread.

Aug 01 15:23:17 [dfmserver: INFO]: [18406:0x1edbb90]: snmp port = 162

Aug 01 15:23:17 [dfmserver: INFO]: [18406:0x1edbb90]: Starting snmpTrapListener thread...

jason_lempka
11,959 Views

Hi Pradeep,

Indeed I do see those lines:

Aug 01 12:59:40 [dfmserver: INFO]: [2172:0x820]: Starting Event Poller thread.

Aug 01 12:59:40 [dfmserver: INFO]: [2172:0xe18]: snmp port = 162

Aug 01 12:59:40 [dfmserver: INFO]: [2172:0xe18]: Starting snmpTrapListener thread...

Interestingly enough, when I opened the log file dfmserver.log, I see many many lines such as the following:

Aug 01 12:32:33 [dfmserver: WARN]: [2152:0x364]: Filtering trap from '10.1.25.180' as there are multiple interfaces with this IP.

Aug 01 12:32:34 [dfmserver: WARN]: [2152:0x364]: Filtering trap from '10.1.25.180' as there are multiple interfaces with this IP.

Aug 01 12:32:36 [dfmserver: WARN]: [2152:0x364]: Filtering trap from '10.1.25.180' as there are multiple interfaces with this IP.

Aug 01 12:32:38 [dfmserver: WARN]: [2152:0x364]: Filtering trap from '10.1.25.180' as there are multiple interfaces with this IP.

When I was running Wireshark, the traps the DFM server was receiving were being sourced from an IP address ona  VIF on my filer.  I was surprised by that.  I had assumed that the trap would be sourced from the management interface on the filer.  I assumed that the filer chose that VIF and that IP address because it's the same subnet as the DFM server.  The DFM server has the filer listed under it's management IP address.

Thanks!

Jason

jason_lempka
11,959 Views

After working with NetApp support, I have everything working in my system.

The bottom line is that because my filer has an interface on the same network as my DFM server it chooses to use that interface to submit the SNMP Traps.
The filer in DFM is configured with a differenet IP Address. The host entry in DFM for this filer was configured with its e0m interface which is on a different network than the DFM server.

We modified the primary address of the host entry on the DFM server for this filer.  We changed the primary address from the e0m interface address to the VIF interface address which lives on the same network as the DFM server (10.1.25.0/24 as shown above).  Once we did that, the traps started being processed on the DFM server and alarms were generated.

Thanks for all of the help!

Jason

GRAEMEOGDEN
11,351 Views

Sorry to resurrect an old thread here but same issue experienced and none of the suggestions posted here are applicable. The volume-autosize event just never triggers in DFM.

Filer SNMP settings

FILER> snmp

contact:

location:

        *** House

authtrap:

        0

init:

        1

traphosts:

        10.10.0.44 (hostname.domain.com) <10.10.0.44>

community:

        ro public

Autogrow Test

FILER> Wed Jul  4 16:22:59 BST [FILER: wafl.vol.autoSize.done:info]: Automatic increase size of volume 'Test_Autogrow' by 26214400 kbytes done.

DFM

C:\>dfm option list snmpTrapListenerEnabled

Option                  Value

----------------------- ------------------------------

snmpTrapListenerEnabled Yes

C:\>dfm report view snmp-traps-all <filer-IP>

There are no trap generated events.

C:\>dfm report view events-history 53715

Severity    Event ID Event    Triggered    Ack'ed By Ack'ed       Deleted By Deleted      Source ID Source

Normal      867077   Volume Growth Rate OK    04 Jul 16:08                                      53715     FILER:/Test_Autogrow

Warning     866773   Volume Growth Rate Abnormal    04 Jul 14:07                            53715     FILER:/Test_Autogrow

Normal      254649   Volume Growth Rate OK    28 Jan 16:35                                     53715     FILER:/Test_Autogrow

Information 238134   Scheduled Snapshots Disabled    20 Jan 17:01                           53715     FILER:/Test_Autogrow

Normal      238133   Volume Space Normal    20 Jan 17:01                                        53715     FILER:/Test_Autogrow

DFM host list also displays the correct IP address.

Any suggestions? Thanks!

Public