Active IQ Unified Manager Discussions
Active IQ Unified Manager Discussions
Hi All
I am trying to get an DFM email alert sent when volumes auto-grow. This is because my Production SQL LUNs are in volumes that are set to grow automatically if they reach their pre-defined threshold. If the SQL volumes needs to autogrow then i would like to be alerted, investigate (with the DBAs) why and if required increase the size of the volumes (or decrease the size if the increase is not warranted).
However my attempt to get this alert through has failed so far. If I "test" the alarm i do get a test email, however the SNMP trap does not seem to be sent from the filers to the admin host where DFM resides (which should trigger the email alert). I receive other email alerts for other events just fine.
I have created a test volume and added it to an Alarm in DFM. I have then enabled the "vol autosize" function on this volume and then filled the volume up with data files. When the volume reaches its capacity threshold I can see a message on the filer console that the volumes does indeed autogrow but am not receiving the email alert about it.
STEPS FOLLOWED:
1)Add the DFM server as traphost to filer using snmp traphost add <dfm-ip-address> and set init to "1".
2)Create a volume of size say 2GB on ontap and let dfm discover it.
3)Enable autosize for newly created volume by vol autosize <vol> -m <size> on eg: vol autosize vol_autosize -m 3g -i 100g on
4)Created an e-mail alert(alarm) for the event " volume-autosized" on group of volumes which my test volume is part of.
5)I have written data on this volume to fill it up and trigger autogrow function (successfully).
6)Now on filer console you can see meesages like "JUNE 28 17:06:02 EST [NASHC01: wafl.vol.autoSize.done:info]: Automatic increase size of volume 'test_vol_autogrow' by 102400 kbytes done.."
However as stated i do not receive the email alert from DFM warning me that the volume has grown in size.
Can anyone advise how to get this email alert working? Thanks in advance.
Regards
WH
Hi,
The problem could be DFM is not receiving the SNMP trap for this event.
Can you paste the output of dfm report view snmp-traps-all <host-ip/host-id>?
Regards
Pradeep L
Thanks Pradeep,
Output to "dfm report view snmp-traps-all" listed below....
C:\>dfm report view snmp-traps-all 192.168.22.91
There are no trap generated events.
Hi,
The problem is clear. DFM is not receiving any traps from that particular storage system. Once DFM receives traps, it will generate the event
and an e-mail alert will be sent for the same.
Regards
Pradeep L
Hi - Yes Pradeep,
I had guessed that the traps were not getting through. Can you help to understand why this is? and how to fix it?
Thanks
WH
Hi Pradeep
The Options output is the same as yours…
Kind Regards
Wayne Hapuku
Enterprise Storage Administrator
Information Technology Directorate
Email: Wayne.Hapuku@det.nsw.edu.au<mailto:Wayne.Hapuku@det.nsw.edu.au>
Deskphone: 02 9302 7556
Level 8 (West), 8 Central Avenue
Australian Technology Park
EVELEIGH NSW 2015
Is the trap listener enabled within DFM?
dfm option list snmpTrapListenerEnabled
Also check the following command in the filer.
sim1> snmp
contact:
location:
Lab-DC1
authtrap:
0
init:
1
traphosts:
192.168.98.10 (192.168.98.10) <192.168.98.10>
community:
ro public
sim1>
Make sure your DFM server is under traphosts and init is set to 1.
Hi Adai,
As stipulated earlier I have added the DFM server as traphost and init is set to 1 (but here is the output anyway).
Filer1> snmp
contact:
location:
St Leonards
authtrap:
0
init:
1
traphosts:
192.168.86.39 (uppstxxxxxx.xxx.mgmt.det) <192.168.86.39>
community:
ro ingarlic
thanks for your reply though.
Hi WH,
This is what I did.
My snmp setting on the filer.
sim1> snmp
contact:
location:
lab-DC
authtrap:
0
init:
1
traphosts:
192.168.98.10 (192.168.98.10) <192.168.98.10>
community:
ro public
sim1>
My DFM settings
C:\>dfm options list snmpTrapListenerEnabled
Option Value
----------------------- ------------------------------
snmpTrapListenerEnabled Yes
C:\>
Then created a volume in the filer.
sim1> vol create auto_grow_test -s none aggr1 25m
Creation of volume 'auto_grow_test' with size 25m on containing aggregate
'aggr1' has completed.
sim1>
Turned on autosize and below are its details.
sim1> vol autosize auto_grow_test
Volume autosize is currently ON for volume 'auto_grow_test'.
The volume is set to grow to a maximum of 40 MB, in increments of 5 MB.
sim1>
Discovered that volume in dfm, forcefully instead of waiting by issuing the below cli.
dfm host discover <filerid/name>
Then kept writing to the volume such that it will autogrow.
Below are the messages from the filer.
sim1*> df -h auto_grow_test
Filesystem total used avail capacity Mounted on
/vol/auto_grow_test/ 26MB 21MB 4768KB 82% /vol/auto_grow_test/
/vol/auto_grow_test/.snapshot 6756KB 0KB 6756KB 0% /vol/auto_grow_test/.snapshot
sim1*>
sim1*> Tue Jul 26 13:23:07 GMT [wafl.vol.autoSize.done:info]: Automatic increase size of volume 'auto_grow_test' by 5120 kbytes done.
sim1*> Tue Jul 26 13:23:17 GMT [wafl.vol.autoSize.done:info]: Automatic increase size of volume 'auto_grow_test' by 2048 kbytes done.
sim1*> Tue Jul 26 13:23:30 GMT [wafl.vol.autoSize.fail:info]: Unable to grow volume 'auto_grow_test' to recover space: Volume cannot be grown beyond maximum growth limit
Tue Jul 26 13:24:00 GMT [monitor.globalStatus.nonCritical:warning]: /vol/auto_grow_test is full (using or reserving 100% of space and 9% of inodes, using 100% of reserve).
Below is the report in DFM where I see the auto-size event.
C:\>dfm report view events-history 228
Severity Event ID Event Triggered Ack'ed By Ack'ed Del
----------- -------- -------------------------------------------------------------- ------------ --------- ------------ ---
Error 325 Volume Full 26 Jul 13:20
Information 324 Volume Autosized 26 Jul 13:20
Information 322 Volume Autosized 26 Jul 13:20
Information 319 Volume Autosized 26 Jul 13:18
Warning 317 Volume Almost Full 26 Jul 13:18
Information 315 Volume Autosized 26 Jul 13:18
Normal 313 No Schedule Conflict between snapshot and snapvault schedules 26 Jul 13:18
Normal 312 No Schedule Conflict between snapshot and SnapMirror schedules 26 Jul 13:18
Normal 311 Snapshots Age: Normal 26 Jul 13:18
Normal 310 Snapshots Count: Normal 26 Jul 13:18
Information 309 Volume Autosized 26 Jul 13:18
Information 307 Volume Autosized 26 Jul 13:18
Normal 304 Volume Space Reserve OK 26 Jul 13:17
Normal 303 Volume Next Snapshot Possible 26 Jul 13:17
Normal 302 Volume First Snapshot OK 26 Jul 13:17
Normal 301 Inodes Utilization Normal 26 Jul 13:17
Normal 300 Volume Space Normal 26 Jul 13:17
Normal 299 Scheduled Snapshots Enabled 26 Jul 13:17
Normal 298 Volume Online 26 Jul 13:17
C:\>
Where 228 is the volume id in dfm.
Hope this helps.
Regards
adai
HI Adai,
Here is the output to your queries….
Also the SNMP trap settings on DFM server..
Kind Regards
Wayne Hapuku
Enterprise Storage Administrator
Information Technology Directorate
Email: Wayne.Hapuku@det.nsw.edu.au<mailto:Wayne.Hapuku@det.nsw.edu.au>
Deskphone: 02 9302 7556
Level 8 (West), 8 Central Avenue
Australian Technology Park
EVELEIGH NSW 2015
Thanks richardsopp
Output to "dfm option list snmpTrapListenerEnabled"
C:\>dfm option list snmpTrapListenerEnabled
Option Value
----------------------- ------------------------------
snmpTrapListenerEnabled Yes
Regards WH
Thanks guys,
Today is my last day at my present employer so will not be contactable at this email addy. The problem is unresolved but is now someone elses issue.
Thanks for your help so far.
WH
I'm actually having the same problem as Wayne, so I've been watching and following this thread.
I originally did not have the snmp configuration set appropriately on my filers. I did not have the traphost set to send traps to the DFM server as I did not realize this was a requirement. So I've updated one of my filers with:
snmp init 1
snmp traphost add <dfmserveraddress>
I have an alarm configured to send me email upon a Volume Autosize event and a Host ColdStart Event. On that filer I have filled a test volume such that an autogrow process happened. I can see that in the messages file on the filer. But I never received the event on the DFM server.
I have installed Wireshark on the DFM server (it's running on Windows) and I sniffed for traffic on port 162. On the filer I issued the command:
snmp init 1
which is supposed to send a coldStart trap to the traphost. In wireshark I see the trap come across the wire. But the DFM Server does not show a trap received:
C:\Windows\system32>dfm options list snmpTrapListenerEnabled
Option Value
----------------------- ------------------------------
snmpTrapListenerEnabled Yes
C:\Windows\system32>dfm report view snmp-traps-all
There are no trap generated events.
Can anyone provide some pointers on what else I should be looking for?
Thanks!
Jason
Hi Jason,
1)Can you confirm that snmp service is running on the windows machine in services.msc?
2)Also, Can you confirm that port 162 is not used by any other processes?
Type netstat –ano, and ensure no process is using port 162.
Regards
Pradeep L
Pradeep,
Thank you for your response.
The Windows SNMP service is configured and running on our DFM server. Here is the output of "netstat -anop udp":
C:\Windows\system32>netstat -anop udp
Active Connections
Proto Local Address Foreign Address State PID
UDP 0.0.0.0:123 *:* 820
UDP 0.0.0.0:161 *:* 2456
UDP 0.0.0.0:162 *:* 2152
UDP 0.0.0.0:500 *:* 764
UDP 0.0.0.0:4500 *:* 764
UDP 0.0.0.0:5355 *:* 912
UDP 10.1.25.148:137 *:* 4
UDP 10.1.25.148:138 *:* 4
UDP 127.0.0.1:51649 *:* 912
UDP 127.0.0.1:57373 *:* 456
UDP 127.0.0.1:61736 *:* 764
UDP 127.0.0.1:62537 *:* 1760
where PID 2456 is snmp.exe as provided by Windows and PID 2152 is dfmserver.exe *32
Here is a netstat output showing one and only one service listening on port 162:
C:\Windows\system32>netstat -ano | findstr 162
UDP 0.0.0.0:162 *:* 2152
My traffic sniffing indicates that the SNMP Trap from my filer is version 1. The Network Credentials for my filers on the DFM server were configured for SNMPv3. I've added a specific entry for the filer's IP address (as indicated in the network capture) for SNMPv1 with the proper community string hoping that would help but so far it has not:
C:\Windows\system32>dfm report view snmp-traps-all
There are no trap generated events.
Thanks!
Jason
Hi Jason,
Do a dfm service stop server and dfm service start server.
In dfmserver.log ,can you see messages like this?
Aug 01 15:23:17 [dfmserver: INFO]: [18406:0xb7f9d6c0]: Starting Event Poller thread.
Aug 01 15:23:17 [dfmserver: INFO]: [18406:0x1edbb90]: snmp port = 162
Aug 01 15:23:17 [dfmserver: INFO]: [18406:0x1edbb90]: Starting snmpTrapListener thread...
Hi Pradeep,
Indeed I do see those lines:
Aug 01 12:59:40 [dfmserver: INFO]: [2172:0x820]: Starting Event Poller thread.
Aug 01 12:59:40 [dfmserver: INFO]: [2172:0xe18]: snmp port = 162
Aug 01 12:59:40 [dfmserver: INFO]: [2172:0xe18]: Starting snmpTrapListener thread...
Interestingly enough, when I opened the log file dfmserver.log, I see many many lines such as the following:
Aug 01 12:32:33 [dfmserver: WARN]: [2152:0x364]: Filtering trap from '10.1.25.180' as there are multiple interfaces with this IP.
Aug 01 12:32:34 [dfmserver: WARN]: [2152:0x364]: Filtering trap from '10.1.25.180' as there are multiple interfaces with this IP.
Aug 01 12:32:36 [dfmserver: WARN]: [2152:0x364]: Filtering trap from '10.1.25.180' as there are multiple interfaces with this IP.
Aug 01 12:32:38 [dfmserver: WARN]: [2152:0x364]: Filtering trap from '10.1.25.180' as there are multiple interfaces with this IP.
When I was running Wireshark, the traps the DFM server was receiving were being sourced from an IP address ona VIF on my filer. I was surprised by that. I had assumed that the trap would be sourced from the management interface on the filer. I assumed that the filer chose that VIF and that IP address because it's the same subnet as the DFM server. The DFM server has the filer listed under it's management IP address.
Thanks!
Jason
After working with NetApp support, I have everything working in my system.
The bottom line is that because my filer has an interface on the same network as my DFM server it chooses to use that interface to submit the SNMP Traps.
The filer in DFM is configured with a differenet IP Address. The host entry in DFM for this filer was configured with its e0m interface which is on a different network than the DFM server.
We modified the primary address of the host entry on the DFM server for this filer. We changed the primary address from the e0m interface address to the VIF interface address which lives on the same network as the DFM server (10.1.25.0/24 as shown above). Once we did that, the traps started being processed on the DFM server and alarms were generated.
Thanks for all of the help!
Jason
Sorry to resurrect an old thread here but same issue experienced and none of the suggestions posted here are applicable. The volume-autosize event just never triggers in DFM.
Filer SNMP settings
FILER> snmp
contact:
location:
*** House
authtrap:
0
init:
1
traphosts:
10.10.0.44 (hostname.domain.com) <10.10.0.44>
community:
ro public
Autogrow Test
FILER> Wed Jul 4 16:22:59 BST [FILER: wafl.vol.autoSize.done:info]: Automatic increase size of volume 'Test_Autogrow' by 26214400 kbytes done.
DFM
C:\>dfm option list snmpTrapListenerEnabled
Option Value
----------------------- ------------------------------
snmpTrapListenerEnabled Yes
C:\>dfm report view snmp-traps-all <filer-IP>
There are no trap generated events.
C:\>dfm report view events-history 53715
Severity Event ID Event Triggered Ack'ed By Ack'ed Deleted By Deleted Source ID Source
Normal 867077 Volume Growth Rate OK 04 Jul 16:08 53715 FILER:/Test_Autogrow
Warning 866773 Volume Growth Rate Abnormal 04 Jul 14:07 53715 FILER:/Test_Autogrow
Normal 254649 Volume Growth Rate OK 28 Jan 16:35 53715 FILER:/Test_Autogrow
Information 238134 Scheduled Snapshots Disabled 20 Jan 17:01 53715 FILER:/Test_Autogrow
Normal 238133 Volume Space Normal 20 Jan 17:01 53715 FILER:/Test_Autogrow
DFM host list also displays the correct IP address.
Any suggestions? Thanks!