Symantec EndPoint Protect, VDI & SnapVault Snapshot size

[diarmuidn]

Greetings all,

We would like to share something with the community to see if we are experiencing something unique or something that is known to others.

We recently moved from using Trend Server Protect to Symantec End Point Protection as our Windows Server Virus Scanning platform with both our VDI and Physical Server environments. Our VDI  environment is built on the following components,

- VMWARE ESX 5.5
- Windows Server 2008 + a small amount of Windows Server 2012.
- Symantec End Point Protection 12.1.5
- DataOntap 7-Mode Versions 8.0.3, 8.1.2, 8.2.2 & Cluster-Mode Version 8.2.2

Points of note are,

DataStores are mounted using different protocols - NFS, FC & iSCSI
We use VSC 5.0 to backup our VMWARE data and SnapVault to move the data offsite.
Deduplication runs daily prior to the VSC job running
VM guest Transient data is hosted within a dedicated Datastore as recommended in NETAPP best practices. Interestingly the volumes hosting transient data are also showing the 200 - 400% increase recorded on the actual Data Hosting volumes.
Impact is seemingly equal regardless of protocol used and Ontap revision ruling our certain known bugs.

Tape backups of Physical Servers have not shown any notable increase.

Our Symantec End Point (SEP) configuration is based on the following principles.

- We keep 90 Virus Definitions. On average 3 updates daily
- Definition updates are delta based as opposed to full (or so I have been informed!)
- Downloads are random and we have no control over when they happen - This means that pattern files downloads can happen post deduplication and pre VSC job instigation.

Since we migrated to SEP we have encountered quite a serious issue - our daily snapvault snapshot sizes have increased by 200 - 400% from where they were before the migration. This is having an impact on the length of time the transfers take causing backup jobs to over run into working hours and it is causing an explosion in storage space consumption. As I am sure you might imagine the scale of this impact is proving rather problematic. For a short period we switched off Pattern File updates and the snapshot sizes reverted the original figures. Once we switched back on pattern file downloads the Snapshot size increase reoccurred.

We have been in touch with both NETAPP Technical Support and Symantec Technical Support both of whom are being helpful. The impression however is that neither vendor has seen an impact from deployment quite like ours. We therefore want to share this experience with the community to see if anyone else has experience anything similar.

Any feedback would be appreciated.

best regards.
D

[hoazem]

Weird issue, wondering no reply so far?

[Uddhav]

send for one typical volume

a) snapshot list

b) snap delta for that volume

c) snapmirror for that volume

d) volume options

e) sis status

[diarmuidn]

Replies are in the attached .txt file.

Note that the issue occurs on all Filers, 7-MODE and C-MODE and began the day the SEP Client was rolled out. We stopped pattern files updates for a short period and the snapshot size went back to normal.  What I am really trying out is whether or not anyone else has ever seen this issue? A 200% - 400% increase in snapshot sizes is hard to miss but I guess changing Virus Scan clients is not that common an activity so we could be unique in that regard.

At this stage we are resigned to the fact that there is little or nothing we can do to reduce this increased overhead from a Storage Perspective. The idea solution would be to have the SEP pattern files located on a volume dedicated to transient data but from what we have been told this is not something that is likely to happen any time soon. Increasing bandwidth and purchasing new Storage is likely to be our only short to medium term option.

[Uddhav]

Yes,  block level file chages inside the volume which comprise of big files,  needs great attention.

You cannot have many VMs or ESXs inside that particular volume.  I have seen the pain ,  people putting many VMs inside the volume level.

On top of that if your SEP pattern files reside in same volume,  it can be trouble.

You can be max out pretty soon.

Dedicating the particular volume is good idea,  also focus on your volume for number of VMs