Hi guys,
We are seeing a subtle but weird problem with deduplication. It seems that, after some time, deduplication stops happening and, over time, the space savings are gradually lost over time due to the churn on the volume. There are no errors or indications that anything is wrong. It is as if deduplication is not running at all.
sis status -l shows that the deduplication job is running as expected (nightly outside business hours), that it found very large amounts of new data to be processed since the last run, and that it found lots of duplicate blocks. But, without indication, the results of the deduplication operation do not seem to take effect. It's as if it is deciding not to apply the deduplication results at the very last step in the process.
When we first noticed this a while ago, we found that re-running the deduplication process from the top (sis start -sd), rather than relying on the incremental dedupe, cleared the problem and the blocks were correctly deduplicated. But then the deduplication process seemed to mysteriously and silently cease again.
The volumes in both cases are regular NFS volumes hosting virtual machines on VMware and Xen. There are daily snapshots being kept with VSC. We noticed these problems when the volumes were 50% full. The deduplication and snapshot schedules are set up carefully so that the deduplication is complete before the snapshots kick in.
Has anyone else seen this and could provide a suggestion about how we could analyze the cause ?
Brendan
