Performance issue when provisioning qtrees in a dataset

IHAC where we have some serious performance issues when provisioning new qtrees in a dataset.

The DFM server is on 4 vCPU VM with 8 GB of memory. There is no contention on the ESX server.

DB size is about 100 MB

I tried to grow dbCache to 1 GB without no change in behaviour. I disabled PerfAdvisor with still no change in behaviour.

Web GUI and NMC GUI are very responsive and CPU is OK most of the time.

The DFM server monitors only 4 controllers. At the current time, there is only 2 vfilers, one only active. The active vfiler contains 16 data volumes (containing 25 qtrees each) managed in a dataset.

The dataset follows a DR Mirror then Backup Protection Policy without a single issue.

All snapmirrors are done once a day in less than 2 minutes

All snapvaults are done once a day in less than 15 minutes.

The only problem we have is when trying to provision a new qtree (or sometimes when changing the quota of an existing one).

When using the NMC GUI, it times out after some time and does nothing.

When using a dfpm CLI command, it works but it takes ages.

What I see in the logs: A Conformance Dry Run occurs immediately but nothing happens for 7-10 minutes, then the provisioning job is launched and done in a few seconds (vol creation, qtree creation, share creation, then creation of protection volumes when necessary). Then we have to wait 5-10 more minutes before the CLI is available again.

We don't see that on another dataset .

I think we will try to add a new dataset but it will add complexity while there is no need for a new one to manage snapmirror and snapvaults since they alway complete in a few minutes.

My question:

- is this behaviour normal or not since it is the only performance issue we with OnCommand and CPU is not full at all ?

- is there a way to at least minimize the time taken to edit existing qtrees ?

The dfm diag is attached.

Here is an extract from the conformance log show what happens when provisioning a new qtree (15 minutes):

Apr 19 12:14:27 [dfmserver: INFO]: [2576:0x1604]: Dataset 'xxxx_Data' (1046) is busy in check and reserve run as conformance tasks are in progress. Deferred conformance run scheduled as username 'NT AUTHORITY\SYSTEM' with user confirmation set to 1.