Hi all,
I have opened this discussion to request your feedback about a performance issue that we have observer on our production environment. I will try to give to you as must information that I know, sorry if the question is too long.
We have bought a new (2) NetApp FAS3240 to replace out old EMC AX4-5. On one of the controller (1) we have created an aggregate with 2 raid groups of 12 disk (2,5" 10k rpm) in each diskgroup with RAID-DP. On the other controller (2) we have created an aggregate with 2 raid groups of 20 disk (2,5" 10k rpm). Both controller are running NetApp Release 8.0.2P6 7-Mode. Servers access to the filer through FC protocol.
On the aggregate 1 of the "Controller 1" we have create several volume an inside of each volume we have created several LUNs following the NetApp best practices. Nowadays this aggregate is dedicated to run a Oracle RAC database 11.2.0.2 with two nodes, one instance per node. Oracle datafiles are on a volume 1.1 Tbytes with 10 LUNs of 100Gbytes each one. Volumen created without snapshot and thin provision.
After the migration to the new storage, at database level, we have observe an increase in the average latency response time. The following example correspond with our load per week with both storages comparing the week before and after migration:
EMC:
Avg
%Time Total Wait wait Waits % DB
Event Waits -outs Time (s) (ms) /txn time
-------------------------- ------------ ----- ---------- ------- -------- ------
db file sequential read 55,997,027 0 460,973 8 4.9 18.5
db file scattered read 10,712,448 0 107,856 10 0.9 4.3
NetApp
Avg
%Time Total Wait wait Waits % DB
Event Waits -outs Time (s) (ms) /txn time
-------------------------- ------------ ----- ---------- ------- -------- ------
db file sequential read 54,807,814 0 662,336 12 4.5 17.0
db file scattered read 11,615,111 0 218,657 19 1.0 5.6
First approach to try to improve the performance was to enable the Read Reallocate function for the volumen used by Oracle datafiles because in two weeks the threshold reach the value 4. With this change we have the threshold lower than 3.
With the information gathered with NetApp Management Console we have identified that we have latencies bigger than 100 ms on some period of time as you can see on the attached graph
Oracle Export Data Pump utility appear to be the root cause of this behavior when it execute a export of our biggest tables (30 Gbytes) using direct path way using two parallel process. This process export his biggest tables between 1:30-2:00, if we remove this process we got an improvement on the latencies, as you can see:
We have detected the same performance issue when executing an index rebuild on our biggest tables with a degree of 4 or moving this segment from one tablespace to another.
On the aggregate 1 of the "Controller 1" we have create a volumen of 1.1 Tbytes with 10 LUNs of 100Gbytes each one to create a Oracle ASM Cluster file system (ACFS). On this volume we write the expdp file output. We have a slow performance also in this volume as you can see on the graphs:
Latency:
Throughput
On "controller 2" on the same a aggregate we have created several volume for deploy a SQL Server 2012 Enterprise Edition used by our BI team
Latency
Throughput
My feeling is that the storage has a poor performance when Oracle execute multiblock IO operations, Oracle could execute bigger (in size) multiblock IO operations than SQL Server because of that the issue only happen with Oracle.
Please, Any one running Oracle with a similar environment has detected any similar issue or could give me any feedback about how can I solve this issue.
Thank you in advance, regards
Public