Weird read / write performance of XFS on NetApp LUN

[PhilDai1]

Hi all,

IHAC met a weird performance issue with XFS:

• Mapped a LUN from a FAS8300 (ONTAP 9.7) to a CentOS 7.8 system
• Aggregate consist of  56 NL-SAS disks
• tested the throughput of LUN ( on RAW device), both read and write throughput > 500MB/s
• Formated that LUN and create a XFS file system and used FIO to test the throughput of file system
  • Write throughput > 500MB/s
  • Read throughput < 100MB/s -- weird

Any thoughts and suggestions?  Thanks in advance!

Best Regards,

Phil

[TMACMD]

I’ll start. How is the lun attached?

 Iscsi, fc, nvme/tcp or nvme/fc?

 How many lifs per node?

[PhilDai1]

Hi TMACMD, 

Thanks for your respond!

It is FC protocol.  2 lifs per node.

Thanks!

-Phil

[TMACMD]

And what was the FIO command/args used for testing?

 Did you try formatting the lun with other formats and test besides XFS?

 It could be some kind of caching issue

[PhilDai1]

Hi TMACMD,

Command was like: 

fio -directory=/test02 -direct=1 -rw=write -bs=128k -size=15G -numjobs=16 -runtime=300 -group_reporting -name=test-write

Single LUN on a volume, was mapped to the Linux Server.

On the Linux Server, they did not create partition or use LVM.

We will try: 

1. create a partition before makes

2. try ext4 file system for comparision

Thanks!

-Phil

[PhilDai1]

Customer has done tests today:

• create partition before make filesystem -- just follow a NetApp KB
• Added ext4 for comparison 
• Used SSD LUN for comparison

Please find results as below (block size 128K):

• No big difference between XFS and Ext4
• It is interesting that read throughput (on XFS) increased to 139MB/s when customer did the read test 2nd time, compared to 75MB/s at the 1st time
• In contrast, read throughput was close to write when customer did the same test on SSD LUN

Any thoughts?