Greetings! I am new to iSCSI and I just installed a new FAS2050 configured with dual filers and all 15K SAS drives. It's configured for Active/Active and the disks are split evenly between the 2 filers. I created a 300 gig volume for VMWare on the first filter (fas2050a).
The clients are ESX4 servers, 2 new HP DL385G6 machines and 2 HP DL380G6 machines.
The switches dedicated to iSCSI are 2 ProCurve 2910AL-24G with 10gbit link between them.
Here are the relevant lines from the switch config.
interface 1
name "FAS2050a_e0a"
flow-control
exit
interface 2
name "FAS2050a_e0b"
flow-control
exit
trunk 1-2 Trk3 LACP
spanning-tree Trk3 priority 4
The DL385s are configured with 8 gbit NIC ports. The DL380s only have 2 (will upgrade later).
The filer fas2050a has both nics connected to one of the switches and are configured as virtual interface vif0 for lacp. Within that virtual inteface I created a virtual interface for vlan 100 and have the switch ports trunked for LACP tagged vlan 100. All interfaces that will be used for iSCSI are setup for an MTU size of 9000. vif0-2 is for our normal "server" vlan segmant and has iSCSI disabled.
fas2050a> ifconfig -a
e0a: flags=80908043<BROADCAST,RUNNING,MULTICAST,TCPCKSUM,VLAN> mtu 9000
ether 02:a0:98:12:b4:f4 (auto-1000t-fd-up) flowcontrol full
trunked vif0
e0b: flags=80908043<BROADCAST,RUNNING,MULTICAST,TCPCKSUM,VLAN> mtu 9000
ether 02:a0:98:12:b4:f4 (auto-1000t-fd-up) flowcontrol full
trunked vif0
lo: flags=1948049<UP,LOOPBACK,RUNNING,MULTICAST,TCPCKSUM> mtu 8160
inet 127.0.0.1 netmask 0xff000000 broadcast 127.0.0.1
ether 00:00:00:00:00:00 (VIA Provider)
vif0: flags=80908043<BROADCAST,RUNNING,MULTICAST,TCPCKSUM,VLAN> mtu 9000
ether 02:a0:98:12:b4:f4 (Enabled virtual interface)
vif0-2: flags=4948043<UP,BROADCAST,RUNNING,MULTICAST,TCPCKSUM,NOWINS> mtu 1500
inet 10.0.4.60 netmask 0xfffffc00 broadcast 10.0.7.255
partner vif0-2 (not in use)
ether 02:a0:98:12:b4:f4 (Enabled virtual interface)
vif0-100: flags=4948043<UP,BROADCAST,RUNNING,MULTICAST,TCPCKSUM,NOWINS> mtu 9000
inet 10.0.100.2 netmask 0xffffff00 broadcast 10.0.100.255
partner vif0-100 (not in use)
ether 02:a0:98:12:b4:f4 (Enabled virtual interface)
Since the DL380s only have 2 nics their configuration is simple so I will keep to that for now.
I hace 2 virtual switches with a single nic assigned to each. vSwitch0 is for the sevice console and VM Network, vSwitch1 is for the iSCSI software HBA. I configured both vSwitch1 and vmnic1 for a MTU size of 9000.
[root@ushat-esx03 ~]# esxcfg-vswitch -l
Switch Name Num Ports Used Ports Configured Ports MTU Uplinks
vSwitch0 32 8 32 1500 vmnic0
PortGroup Name VLAN ID Used Ports Uplinks
VM Network 0 5 vmnic0
Service Console 0 1 vmnic0
Switch Name Num Ports Used Ports Configured Ports MTU Uplinks
vSwitch1 64 3 64 9000 vmnic1
PortGroup Name VLAN ID Used Ports Uplinks
iSCSI_VMkernel0 100 1 vmnic1
vmnic1 has been properly bound to the iSCSI software hba.
[root@ushat-esx03 ~]# esxcli swiscsi nic list -d vmhba33
vmk0
pNic name: vmnic1
ipv4 address: 10.0.100.30
ipv4 net mask: 255.255.255.0
ipv6 addresses:
mac address: 00:22:64:c2:2d:9e
mtu: 9000
toe: false
tso: true
tcp checksum: false
vlan: true
link connected: true
ethernet speed: 1000
packets received: 283013
packets sent: 146301
NIC driver: bnx2
driver version: 1.6.9
firmware version: 1.9.6
The test is copying very large files such as ISOs or server images from machine to machine through the local vswitch, or "migrating" virtual machines from the ESX server's fast local storage to the FAS2050 and back.
Here is a sample sysstat output from the filer while copying a 5 gig image file from a file server on the local network to a virtual machine hard drive on the SAN. The numbers are often much lower when migrating a VM from local to iSCSI.
CPU NFS CIFS HTTP Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk FCP iSCSI FCP kB/s iSCSI kB/s
in out read write read write age hit time ty util in out in out
90% 0 0 0 698 72751 409 1168 45316 0 0 4s 99% 58% F 23% 0 698 0 0 72071 34
87% 0 0 0 678 70531 364 492 44516 0 0 4s 99% 56% F 25% 0 678 0 0 69861 0
90% 0 0 0 559 58775 345 968 44668 0 0 4s 99% 66% F 31% 0 559 0 0 58205 34
90% 0 0 0 685 70433 399 596 43224 0 0 4s 99% 52% F 23% 0 685 0 0 69761 34
91% 0 0 0 533 54872 361 380 43240 0 0 4s 100% 68% F 25% 0 533 0 0 54290 34
91% 0 0 0 692 72061 373 408 43368 0 0 4s 100% 64% F 22% 0 692 0 0 71420 0
77% 0 0 0 590 61025 351 756 43404 0 0 4s 99% 49% F 24% 0 590 0 0 60368 34
91% 0 0 0 571 59405 315 996 43720 0 0 4s 99% 65% F 29% 0 571 0 0 58902 0
89% 0 0 0 708 72635 408 580 43296 0 0 4s 99% 53% F 24% 0 708 0 0 71925 34
91% 0 0 0 634 66449 375 432 47828 0 0 4s 99% 62% Fn 27% 0 634 0 0 65809 34
90% 0 0 0 628 64908 368 611 54985 0 0 4s 99% 64% Fn 29% 0 628 0 0 64312 34
90% 0 0 0 666 68360 352 1292 54868 0 0 4s 100% 63% Ff 32% 0 666 0 0 67641 0
92% 0 0 0 566 59267 351 1944 41020 0 0 4s 99% 53% Fs 30% 0 566 0 0 58760 34
97% 0 0 0 345 34418 187 328 16743 0 0 4s 100% 24% : 12% 0 345 0 0 34022 11
89% 0 0 0 720 73894 416 804 38240 0 0 4s 99% 69% Ff 21% 0 720 0 0 73254 34
88% 0 0 0 616 63528 328 590 53546 0 0 4s 99% 59% F 29% 0 616 0 0 62879 0
89% 0 0 0 515 51532 365 412 45976 0 0 4s 99% 58% F 30% 0 515 0 0 51021 69
89% 0 0 0 523 54084 305 820 40496 0 0 5s 99% 63% Ff 33% 0 523 0 0 53599 0
83% 0 0 0 615 64082 365 564 48376 0 0 4s 99% 66% Fv 27% 0 615 0 0 63458 34
88% 0 0 0 681 70214 361 864 44672 0 0 4s 99% 58% F 25% 0 681 0 0 69562 0
It seems the performance tops out around 60-70 MB/s with rather high CPU usage on the filer. My understanding was we should see closer to 120 MB/s when using gigabit and jumbo frames. When disabling jumbo frames there is hardly any impact on performance.
The performance is a little better when testing on the DL385s where I have 4 nics dedicated to 4 seperate VMkernels with round robin providing 4 active paths (following TR-3749 as a configuration guide)--about 80-90 MB/s.
Am I right to assume we should be seeing quite a bit more throughput from this configuration? I was hoping to see >120 megs/sec since I have 2 gigabit nics in the FAS2050 filter in a LACP trunk, and 4 nics using round robin on the ESX servers AND using jumbo frames.
Any suggestions?