Hello
We have just discovered that we have very slow performance on our SAN.
This is our environment :
4 - PS6010 and 1 - PS6110 - firmware 6.011
All are full with 15k sas disk and RAID 50, we have 10GB controllers , MTU 9000
We use DELL N4064f switches in stack , with MTU 9000 , flow control, STP disabled as recommended
clients are Dell servers with ixgbe controllers
OS is Debian Jessie ( but tested with redhat 6 )
we use openiscsi :
node.session.timeo.replacement_timeout = 86400
node.conn[0].timeo.login_timeout = 15
node.conn[0].timeo.logout_timeout = 15
node.conn[0].timeo.noop_out_interval = 0
node.conn[0].timeo.noop_out_timeout = 0
node.session.err_timeo.abort_timeout = 15
node.session.err_timeo.lu_reset_timeout = 30
node.session.err_timeo.tgt_reset_timeout = 30
node.session.initial_login_retry_max = 8
node.session.cmds_max = 1024
node.session.queue_depth = 128
node.session.xmit_thread_priority = -20
node.session.iscsi.InitialR2T = No
node.session.iscsi.ImmediateData = Yes
node.session.iscsi.FirstBurstLength = 262144
node.session.iscsi.MaxBurstLength = 16776192
node.conn[0].iscsi.MaxRecvDataSegmentLength = 262144
node.conn[0].iscsi.MaxXmitDataSegmentLength = 0
discovery.sendtargets.iscsi.MaxRecvDataSegmentLength = 32768
and some tuning in the OS via udev:
ACTION=="add", SUBSYSTEM=="scsi" , RUN+="/bin/sh -c 'echo 60 > /sys$$DEVPATH/timeout'"
SUBSYSTEM=="block", ACTION=="add|change", KERNEL=="sd[a-z]", ATTR{queue/read_ahead_kb}="2048", ATTR{queue/max_sectors_kb}="4096", ATTR{queue/nr_requests}="1024" , ATTR{queue/scheduler}="noop"
if we use IOperf to test the network card between servers all is ok : more than 9GB/s
if we test the read / write speed of the SAN it's very slow compared that we can expect :
root@server1:~# dd if=/dev/sdc of=/dev/null ***=4k count=500000
500000+0 records in
500000+0 records out
2048000000 bytes (2.0 GB) copied, 20.4069 s, 100 MB/s
root@server1:~# dd if=/dev/zero of=/root/testfile ***=4k count=500000 oflag=direct
500000+0 records in
500000+0 records out
2048000000 bytes (2.0 GB) copied, 429.087 s, 4.8 MB/s
root@xen0dmzi10:~# dd if=/dev/zero of=/root/testfile ***=4M count=500 oflag=direct
500+0 records in
500+0 records out
2097152000 bytes (2.1 GB) copied, 29.7126 s, 70.6 MB/s
In SanHQ , network and IO are very low ( 2% for network )
If you have any advise ?
I'm open to any test.