raidz vs raidz2 performance

Added by Martin Hlavaty over 2 years ago

Hello,

I'm having issues with performance of my raidz2 volume. It consists of 2x Samsung HD203WI and 4x Samsung HD204UI. When testing locally (using dd), i get following speeds:

dd if=/dev/zero of=32G bs=1M count=32768 34359738368 bytes (34 GB) copied, 513.3 seconds, 66.9 MB/s

dd if=32G of=/dev/null bs=1M 34359738368 bytes (34 GB) copied, 137.196 seconds, 250 MB/s

But when testing on volume mounted via iSCSI on linux guest in ESXi, i get read speed of 22,5MB/s and write speed 48.4 MB/s. Also, when reading/writing, cpu in linux guest gets to 100% load (%wa - waiting for io part according to top), making system extremly laggy.

I also have raidz volume consisting of 4x Samsung HD102. Local tests gives me 250/90MBps speeds and iSCSI tests gets to speeds 110/44MBps. While reading, cpu is at 45% of its usage, while writing it is somewhere around 70%.

Configuration is following:

NAS: AMD Athlon x2 250e, Asus M5A88-V EVO, 8GB DDR3, Intel 572 NIC (EXPI9400PTBLK), LSI 9211-8i card, Nexenta 3.1.1 Server: Intel Core 2 Quad 9550s, Intel DQ45CB, 8GB DDR2, 2x Intel 572 NIC (EXPI9400PTBLK), ESXi 5

When I checked smart attributes, one disk had 3 blocks in CurrentPendingSector. After getting this hdd offline, i got results presented above. Before this, local write was at half of that speed. Getting another disk offline didnt affect speeds in any direction.

Also, how does RawReadError_Rate attribute affects hard disk, when count is e.g. 30000 ? Sata cables are new (2months old)

My question is, what is causing slowness of raidz2 pool, and is it also affecting the iSCSI read speed?

martin


Replies

RE: raidz vs raidz2 performance - Added by Jeff Gibson over 2 years ago

As i was reading this I had the following questions: Do you have compression turned off on your pool? If you don't then your /dev/zero test is going to be inaccurate and likely quite inflated When you setup your zdev for iSCSI did you leave the write caching on or did you turn it off? Is your NAS the linux system you mention going to 100% usage? If so the x2 might be under powered to push that type of net traffic and you may want to do some net tuning to see what kind of bandwidth you can get between the hosts without the iSCSI layer on top of it. This also will help you track down MTU issues if you're trying to use jumbo frames (which may help reduce the CPU load if you can use it).

And here are my general comments: I've never tested it, but have read mixing different devices in a single vdev is a bad idea. With that said your numbers don't seem horrible on your samsung pool (since I assume you don't have any log or cache devices). With a single raidz2 vdev you'll only get the write performance of a single disk (which I believe is being influenced by the write-back cache at this point). Reads should be spread accross all the disks, but if one of them is throwing the RawReadErrors then it's probably having to stall at that disk for longer than it's supposed to. Try posting your iostat -xcn while doing the tests.

RE: raidz vs raidz2 performance - Added by Martin Hlavaty over 2 years ago

Hello Jeff,

compression is turned off, writeback cache on zvols is on, dont have any ssd for caching. I thought when the filesize is big enough (4times size of the RAM), the results arent influenced by cache. Linux system i mention is guest on ESXi server (different hw than NAS). CPU on NAS gets to 60-70% (according to graph on status page) while testing. I have phenom II x2 ready for exchange, but didnt feel the need as cpu on NAS wasnt getting to 100% of its usage. NICs should have tcp offloading, and my iperf bw tests resulted in 800-950mbps. Im not using jumbo frames, didnt think it would add that much performance (mistake?).

iostat while writing zeroes simultaneously to both raidz and raidz2 volume:

admin@maslo:~$ iostat -xcn
     cpu
 us sy wt id
  3  8  0 89
                    extended device statistics
    r/s    w/s   kr/s   kw/s wait actv wsvc_t asvc_t  %w  %b device
   32.2    3.2  248.7   24.0  0.0  0.0    0.8    0.6   1   1 c1t1d0
   32.3    3.0  248.1   23.7  0.0  0.0    0.7    0.5   1   1 c1t2d0
   32.3    3.2  248.6   24.1  0.0  0.0    0.6    0.5   1   1 c1t4d0
   31.4    3.0  249.6   23.7  0.0  0.0    0.7    0.5   1   1 c1t5d0
    1.0   13.6    8.2   86.7  0.0  0.0    0.1    0.4   0   0 c1t0d0
    4.9   33.3   35.3  551.9  0.0  0.0    0.0    1.0   0   1 c6t50024E9003BE04C1d0
    5.9   31.7   39.2  550.3  0.0  0.1    0.0    1.4   0   2 c6t50024E9003BE04CAd0
    4.3   30.1   18.9  526.5  0.0  0.1    0.0    3.9   0   4 c6t50024E90047BE925d0
    4.7   33.1   26.6  551.9  0.0  0.1    0.0    2.9   0   3 c6t50024E90047BE942d0
    6.1   32.9   37.4  550.4  0.0  0.1    0.0    2.2   0   2 c6t50024E9204BC3E2Dd0
    5.0   32.1   34.5  549.9  0.0  0.1    0.0    3.7   0   4 c6t50024E9204BC3E2Fd0

and here is output of zpool iostat

admin@maslo:~$ zpool iostat 1
               capacity     operations    bandwidth
pool        alloc   free   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
data2pool   2.58T  1.04T    100      7   781K  70.6K
datapool    4.89T  5.98T     13    212   161K  2.14M
syspool     1.61G  2.08G      0     11  8.02K  86.7K
----------  -----  -----  -----  -----  -----  -----
data2pool   2.58T  1.04T      0  1.08K      0   120M
datapool    4.89T  5.98T      0    461  7.99K  48.7M
syspool     1.61G  2.08G      0      0  52.9K      0
----------  -----  -----  -----  -----  -----  -----
data2pool   2.58T  1.04T      0  1.31K  7.99K   118M
datapool    4.89T  5.98T      4     93  40.0K   165K
syspool     1.61G  2.08G     19    117  1.24M   380K
----------  -----  -----  -----  -----  -----  -----
data2pool   2.58T  1.04T      0    710      0  67.8M
datapool    4.89T  5.98T      7    739  50.7K  90.5M
syspool     1.61G  2.08G     15     23  62.9K   578K
----------  -----  -----  -----  -----  -----  -----
data2pool   2.58T  1.04T      3  1.05K  20.8K   108M
datapool    4.89T  5.98T      1    564  7.98K  47.8M
syspool     1.61G  2.08G      0      0      0      0
----------  -----  -----  -----  -----  -----  -----
data2pool   2.58T  1.04T      0  1.36K      0   167M
datapool    4.89T  5.98T      2      0  15.0K      0
syspool     1.61G  2.08G      0      0      0      0
----------  -----  -----  -----  -----  -----  -----
data2pool   2.58T  1.04T      0    626      0  61.8M
datapool    4.89T  5.98T      0      0      0      0
syspool     1.61G  2.08G      0      0      0      0
----------  -----  -----  -----  -----  -----  -----
data2pool   2.58T  1.04T      0  1.43K      0   174M
datapool    4.89T  5.98T     15    355   111K  35.1M
syspool     1.61G  2.08G      0    158      0  1.28M
----------  -----  -----  -----  -----  -----  -----

Why speeds on raiz2 (datapool) pool so much fluctuates, while raidz (data2pool) is more stable? Testing pools separately gives me same results.

Martin

Content-Type: text/html; charset=utf-8 Set-Cookie: _redmine_session=BAh7BiIKZmxhc2hJQzonQWN0aW9uQ29udHJvbGxlcjo6Rmxhc2g6OkZsYXNoSGFzaHsABjoKQHVzZWR7AA%3D%3D--cebfb08d300a85bd88dafd1422210ebe7c9a5873; path=/; HttpOnly Status: 500 Internal Server Error X-Powered-By: Phusion Passenger (mod_rails/mod_rack) 2.0.3 ETag: "3c4e493e3c1ae35d2626c30e9b6f4018" X-Runtime: 543ms Content-Length: 13682 Cache-Control: private, max-age=0, must-revalidate redMine 500 error

Internal error

An error occurred on the page you were trying to access.
If you continue to experience problems please contact your redMine administrator for assistance.

Back