Hi Reuben,

You should probably be most concerned about your random reads. Those are the ones that can impact performance the most. If you run Exchange 2007 there are quite a lot of random reads, Exchange 2010 not so much. VMware usually does a lot of small random io as well.

I see no reason to go for multiple raidz2 pools, you're performance will be better with 1 pool consisting of multiple raidz2 vdev's. If you're thinking of multiple pools because you want multiple ZeusRam's as log devices you can always stripe multiple ZeusRam's in 1 pool.

So let's try to compare these two scenario's:

1. 18 3-way mirror vDevs, ZeusRam (preferably mirrored), ZeusIOPS (not mirrored)
2. 9 6-disk Raidz2 vDevs, ZeusRam (preferably mirrored), ZeusIOPS (not mirrored)

Performance: With option 1 you have the write speed of 18 disks, and the read speed of 54 disks. With option 2 you have the write speed of 9 disks, and the read speed of 9 disks. Depending on the workload you can get a lot more bandwith, but for small random io i believe you should calculate using these numbers.

Capacity: With option 1 you have 18 disks worth of capacity. With option 2 you have 36 disks worth of capacity.

Basically it really depends on you're working set. If your entire workig set is in ARC giving you a hit ratio of (for example) 97% the few reads that will reach the actual disks should be no problem for option 2. If you have a big working set and a lousy hit ratio option 2 will probably not be fast enough.

Without actual workload numbers or guesstimates (exchange average load per user calculations etc) it is near impossible to tell you what the best option is. If I had to design this without any more info than provided here I would go for 18 3-way mirrors (assuming double-disk failure protection is a must).

One tip though: Max out the memory. Adding more memory is the cheapest and fastest way to improve your performance. Hitting the ARC is always faster than hitting L2ARC, and nowadays memory is pretty cheap.

regards,

Brenn

p.s. interested to see what others here think about this..

Hello Breen, Thank you for that very detailed write up.

This is a green fields install. The Sharepoint team is demanding high performing iSCSI storage. I was thinking the 18 3-way setup as well. The server has 92GB Ram. I think even though I am using 7.2K Disks, these are SAS with dual pathing and I am front loading the HHD array with Mirrored and Striped ZeusRam SSDs and Striped ZeusIOPs SSD and the server has two LSI 9200-8E in it, the storage should be well performing.

With the disks being 1TB, the 3-Way Mirror is the best way to go in case there is a multiple disk failure during a resilver and disks are cheap :)

I have MS Exchange 2010, It is currently running from within vmdk's and I am not having any issues with the performance of the system.

Thank you for your time and detailed information

Cheers R