Most of the PCI-e SSDs that are out there are actually PCI-e RAID controllers with SATA SSDs on-board. The ones that are native PCI-e tend to be VERY expensive and need special drivers.

Dell servers shouldn't have issues with SATA SSDs unless you also have SAS drives installed. If that's the case you will need interposers. If you've got some SAS drives installed then give the Seagate Pulsar 2 (MLC) or XT.2 (SLC) a try. I believe the points of entry for the 100GB versions of those are about $1000 and $1800 respectively.

The fine people at fusionio gave us drivers for their pci ssds to work with nexenta. We never finished to project though, I do have some of their ssd left lying around, could try them out. As far as performance goes, the gen1 of their cards still outperforms pretty much any sas ssd ive seen, the controllers are so much better at 4k. But as Jason said, they are expensive.

Anecdotally we have seen really nice performance with zeus ram, they also have the zeus iops for l2arc..

Thanks for the helpful responses. As far as I can tell, the ZeusIOPs seems to basically cost more than DRAM.

So here's a question: does one actually need an L2ARC at all? Right now, I have 32 GB of RAM in my Dell server. I could go up to 72 GB for around $400, which is a lot cheaper than, say, $800-$1000 for Pulsar or Talos at 100 GB. Given that you have to use some of the ARC to reference the L2ARC, is there any point in bothering with the L2ARC? Specifically, my main pool is small (only 2 TB total), and I think I'll avoid dedup this time around; however, more generally, if the flash drives cost as much as DRAM, isn't it better (faster, no need to reference L2ARC in the ARC) just to add more RAM and forget about the L2ARC altogether?

Thanks!

Peter

RAM would be better than a SSD, if you have the room for it. The SSD L2ARC is a good way to work around having a system where you only have 4 or 6 DIMM slots and a max of 16-24GB of RAM. This is especially important when you use dedup as I believe it's limited to 1/4 of your ARC.

Regarding the ZIL, I'd use an SSD there, but be mindful of the size and the potential throughput of your system (10Gbe or FC links can write a lot more per second than 1Gbe). I don't believe ZFS will write more to the ZIL than ARC/2, so if you've got 32GB of RAM, there's little point in going for a 100GB SSD.

The benefits of zil and l2arc are to take inexpensive disk and make them seem faster. You stage io's into the pool of slower disks. Writes are usually more expensive than read, so zil is going to give you the best bang for the buck.

L2arc is good when you have a large read workload. Database's for example or anytime you have a working set size that is larger than ram. If your workload fits into ram, than... great, ram is still much faster than ssd.

At the moment, I'm using a DDRDrive for ZIL, and have 48 GB of RAM (the pool is about 1.5 TB, and I have four pairs of 7200 RPM NL SAS drives to hold the data). At the moment, deduplication is off. Under these conditions, do you see any reason to get an SSD for L2ARC?

What's your arc hit rate?

what's the best way to determine the ARC hit rate? Just reading the main status web page, or is there a better command-line way to do so?

I ended up buying another 24 GB of RAM, so I'll shortly have a total of 72 GB.

You can download an arcstat script from https://blogs.oracle.com/realneel/entry/zfsarcstatistics or arc_summary from http://www.cuddletech.com/blog/pivot/entry.php?id=979

Arc stats are actually built into the tool also, in nmc you can

show performance arc

So here's the output for my ARC:

nmc@server:/$ show performance arc                                        
Press Ctrl-C to interrupt. See usage (-h) for details.

    Time  read  miss  miss%  dmis  dm%  pmis  pm%  mmis  mm%  arcsz     c  
13:14:40    7M  142K      1   72K    1   69K   16   86K    2     9G   50G  
13:14:41     0     0      0     0    0     0    0     0    0     9G   50G  
13:14:42     0     0      0     0    0     0    0     0    0     9G   50G  
13:14:43     0     0      0     0    0     0    0     0    0     9G   50G  
13:14:44     0     0      0     0    0     0    0     0    0     9G   50G  
13:14:45     0     0      0     0    0     0    0     0    0     9G   50G  
^C
Current ARC Size   Min ARC Size(zfs_arc_min)   Max ARC Size(zfs_arc_max)  
9071MB             6013MB                      48107MB                    

Cache hits and misses (total):
  Cache Hits:                 93%
  Cache Misses:                1%
Cache hits by type:
  Demand Data:                57%
  Prefetch Data:               0%
  Demand Metadata:            38%
  Prefetch Metadata:           4%
Cache misses by type:
  Demand Data:                 8%
  Prefetch Data:              31%
  Demand Metadata:            42%
  Prefetch Metadata:          17%

So your current arc is only 9GB and you have available 50GB.. Your hit rate is very high so you seem to be caching most of what you need in ram.