To use HA plugin you need to have two NexentaStor Appliances and one storage shared between the two Appliances.

HA Plugin User Guide

There are in fact two options for HA with Nexenta - Basic HA which is two (or more) appliances that have their own storage that gets replicated across and a shared IP that clients connect to. From what I can gather you need the Auto-CDP plugin to do block-level synchronization of the data across and the Basic HA plugin to do manual failovers of said shared IP from appliance to appliance.

Then there is the full-on active-active cluster solution that Roman mentions which is two (or more) appliances set up in front of shared storage. That one gets pricey fast - need gold support, need to pay for Nexenta to come set it up, need to fork over quite a chunk of change for the licenses, and then there's the yearly support fees, too. That said, it is a non-trivial system to set up, I suppose, and if you need that sort of redundancy it's probably going to be for a larger system (where the pricing for Nexenta alone is no longer such a stand-out that it makes you want to scream "rape!". ;) )

So what constitutes shared storage between the 2? This seems like a catch 22 since my goal is to use NexentaStor as the shared storage for my cluster! :) Brings up some new questions. Thanks for the information so far. It definitely helps.

1. Can you give an example of shaed stoage that NexentaSto woks well with (are we talking about a FC shelf that can connect to both chassis or something else)?
2. So even though Auto-CDP is synchonous, it is not fast enough to be a second path for the same LUN (ie. 2 NexentaStor servers with local ZFS pools serving the same iSCSI LUN for iSCSI multipath)?
3. With Auto-CDP and basic HA, do we know what happens to vSphere VMs residing on iSCSI when a manual failover happens? I assume the local FS go read only and VMs may have to be rebooted after a failover...

I'm trying to evaluate the costs involved in doing active/active HA (using essentially 3 machines, 2 NexentaStor and 1 "shared storage" machine) and just Auto-CDP and basic HA. So far, it sounds like active/active HA might defeat the purpose as the shared storage will still be a single point of failure :\

• To use HA Plugin you need to have shared storage (via FC, iSCSI, SAS). (you get automatic and manual failover functionality)
• To use Auto-CDP plugin (also to create Basic HA you need to have Simple-Failover plugin, because AutoCDP provides only replication functionality) you need to have two storages and fast replication channel (you get only manual failover functionality)

There are redundant connections available for just about all the ways to hook external storage up, so there's no reason why the external storage would have to be single-point-of-failure. FC and SAS can be built basically like a network, and iSCSI obviously is. How you'd go about tying dual external storage boxes together and getting it to work with Nexenta and ZFS is another matter (rather, doing the hardware connections is not hard but getting Nexenta to talk to it is not something I know well enough to speculate further on.) The Sun Clusters I've interacted with personally over the years were SCSI or FC, but they ran Veritas Cluster FS which is built for parallel access, how ZFS compares to that is not something I've ever so much as looked at personally.

But there's probably a reason why Nexenta insists on both gold licenses and the purchase of professional services for doing active-active - clusters get complex in a hurry.

Talk to Nexenta sales and ask them to give you a hand figuring it out. :)

I guess I'll give them a call when I'm closer to a purchase decision. Just trying to find a real world example. Any other solutions I see that have multiple heads for multipath to the storage, already employ their own iSCSI/FC targets, which negates the need for NexentaStor anyways. It seems like a waste to spend 5 figures on a shared storage setup and then add NexentaStor to it which is just adding cost and complexity... This concept confuses me.

Well, a quick search for "external jbod sas" gives a number of external enclosures that do SAS. This should in theory work to build redundant solutions with the disks managed by Nexenta, but I'm guessing wildly here and there are probably some gotchas I'm not thinking about since I haven't thought a lot about it and certainly have no experience trying to build something like that. But drop an email to sales@nexenta.com and ask for feedback, it's why they draw a salary after all - to assist people considering a purchase.

ZFS likes to add whole disks as-is to the pools, or rather that's the best way to do it, but it's not limited to that so I'm thinking it's not even necessary to present raw disks to Nexenta to get most of the benefits of ZFS like checksumming, snapshots, self-healing... but again, just speculating.

Andy Shinn wrote:

I guess I'll give them a call when I'm closer to a purchase decision. ... Just trying to find a real world example. Any other solutions I see that have multiple heads for multipath to the storage, already employ their own iSCSI/FC targets, which negates the need for NexentaStor anyways. It seems like a waste to spend 5 figures on a shared storage setup and then add NexentaStor to it which is just adding cost and complexity... This concept confuses me.

There is a much cheaper example, the SuperMicro SBB (Storage Bridge Bay) [[http://www.supermicro.com/products/nfo/sbb.cfm]].

It is a 3U chassis which has 16 HDD and 2 Controllers (Intel CPU PC which can run Nexenta). So you get 2 heads, shared SAS storage at a cost of less than $5000.

Best of all, it is listed on the Nexenta HCL.

Sean, thanks for the suggestion. I've actually checked out the SBB and it looks really cool :)

My concerns are more around the software/multipathing to the backend storage. I actually had a meeting last week with Dell about their EqualLogic product. I may be out of line trying to compare NexentaStor with EqualLogic (the price points are wildly different). But the feature that I was trying to figure out how to do was multiple paths between both heads. The EqualLogic product is essentially storage + 2 controllers. One controller can fail, and in a matter of seconds, the 2 paths to the storage become 1 path and the commands take a shift. There is a moment of iSCSI deadlock. But usually timeouts on VMs and hosts are high enough to cope with the path failover in realtime.

So essentially, I'd like to do something like this in NexentaStor. It appears that the SBB mid-plane may be the key to this (it says it can work in active-active mode with both mid-plane controllers talking to the storage at the same time!). I guess my next step is figuring out how NexentaStor works with this and is "mid-plane" aware.

Has anyone used the Supermicro SBB and care to share?

Andy Shinn wrote:

But the feature that I was trying to figure out how to do was multiple paths between both heads. The EqualLogic product is essentially storage + 2 controllers. One controller can fail, and in a matter of seconds, the 2 paths to the storage become 1 path and the commands take a shift....

One thing to keep in mind -- it is not something that most of us are comfortable with -- the SCSI/SAS protocol is essentially a network protocol/bus, which allows for multiple hosts to send commands to devices over the bus. In case you didn't know LSI has a SAS switch which allows up to 16 hosts to share a SAS infrastructure.

The "trick" is in managing the SAS device assignment/allocation between the hosts.

The EqualLogic IO controllers are very "simple" SAS controllers, whereas the SBB has "smart" PCs in the IO controllers locations. There are option SuperMicro SAS controllers which would allow you to create a duplicate disk SAS controller if you wanted.

So essentially, I'd like to do something like this in NexentaStor. It appears that the SBB mid-plane may be the key to this (it says it can work in active-active mode with both mid-plane controllers talking to the storage at the same time!). I guess my next step is figuring out how NexentaStor works with this and is "mid-plane" aware.

The mid-plane is actually a 10GB/s ethernet connection which provides a high-speed heartbeat channel between the 2 controller modules.

It is the SAS Expander(s) in the HDD backplane and the SAS controller on the controlers which provides the shared disk solution.

As you might tell, I have been looking at the SBB solution for quite a while. ;-)