Matt this sounds like iscsi is working as designed with VAAI extensions. Does your NFS datastore indicate it has hardware acceleration (I'm pretty sure it's just iSCSI that has that option)?

Jeff Gibson wrote:

Matt this sounds like iscsi is working as designed with VAAI extensions. Does your NFS datastore indicate it has hardware acceleration (I'm pretty sure it's just iSCSI that has that option)?

Hi Jeff,

Nope, sorry there are no VAAI extentions running here, this is a Nexenta Community Edition install only, VAAI comes to paying customers only. I don't see any indications of hardware acceleration for the NFS datastore.

To reiterate the above: when going from a 2 disk mirror (HW RAID) to a 4 x 2 disk mirror (ZFS with cache+zil), the template is cloned successfully in about 30 seconds. So i expected that when cloning the same template from the 4x2 mirror to the same 4x2 mirror i will get at LEAST the same or better performance as the first test... and when using iSCSI it is better performance (only 20 seconds for the whole clone operation to complete). Instead with NFS, it takes almost 2 minutes. I have narrowed this down somewhat to NFS's read performance.

In case anyone is wondering, i am "stuck" on this issue due to the following pros/cons:

iSCSI

Pros

• -vmware's multipath io gives ~40% improvement over NFS on sequential read peformance as shown by iometer
• -cloning from template is significantly faster (as described above)

Cons

• -seems to hang a big Win 2008 VM of mine after transmitting ~300 GB of a ~800GB data. This is repeatable, i can demonstrate to someone from Nexenta if requested. I can't determine if it is Nexenta or VMware's fault, but it definitely only occurs with iSCSI.
• -VMWare does not handle iscsi failures gracefully... poor LUN connectivity will hang the entire hypervisor on ESXi 4.1, 4.1u2, 5.0u1 installs
• -VMWare does not "try again" if an iSCSI mounted datastore is not immediately available when the hypervisor is booting (e.g. in my case when Nexenta VM is not available for several minutes after ESXi host is booted, and therefoe the iSCSI taget isn't available yet)

NFS

Pros

• -VMWare's NFS seems to be more tolerant of network failures, exported NFS shares are mounted ~3 minutes afte ESXi is booted so i can autostart vms located on the NFS based datastore
• -has never cashed/hung vms as pedictably as iscsi has
• -more flexible for VMware 4.x that have 2TB LUN limits

Cons

• -slower sequential throughput
• -vmware does not show perfomance statistics for NFS based datastores in per-vm perfomancer monitor
• -inexplicable latency accoding to VMware system-wide pefomancer monitor, but iometer does not reflect the same numbers
• -no multipath io means a physical install is limited to single NIC only (no multipath io = can't aggregate pefomance fom multiple 1 Gb NICs without being a PITA)

Yeah, Matt, I feel your pain. The unstable iSCSI target under heavy load hosed me once, and that was enough. The entire hypervisor was dead in the water and I had to hit the reset button. There was NOT happiness and joy here at that point... NFS has been 1000% stable and easy.

Matt,

VAAI does work on Nexenta's iSCSI LUNs on VMFS datastores.

see link: http://blog.solori.net/2011/08/03/short-take-nexenta-3-1-adds-vaai-support-auto-sync-resume/

It would appear that it is still unsupported on NFS datastores. That's likely the reason for your fast performance on iSCSI and slow performance on NFS.

I ended up going 10GbE with NFS on our Nexenta build to get the speed we needed on the network. A clone that was taking 6 hours+ on a 1Gbps link now takes ~55-65 minutes.

All that being said, we went NFS 100% on our Nexenta solution because I need stability first and foremost and I do not trust the issues with iSCSI on Nexenta at this time.

It has been a while so I thought I would check in...

• Nexenta's own best pactices documents ecommend the use of NFS (http://info.nexenta.com/s/nexenta/images/5000-nxs-v0.0-000002-Anxstovmwaebestpactices.pdf, page 12)
• VMware released support for Hardware Acceleration for NAS devices (i.e. NFS) a year and a half ago. (http://www.vmware.com/files/pdf/techpaper/Whats-New-VMware-vSphere-50-Storage-Technical-Whitepaper.pdf, page 13)

So when will Nexenta support the VMWare NAS VAAI storage primitives (especially the Full File Clone)?

Nexenta has had a long time to come out with support for this option. Is there simply no customers driving this feature?

No comments from Nexenta (Derek Glover)?

I saw references from over a year ago that Nexenta was "working on" this feature... so whats the deal?

Checking on it...

We are looking into when this can be integrated, tested and released. It will likely come after the upcoming 4.0 with Illumos shift.

Thanks Derek... I realize this was scheduled for this past summer and was delayed, but can you comment on the new expected date for the 4.0 release?

By the way, if you testers, sign me up :)

I don't have a date for release yet. But on the beta subject, check back in 1-2 weeks, hoping to have a public beta available soon.

It has been 3 weeks :)

Nexenta CE 3.1.3 connected to and VMWare 4.1 (no VAAI) or 5.0 via NFS is still inexplicably slow compared to iscsi. Everything occurring over 1gbe Intel NICs.