My Nexenta ship went down after changing BIOS settings - sending message in a bottle

Added by Xenophon Costeas over 2 years ago

Hi, and greetings to all. Apologies for the long post that follows, but I hope someone can help.

*warning! nubie alert!

My situation: ASUS p55/i5 with 8Gb of RAM running NS CE 3.0.4 fresh install -- hardware bought new, because I had a lot of confidence in Nexenta! Six (consumer-grade 1Tb) disks are connected to the built-in SATA ports, and the system boots from a 2-disk array on a SIL3124-compatible RAID card. Installed ~6 mos ago, and running fine (RAIDZ-2) except for the occasional freeze.

Last week I decided (argggh!) to change the BIOS to AHCI mode, just in case it improves stability. Sure enough, I did not take any precautions (e.g. exporting the data pool or backing up :-(, and, predictably, the system was completely borked.

Up until Feb 23rd, the zpool ("opel") showed up as follows, with BIOS in "SATA Enhanced - IDE" mode:

NAME        STATE     READ WRITE CKSUM
opel        ONLINE       0     0     0
  raidz2-0  ONLINE       0     0     0
    c1d0    ONLINE       0     0     0
    c1d1    ONLINE       0     0     0
    c2d0    ONLINE       0     0     0
    c2d1    ONLINE       0     0     0
    c3d0    ONLINE       0     0     0
    c4d0    ONLINE       0     0     0

Subsequently, however, all efforts to re-import the pool failed (in AHCI mode), even after restoring the system to the non-AHCI mode that was active previously:

root@Nexenta:/export/home/admin# zpool import

    opel        UNAVAIL  missing device
      raidz2-0  ONLINE
        c1d0    ONLINE
        c1d1    ONLINE
        c2d0    ONLINE
        c2d1    ONLINE
        c3d0    ONLINE
        c4d0    ONLINE`

Well, I tried all force-import options, to no avail (even after reinstalling NexentaStor CE in both ACHI and non-AHCI flavors):

root@Nexenta:/export/home/admin# zpool import -f opel

cannot import 'opel': one or more devices is currently unavailable Destroy and re-create the pool from a backup source.

Even tried last-resort commands, like:

root@Nexenta:/export/home/admin# zpool import -fF opel

cannot import 'opel': one or more devices is currently unavailable Destroy and re-create the pool from a backup source.

root@Nexenta:/export/home/admin# zpool import -fFD opel

cannot import 'opel': no such pool available

root@Nexenta:/export/home/admin# zpool import -fFX opel

system crashed <<<<

Connection to 192.168.1.101 closed.

Now was the time for some (nubie) trouble-shooting, so I tried some diagnostic commands -

root@Nexenta:/export/home/admin# zdb -l /dev/rdsk/c1d0

LABEL 0

failed to unpack label 0

LABEL 1

failed to unpack label 1

LABEL 2

failed to unpack label 2

LABEL 3

failed to unpack label 3

Doesn't look good, but I figured that maybe the change in device names (SATA vs. IDE) somehow hid the labels, and ZFS would be smart enough to figure that out! As a last-resort, I installed OpenIndiana (hope that doesn't offend Anil :-) and tried re-importing. This, again, caused a crash.

Hmm, now I noticed some device errors in the logs, relating to an old Samsung disk -

Feb 27 22:38:47 Nexenta gda: [ID 107833 kern.warning] WARNING: /pci@0,0/pci-ide@1f,5/ide@1/cmdk@0,0 (Disk1):

Feb 27 22:38:47 Nexenta Error for command 'read sector' Error Level: Informational

Feb 27 22:38:47 Nexenta gda: [ID 107833 kern.notice] Sense Key: aborted command

Feb 27 22:38:47 Nexenta gda: [ID 107833 kern.notice] Vendor 'Gen-ATA ' error code: 0x3

Feb 27 22:39:09 Nexenta scsi: [ID 107833 kern.warning] WARNING: /pci@0,0/pci-ide@1f,5/ide@0 (ata2):

Well, a RAIDZ2 is supposed to survive 2-disk failures, so I removed it. This time, the OpenIndiana system completed a "zpool import -fFX" command after running for a couple of hours, but now the zpool is "FAULTED".

Is there any way to access my (family video) data? If yes, I pledge I'm gonna buy an appliance!

With greetings from Athens,

Xen

P.S. Please don't flame me for nubie mistakes, lessons must be learned somehow!

Nexenta_logs_TRYING - Trying to recover! (404.6 KB)


Replies

RE: My Nexenta ship went down after changing BIOS settings - sending message in a bottle - Added by Xenophon Costeas over 2 years ago

Success!

After managing to botch my NexentaStor CE server, I installed Solaris Express 11 and ran "zpool import -fFX" on the faulted pool. It managed to bring my data back! Kudos!

RE: My Nexenta ship went down after changing BIOS settings - sending message in a bottle - Added by Xenophon Costeas over 2 years ago

(Talking to myself)

This was the most serious incident I had with NSCE, and luckily things turned out well. Anyone finding him/herself in a similar predicament, have a look at the following post from the sister forum at Nexenta.org:

http://nexenta.org/boards/1/topics/2139

Case closed.