My Nexenta ship went down after changing BIOS settings - sending message in a bottle
Added by Xenophon Costeas over 2 years ago
Hi, and greetings to all. Apologies for the long post that follows, but I hope someone can help.
*warning! nubie alert!
My situation: ASUS p55/i5 with 8Gb of RAM running NS CE 3.0.4 fresh install -- hardware bought new, because I had a lot of confidence in Nexenta! Six (consumer-grade 1Tb) disks are connected to the built-in SATA ports, and the system boots from a 2-disk array on a SIL3124-compatible RAID card. Installed ~6 mos ago, and running fine (RAIDZ-2) except for the occasional freeze.
Last week I decided (argggh!) to change the BIOS to AHCI mode, just in case it improves stability. Sure enough, I did not take any precautions (e.g. exporting the data pool or backing up :-(, and, predictably, the system was completely borked.
Up until Feb 23rd, the zpool ("opel") showed up as follows, with BIOS in "SATA Enhanced - IDE" mode:
NAME STATE READ WRITE CKSUM
opel ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
c1d0 ONLINE 0 0 0
c1d1 ONLINE 0 0 0
c2d0 ONLINE 0 0 0
c2d1 ONLINE 0 0 0
c3d0 ONLINE 0 0 0
c4d0 ONLINE 0 0 0
Subsequently, however, all efforts to re-import the pool failed (in AHCI mode), even after restoring the system to the non-AHCI mode that was active previously:
root@Nexenta:/export/home/admin# zpool import
opel UNAVAIL missing device
raidz2-0 ONLINE
c1d0 ONLINE
c1d1 ONLINE
c2d0 ONLINE
c2d1 ONLINE
c3d0 ONLINE
c4d0 ONLINE`
Well, I tried all force-import options, to no avail (even after reinstalling NexentaStor CE in both ACHI and non-AHCI flavors):
root@Nexenta:/export/home/admin# zpool import -f opel
cannot import 'opel': one or more devices is currently unavailable
Destroy and re-create the pool from
a backup source.
Even tried last-resort commands, like:
root@Nexenta:/export/home/admin# zpool import -fF opel
cannot import 'opel': one or more devices is currently unavailable
Destroy and re-create the pool from
a backup source.
root@Nexenta:/export/home/admin# zpool import -fFD opel
cannot import 'opel': no such pool available
root@Nexenta:/export/home/admin# zpool import -fFX opel
system crashed <<<<
Connection to 192.168.1.101 closed.
Now was the time for some (nubie) trouble-shooting, so I tried some diagnostic commands -
root@Nexenta:/export/home/admin# zdb -l /dev/rdsk/c1d0
LABEL 0
failed to unpack label 0
LABEL 1
failed to unpack label 1
LABEL 2
failed to unpack label 2
LABEL 3
failed to unpack label 3
Doesn't look good, but I figured that maybe the change in device names (SATA vs. IDE) somehow hid the labels, and ZFS would be smart enough to figure that out! As a last-resort, I installed OpenIndiana (hope that doesn't offend Anil :-) and tried re-importing. This, again, caused a crash.
Hmm, now I noticed some device errors in the logs, relating to an old Samsung disk -
Feb 27 22:38:47 Nexenta gda: [ID 107833 kern.warning] WARNING: /pci@0,0/pci-ide@1f,5/ide@1/cmdk@0,0 (Disk1):
Feb 27 22:38:47 Nexenta Error for command 'read sector' Error Level: Informational
Feb 27 22:38:47 Nexenta gda: [ID 107833 kern.notice] Sense Key: aborted command
Feb 27 22:38:47 Nexenta gda: [ID 107833 kern.notice] Vendor 'Gen-ATA ' error code: 0x3
Feb 27 22:39:09 Nexenta scsi: [ID 107833 kern.warning] WARNING: /pci@0,0/pci-ide@1f,5/ide@0 (ata2):
Well, a RAIDZ2 is supposed to survive 2-disk failures, so I removed it. This time, the OpenIndiana system completed a "zpool import -fFX" command after running for a couple of hours, but now the zpool is "FAULTED".
Is there any way to access my (family video) data? If yes, I pledge I'm gonna buy an appliance!
With greetings from Athens,
Xen
P.S. Please don't flame me for nubie mistakes, lessons must be learned somehow!
Nexenta_logs_TRYING - Trying to recover! (404.6 KB)
Replies
RE: My Nexenta ship went down after changing BIOS settings - sending message in a bottle - Added by Xenophon Costeas over 2 years ago
Success!
After managing to botch my NexentaStor CE server, I installed Solaris Express 11 and ran "zpool import -fFX" on the faulted pool. It managed to bring my data back! Kudos!
RE: My Nexenta ship went down after changing BIOS settings - sending message in a bottle - Added by Xenophon Costeas over 2 years ago
(Talking to myself)
This was the most serious incident I had with NSCE, and luckily things turned out well. Anyone finding him/herself in a similar predicament, have a look at the following post from the sister forum at Nexenta.org:
http://nexenta.org/boards/1/topics/2139
Case closed.