VxVM - Fixing plex in DISABLED IOFAIL state

Filesystem is un-mountable because of plex in DISABLED IOFAIL state:


The filesystem fails to mount 


(HostSol10:/root)# mount -F vxfs /dev/vx/dsk/DGDB1/VOLFSARC /DGDB1
UX:vxfs mount: ERROR: V-3-20003: Cannot open /dev/vx/dsk/DGDB1/VOLFSARC: No such device or address
UX:vxfs mount: ERROR: V-3-24996: Unable to get disk layout version
(HostSol10:/root)#


The volume is shown as in disabled state

(HostSol10:/root)# vxprint -g DGDB1 -v
TY NAME         ASSOC        KSTATE   LENGTH   PLOFFS   STATE    TUTIL0  PUTIL0
v  VOLFSARC fsgen      DISABLED 9013559296 -      ACTIVE   -       -
(HostSol10:/root)#

Starting the volume also fails

(HostSol10:/root)# vxvol -g DGDB1 startall
VxVM vxvol ERROR V-5-1-1198 Volume VOLFSARC has no CLEAN or non-volatile ACTIVE plexes
(HostSol10:/root)#

Detailed status of the volume shows DISABLED IOFAIL state

(HostSol10:/root)# vxprint -htg DGDB1 -p | more
PL NAME         VOLUME       KSTATE   STATE    LENGTH   LAYOUT    NCOL/WID MODE
SD NAME         PLEX         DISK     DISKOFFS LENGTH   [COL/]OFF DEVICE   MODE
SV NAME         PLEX         VOLNAME  NVOLLAYR LENGTH   [COL/]OFF AM/NM    MODE
SC NAME         PLEX         CACHE    DISKOFFS LENGTH   [COL/]OFF DEVICE   MODE

pl VOLFSARC-01 VOLFSARC DISABLED IOFAIL 9013559296 CONCAT -    RW
sd DISK-222A-01 VOLFSARC-01 DISK-222A 0 209643136 0 emc0_222a ENA
sd DISK-222B-01 VOLFSARC-01 DISK-222B 0 209643136 209643136 emc0_222b ENA
sd DISK-222C-01 VOLFSARC-01 DISK-222C 0 209643136 419286272 emc0_222c ENA
sd DISK-222D-01 VOLFSARC-01 DISK-222D 0 209643136 628929408 emc0_222d ENA
sd DISK-222E-01 VOLFSARC-01 DISK-222E 0 209643136 838572544 emc0_222e ENA
sd DISK-222F-01 VOLFSARC-01 DISK-222F 0 209643136 1048215680 emc0_222f ENA
sd DISK-2228-01 VOLFSARC-01 DISK-2228 0 209643136 1257858816 emc0_2228 ENA
sd DISK-2229-01 VOLFSARC-01 DISK-2229 0 209643136 1467501952 emc0_2229 ENA
(HostSol10:/root)#

Once all is verified and no errors seen in disk or SAN, clean the plex

(HostSol10:/root)# vxmend fix clean VOLFSARC-01

(HostSol10:/root)# vxprint -htg DGDB1 -v | grep v
v  VOLFSARC -          DISABLED ACTIVE   9013559296 SELECT  -        fsgen
(HostSol10:/root)#
(HostSol10:/root)#

There seems to be some disk reporting failing status. failing status is differnt from failed status reported by vxvm. The disk showing failing status may be because of a transient error rather than a truly failing disk. In this case, you can simply clear the status. If the failing status continues to reappear for the same disk, it may be a sign that there is genuine, hardware problem with the disk, or with the SAN connectivity

(HostSol10:/root)# vxdisk list | grep -i fai
emc0_24ab    auto:cdsdisk    DISK-DB3-24AB  DGDB3   online thinrclm failing
emc0_24a3    auto:cdsdisk    DISK-DB3-24A3  DGDB3   online thinrclm failing
emc0_38c0    auto:cdsdisk    DISK-DB4-38c0  DGDB4   online thinrclm failing
emc0_39a0    auto:cdsdisk    DISK-DB6-39A0  DGDB6   online thinrclm failing
(HostSol10:/root)#


The volume can be forcefully started and the FS mounted.

(HostSol10:/root)#
(HostSol10:/root)# vxvol -g DGDB1 -f start VOLFSARC
(HostSol10:/root)#
(HostSol10:/root)# vxprint -g DGDB1 -v
TY NAME         ASSOC        KSTATE   LENGTH   PLOFFS   STATE    TUTIL0  PUTIL0
v  VOLFSARC fsgen      ENABLED  9013559296 -      ACTIVE   -       -
(HostSol10:/root)#

VxVM - vxconfigd debug mode operation

VERITAS Volume Manager (tm) provides the option of logging console output to a file. The Volume Manager configuration daemon vxconfigdcontrols whether such logging is turned on or off, which by default, is disabled. If enabled, the default log file is vxconfigd.log, and its location varies by operating system:

SolHost10:/root#vxconfigd -x 6 -k -x log -x logfile=/tmp/vxconfigd.out

VxVM vxconfigd DEBUG V-5-1-24577

VOLD STARTUP pid=54427 debug-level=6 logfile=/tmp/vxconfigd.out

VxVM vxconfigd DEBUG V-5-1-681 IOCTL GET_VOLINFO: return 0(0x0)

VxVM vxconfigd DEBUG V-5-1-23909 Kernel version 5.1_SP1

VxVM vxconfigd DEBUG V-5-1-681 IOCTL KTRANS_ABORT: failed: errno=22 (Invalid argument)

VxVM vxconfigd DEBUG V-5-1-681 IOCTL GET_KMEM id=0 size=84: return 0(0x0)

        Results: got 84 bytes

VxVM vxconfigd DEBUG V-5-1-681 IOCTL GET_KMEM id=2 size=6384: return 0(0x0)

        Results: got 6384 bytes

VxVM vxconfigd DEBUG V-5-1-681 IOCTL GET_KMEM id=1 size=0: return 0(0x0)

        Results: got 0 bytes

VxVM vxconfigd DEBUG V-5-1-681 IOCTL SET_KMEM id=1 size=0: return 0(0x0)

VxVM vxconfigd DEBUG V-5-1-681 IOCTL SET_KMEM id=0 size=84: return 0(0x0)

VxVM vxconfigd DEBUG V-5-1-5657 mode_set: oldmode=none newmode=enabled

VxVM vxconfigd DEBUG V-5-1-5656 mode_set: locating system disk devices

VxVM vxconfigd DEBUG V-5-1-681 IOCTL GET_DISKS rids: 0.17: failed: errno=2 (No such file or directory)

VxVM vxconfigd DEBUG V-5-1-5477 find_devices_in_system: locating system disk devices

VxVM vxconfigd DEBUG V-5-1-16309 ddl_find_devices_in_system: Thread pool initialization succeed

VxVM vxconfigd DEBUG V-5-1-16366 devintf_find_soldevices: Using libdevinfo for scanning device tree

VxVM vxconfigd DEBUG V-5-1-14578 ctlr_list_insert: Creating entry for c-1, state=V

VxVM vxconfigd DEBUG V-5-1-5294 ctlr_list_insert: check for c-1

VxVM vxconfigd DEBUG V-5-1-14579 ctlr_list_insert: entry already present for c-1, state=V

VxVM vxconfigd DEBUG V-5-1-5294 ctlr_list_insert: check for c-1

VxVM vxconfigd DEBUG V-5-1-14579 ctlr_list_insert: entry already present for c-1, state=V

VxVM vxconfigd DEBUG V-5-1-21563 ddl_add_hba: Added hba c1

VxVM vxconfigd DEBUG V-5-1-21567 ddl_add_port: Added port c1_p0 under hba c1

VxVM vxconfigd DEBUG V-5-1-21569 ddl_add_target: Added target c1_p0_t0 under port c1_p0

.....

...

VxVM vxconfigd DEBUG V-5-1-14886 ddl_vendor_info: nvlist[3] is ANAME=PILLAR-AXIOM

VxVM vxconfigd DEBUG V-5-1-14885 ddl_vendor_info: name = ANAME values = 1

VxVM vxconfigd DEBUG V-5-1-14884 ddl_vendor_info: library: libvxpillaraxiom.so name: ANAME value[0]: PILLAR-AXIOM

VxVM vxconfigd DEBUG V-5-1-14886 ddl_vendor_info: nvlist[4] is ASL_VERSION=vm-5.1.100-rev-1

VxVM vxconfigd DEBUG V-5-1-14885 ddl_vendor_info: name = ASL_VERSION values = 1

VxVM vxconfigd DEBUG V-5-1-14884 ddl_vendor_info: library: libvxpillaraxiom.so name: ASL_VERSION value[0]: vm-5.1.100-rev-1

VxVM vxconfigd DEBUG V-5-1-14882 ddl_vendor_info exits with success for library = libvxpillaraxiom.so

VxVM vxconfigd DEBUG V-5-1-0 Check ASL - libvxpp.so

VxVM vxconfigd DEBUG V-5-1-14563 checkasl: ASL Key file - /etc/vx/aslkey.d/libvxpp.key

VxVM vxconfigd DEBUG V-5-1-14880 ddl_vendor_info entered for library = libvxpp.so

VxVM vxconfigd ERROR V-5-1-0 Segmentation violation - core dumped

SolHost10:/root#

The higher the debug level (0-9), the greater the output

Solaris - Booting into a zfs root from ok prompt

If the root is in zfs, to boot into a specific Boot Environment, below process can be used

In our case we have 2 BE's but one is not visible 

{0} ok boot -L
Boot device: /pci@400/pci@0/pci@8/scsi@0/disk@0:a  File and args: -L
1 sol10u8
Select environment to boot: [ 1 - 1 ]: 1

To boot the selected entry, invoke:
boot [] -Z rpool/ROOT/sol10u8


Program terminated

However, we can still invoke it from ok prompt, if we know the zfs root fs name.


{0} ok boot -Z rpool/ROOT/sol10u9


T5240, No Keyboard
Copyright (c) 1998, 2012, Oracle and/or its affiliates. All rights reserved.
OpenBoot 4.33.6.b, 130848 MB memory available, Serial #94648388.
Ethernet address 0:21:28:a4:38:44, Host ID: 85a43844.



Boot device: /pci@400/pci@0/pci@8/scsi@0/disk@0:a  File and args: -Z rpool/ROOT/sol10-u9
SunOS Release 5.10 Version Generic_147440-11 64-bit
Copyright (c) 1983, 2012, Oracle and/or its affiliates. All rights reserved.
WARNING: /scsi_vhci/ssd@g60060e80153269000001326900002023 (ssd16):

        Corrupt label; wrong magic number