VxVM - Fixing plex in DISABLED IOFAIL state

Filesystem is un-mountable because of plex in DISABLED IOFAIL state:


The filesystem fails to mount 


(HostSol10:/root)# mount -F vxfs /dev/vx/dsk/DGDB1/VOLFSARC /DGDB1
UX:vxfs mount: ERROR: V-3-20003: Cannot open /dev/vx/dsk/DGDB1/VOLFSARC: No such device or address
UX:vxfs mount: ERROR: V-3-24996: Unable to get disk layout version
(HostSol10:/root)#


The volume is shown as in disabled state

(HostSol10:/root)# vxprint -g DGDB1 -v
TY NAME         ASSOC        KSTATE   LENGTH   PLOFFS   STATE    TUTIL0  PUTIL0
v  VOLFSARC fsgen      DISABLED 9013559296 -      ACTIVE   -       -
(HostSol10:/root)#

Starting the volume also fails

(HostSol10:/root)# vxvol -g DGDB1 startall
VxVM vxvol ERROR V-5-1-1198 Volume VOLFSARC has no CLEAN or non-volatile ACTIVE plexes
(HostSol10:/root)#

Detailed status of the volume shows DISABLED IOFAIL state

(HostSol10:/root)# vxprint -htg DGDB1 -p | more
PL NAME         VOLUME       KSTATE   STATE    LENGTH   LAYOUT    NCOL/WID MODE
SD NAME         PLEX         DISK     DISKOFFS LENGTH   [COL/]OFF DEVICE   MODE
SV NAME         PLEX         VOLNAME  NVOLLAYR LENGTH   [COL/]OFF AM/NM    MODE
SC NAME         PLEX         CACHE    DISKOFFS LENGTH   [COL/]OFF DEVICE   MODE

pl VOLFSARC-01 VOLFSARC DISABLED IOFAIL 9013559296 CONCAT -    RW
sd DISK-222A-01 VOLFSARC-01 DISK-222A 0 209643136 0 emc0_222a ENA
sd DISK-222B-01 VOLFSARC-01 DISK-222B 0 209643136 209643136 emc0_222b ENA
sd DISK-222C-01 VOLFSARC-01 DISK-222C 0 209643136 419286272 emc0_222c ENA
sd DISK-222D-01 VOLFSARC-01 DISK-222D 0 209643136 628929408 emc0_222d ENA
sd DISK-222E-01 VOLFSARC-01 DISK-222E 0 209643136 838572544 emc0_222e ENA
sd DISK-222F-01 VOLFSARC-01 DISK-222F 0 209643136 1048215680 emc0_222f ENA
sd DISK-2228-01 VOLFSARC-01 DISK-2228 0 209643136 1257858816 emc0_2228 ENA
sd DISK-2229-01 VOLFSARC-01 DISK-2229 0 209643136 1467501952 emc0_2229 ENA
(HostSol10:/root)#

Once all is verified and no errors seen in disk or SAN, clean the plex

(HostSol10:/root)# vxmend fix clean VOLFSARC-01

(HostSol10:/root)# vxprint -htg DGDB1 -v | grep v
v  VOLFSARC -          DISABLED ACTIVE   9013559296 SELECT  -        fsgen
(HostSol10:/root)#
(HostSol10:/root)#

There seems to be some disk reporting failing status. failing status is differnt from failed status reported by vxvm. The disk showing failing status may be because of a transient error rather than a truly failing disk. In this case, you can simply clear the status. If the failing status continues to reappear for the same disk, it may be a sign that there is genuine, hardware problem with the disk, or with the SAN connectivity

(HostSol10:/root)# vxdisk list | grep -i fai
emc0_24ab    auto:cdsdisk    DISK-DB3-24AB  DGDB3   online thinrclm failing
emc0_24a3    auto:cdsdisk    DISK-DB3-24A3  DGDB3   online thinrclm failing
emc0_38c0    auto:cdsdisk    DISK-DB4-38c0  DGDB4   online thinrclm failing
emc0_39a0    auto:cdsdisk    DISK-DB6-39A0  DGDB6   online thinrclm failing
(HostSol10:/root)#


The volume can be forcefully started and the FS mounted.

(HostSol10:/root)#
(HostSol10:/root)# vxvol -g DGDB1 -f start VOLFSARC
(HostSol10:/root)#
(HostSol10:/root)# vxprint -g DGDB1 -v
TY NAME         ASSOC        KSTATE   LENGTH   PLOFFS   STATE    TUTIL0  PUTIL0
v  VOLFSARC fsgen      ENABLED  9013559296 -      ACTIVE   -       -
(HostSol10:/root)#

Resize a FS with space from a specific disk

In a disk group, where some volumes are stripped and some are concat, while adding disk space to a concat volume, care needs to be taken.

For STRIPE volumes, when adding a new disk, the whole stripe has to be relayedout.

Solaris1:/root# vxprint -htg DG-DBDG | grep -i STRIP

pl VOL-DBDG-ARCH01-02 VOL-DBDG-ARCH01 ENABLED ACTIVE 1822425088 STRIPE 11/8192 RW

pl VOL-DBDG-DTA01-02 VOL-DBDG-DTA01 ENABLED ACTIVE 678232064 STRIPE 8/8192 RW

pl VOL-DBDG-DTA04-02 VOL-DBDG-DTA04 ENABLED ACTIVE 3091324928 STRIPE 19/8192 RW

pl VOL-DBDG-DTA05-02 VOL-DBDG-DTA05 ENABLED ACTIVE 2910044160 STRIPE 18/8192 RW

pl VOL-DBDG-DTA06-02 VOL-DBDG-DTA06 ENABLED ACTIVE 2910044160 STRIPE 18/8192 RW

pl VOL-DBDG-DTA07-02 VOL-DBDG-DTA07 ENABLED ACTIVE 2910044160 STRIPE 18/8192 RW

pl VOL-DBDG-ID01-02 VOL-DBDG-ID01 ENABLED ACTIVE 671121408 STRIPE 12/8192 RW

pl VOL-DBDG-ID04-02 VOL-DBDG-ID04 ENABLED ACTIVE 1343619072 STRIPE 16/8192 RW

pl VOL-DBDG-ID05-02 VOL-DBDG-ID05 ENABLED ACTIVE 1121992704 STRIPE 14/8192 RW

pl VOL-DBDG-ID06-02 VOL-DBDG-ID06 ENABLED ACTIVE 1121992704 STRIPE 14/8192 RW

pl VOL-DBDG-ID07-02 VOL-DBDG-ID07 ENABLED ACTIVE 1121992704 STRIPE 14/8192 RW

pl VOL-DBDG-MIDO01-02 VOL-DBDG-MIDO01 ENABLED ACTIVE 62914560 STRIPE 8/8192 RW

pl VOL-DBDG-RBS01-02 VOL-DBDG-RBS01 ENABLED ACTIVE 381681664 STRIPE 13/8192 RW

pl VOL-DBDG-RDO01-02 VOL-DBDG-RDO01 ENABLED ACTIVE 62914560 STRIPE 8/8192 RW

pl VOL-DBDG-TMP01-02 VOL-DBDG-TMP01 ENABLED ACTIVE 1006632960 STRIPE 12/8192 RW

For CONCAT volumes, it is straight extension but the disk that was added needs to be used only for this volume and shouldn't mix with the STRIPE volumes.

Solaris1:/root# vxprint -htg DG-DBDG | grep -i CONCAT

pl VOL-DBDG-bkp01-02 VOL-DBDG-bkp01 ENABLED ACTIVE 63955712 CONCAT - RW

pl VOL-DBDG-ptr-02 VOL-DBDG-ptr ENABLED ACTIVE 4194304 CONCAT - RW

pl VOL-DBDG-mon-02 VOL-DBDG-mon ENABLED ACTIVE 2097152 CONCAT -    RW

Solaris1:/root#

Solaris1:/root# df -h /DG-DBDG/bkp01

Filesystem             size   used  avail capacity  Mounted on

/dev/vx/dsk/DG-DBDG/VOL-DBDG-bkp01

                        13G   4.6G   8.3G    36%    /DG-DBDG/bkp01

Solaris1:/root#

Solaris1:/root#

Solaris1:/root# vxprint -htg DG-DBDG -v VOL-DBDG-bkp01

V  NAME         RVG/VSET/CO  KSTATE   STATE    LENGTH   READPOL   PREFPLEX UTYPE

PL NAME         VOLUME       KSTATE   STATE    LENGTH   LAYOUT    NCOL/WID MODE

SD NAME         PLEX         DISK     DISKOFFS LENGTH   [COL/]OFF DEVICE   MODE

SV NAME         PLEX         VOLNAME  NVOLLAYR LENGTH   [COL/]OFF AM/NM    MODE

SC NAME         PLEX         CACHE    DISKOFFS LENGTH   [COL/]OFF DEVICE   MODE

DC NAME         PARENTVOL    LOGVOL

SP NAME         SNAPVOL      DCO

EX NAME         ASSOC        VC                       PERMS    MODE     STATE

v  VOL-DBDG-bkp01 -    ENABLED  ACTIVE   28304128 SELECT    -        fsgen

pl VOL-DBDG-bkp01-02 VOL-DBDG-bkp01 ENABLED ACTIVE 28304128 CONCAT - RW

sd DSK-DBDG-2B04-01 VOL-DBDG-bkp01-02 DSK-DBDG-2B4 0 28304128 0 hp_p95000_2b4 ENA

Solaris1:/root#

Solaris1:/root#

Solaris1:/root# /etc/vx/bin/vxdisksetup -i hp_p95000_2f9

Solaris1:/root# vxdisk -o alldgs -e list | grep -i 2f9

hp_p95000_2f9 auto:cdsdisk   -            -           online thinrclm      c2t50060E80164EF966d80s2 tprclm fc tc-pvol pvol

Solaris1:/root#

Solaris1:/root#

Solaris1:/root# vxdg -g DG-DBDG adddisk DSK-DBDG-2F9=hp_p95000_2f9

Solaris1:/root#

Solaris1:/root# /etc/vx/bin/vxresize -g DG-DBDG VOL-DBDG-bkp01 +17g DSK-DBDG-2F9

Solaris1:/root#

Solaris1:/root# df -h /DG-DBDG/bkp01

Filesystem             size   used  avail capacity  Mounted on

/dev/vx/dsk/DG-DBDG/VOL-DBDG-bkp01

                        30G   4.6G    24G    17%    /DG-DBDG/bkp01

Solaris1:/root#

Solaris1:/root# vxprint -htg DG-DBDG -v VOL-DBDG-bkp01

V  NAME         RVG/VSET/CO  KSTATE   STATE    LENGTH   READPOL   PREFPLEX UTYPE

PL NAME         VOLUME       KSTATE   STATE    LENGTH   LAYOUT    NCOL/WID MODE

SD NAME         PLEX         DISK     DISKOFFS LENGTH   [COL/]OFF DEVICE   MODE

SV NAME         PLEX         VOLNAME  NVOLLAYR LENGTH   [COL/]OFF AM/NM    MODE

SC NAME         PLEX         CACHE    DISKOFFS LENGTH   [COL/]OFF DEVICE   MODE

DC NAME         PARENTVOL    LOGVOL

SP NAME         SNAPVOL      DCO

EX NAME         ASSOC        VC                       PERMS    MODE     STATE

v  VOL-DBDG-bkp01 -    ENABLED  ACTIVE   63955712 SELECT    -        fsgen

pl VOL-DBDG-bkp01-02 VOL-DBDG-bkp01 ENABLED ACTIVE 63955712 CONCAT - RW

sd DSK-DBDG-2B04-01 VOL-DBDG-bkp01-02 DSK-DBDG-2B4 0 28304128 0 hp_p95000_2b4 ENA

sd DSK-DBDG-2F98-01 VOL-DBDG-bkp01-02 DSK-DBDG-2F9 0 35651584 28304128 hp_p95000_2f9 ENA

Solaris1:/root#