Recovering VxVM Volumes - Disk is LOST or Plex Status is RECOVER

Recovering VxVM Volumes - Disk is LOST or Plex Status is RECOVER 

Step 1. identify the lost disks

   SolarisHost1# vxdisk list
   ...
   - - c4t2d4 SolarisHost1-Application5 failed was:EMC_CLARiiON0_4
   - - DG-application03 DG-application failed was:EMC_CLARiiON0_5

Step 2. for each lost disk, find the device

   SolarisHost1# vxdisk -o alldgs list | grep DG-application
   ...
   EMC_CLARiiON1_5 auto:sliced     -           (DG-application) online
   ...
Note: the disk should appear as online with a seemingly deported diskgroup, for example ””(DG-application)””.


Step 3. reinsert the disk in the Veritas config

   SolarisHost1# vxdg -g DG-application -k adddisk =

   SolarisHost1# vxdg -g DG-application -k adddisk DG-application03=EMC_CLARiiON1_5

Note: ”“DG-application03”” is the Veritas disk name and ”“EMC_CLARiiON1_5”” is the associated device.


Step 4. find the broken Veritas plexes in the diskgroup

   SolarisHost1# vxprint -g DG-application -th | grep DISABLED | grep RECOVER

Step 5. for each broken plex, clear the error flags

   SolarisHost1# vxmend -g DG-application -o force off Application-01
   SolarisHost1# vxmend -g DG-application on Application-01
   SolarisHost1# vxmend -g DG-application fix clean Application-01

Step 6. Start all volumes in the diskgroup

   SolarisHost1# vxvol -g DG-application startall

Step 7. Mount the filesystems

Optionally: do a fsck first! 

Also applied when PLEX hang in RECOVER state :

v  MyDG1   fsgen        DISABLED 1048576  -        ACTIVE   -       -
pl MyDG1-01 MyDG1  DISABLED 1048576  -        RECOVER  -       -
sd MyDG1-04 MyDG1-01 ENABLED 1048576 0        -        -       -

Force a Panic and collect a core dump - M Series SPARC

Force a Panic and collect a core dump

Scenario: System not reachable and no response through console.
Action Taken: Initate Forced Panic and forced savecore

Send break signal from chassis to the domain. (2 ways)

Before that check the domainmode. Should be on
setdomainmode -d 00 -m secure=on/off

Ways to send break.
reset -d 0 panic 
sendbreak -d 0 -y 



####Send break to domain 0

XSCF> sendbreak -d 0
Send break signal to DomainID 0?[y|n] :y
XSCF>

#### Open a new session and connect to console again

XSCF> console -f -d 0
Connect to DomainID 0?[y|n] :y 


### System reaches OK prompt. Give sync to force coredump

{a7} ok sync
panic[cpu167]/thread=2a174821ca0: sync initiated
sched: software trap 0x7f
pid=0, pc=0xf005d18c, sp=0x2a174820cb1, tstate=0x4400001407, context=0x0
g1-g7: 10511c4, 18de000, 60, 0, 0, 0, 2a174821ca0
00000000fdb79cd0 unix ync_handler+144 (182e400, f7, 3, 1, 1, 109f400)
%l0-3: 0000000001893e80 00000000018dddd0 00000000018ddc00 000000000000017f
%l4-7: 00000000018c1000 0000000000000000 00000000018bac00 0000000000000037
00000000fdb79da0 unix:vx_handler+80 (fdb02078, 183e038, 7fffffffffffffff, 1, 183e140, f006d515)
%l0-3: 000000000183e140 0000000000000000 0000000000000001 0000000000000001
%l4-7: 000000000182ec00 00000000f0000000 0000000001000000 0000000001019734
00000000fdb79e50 unix:callback_handler+20 (fdb02078, fdfea400, 0, 0, 0, 0)
%l0-3: 0000000000000016 00000000fdb79701 0000000000000000 0000000000000000
%l4-7: 0000000000000000 0000000000000000 0000000000000000 0000000000000001
syncing file systems... 570 568 568 568 568 568 568 568 568 568 568 568 568 568 568 568 568 568 568 568 568 568 done (not 

all i/o completed)
dumping to /dev/md/dsk/d11, offset 21476081664, content: kernel
100% done: 2916208 pages dumped, compression ratio 2.90, dump succeeded

### System reboots to init level 3
rebooting...
Resetting...
.POST Sequence 01 CPU Check
LSB#02 (XSB#01-0): POST 2.11.0 (2009/06/18 09:30)
LSB#06 (XSB#03-1): POST 2.11.0 (2009/06/18 09:30)
LSB#07 (XSB#03-2): POST 2.11.0 (2009/06/18 09:30)
LSB#03 (XSB#02-0): POST 2.11.0 (2009/06/18 09:30)
LSB#01 (XSB#00-1): POST 2.11.0 (2009/06/18 09:30)
LSB#04 (XSB#02-1): POST 2.11.0 (2009/06/18 09:30)
POST Sequence 02 Banner

IPMP failover

IPMP failover

(From the man page)

NAME

     if_mpadm - change operational status of interfaces within  a

     multipathing group

SYNOPSIS

     /usr/sbin/if_mpadm -d interface_name

     /usr/sbin/if_mpadm -r interface_name

If the interface is operational, you can use if_mpadm -d to  detach or  off-line  the  interface. If the interface is off-lined, use if_mpadm -r to revert it to its original state. When a network interface is off-lined,  all  network  access fails  over  to a different interface in the IP multipathing group. Any addresses that do not failover are brought  down. Addresses marked  with IFF_NOFAILOVER do not failover. They are marked down. After an interface is off-lined, the system  will  not use  the  interface for any outbound or inbound traffic, and the interface can be safely removed from the system  without any loss of network access.

The if_mpadm utility can be applied only to interfaces  that are part of an IP multipathing group.

Here the backup ip 174.48.22.36 is in IPMP group backup-lan


(SolarisHost1:/root)# ifconfig -a
lo0: flags=2001000849 mtu 8232 index 1
        inet 127.0.0.1 netmask ff000000
igb2: flags=69000842 mtu 0 index 2
        inet 0.0.0.0 netmask 0
        groupname backup-lan
        ether 3d:4a:97:76:8d:00
igb722000: flags=269000842 mtu 0 index 3
        inet 0.0.0.0 netmask 0
        groupname main-lan
        ether 3d:4a:97:76:8d:01
igb1: flags=1000843 mtu 1500 index 4
        inet 174.48.22.36 netmask fffffe00 broadcast 10.48.233.255
        groupname backup-lan
        ether 3d:4a:97:76:8d:0e
igb1:1: flags=1000843 mtu 1496 index 4
        inet 0.0.0.0 netmask ff000000 broadcast 0.255.255.255
igb722000: flags=201000843 mtu 1500 index 5
        inet 174.10.11.201 netmask ffffff00 broadcast 10.120.10.255
        groupname main-lan
        ether 3d:4a:97:76:8d:0f
igb722000:1: flags=201000843 mtu 1496 index 5
        inet 0.0.0.0 netmask ff000000 broadcast 0.255.255.255
(SolarisHost1:/root)#


We use if_mpadm to do a manual failover between an IPMP group.



The failover is performed now

(SolarisHost1:/root)# if_mpadm -d igb

Here the ip is now failed over to the other interface igb2.

(SolarisHost1:/root)# ifconfig -a

lo0: flags=2001000849 mtu 8232 index 1

        inet 127.0.0.1 netmask ff000000

igb2: flags=21000843 mtu 1496 index 2

        inet 0.0.0.0 netmask ff000000 broadcast 0.255.255.255

        groupname backup-lan

        ether 3d:4a:97:76:8d:00

igb2:1: flags=21000843 mtu 1496 index 2

        inet 174.48.22.36 netmask fffffe00 broadcast 10.48.233.255

igb722000: flags=269000842 mtu 0 index 3

        inet 0.0.0.0 netmask 0

        groupname main-lan

        ether 3d:4a:97:76:8d:01

igb1: flags=89000842 mtu 0 index 4

        inet 0.0.0.0 netmask 0

        groupname backup-lan

        ether 3d:4a:97:76:8d:0e

igb722000: flags=201000843 mtu 1500 index 5

        inet 174.10.11.201 netmask ffffff00 broadcast 10.120.10.255

        groupname main-lan

        ether 3d:4a:97:76:8d:0f

igb722000:1: flags=201000843 mtu 1496 index 5

        inet 0.0.0.0 netmask ff000000 broadcast 0.255.255.255

(SolarisHost1:/root)#

Reattaching the ip back to its original interface

(SolarisHost1:/root)# if_mpadm -r igb

(SolarisHost1:/root)# ifconfig -a

lo0: flags=2001000849 mtu 8232 index 1

        inet 127.0.0.1 netmask ff000000

igb2: flags=69000842 mtu 0 index 2

        inet 0.0.0.0 netmask 0

        groupname backup-lan

        ether 3d:4a:97:76:8d:00

igb722000: flags=269000842 mtu 0 index 3

        inet 0.0.0.0 netmask 0

        groupname main-lan

        ether 3d:4a:97:76:8d:01

igb1: flags=1000843 mtu 1500 index 4

        inet 174.48.22.36 netmask fffffe00 broadcast 10.48.233.255

        groupname backup-lan

        ether 3d:4a:97:76:8d:0e

igb1:1: flags=1000843 mtu 1496 index 4

        inet 0.0.0.0 netmask ff000000 broadcast 0.255.255.255

igb722000: flags=201000843 mtu 1500 index 5

        inet 174.10.11.201 netmask ffffff00 broadcast 10.120.10.255

        groupname main-lan

        ether 3d:4a:97:76:8d:0f

igb722000:1: flags=201000843 mtu 1496 index 5

        inet 0.0.0.0 netmask ff000000 broadcast 0.255.255.255

(SolarisHost1:/root)#

Sendmail - load average

Sendmail - load average

The mail queue is full and no more mail is delivered from the system. 

SolarisBox1:/root# mailq

                /var/spool/mqueue (845 requests)

----Q-ID---- --Size-- -----Q-Time----- ------------Sender/Recipient------------

q86892K23102X       0 Thu Sep  6 10:09 root

                                       schweitzer@mydomain.com

q8687N218303X       0 Thu Sep  6 10:07 root

                                       schweitzer@mydomain.com

q8688X121871X       0 Thu Sep  6 10:08 root

                                       schweitzer@mydomain.com

q868FH944906X       0 Thu Sep  6 10:15 root

                                       schweitzer@mydomain.com

....., Very High

SolarisBox1:/root# tail -f /var/log/syslog

Sep  6 10:21:03 SolarisBox1 sendmail[48275]: [ID 702911 mail.info] accepting connections again for daemon MTA-IPv4

Sep  6 10:21:03 SolarisBox1 sendmail[48275]: [ID 702911 mail.info] accepting connections again for daemon MTA-IPv6

Sep  6 10:21:03 SolarisBox1 sendmail[48275]: [ID 702911 mail.info] accepting connections again for daemon MSA

Sep  6 10:21:10 SolarisBox1 sendmail[48275]: [ID 702911 mail.info] rejecting connections on daemon MTA-IPv4: load average: 192

Sep  6 10:21:10 SolarisBox1 sendmail[48275]: [ID 702911 mail.info] rejecting connections on daemon MTA-IPv6: load average: 192

Sep  6 10:21:10 SolarisBox1 sendmail[48275]: [ID 702911 mail.info] rejecting connections on daemon MSA: load average: 192

Sep  6 10:21:10 SolarisBox1 sendmail[8293]: [ID 801593 mail.info] NOQUEUE: [10.1.5.17] did not issue MAIL/EXPN/VRFY/ETRN during connection to MTA-IPv4

SolarisBox1:/root# mailx -v -s "test" schweitzer@mydomain.com

EOT

SolarisBox1:/root# schweitzer@mydomain.com... queued

Tried restarting sendmail. No luck

Sep  6 10:15:48 SolarisBox1 sendmail[46426]: [ID 801593 mail.info] NOQUEUE: [10.3.5.16] did not issue MAIL/EXPN/VRFY/ETRN during connection to MTA-IPv4

Sep  6 10:16:16 SolarisBox1 sendmail[48275]: [ID 702911 mail.info] starting daemon (8.11.7p3+Sun): SMTP+queueing@00:15:00

Sep  6 10:16:16 SolarisBox1 sendmail[48275]: [ID 702911 mail.info] runqueue: Skipping queue run -- load average too high

Sep  6 10:16:17 SolarisBox1 sendmail[48395]: [ID 801593 mail.info] NOQUEUE: [10.1.5.16] did not issue MAIL/EXPN/VRFY/ETRN during connection to MTA-IPv4

Sep  6 10:16:18 SolarisBox1 sendmail[48572]: [ID 801593 mail.info] NOQUEUE: [10.3.5.16] did not issue MAIL/EXPN/VRFY/ETRN during connection to MTA-IPv4

Load average is reported as high

Sep  6 10:21:25 SolarisBox1 sendmail[48275]: [ID 702911 mail.info] rejecting connections on daemon MTA-IPv4: load average: 193

Sep  6 10:21:25 SolarisBox1 sendmail[48275]: [ID 702911 mail.info] rejecting connections on daemon MTA-IPv6: load average: 193

Sep  6 10:21:25 SolarisBox1 sendmail[48275]: [ID 702911 mail.info] rejecting connections on daemon MSA: load average: 193

Sep  6 10:21:40 SolarisBox1 sendmail[48275]: [ID 702911 mail.info] rejecting connections on daemon MTA-IPv4: load average: 192

Sep  6 10:21:40 SolarisBox1 sendmail[48275]: [ID 702911 mail.info] rejecting connections on daemon MTA-IPv6: load average: 192

Sep  6 10:21:40 SolarisBox1 sendmail[48275]: [ID 702911 mail.info] rejecting connections on daemon MSA: load average: 192

Sep  6 10:21:55 SolarisBox1 sendmail[48275]: [ID 702911 mail.info] rejecting connections on daemon MTA-IPv4: load average: 194

Sep  6 10:21:55 SolarisBox1 sendmail[48275]: [ID 702911 mail.info] rejecting connections on daemon MTA-IPv6: load average: 194

Sep  6 10:21:55 SolarisBox1 sendmail[48275]: [ID 702911 mail.info] rejecting connections on daemon MSA: load average: 194

Sep  6 10:21:55 SolarisBox1 sendmail[10488]: [ID 702911 mail.info] runqueue: Skipping queue run -- load average too high

SolarisBox1:/root# grep QueueLA /etc/mail/sendmail.cf

O QueueLA=128

SolarisBox1:/root#

Temporarily edit the queue size to above the load and restart sendmail

# load average at which we just queue messages

O QueueLA=228

# load average at which we refuse connections

O RefuseLA=292

SolarisBox1:/root# /etc/init.d/sendmail stop

SolarisBox1:/root# /etc/init.d/sendmail start

SolarisBox1:/root#

Sep  6 10:27:19 SolarisBox1 sendmail[30901]: [ID 702911 mail.info] rejecting connections on daemon MSA: load average: 202

Sep  6 10:27:34 SolarisBox1 sendmail[30901]: [ID 702911 mail.info] rejecting connections on daemon MTA-IPv4: load average: 200

Sep  6 10:27:34 SolarisBox1 sendmail[30901]: [ID 702911 mail.info] rejecting connections on daemon MTA-IPv6: load average: 200

Sep  6 10:27:34 SolarisBox1 sendmail[30901]: [ID 702911 mail.info] rejecting connections on daemon MSA: load average: 200

Sep  6 10:27:59 SolarisBox1 sendmail[33515]: [ID 702911 mail.info] starting daemon (8.11.7p3+Sun): SMTP+queueing@00:15:00

Sep  6 10:28:00 SolarisBox1 sendmail[33540]: [ID 801593 mail.info] NOQUEUE: [10.3.1.17] did not issue MAIL/EXPN/VRFY/ETRN during connection to MTA-IPv4

Sep  6 10:28:01 SolarisBox1 sendmail[33571]: [ID 801593 mail.info] NOQUEUE: [10.1.5.16] did not issue MAIL/EXPN/VRFY/ETRN during connection to MTA-IPv4

Sep  6 10:28:02 SolarisBox1 sendmail[33653]: [ID 801593 mail.info] NOQUEUE: [10.1.5.17] did not issue MAIL/EXPN/VRFY/ETRN during connection to MTA-IPv4

Sep  6 10:28:02 SolarisBox1 sendmail[33702]: [ID 801593 mail.info] NOQUEUE: [10.3.5.16] did not issue MAIL/EXPN/VRFY/ETRN during connection to MTA-IPv4

Sep  6 10:28:05 SolarisBox1 sendmail[33922]: [ID 801593 mail.info] NOQUEUE: [10.3.5.17] did not issue MAIL/EXPN/VRFY/ETRN during connection to MTA-IPv4

Sep  6 10:28:06 SolarisBox1 sendmail[34017]: [ID 801593 mail.info] NOQUEUE: [10.1.5.16] did not issue MAIL/EXPN/VRFY/ETRN during connection to MTA-IPv4

Sep  6 10:28:07 SolarisBox1 sendmail[34139]: [ID 801593 mail.info] NOQUEUE: [10.1.5.17] did not issue MAIL/EXPN/VRFY/ETRN during connection to MTA-IPv4

Sep  6 10:28:07 SolarisBox1 sendmail[34197]: [ID 801593 mail.info] NOQUEUE: [10.3.5.16] did not issue MAIL/EXPN/VRFY/ETRN during connection to MTA-IPv4

Sep  6 10:28:08 SolarisBox1 sendmail[33516]: [ID 801593 mail.info] q86892K23102: to=schweitzer@mydomain.com, ctladdr=root (0/1), delay=00:19:06, xdelay=00:00:00, mailer=relay, pri=120058, 

relay=mailrelay [10.5.253.35], dsn=2.0.0, stat=Sent ( 201209060809.q86892K23102@SolarisBox1.sub.mydomain.com Queued mail for delivery)

Sep  6 10:28:09 SolarisBox1 sendmail[33516]: [ID 801593 mail.info] q8687N218303: to=schweitzer@mydomain.com, ctladdr=root (0/1), delay=00:20:46, xdelay=00:00:01, mailer=relay, pri=120058, 

relay=mailrelay [10.5.253.35], dsn=2.0.0, stat=Sent ( 201209060807.q8687N218303@SolarisBox1.sub.mydomain.com Queued mail for delivery)

SolarisBox1:/root#

SolarisBox1:/root# mailq

/var/spool/mqueue is empty

SolarisBox1:/root#

Messages delivered.

Now change back to the default value

# load average at which we just queue messages

O QueueLA=128

# load average at which we refuse connections

O RefuseLA=192

SolarisBox1:/root# /etc/init.d/sendmail stop

SolarisBox1:/root# /etc/init.d/sendmail start

SolarisBox1:/root#

SolarisBox1:/root#

NFS4 nobody permissions

NFS4 nobody permissions

When the NFS share is mounted in a client, the permissions are displayed as nobody. If this is in NFS4, it is because of the new representation of users and group information between the systems. 

(MyClient:/)# mount -F nfs MyServer:/root/application/archive /application/archive

nfs mount: mount: /application/archive: Permission denied

(MyClient:/)# nslookup MyClient

Server:         175.21.86.11

Address:        175.21.86.11#53

Name:   MyClient.bc

Address: 177.10.7.11

The share is not ok.

(MyServer:/)# vi /etc/dfs/dfstab

"/etc/dfs/dfstab" 12 lines, 763 characters

#       Place share(1M) commands here for automatic execution

#       on entering init state 3.

#

#       Issue the command '/etc/init.d/nfs.server start' to run the NFS

#       daemon processes and the share commands, after adding the very

#       first entry to this file.

#

#       share [-F fstype] [ -o options] [-d ""] [resource]

#       .e.g,

#       share  -F nfs  -o rw=engineering  -d "home dirs"  /export/home2

#

share -F nfs -o rw=MyClient.bc,anon=0 /root/application/archive

~

~~

~

"/etc/dfs/dfstab" 12 lines, 775 characters

(MyServer:/)#

(MyServer:/)# shareall

(MyServer:/)#

Now the dir is shared.

In the client, NFS is now mounted

(MyClient:/)# mount -F nfs MyServer:/root/application/archive /application/archive

(MyClient:/)#

(MyClient:/)#

(MyClient:/)# df -h /application/archive

Filesystem             size   used  avail capacity  Mounted on

MyServer:/root/application/archive

                       880G   720G   153G    83%    /application/archive

(MyClient:/)# grep /application/archive /etc/mnttab

MyServer:/root/application/archive     /application/archive   nfs     rw,nodevices,xattr,zone=MyClient,dev=59412eb    1361790688

(MyClient:/)# ls -ld /application/archive

drwxr-xr-x+ 36 nobody   nobody      1024 Oct  5 18:55 /application/archive

Solaris handles one NFSv4 domain. 

If the client or server receives an user/group string that does not match its domain, it will map that user/group into uid/gid "nobody" (60001).

(MyClient:/)# grep NFSMAPID_DOMAIN /etc/default/nfs

NFSMAPID_DOMAIN=mywrongdomain.com //wrong domain

(MyClient:/)#

(MyClient:/)#

(MyClient:/)# cp /etc/default/nfs /etc/default/nfs.old

(MyClient:/)# vi /etc/default/nfs

# Specifies to nfsmapid daemon that it is to override its default

# behavior of using the DNS domain, and that it is to use 'domain' as

# the domain to append to outbound attribute strings, and that it is to

# use 'domain' to compare against inbound attribute strings.

NFSMAPID_DOMAIN=nfscorrectdomain.nfs //Correct nfs domain that can map the user/group

~

~

Restart the nfs mapid

(MyClient:/)# svcs -a|grep mapid

online         Feb_07   svc:/network/nfs/mapid:default

(MyClient:/)# svcadm restart /network/nfs/mapid

(MyClient:/)#

(MyClient:/)# svcs -a|grep mapid

online         12:13:36 svc:/network/nfs/mapid:default

(MyClient:/)#

(MyClient:/)# ls -ld /application/archive

drwxr-xr-x+ 36 applicationload applicationgr      1024 Oct  5 18:55 /application/archive

(MyClient:/)#

Ref: nfsmapid_domain from oracle blog

rsh and rlogin port secrets

rsh and rlogin are two utilities for establishing and executing commands on remote host.
It is insecure and doesn't encrypt the data it transfers. 


The default port used by rsh is 514
The default port used by rlogin is 513


The issue:
Firewall has blocked 513, but 514 is open




We observe the following behavior:


//No response. snooping shows no response from remote host 10.56.41.11

sunhost1:/# rsh 10.56.41.11
^Csunhost1:/#


//We have got the desired result

sunhost1:/# rsh 10.56.41.11 "uname -a"
SunOS MDSSMP01 5.10 Generic_127127-11 sun4v sparc SUNW,Sun-Fire-T200
sunhost1:/#


Why is the difference?
As per MAN page of rsh - If you omit command, instead of executing a single  command, rsh logs you in on the remote host using rlogin(1).


Thus the rsh without a sub-command has used rlogin and thus port 513 was used which resulted in packets dropped by firewall.

The savemail panic

The savemail panic message is noted when sending mails.


Below messages are found in syslog.


(sunstation1:/)# tail /var/log/syslog
Jun  8 10:20:01 sunstation1 sendmail[1067]: [ID 577507 mail.debug] tid= 1: serverAddr=45.222.28.206:20389
Jun  8 10:20:01 sunstation1 sendmail[1067]: [ID 939703 mail.debug] tid= 1: AuthType=1
Jun  8 10:20:01 sunstation1 sendmail[1067]: [ID 142272 mail.debug] tid= 1: TlsType=0
Jun  8 10:20:01 sunstation1 sendmail[1067]: [ID 537450 mail.debug] tid= 1: SaslMech=0
Jun  8 10:20:01 sunstation1 sendmail[1067]: [ID 625532 mail.debug] tid= 1: SaslOpt=0
Jun  8 10:20:01 sunstation1 sendmail[1067]: [ID 639905 mail.debug] tid= 1: userID=cn=proxyagent,ou=profile,dc=dev,dc=mobile,dc=belgacom,dc=be
Jun  8 10:20:01 sunstation1 sendmail[1067]: [ID 801593 mail.info] p588K1BN001067: p588K1BO001067: return to sender: aliasing/forwarding loop broken
Jun  8 10:20:01 sunstation1 sendmail[1067]: [ID 801593 mail.notice] p588K1BO001067: setsender: : invalid or unparsable, received from localhost
Jun  8 10:20:01 sunstation1 sendmail[1067]: [ID 801593 mail.alert] p588K1BN001067: Losing ./qfp588K1BN001067: savemail panic
Jun  8 10:20:01 sunstation1 sendmail[1067]: [ID 801593 mail.crit] p588K1BN001067: SYSERR(root): savemail: cannot save rejected email anywhere

(sunstation1:/)# mailx -s "Test" aaron@domain.com
EOT
(sunstation1:/)# Jun  8 10:33:35 sunstation1 sendmail[15981]: [ID 801593 mail.alert] p588XZAI015981: Losing ./qfp588XZAI015981: savemail panic



Can be rectified by removing the ldap datasource from nsswitch.conf file.

(sunstation1:/)# cat /etc/nsswitch.conf | grep ali
aliases:    files ldap
(sunstation1:/)#


automount:  files ldap
aliases:    files
"/etc/nsswitch.conf" 49 lines, 1416 characters
(sunstation1:/)#



Restart sendmail services

(sunstation1:/)# svcs -a | grep sendmail
online         Mar_28   svc:/network/smtp:sendmail
online         Mar_28   svc:/network/sendmail-client:default
(sunstation1:/)#

(sunstation1:/)# svcadm restart svc:/network/smtp:sendmail
(sunstation1:/)# svcadm restart svc:/network/sendmail-client:default

(sunstation1:/)# mailx -s "Test" aaron@domain.com
EOT
(sunstation1:/)#

VxVM extension - stripped volume

Issue encountered while extending a volume even when there is enough space available in the DG.


sunstation1:/root# vxassist -g DG-DB001 maxsize
Maximum volume size: 326191104 (159273Mb)
sunstation1:/root#
sunstation1:/root#
sunstation1:/root# /etc/vx/bin/vxresize -g DG-DB001 VOL-DB001-data1 +50g
VxVM vxassist ERROR V-5-1-436 Cannot allocate space to grow volume to 1498001965 blocks
VxVM vxresize ERROR V-5-1-4703 Problem running vxassist command for volume VOL-DB001-data1, in diskgroup DG-DB001
sunstation1:/root#
sunstation1:/root#
sunstation1:/root# /etc/vx/bin/vxresize -g DG-DB001 VOL-DB001-data1 +5g
sunstation1:/root# /etc/vx/bin/vxresize -g DG-DB001 VOL-DB001-data1 +5g
VxVM vxassist ERROR V-5-1-436 Cannot allocate space to grow volume to 1424601645 blocks
VxVM vxresize ERROR V-5-1-4703 Problem running vxassist command for volume VOL-DB001-data1, in diskgroup DG-DB001
sunstation1:/root#
sunstation1:/root#
sunstation1:/root#
sunstation1:/root# vxassist -g DG-DB001 maxsize
Maximum volume size: 305178624 (149013Mb)
sunstation1:/root#
sunstation1:/root#


Reason: Stripped volume


sunstation1:/root# vxprint -htqg DG-DB001 VOL-DB001-data1
v  VOL-DB001-data1 -       ENABLED  ACTIVE   1414115885 SELECT  VOL-DB001-data1-01 fsgen
pl VOL-DB001-data1-01 VOL-DB001-data1 ENABLED ACTIVE 1414164480 STRIPE 8/512 RW
sd DSK-DG-DB001-110d-01 VOL-DB001-data1-01 DSK-DG-DB001-110d 3328 170826240 0/0 xp24k1_110d ENA
sd DSK-DG-DB001-1369-02 VOL-DB001-data1-01 DSK-DG-DB001-1369 2107648 3317760 0/170826240 xp24k1_1369 ENA
sd DSK-DG-DB001-1369-04 VOL-DB001-data1-01 DSK-DG-DB001-1369 110288128 2626560 0/174144000 xp24k1_1369 ENA
sd DSK-DG-DB001-110f-01 VOL-DB001-data1-01 DSK-DG-DB001-110f 3328 170826240 1/0 xp24k1_110f ENA
sd DSK-DG-DB001-136B-02 VOL-DB001-data1-01 DSK-DG-DB001-136B 20969728 5944320 1/170826240 xp24k1_136b ENA
sd DSK-DG-DB001-1100-01 VOL-DB001-data1-01 DSK-DG-DB001-1100 3328 170826240 2/0 xp24k1_1100 ENA
sd DSK-DG-DB001-1112-03 VOL-DB001-data1-01 DSK-DG-DB001-1112 21292288 5944320 2/170826240 xp24k1_1112 ENA
sd DSK-DG-DB001-1101-01 VOL-DB001-data1-01 DSK-DG-DB001-1101 3328 170826240 3/0 xp24k1_1101 ENA
sd DSK-DG-DB001-110e-02 VOL-DB001-data1-01 DSK-DG-DB001-110e 49024768 5944320 3/170826240 xp24k1_110e ENA
sd DSK-DG-DB001-1102-01 VOL-DB001-data1-01 DSK-DG-DB001-1102 3328 170826240 4/0 xp24k1_1102 ENA
sd DSK-DG-DB001-110c-02 VOL-DB001-data1-01 DSK-DG-DB001-110c 83891968 5944320 4/170826240 xp24k1_110c ENA
sd DSK-DG-DB001-1103-01 VOL-DB001-data1-01 DSK-DG-DB001-1103 3328 170826240 5/0 xp24k1_1103 ENA
sd DSK-DG-DB001-1368-02 VOL-DB001-data1-01 DSK-DG-DB001-1368 135232768 5944320 5/170826240 xp24k1_1368 ENA
sd DSK-DG-DB001-1104-01 VOL-DB001-data1-01 DSK-DG-DB001-1104 3328 170826240 6/0 xp24k1_1104 ENA
sd DSK-DG-DB001-110b-02 VOL-DB001-data1-01 DSK-DG-DB001-110b 164723968 5944320 6/170826240 xp24k1_110b ENA
sd DSK-DG-DB001-1105-01 VOL-DB001-data1-01 DSK-DG-DB001-1105 3328 170826240 7/0 xp24k1_1105 ENA
sd DSK-DG-DB001-1111-03 VOL-DB001-data1-01 DSK-DG-DB001-1111 165430528 5399040 7/170826240 xp24k1_1111 ENA
sd DSK-DG-DB001-1640-01 VOL-DB001-data1-01 DSK-DG-DB001-1640 3328 545280 7/176225280 xp24k1_1640 ENA
sunstation1:/root#
sunstation1:/root#
sunstation1:/root# vxassist -g DG-DB001 maxgrow VOL-DB001-data1
Volume VOL-DB001-data1 can be extended by 4163584 to: 1418279469 (692519Mb+557 sectors)
sunstation1:/root#






Striped volumes must have enough devices and free space to grow all columns in parallel Solution.


Volume Manager has the following internal restrictions regarding the extension of striped volume columns:
Device(s) used in one column cannot be used in any other columns in that volume
All stripe columns must be grown in parallel



To resolve this issue you must add enough storage devices to satisfy the above constraints or use a relayout operation to convert the volume's column count.

ZFS Mounting issue

 

ZFS file systems defined:

 

(zone01:/)# zfs list | grep appco

zone01-pool/zone01/dataset/appco               60.5K  99.9M  60.5K  /appco

zone01-pool/zone01/dataset/appcocow             494K  99.5M   494K  /appco/cow

zone01-pool/zone01/dataset/appcows             418K   800M   418K  /appco/ws

(zone01:/)#

(zone01:/)#

Mounted:

 

(zone01:/)# df -k | grep appco

zone01-pool/zone01/dataset/appco  102400      60  102339     1%    /appco

zone01-pool/zone01/dataset/appcows  819200     417  818782     1%    /appco/ws

(zone01:/)#

(zone01:/)#

(zone01:/)# zfs get all zone01-pool/zone01/dataset/appcocow

NAME                              PROPERTY              VALUE                  SOURCE

zone01-pool/zone01/dataset/appcocow  type                  filesystem             -

zone01-pool/zone01/dataset/appcocow  creation              Mon Jan  3 12:04 2011  -

zone01-pool/zone01/dataset/appcocow  used                  494K                   -

zone01-pool/zone01/dataset/appcocow  available             99.5M                  -

zone01-pool/zone01/dataset/appcocow  referenced            494K                   -

zone01-pool/zone01/dataset/appcocow  compressratio         1.00x                  -

zone01-pool/zone01/dataset/appcocow  mounted               no                     -

zone01-pool/zone01/dataset/appcocow  quota                 100M                   local

zone01-pool/zone01/dataset/appcocow  reservation           100M                   local

zone01-pool/zone01/dataset/appcocow  recordsize            128K                   default

zone01-pool/zone01/dataset/appcocow  mountpoint            /appco/cow              local

zone01-pool/zone01/dataset/appcocow  sharenfs              off                    default

zone01-pool/zone01/dataset/appcocow  checksum              on                     default

zone01-pool/zone01/dataset/appcocow  compression           off                    default

zone01-pool/zone01/dataset/appcocow  atime                 on                     default

zone01-pool/zone01/dataset/appcocow  devices               on                     default

zone01-pool/zone01/dataset/appcocow  exec                  on                     default

zone01-pool/zone01/dataset/appcocow  setuid                on                     default

zone01-pool/zone01/dataset/appcocow  readonly              off                    default

zone01-pool/zone01/dataset/appcocow  zoned                 on                     inherited from zone01-pool/zone01/dataset

zone01-pool/zone01/dataset/appcocow  snapdir               hidden                 default

zone01-pool/zone01/dataset/appcocow  aclmode               groupmask              default

zone01-pool/zone01/dataset/appcocow  aclinherit            restricted             default

zone01-pool/zone01/dataset/appcocow  canmount              on                     default

zone01-pool/zone01/dataset/appcocow  shareiscsi            off                    default

zone01-pool/zone01/dataset/appcocow  xattr                 on                     default

zone01-pool/zone01/dataset/appcocow  copies                1                      default

zone01-pool/zone01/dataset/appcocow  version               4                      -

zone01-pool/zone01/dataset/appcocow  utf8only              off                    -

zone01-pool/zone01/dataset/appcocow  normalization         none                   -

zone01-pool/zone01/dataset/appcocow  casesensitivity       sensitive              -

zone01-pool/zone01/dataset/appcocow  vscan                 off                    default

zone01-pool/zone01/dataset/appcocow  nbmand                off                    default

zone01-pool/zone01/dataset/appcocow  sharesmb              off                    default

zone01-pool/zone01/dataset/appcocow  refquota              none                   default

zone01-pool/zone01/dataset/appcocow  refreservation        none                   default

zone01-pool/zone01/dataset/appcocow  primarycache          all                    default

zone01-pool/zone01/dataset/appcocow  secondarycache        all                    default

zone01-pool/zone01/dataset/appcocow  usedbysnapshots       0                      -

zone01-pool/zone01/dataset/appcocow  usedbydataset         494K                   -

zone01-pool/zone01/dataset/appcocow  usedbychildren        0                      -

zone01-pool/zone01/dataset/appcocow  usedbyrefreservation  0                      -

zone01-pool/zone01/dataset/appcocow  logbias               latency                default

(zone01:/)#

(zone01:/)# mkdir /appco/cow

mkdir: Failed to make directory "/appco/cow"; File exists

(zone01:/)#

(zone01:/)# mount zone01-pool/zone01/dataset/appcocow

mount: Mount point cannot be determined

(zone01:/)#

(zone01:/)#

Filesystem /appco/cow is not mounted:

 

(zone01:/)# df -k /appco/cow

Filesystem            kbytes    used   avail capacity  Mounted on

zone01-pool/zone01/dataset/appco

                      102400      60  102339     1%    /appco

(zone01:/)#

 

Traditional mounting doesn't work because of zfs current property:

 

(zone01:/)# mount -F zfs zone01-pool/zone01/dataset/appcocow /appco/cow

filesystem 'zone01-pool/zone01/dataset/appcocow' cannot be mounted using 'mount -F zfs'

Use 'zfs set mountpoint=/appco/cow' instead.

If you must use 'mount -F zfs' or /etc/vfstab, use 'zfs set mountpoint=legacy'.

See zfs(1M) for more information.

(zone01:/)#

(zone01:/)#

 

Mount is not success even after setting the mount point:

 

(zone01:/)# zfs set mountpoint=/appco/cow zone01-pool/zone01/dataset/appcocow

(zone01:/)#

(zone01:/)# df -k /appco/cow

Filesystem            kbytes    used   avail capacity  Mounted on

zone01-pool/zone01/dataset/appco

                      102400      60  102339     1%    /appco

(zone01:/)#

 

(zone01:/)# df -k | grep appco

zone01-pool/zone01/dataset/appco  102400      60  102339     1%    /appco

zone01-pool/zone01/dataset/appcows  819200     417  818782     1%    /appco/ws

(zone01:/)#

(zone01:/)#

 

Set the mountpoint option as legacy:

 

(zone01:/)# zfs set mountpoint=legacy zone01-pool/zone01/dataset/appcocow

(zone01:/)#

(zone01:/)#

(zone01:/)# mount -F zfs zone01-pool/zone01/dataset/appcocow /appco/cow

(zone01:/)#

(zone01:/)#

(zone01:/)# df -k /appco/cow

Filesystem            kbytes    used   avail capacity  Mounted on

zone01-pool/zone01/dataset/appcocow

                      102400     494  101905     1%    /appco/cow

Successfully mounted

 

(zone01:/)# df -k | grep appco

zone01-pool/zone01/dataset/appco  102400      60  102339     1%    /appco

zone01-pool/zone01/dataset/appcows  819200     417  818782     1%    /appco/ws

zone01-pool/zone01/dataset/appcocow  102400     494  101905     1%    /appco/cow

(zone01:/)#

 

Set the property back

 

(zone01:/)# zfs set mountpoint=/appco/cow zone01-pool/zone01/dataset/appcocow

(zone01:/)# df -k /appco/cow

Filesystem            kbytes    used   avail capacity  Mounted on

zone01-pool/zone01/dataset/appcocow

                      102400     494  101905     1%    /appco/cow

(zone01:/)#

or mount using zfs mount option

(zone01:/)# zfs mount zone01-pool/zone01/dataset/appcocow