Solaris 10 patching best practices

We have a mammoth task every quarter to patch all our Unix servers, and every month if the servers are in DMZ. With time we have formulated a couple of steps that makes life easier during patching of Sun servers.

1. We apply the same patch cluster on every Sun server for a particular quarter, no matter when we patch the server during a particular quarter .

2. Once the patch is downloaded at the centralized location, it is distributed to every server on the / directory. This may sound a little off track as the patch cluster consumes a good 1GB of space and the patching process might fail if we do not have sufficient disk space on our / and /var file system, but we had faced some NFS issues before where we had a lot of trouble bringing up the system after the nfs server crash in middle of the patching process.

3. We inform the database team before hand (1 week) about the patch cluster release and also share with the them release notes, for them to identify any issues with the database and the current patch cluster. If there is any set of point patch for the database that needs to be applied post server patching then they plan the production downtime in addition to ours.

4. Get the downtime approved from the Business Unit and all the changes approved through the change control board of our organization. We take a blanket downtime approval from the change control board and later follow up with the Business unit to approve the downtime as per our schedule. If their is some critical release then it might get postponed to subsequent weekend, but never post the patching quarter. In the worst case if we cannot patch the server, then this is raised as an exception with the audit team and has to be approved by business unit.

5. Before the patching starts (preferably 1 day before), we use to run a script that collects system information like vfstab, veritas information etc. We take backup of the critical system folder/files and put it on a different server. We also ensure the solaris DVD is inserted into the server, just in case we have to boot off from the DVD.

6. Send out the notification to the user about the system un availability during the patching window. This is usually comunicated by Helpdesk.

7. The implementation part is basically a sequence of steps which I will touch in brief:
  • Break the mirror, if you are using SVM. Detach it completely (metadetach, metaclear)
  • Mount the mirror and change /mnt/etc/vfstab entry from md device to native device mapping i.e /dev/dsk/c1t0d0s0 and hash out other entries
  • Comment out the rootdev entry from the system file of the mirror disk.
  • Without unmounting the mirror go to OK prompt, just to ensure boot-archive gets updated on the mirror disk as well. If you haven't run installboot on the mirror disk, please do it so to make it bootable.
  • Boot off from the mirror disk, to ensure that the second disk is bootable in case we run into any issues post patching.
  • Go back to OK prompt and boot from first disk, initially it was required to be in single user mode but now you can patch the server normal runlevel also
  • Start the patching process
  • Reboot
  • Wait for a week to system to stablize and if no issues are reported, reattach the mirror back. If their are issues then you have the second disk to boot from.
  • Uncomment out the vfstab entries and mount all the file system.
  • Send out the notification to the DBA's to start the database.

8. Update the logbook.

Comments

  1. Sudarshan4unixSeptember 30, 2010 at 6:23 PM

    Very helpful information summarizing steps in brief which you can not get in any blogs.

    ReplyDelete
  2. satheeshDecember 8, 2010 at 11:46 PM

    hi rupesh,
    tanks a lot.
    i was looking for the same content, procedure and i got it. I hope in IBM they use the same strategy.??
    chck out my blog too. http://unixadvice.blogspot.com
    Satheesh

    ReplyDelete

Post a Comment

Popular posts from this blog

zpool and power path partial compatibility

Is power path fully supported with zpools, no. It's partial. You can have the pseudo devices assigned to a pool but it uses the native ctd disk only. Recently I lost 2 paths from the vnx array to m9000 server running Solaris 11.1. When I ran the zpool status -x command some of the pools were online while some of them were un available. Power path was reporting 2 paths only but it was active. So ideally the zpools should be online. But it's not. I enabled the remaining 2 paths and all became online. Internally the pools were pointing to one of the ctd devices that belonged to dead path and it was not coming online. Also few of the emcpower devices were renamed, so I had to do zpool import -d with the alternate disk. It seems power path is not completely supported now but may become in future.
Read more

Recoverpoint replication reporting with vmax splitter

Guys, Have you ever thought of automating the RP replication status and send status via email or post it to any reporting platform. Since the splitter impacts the VMAX FA's and there is an overhead of 13% on the utilization its good idea to see if the replication is driving the utilization of vmax FA. There are ways of pulling iops/read/write using symstat but its utilization (Director CPU utilization etc) which plays a significant role. As it crosses 80% we start seeing the increase in device response time. However, to measure FA utilization we will have to go through SMAS UI then pull out the reports. Its manual effort. But there is a way to automate it through REST API. You can get the details what type of objects are supported by pointing to : https://smc server :8443/univmax/restapi Coming to the point, getting recoverpoint consistency group status fully automated doesn't seems to be tough. Lets get through the steps. 1. Hash your SSH keys of the unix host to ...
Read more

Weird problem post power maintenance, no Powerpath pseudo name for Luns

We had the scheduled power maintenance activity in our data center. Every team was notified and we powered doen the servers through cron job. The servers were powered down first then storage (CX-380). And the reverse was followed during startup. After the power shutdown, the SAN was booted up. we forgot to disable the the snapshots. They were still active. Once it came up, the sessions were not there, but snapshot was active. Tried to deactivate the snapshot, failed with the error the "Failing Command: K10SnapCopyAdmin DBid 0 Op 1051. 71 00 80 43" Tried to create a session and attach the snapshot to the session, still unable to deactivate it, neither was able to attach to any other session. Looked at powerlink: It says the command issued to the snapshot is not owned by the SP. I trespassed all the snapshot source lun to SPA, and then tried to deactivate, nothing happened. Rebooted the SP's one by one through the management console. (http:/SP-IP/setup) Then I was able to...
Read more