Weird problem post power maintenance, no Powerpath pseudo name for Luns
We had the scheduled power maintenance activity in our data center. Every team was notified and we powered doen the servers through cron job. The servers were powered down first then storage (CX-380). And the reverse was followed during startup.
After the power shutdown, the SAN was booted up.
we forgot to disable the the snapshots. They were still active.
Once it came up, the sessions were not there, but snapshot was active.
Tried to deactivate the snapshot, failed with the error the "Failing Command: K10SnapCopyAdmin DBid 0 Op 1051. 71 00 80 43"
Tried to create a session and attach the snapshot to the session, still unable to deactivate it, neither was able to attach to any other session.
Looked at powerlink: It says the command issued to the snapshot is not owned by the SP.
I trespassed all the snapshot source lun to SPA, and then tried to deactivate, nothing happened.
Rebooted the SP's one by one through the management console. (http:/SP-IP/setup)
Then I was able to deactivate the snapshot, and I recreated it back.
All set on the clariion.
Moved to the server, that was powered off before the SAN and came up after SAN, when the snapshot problem was there.
All the snapshot luns were present without the emcpower values.
Veritas was not seeing the luns (no emcpower values)
Ran, powermt check, powercf -q ; nothing.
Decided to recreate the device tree on the operating system (Solaris 10 on Oracle E2900).
1. Removed the snapshot luns from the server storage group on Clariion.
2. cfgadm -c configure c2; cfgadm -c configure c3; devfsadm -Cv; powercf -q
Nothing happened, same issue.
3.Powermt check force; powercf -q ; removed 16 paths
Powermt check force; powercf -q; another 8 paths...... It went on for 3 or 4 times (don't know why)
But powercf -q never removed the paths mainly emcpower8,9,10,11 (snapshot luns), it was not available in powermt display option.
4.emcpadm print_mappings (it still showed emcpower8,9,10,11)
Again repeated cfgadm....
5. This time the path got deleted. Removed the entries from /etc/vx/disk.info file for emcpower8,9,10,11.
6. Deleted the older entries in /dev/vx/dmp and /dev/vx/rdmp
7. Deleted the entries from /dev/rdsk/emc and /dev/dsk/emcp (the files were not their)
8. Again presented the luns back to the storage group and issued cfgadm command same as in step 2, surprise powercf -q didn't found any new luns.
9. Powermt display dev=all, the values are still there with no pseudo name, back to square one.
Again, removed the snapshots from the storage group, and did the cleanup from step 2 to 5.
Now I presented 4 dummy luns of 100Mb each and did step 2. Powerpath found 4 luns and assigned them emcpower2,14,15,21. It didn’t assigned 8-11, so its still being used.
10. emcpadm renamepseudo -s emcpower2 -t emcpower8 ; renaming the new pseudo names to old one.
It complained the target emapcpower values are in use; powermt check reconfig
And again repeated 10 for emcpower2-5; success.
11. Back to step 8
12. We found the luns and were assigned emcpower2-9, all set; vxdctl enable; vxdisk list ; and the luns were in error condition, it said no path to device.
13. Checked the powermt display dev=all command, and all the paths were active
14. Issued the command vxconfigd -K, lot of errors VxVM vxconfigd ERROR V-5-1-8645 Error in claiming /dev/rdsk/......
15. vxdisk list; surprise the luns are online now.
After the power shutdown, the SAN was booted up.
we forgot to disable the the snapshots. They were still active.
Once it came up, the sessions were not there, but snapshot was active.
Tried to deactivate the snapshot, failed with the error the "Failing Command: K10SnapCopyAdmin DBid 0 Op 1051. 71 00 80 43"
Tried to create a session and attach the snapshot to the session, still unable to deactivate it, neither was able to attach to any other session.
Looked at powerlink: It says the command issued to the snapshot is not owned by the SP.
I trespassed all the snapshot source lun to SPA, and then tried to deactivate, nothing happened.
Rebooted the SP's one by one through the management console. (http:/SP-IP/setup)
Then I was able to deactivate the snapshot, and I recreated it back.
All set on the clariion.
Moved to the server, that was powered off before the SAN and came up after SAN, when the snapshot problem was there.
All the snapshot luns were present without the emcpower values.
Veritas was not seeing the luns (no emcpower values)
Ran, powermt check, powercf -q ; nothing.
Decided to recreate the device tree on the operating system (Solaris 10 on Oracle E2900).
1. Removed the snapshot luns from the server storage group on Clariion.
2. cfgadm -c configure c2; cfgadm -c configure c3; devfsadm -Cv; powercf -q
Nothing happened, same issue.
3.Powermt check force; powercf -q ; removed 16 paths
Powermt check force; powercf -q; another 8 paths...... It went on for 3 or 4 times (don't know why)
But powercf -q never removed the paths mainly emcpower8,9,10,11 (snapshot luns), it was not available in powermt display option.
4.emcpadm print_mappings (it still showed emcpower8,9,10,11)
Again repeated cfgadm....
5. This time the path got deleted. Removed the entries from /etc/vx/disk.info file for emcpower8,9,10,11.
6. Deleted the older entries in /dev/vx/dmp and /dev/vx/rdmp
7. Deleted the entries from /dev/rdsk/emc and /dev/dsk/emcp (the files were not their)
8. Again presented the luns back to the storage group and issued cfgadm command same as in step 2, surprise powercf -q didn't found any new luns.
9. Powermt display dev=all, the values are still there with no pseudo name, back to square one.
Again, removed the snapshots from the storage group, and did the cleanup from step 2 to 5.
Now I presented 4 dummy luns of 100Mb each and did step 2. Powerpath found 4 luns and assigned them emcpower2,14,15,21. It didn’t assigned 8-11, so its still being used.
10. emcpadm renamepseudo -s emcpower2 -t emcpower8 ; renaming the new pseudo names to old one.
It complained the target emapcpower values are in use; powermt check reconfig
And again repeated 10 for emcpower2-5; success.
11. Back to step 8
12. We found the luns and were assigned emcpower2-9, all set; vxdctl enable; vxdisk list ; and the luns were in error condition, it said no path to device.
13. Checked the powermt display dev=all command, and all the paths were active
14. Issued the command vxconfigd -K, lot of errors VxVM vxconfigd ERROR V-5-1-8645 Error in claiming /dev/rdsk/......
15. vxdisk list; surprise the luns are online now.
Comments
Post a Comment