Move LVM from multiple local and iscsi disk to SAN

We are setting up our DR site. For replication we are using EMC Recoverpoint appliance. The license what we have works for only SAN volumes on EMC arrays namely VNX in our case. The database and file shares which are on SAN have no problem however, the critical file shares that are on local storage or Dell iscsi storage have an issue. It sounds easy to move them to SAN and start replication but not so. Consider below:

1. The LVM housing the file shares on a Dell server contains PV from local disk partition (yes not complete disk but like /dev/sda10 on extended partition) and other full disk from iscsi storage /dev/sdq and /dev/sdi
2. Total size is ~400 GB with 100 GB from local disk partition (total local disk size is 300 GB) and 300 (200 + 100) from iscsi. I know its wrong but it was done long time ago and we were running out of space. 3. The dell server does not have FC connection.
4. The file system structure contains 32000 nested directory structure with approx 1.2 billion files on JFS volumes. Copy is ruled out, it takes 7 days for a full scan.
5. Normal dd of the volume will not work, as it spans multiple disk. We tried it, doesn't works with the identical size disk also.

Solution:

1. Find a system with FC connections and jfs compiled in with the kernel. We will call it SANA.
2. Carve out 3 Luns of 300 GB, 200 Gb and 100 Gb luns on SAN. Add another 400 GB temporary space for file copy. Mount it as /tmp/san on SANA
3. Share the 400GB space over NFS from FC server to the original jfs server, mount it on production as /tmp/t (name it whatever you feel like).
4. Login to original jfs, Take partition dump and sfdisk output of local disk: dd if=/dev/sda of=/tmp/t/sda.mbr bs=512 count=1 sfdisk /dev/sda > /tmp/t/sda.sfdisk
5. Take for other iscsi disk as well: dd if=/dev/sdi of=/tmp/t/sdi.mbr bs=512 count=1 dd if=/dev/sdq of=/tmp/t/sdq.mbr bs=512 count=1
6. Login to SANA, Copy the partition on SAN disk, assuming /dev/san1 (equiv to 300 GB local) /dev/san2 (equiv 200 GB iscsi) and /dev/san3 (equiv 100 GB iscsi)
     dd if=/tmp/san/sda.mbr of=/dev/san1 bs=512 count=1
     sfdisk /dev/san1 < /tmp/san/sda.sfdisk (This will copy the extended partition table)
     dd if=/tmp/san/sdi.mbr of=/dev/san2 bs=512 count=1
     dd if=/tmp/san/sdq.mbr of=/dev/san3 bs=512 count=1
7. We are done with creating the partition table on SANA, back to original server.
8. Find the block size of the jfs volume using jfs_tune -l , it will be 4096 please check 
9. If downtime is approved, then unmount the file system on original server and disable the vg with vgchange -ay . or if you know nobody is accessing it then proceed below. It will be good if you disable it. 
9. Issue the dd command to copy the image to NFS mount point: 
     dd if=/dev/sda10 of=/tmp/t/sda10.img bs=4096 (we are copying only the partition) 
     dd if=/dev/sdq of=/tmp/t/sdq.img bs=4096 
     dd if=/dev/sdi of=/tmp/t/sdi.img bs=4096 
10. Wait till the copy is over, once done login to SANA 
11. Please ensure you are copying the image to correct disk/partition 
     dd if=/tmp/san/sda10.img /dev/san1-10 bs=4096 (correct the 10th partition on /dev/san1, as it was on original server) 
     dd if=/tmp/san/sdq.img of=/dev/san2 bs=4096 
     dd if=/tmp/san/sdi.img of=/dev/san2 bs=4096 
12. Once done, do a pvscan to see the volume group is visible 
13. Activate the volume vgchange -ay  
14. Run fsck.jfs on the newly created volume and mount it (mount -t jfs ) 

Viola, the volume has been moved to SAN and we are ready for replication and it tool cool 9 hours to do so (actual activity was 30 minutes, rest was data copy :)) 
We will use the same process to move other local volumes to SAN to create a perfect DR.

Comments

Post a Comment

Popular posts from this blog

zpool and power path partial compatibility

Is power path fully supported with zpools, no. It's partial. You can have the pseudo devices assigned to a pool but it uses the native ctd disk only. Recently I lost 2 paths from the vnx array to m9000 server running Solaris 11.1. When I ran the zpool status -x command some of the pools were online while some of them were un available. Power path was reporting 2 paths only but it was active. So ideally the zpools should be online. But it's not. I enabled the remaining 2 paths and all became online. Internally the pools were pointing to one of the ctd devices that belonged to dead path and it was not coming online. Also few of the emcpower devices were renamed, so I had to do zpool import -d with the alternate disk. It seems power path is not completely supported now but may become in future.
Read more

Recoverpoint replication reporting with vmax splitter

Guys, Have you ever thought of automating the RP replication status and send status via email or post it to any reporting platform. Since the splitter impacts the VMAX FA's and there is an overhead of 13% on the utilization its good idea to see if the replication is driving the utilization of vmax FA. There are ways of pulling iops/read/write using symstat but its utilization (Director CPU utilization etc) which plays a significant role. As it crosses 80% we start seeing the increase in device response time. However, to measure FA utilization we will have to go through SMAS UI then pull out the reports. Its manual effort. But there is a way to automate it through REST API. You can get the details what type of objects are supported by pointing to : https://smc server :8443/univmax/restapi Coming to the point, getting recoverpoint consistency group status fully automated doesn't seems to be tough. Lets get through the steps. 1. Hash your SSH keys of the unix host to ...
Read more

Weird problem post power maintenance, no Powerpath pseudo name for Luns

We had the scheduled power maintenance activity in our data center. Every team was notified and we powered doen the servers through cron job. The servers were powered down first then storage (CX-380). And the reverse was followed during startup. After the power shutdown, the SAN was booted up. we forgot to disable the the snapshots. They were still active. Once it came up, the sessions were not there, but snapshot was active. Tried to deactivate the snapshot, failed with the error the "Failing Command: K10SnapCopyAdmin DBid 0 Op 1051. 71 00 80 43" Tried to create a session and attach the snapshot to the session, still unable to deactivate it, neither was able to attach to any other session. Looked at powerlink: It says the command issued to the snapshot is not owned by the SP. I trespassed all the snapshot source lun to SPA, and then tried to deactivate, nothing happened. Rebooted the SP's one by one through the management console. (http:/SP-IP/setup) Then I was able to...
Read more