My world of Unix and Storage

Is power path fully supported with zpools, no. It's partial. You can have the pseudo devices assigned to a pool but it uses the native ctd disk only. Recently I lost 2 paths from the vnx array to m9000 server running Solaris 11.1. When I ran the zpool status -x command some of the pools were online while some of them were un available. Power path was reporting 2 paths only but it was active. So ideally the zpools should be online. But it's not. I enabled the remaining 2 paths and all became online. Internally the pools were pointing to one of the ctd devices that belonged to dead path and it was not coming online. Also few of the emcpower devices were renamed, so I had to do zpool import -d with the alternate disk. It seems power path is not completely supported now but may become in future.

Guys, Have you ever thought of automating the RP replication status and send status via email or post it to any reporting platform. Since the splitter impacts the VMAX FA's and there is an overhead of 13% on the utilization its good idea to see if the replication is driving the utilization of vmax FA. There are ways of pulling iops/read/write using symstat but its utilization (Director CPU utilization etc) which plays a significant role. As it crosses 80% we start seeing the increase in device response time. However, to measure FA utilization we will have to go through SMAS UI then pull out the reports. Its manual effort. But there is a way to automate it through REST API. You can get the details what type of objects are supported by pointing to : https://smc server :8443/univmax/restapi Coming to the point, getting recoverpoint consistency group status fully automated doesn't seems to be tough. Lets get through the steps. 1. Hash your SSH keys of the unix host to ...

With flare 30, pool is the new feature.and mixed pool is something totally new. You can combine multiple disk type in a common raid configuration and put it inside the pool. Enable auto-tiering and you are promised for a good performance by moving data across different tiers, the catch (its not real time, unlike VMAX). If you have totally unpredicted I/O then by the time it is decided to move the data above or below the tiers the I/O profile gets completely changed. In the next version of he flare, you can have disks in different raid type inside a single pool. What goes inside the pool, how the data is striped or written is something internal to EMC. The very idea is to give customers a very comfortable way of handling the storage leaving the pain to EMC. However, things are always not green as it looks. This works well when you have predictable I/O, you know what is going inside and when. What if the dynamics changes every now and then and you do not have control over the I/O ...

Version 3 maintains persistent locks. When a file system is unshared and shared back the locks are not destroyed so the state remains the same as if nothing has changed. Version 4 removes any state once the file system is unshared. All the state information is removed and any attempt to access the file again once shared back results in I/O error. The state does not change if it was unshared for some share option changes. Pseudo file system name space is maintained in version 4. In version 3 clients must request for individual mount of the entire exported share. Version 4 client gets a seamless view of all the exported file system. Path traversal was not allowed in version 3. Version 4 supports volatile file handles apart from persistent file handle. Server generates a unique file handle to every file. The file handle will change if the file is deleted and replaced by another file however will remain the same if server reboots or file was renamed. In oracle version 4 client con...

If you have to transfer a huge number of files lets say 50000 over WAN as one time data copy. Each of the files are of 3-4 Mb. We have couple of options like tar (will need extra space; unless you pipe it through ssh), scp (will be sequential), or use rsync. What I have done here, is used the --files-from option of rsync which specifies which files to transfer. On the source, create the file list 50000 lines (huge number) with ls > /tmp/a, then split the files based on line count. The line count is the deciding factor, if you have DS3 or higher make it smaller. Smaller line count will determine the concurrency or the number of simultaneous sessions that can start. split -l 1000 /tmp/a This will create 50000/1000 number of files starting xaa (you can decide the starting prefix, this is default) in the present working directory assuming its /tmp. Once done use the below either as single commands or put them in a script and start your job. nohup /root/rsync/bin/rsync -avz -...

Have you come across very silly problem while mounting NFS share. It just comes out of no where and refuses to go. Hours of efforts are wasted when we find the exact issue and seems like banging our head on the doors. I have summarized some of them, I hope you might be having your set of goodies as well. If you are getting an error like permission denied on the client server (90% of the issue falls here), then check in sequence what has to be done: 1. If the server has multiple IP address, it can no surprises. Do a traceroute from the server to the client and see which interface is being used to connect. Of course please check first whether the client is pinging or not. Once you have the interface detail, this IP will be used on the clients to map the network share. 2. See if you can resolve the client IP address using DNS, if you can then use dns name in the share access list on the NFS server. 3. If the client has multiple interface, then do a traceroute to the IP found in 1...

It started on Friday evening, yes Friday for all but not for me. Waiting for the EMC engineer to do a webex and configure our Recoverpoint appliance. Normally, if you are planning for DR site (big project indeed) for the last 6 months; how much time will you give for the lifeline to work yes, data replication. 1 month would be a decent guess, we have to do in 2 days. Don't laugh, very serious. We had done the rack and stack and all the cabling beforehand. It started around 9 PM and went on till 1 AM. All set for the testing. We just cross checked whether every IP is reachable or not. Viola, the wan IP was not pinging. It was on a different VLAN and was not in the firewall rules. Called up firewall engineer, didn't picked up the call; another same result. Escalated to manager, his phone was busy. called up project manager, he was furious (had to be). Escalated to Director, Information security and he confirmed will do something. Never ending wait started, finally around 3...

We are setting up our DR site. For replication we are using EMC Recoverpoint appliance. The license what we have works for only SAN volumes on EMC arrays namely VNX in our case. The database and file shares which are on SAN have no problem however, the critical file shares that are on local storage or Dell iscsi storage have an issue. It sounds easy to move them to SAN and start replication but not so. Consider below: 1. The LVM housing the file shares on a Dell server contains PV from local disk partition (yes not complete disk but like /dev/sda10 on extended partition) and other full disk from iscsi storage /dev/sdq and /dev/sdi 2. Total size is ~400 GB with 100 GB from local disk partition (total local disk size is 300 GB) and 300 (200 + 100) from iscsi. I know its wrong but it was done long time ago and we were running out of space. 3. The dell server does not have FC connection. 4. The file system structure contains 32000 nested directory structure with approx 1.2 billion fi...

Somebody might have done it, don't know. Some one might think we are crazy and its impossible; but we are doing it. Please add its not one to one correspondence but 100 percent DR with state of the art technology at our DR site. I will just give you a glimpse of what we are doing and how. Our production is a mix bag of: 1. Dell servers for intel needs like Oracle databases (11g rac), MySql on Linux and MsSql on windows, total around 80 non virtualized servers. 2. Dell servers for other application like tomcat,jboss,iis,file srrvers etc which are virtualized, total 300 physical and 1200 virtual instance. 3. Roughly 50 percent are virtualized. 4. Dell servers which makes a 80 node citrix farm 5. Dialers and call recorders for 4000 users 6. We are using exchange with 8000 users 7. We are handling storage of 1.4 pb on EMC vnx and NS480 along with 100 TB on Dell MD3000i storage 8. 80 inhouse developed application running 24x7 9. Core databases running on Oracle M8000 and M5000's wit...

Wow, looks challenging and it's indeed. So, we have finally decided to move our DR data center from India to US. The location I cannot disclose however its almost 50% of the original data center at Atlanta. The designing phase is over except for the data center power and cabling and I believe it happens given the short time frame of 45 days (don't laugh, it got extended by 6 months and I feel another 3 months more it deserves). There are few things which I don't know why it happens every time we try to do something big. 1. Specification keeps on changing, regarding the list of applications for which we are building our DR. 2. RPO and RTO is not decided and simply we are doing an infrastructure DR not true DR. There is no BCP. 3. We are trying new technology Cisco UCS vs discrete Dell servers and going for full virtualization at DR. We are simply doing too many things and rather experimenting new technologies at data center. 4. How the data will get replicated was decided po...

If you have enabled performance logging on EMC VNX, then it is expected to display all the statistics including the SP and FAST cache. However, with the default configuration of statistics logging, you will not be able to see. We have to go to the dashboard of statistics and click customize then check Advance. The advance features of the performance statistics will be visible now.

Sometime have you came across a situation where you want to create psrset but it doesn't works. Please check if pool facility is active or not. If its active then you won't be able to create psrset. svcs -a | egrep pool. Disable pool and create psrset again.

In Linux we can find the port to pid mapping, using netstat -np or using lsof. However, this facility is not available in Solaris. We will have to write a small perl/shell script to find the mapping. Script : ##################### #!/bin/bash export PORT=0 for name in `ps -aef | awk '{print $2}'` do export PORT=`pfiles $name | grep sockname | awk '{print $5}' 2>/dev/null` if [ -n "$PORT" ]; then echo "PID = $name PORT= $PORT" fi done unset PORT exit ####################

We had the scheduled power maintenance activity in our data center. Every team was notified and we powered doen the servers through cron job. The servers were powered down first then storage (CX-380). And the reverse was followed during startup. After the power shutdown, the SAN was booted up. we forgot to disable the the snapshots. They were still active. Once it came up, the sessions were not there, but snapshot was active. Tried to deactivate the snapshot, failed with the error the "Failing Command: K10SnapCopyAdmin DBid 0 Op 1051. 71 00 80 43" Tried to create a session and attach the snapshot to the session, still unable to deactivate it, neither was able to attach to any other session. Looked at powerlink: It says the command issued to the snapshot is not owned by the SP. I trespassed all the snapshot source lun to SPA, and then tried to deactivate, nothing happened. Rebooted the SP's one by one through the management console. (http:/SP-IP/setup) Then I was able to...

We came across a weird problem in Nagios, after it was working for almost an year. Searched the web, but didn't got any head way for the problem. Every body were complaining about the issue but the resolution were not clear. I started looking the source code, and found the function in util.c, there were different error conditions but got a hint what the problem could be. So, all set i started the nagios service and ran strace against the pid. Soon after about 40 seconds, it vanished. Forgot to mention we were running nagios on RHEL 5.5 32 bit linux. I checked the various options for the nagios process. There is one option for running it in the foreground. The syntax check was fine. As soon, it was started on the foreground, it dumped with the error message "File limit ....". Couldn't remember the exact output. There are 2 files which could be the problem, one is nagios.log or nagios.debug. Both were OK. Their is another file perfcheck.pipe or perfmon.pipe (something o...

The thought process has gone into the management of moving the production database from Solaris running on Oracle M8000 to IBM AIX. The initial cost of providing the clustered solution and the DR is coming to $2.5 million. Oracle were taking us for a ride with the discounts on M8000. The 50% agressive discount offering on M8000 comes at the same cost of M8000 that we bought last year. Along with the Veritas storage foundation and the support cost, its too expensive as a solution. We have the quotes from IBM for their p750, however we feel this is not an apples to apple comparison as the p750 lacks the RAS features. Th better would be the comparison with p595 or p785. We have to get the POC done and get the actual working configuration. Its a long way to go, since our aplication compatibility is yet to be established with AIX 6.1. Also, we are in the process of moving from 32 bit progress database to OpenEdge 64 bit. The dynamics will change with respect to cpu and memory utilization on...

A tough task is in my hands to decide between the 3 different storage vendors for our storage requirement. We have been EMC customers for last 10 yrs, but now would like go around and see what others are offering. The requirement is not huge ~260 Tb storage and ~80000 IOPS, which I believe can be meet by any mid-tier storage offering. We have VNX7500 from EMC with FAST and FAST-vp loaded with SSD's and 48GB ram (shared) and single socket CPU (Quad core) and 600 GB of SSD as L2 cache blah blah. The backend is SAS, that is a shift from CX line where we have FC. This also means we cannot re-use the existing hardware from EMC. The IOPS pattern hovers between 8000 to 40000 currently and will soon scale upto 80000 IOPS. Hitachi has a offering AMS2500, which is also SAS with 32 GB cache and Active/Active controller (difference from EMC). The maintenance cost is attractive almost 6-8% of the array cost. Hitachi also does a buy back. IBM has V7000, still to get some input on this. Will upda...

We have a peculiar situation where running multiple snapview clone sync results in I/O wait on the production application. This is a business requirement where we have to take a backup of the production copy before running the EOD (End of day) process and one copy immediately after the EOD (End of day). The duration for EOD varies between 45 minutes to 1 hour. This is the time when the I/O is maximum. The EOD period is not sufficient to schedule snapview sync for the post EOD after the split of Pre-EOD. Their is always an overlap, and that is the period when the user face extreme slowness. We can stripe the luns over multiple Raid groups for the production luns but space constraint prevents us from doing this. The solution what we found was to go for snapshot for Post EOD and with the normal sync for the EOD. 1. We add the luns to the reserved clone group. It depends on the size of the snapshot and the percentage change. 2. We create snapshot on the source luns and add it to the destin...

How you plan to use Linux LVM with a multipathing software like EMc powerpath? The trick is to mask out the non emcpower devices from LVM view. This is to ensure any path failure is handled by the multipathing software rather than LVM itself. The underlying path management complexity is offloaded to multipathing software. When the command pvscan, is issued you see only the emcpower devices. You will have to enable the filter for accepting the emcpower devices and reject all the non emcpower devices inside /etc/lvm/lvm.conf file filter =[ "a|/dev/emcpower.*|", "r|/dev/sd*|", "r|/dev/cdrom|" ] Ensure only one filter is enabled.

Have you ever came across a situation when nslookup works for name resolution but ping or traceroute does not work. Traditional unix commands does not rely on dns for name resolution unless specifically instructed via /etc/nsswitch.conf file. You will have to make an entry in front of hosts parameter: hosts files dns