Configuring Netapp filer for rsh connections

So you need to connect to your Netapp filer via rsh. Maybe for example you wish to coordinate the snapshot on a volume with some other specific events in a script, as I did to perform consistent Oracle database backups.

So how do you set it up for this?

First we need to check that the Netapp settings allow rsh, it is allowed by default but it is worth us checking. Login to the filer via telnet and checks the rsh.enable parameter:

Netapp_filer> options rsh.enable
rsh.enable on

If it is set to off, we will need to turn it on with:

Netapp_filer> options rsh.enable=on

Now we need to allow our host to connect via rsh. We will also define it for a specific user as we don't want to.make the system too insecure by letting everyone run system commands on the filer!

First we need read/write access to the root filesystem on the Netapp, so we need to share the /vol/vol0 volume with read/write permissions through the nfs section of the GUI - I am not going to document that step here as it is a common task when setting up nfs shares. I will however remind you to hit the "export all" button once you have set it up, otherwise you will be left scratching your head as to why you cannot mount the filesystem as expected!

Now navigate to the /etc directory of this filesystem, and make a backup of the hosts and hosts.equiv files:

Linux> cd /vol0/etc
Linux> cp hosts hosts_080211
Linux> cp hosts.equiv hosts.equiv_080211

We know need to add our host to the hosts file, I.e. add the line:

1.2.3.4 our_host

Then we need to add the hostname and account name to the hosts.equiv file. Here we are using the Oracle account so the line we will add is:

our_host oracle

Once this is complete, we should be able to confirm the rsh command is working. To test try the following command whilst logged into the oracle account on our_host:

rsh -l root Netapp_filer df -k

This should return the df output of the filesystem on the Netapp filer.

Single Oracle listener on multiple networks

I had the requirement today to move the application connection to our production database over to the SAN network, rather than over our internal network in the hope of seeing increased performance. I also wanted to keep the listener on the internal network, as our SAN network is segregated from the rest of the network, therefore by keeping the internal network connection it can be used to connect by the DBAs for administrative purposes.

I started by googling for the solution but found this surprisingly difficult to find the solution! Therefore I am now blogging about it!

To start with we ensure that both ip addresses for our Oracle server are in the /etc/hosts file, eg:

1.2.3.4 host_net1
2.3.4.5 host_net2

We then take our existing listener.ora file:

LISTENER =
  (DESCRIPTION_LIST =
    (DESCRIPTION =
      (ADDRESS_LIST =
        (ADDRESS =
(PROTOCOL = TCP)(HOST = host_net1)(PORT = 1521))
      )
      (ADDRESS_LIST =
        (ADDRESS =
(PROTOCOL = IPC)(KEY = EXTPROC))
      )
    )
  )
SID_LIST_LISTENER =
  (SID_LIST =
    (SID_DESC =
      (SID_NAME = PLSExtProc)
      (ORACLE_HOME =
/home/oracle/product/10.2.0/db_1)
      (PROGRAM = extproc)
    )
    (SID_DESC =
      (GLOBAL_DBNAME =TESTSID)
      (ORACLE_HOME =/home/oracle/product/10.2.0/db_1)
      (SID_NAME =TESTSID)
    )
  )

Here we have a listener listening on host host_net1 on port 1521, for database sid TESTSID.

We now want to add the listener listening on the other ip address, host_net2. As this is another connection, we will need to change the port as well, so we will therefore choose port 1526, as this is generally recognised as the alternative listener port.

To edit, the listener.ora we need to locate the port line in the existing file, ie:

(PROTOCOL = TCP)(HOST = host_net1)(PORT = 1521))

And the copy and paste this line, replacing the HOST and PORT, so our completed file will now look like this:

LISTENER =
  (DESCRIPTION_LIST =
    (DESCRIPTION =
      (ADDRESS_LIST =
        (ADDRESS =
(PROTOCOL = TCP)(HOST = host_net1)(PORT = 1521))
(PROTOCOL = TCP)(HOST = host_net2)(PORT = 1526))
      )
      (ADDRESS_LIST =
        (ADDRESS =
(PROTOCOL = IPC)(KEY = EXTPROC))
      )
    )
  )
SID_LIST_LISTENER =
  (SID_LIST =
    (SID_DESC =
      (SID_NAME = PLSExtProc)
      (ORACLE_HOME =
/home/oracle/product/10.2.0/db_1)
      (PROGRAM = extproc)
    )
    (SID_DESC =
      (GLOBAL_DBNAME =TESTSID)
      (ORACLE_HOME =/home/oracle/product/10.2.0/db_1)
      (SID_NAME =TESTSID)
    )
  )

Finally save the file and then stop and start the listener in the usual manner.

You should then be able to confirm you are able to connect to the database via sqlnet from both networks.

Vodafone Sure Signal - first impressions

Having been used to having a very good O2 signal in the house, it came as a shock to get frequent long periods of no signal on my new company Vodafone mobile.

Looking at their coverage map it indicated perfect coverage indoors and outdoors - ie nothing like what I was experiencing!

On the page was a link about improving your indoor signal, so I clicking this and it took me to a page about sure signal. A box of similar size to a broadband router which effectively acts as a private Vodafone mast in your house, using the internet to provide the link into Vodafone's core network. It allows you to register upto 32 numbers into it, and then use upto 4 of those numbers at any one time.

Perfect, I thought and so I checked the minimum criteria:

1. 3g phone - tick
2. Broadband speed above 1Mbps - tick

So I went to the Vodafone shop and bought one for £50. I brought it home and plugged it in, power cord into the wall, and supplied Ethernet cable into the back of the router - easy so far!

Now to register the unit and setup the numbers. The instructions told me to register it at this page - so I follow the instructions and attempt to create a new account - however this then got referred to Customer Services, because my number is on a corporate account. 24 hours later I get an email asking the lead name on the account to give them a call!

There must be an easier way than this! So I check the suresignal forum here and find a post detailing a dedicated registration page for business customers

I therefore filled in this form and the next page says you should receive a txt message within 24 hours to say the unit has been activated. That was Friday lunchtime and it is now Monday lunchtime and no txt message. So I call the helpdesk on 08454 402104 to see what is going on. When I got through to the correct department, I was told the period to wait is 72 hours, but she would do it now for me. So I gave all the details again, and got the system activated and received the txt message a couple of minutes after ending the call.

I did however explain I wanted to add 1 additional number, which the operator informed me was not possible at this time. However 5 minutes after ending the call, she rang back saying she was on the same screen and as it was now activated, the additional number screen was available and that she would do it for me - a nice touch.

I came home later and now find my mobile signal strength bar is at full strength. I've had no dropouts all evening - so it does appear that the suresignal box us going exactly what it is supposed to!

Overall first impressions are therefore a good piece if kit, which does what it says it will, however the registration process needs to be much simpler and quicker! 7 out of 10

Netapp FAS3100 series - hardware changes! I'm impressed

So, in preparation for taking our newly acquired FAS3140 into production, we have purchased two new 4 port nics, one to go in each controller of the active/active configuration.

Whilst the Netapp is not yet in production, I thought I would attempt this task, as if it was so as to gain a better understanding of the process in case I need to do it later when it is actually serving production data. Well the task was surprisingly easy and completed without any downtime to the unit! To say I am impressed is an understatement! With Netapp you really get what you pay for.

So first things first, to complete this exercise you need space at the back of the rack behind the FAS3140 controllers as you will (in turn) be removing each of the controllers through the back of the cabinet.

When I surveyed our site I found a jumble of network cables passing the back of the FAS3140 going upto our Esx hosts at the top of the rack!

This was not suprising as we have not been careful with this space in the past as access here has not been required! I had on my list of jobs some setting changes to each Esx host, that would require me to reboot each Esx host, so I thought I would do that first and sort out each hosts network cables during the reboot exercise. These changes will be covered in another blog article!

So space behind the rack is now not a problem, so we can continue with the hardware change!

First, a little bit of useful knowledge to ease the task! The expansion cards on each FAS3100 series controller are labelled from the top 1 to 4 at the bottom. When adding expansion nic cards as in this article, we can use this information and know the names of the new network ports before we add the card! So as we are adding a 4 port nic to the bottom expansion port (4) our ports will be called e4a, e4b, e4c and e4d. As we were currently using a vif on each controller on ports e0a and e0b we could make the network changes prior to the hardware change and then get it picked up automatically later!

So we edit the /etc/rc file on the first controller changing any vif references from e0a and e0b to e4a, e4b, e4c and e4d. Then save this file. NOTE: changes to this file are read only at boot so it is safe to make these changes whilst the system is running.

We now need to takeover the first controller, so that the second controller assumes its identity and serves its data. You can either do this on the telnet session of the second controller or as I did by using the Netapp systems manager GUI downloadable from the Netapp website.

Once this is done you will now be serving all data from the second controller, it is as if the first controller does not exist!

We are now in a position to install the new card on the first controller. So first we have to remove the controller.

To do this locate it in the rack and remove all the network cables to it, along with the interconnect to the shelves. NOTE: DO NOT REMOVE THE POWER TO THE UNIT OR CABLES FROM THE OTHER CONTROLLER!

It is useful to attach the removed cables to the cable management tray to aid in locating the correct cables when re-cabling later. Then remove the cable management tray and move it out of the way of the controller.

Next you need to locate the cam handle screw on the back of the unit. This is fairly near the top and slightly left of centre. Undo this screw and you should feel the handle drop down partway. Grab hold of this handle and pull it down the rest of the way. Now hold the handle and pull the controller out of the rack. NOTE: the controller will stop partway and you will need to release the holding tab on the left hand side.

Complete the removal of the unit being careful not to let the furthest edge from you drop as it clears the chassis!

Now open up the right rear side to gain access to the expansion slots. As explained earlier you should now see 4 slots labelled from the top 1 to bottom 4.

I inserted my card into slot 4 and then put the side back on.

Next slide the controller back into the chassis taking care not to trap any cables in the process. Finally push the cam handle back firmly and tighten the cam handle screw up.

Recable the controller and reaffix the Cable management tray. NOTE: as I went from 2 ports on the old vif to 4 ports on the new vif I had to move the 2 network cables from the old ports to the ports on the new card and then cable an additional 2 to the other 2 ports on the new card.

Finally as the original vif was configured as a portgroup on the switches it was necessary to adjust this so that it went from containing 2 to 4 ports.

Then it was a case of going back to the systems manager GUI and instigating a giveback. This basically tells the system that the error on the first controller is fixed and that it is ready to serve data again. This process takes a few seconds as it effectively reboots the first controller - thereby reading the /etc/rc file we changed earlier.

Once this has completed, we confirm that the vif is indeed configured as expected and that everything is functioning correctly.

Once we are happy we are ready to perform the steps again, this time on the second controller, with the first once assuming control of the whole unit!

So that's all there is to it, hardware changes can be made with the filer staying up and continuously serving data. The upgrade is also very quick and easy. I am very impressed with this bit of kit!

Laser printer cost per page calculation

So I find that I now need to replace 3 of the 4 toner cartridges in my home laser printer (samsung clx-3170 series), I thought I would work out what it has cost per page to run, so that I can either justify its replacement with a cheaper to run model, or shock me and the rest of the family into printing less!

The Samsung has a neat "supplies information report" accessible through the printer setup, reports menu, so it is this that I have used as a basis of my calculations.

The report, shows the following information that we need:

item	remaining	no of pages
yellow toner	10%	1265
magenta toner	10%	1260
cyan toner	11%	1258
black toner	35%	1140
imaging unit	52%	11719

So I first take the figures and work out a "whole of life" page count for each part, using the formula (page count/(100-remaining))x100 - this gives the figures below:

item	whole of life
yellow toner	1406
magenta toner	1400
cyan toner	1398
black toner	1754
imaging unit	24414

So, looking at the figures above it seemed reasonable to base the initial costs calculation on 1400 pages, as 3 of the 4 toner cartridges will print this amount (give or take s few sheets) during their whole life. I therefore worked out how much life would be used by the black toner and the imaging unit to print these 1400 pages.

I used the formula (1400/whole of life)x100 to work this out. This gave the following figures:

item	used percentage
black toner	80%
imaging unit	6%

I then went online to find current prices for the supplies and then equated these to the price for 1400 pages, using either the whole cost (for the 3 colours) or the percentage above, for the other supplies, giving:

item	online price (£)	cost per 1400 pages
yellow toner	26.45	26.45
magenta toner	26.45	26.45
cyan toner	26.45	26.45
black toner	26.45	21.16
imaging unit	93.14	5.59

So totalling up the whole of life costs gives a cost of £106.10 for 1400 pages, I.e. a cost of 7.58 pence per page

[NOTE: for the purposes of this calculation I have excluded the cost of the waste unit in the printer as last time I looked this was under £10 to buy and last time I opened the printer up it was very nearly empty, so I therefore believe this will have negligible impact on the cost per page price. I have also excluded printer paper costs as these are easily calculated separately]

I really did not have a feel for how much the printer was costing to run, however it does appear that I am exceeding the print life for each of the toner cartridges specified by Samsung which is pleasing.

I have also looked on the internet and our costs are broadly in line with calculations done with other laser/inkjet printers.

Whilst it does come as a shock to get a three figure bill to replace multiple laser toner cartridges this is offset by the reduced frequency in which they occur compared with replacing inkjet cartridges!

This led me on to working out an annual running cost for the printer. The same supplies report showed when the supplies were first installed in the printer and how many prints they have used, so we can work out an annual number per toner per year:

item	installed	pages	pages per year
yellow toner	30/03/2010	1265	1687
magenta toner	30/03/2010	1260	1680
cyan toner	30/03/2010	1258	1677
black toner	05/09/2010	1140	3420
imaging unit	26/11/2009	11719	10817

So with these figures and the whole life figures in one of the tables above I can work out the cost for each supply per year:

item	unit cost	multiplier	annual cost
yellow toner	26.45	1.205	31.87
magenta toner	26.45	1.200	31.74
cyan toner	26.45	1.197	31.66
black toner	26.45	1.949	51.55
imaging unit	93.14	0.443	41.26

So totalling up these costs gives an annual cost of £188.08 or a monthly cost of £15.67

Again these costs exclude the waste disposal unit and paper.

Time to have words with the rest of the family about the running costs and get them to print more efficiently, e.g. by using draft and grey-scale!

Virtual machine problems - Hung vm esx 3.5

Why does it always happen like this! Major problems with a virtual machine just as I am about to take the afternoon off! :(

I get an email from our early warning system (nagios - very good by the way http://www.nagios.org ) that one of our virtual machines is down.

So I check and it is correct, it is not responding to ping requests, so I try to get onto the virtual console and get no response! :(

Ok, i'll reset that virtual machine and all will be fine, or so i thought! The reset gets to 95% complete and then just hangs there!

I look on the VMWare website and found this suggesting that the vm should be reset by restarting the management agent services. So I decide to do this, however before I do this, I am going to make sure that this is the only vm on this physical esx 3.5 host, so as to not make the situation worse, so first things first, I set the host into "maintenance mode" and vmotion all the other vms to other hosts in the cluster, once this is done I then have to cancel the maintenance mode request as it can not handle the stuck vm.

I am now in a position to restart the management agents without worry, so I putty to the esx host as root and run the command:

#> service mgmt-vmware restart

Once this completes, I noticed that the host has become disconnected in virtual center, so I await it to become reconnected, whilst waiting I see the reset virtual machine task had been killed.

After a few minutes, the host is reconnected, so I try to power on the vm, but it fails and the vm is now shown in virtual center as invalid! Ok, back to the internet, I now find this article, which suggests I should remove the vm from the inventory and re-add it, so this is what I do.

I attempt the power on again, and it is now stuck at 1% of " registering virtual machine with the host". Ok, it must now be time to raise a priority 1 support call with VMWare as this is a production vm. So I speak to vmware and we setup a webex to find out what is wrong. The support guy, finds a swap file in existence in its directory on the nfs volume, so he removes this. He also finds locks on the files, but this is not reported properly and thinks this could just be a limitation of using nfs volumes with vmware. He carries on and relocates the vm on another physical esx host using vmotion and then tries to power it on. No joy, so he concludes it lust be an issue with permissions and the nfs volume - I am not convinced with this as we have over 100 vms on this volume and this is the only one with a problem. Anyway he goes away!

So, I think let's see what happens if we copy the vm files to a new directory on the storage and then try and add this into the inventory. So I copy the files and change the directory name within the .vmx file. I then add this new fileset to the inventory. I try to power on the vm, bug it now complains about a lock on one of the vmdk files.

Hmmm, what now!

One last thing we can try before resorting to the backup. This started on one particular host, I wonder if this had got its nfs locked screwed up and a reboot would cure it? So I request this particular host to go into "maintenance mode", which now succeeds as the problematic vm is now "down", so I then request a reboot of this host through virtual center.

I wait the few minutes it takes to complete, and then take it out of maintenance mode. I now try to restart the problematic vm, and guess what - it starts corectly - much to my relief! :)

Netapp Autosupport and Aggregates

Ok, so first day back in the office after the Xmas break.

Guess what - I go into the computer room, orange lights greet me on the new Netapp.  Hmmm somethings up, a closer inspection then reveals an orange light on one of the disks!  Great a failed disk!

I now remember in my email I had a failed to deliver message on an email sent over the christmas period to Netapp autosupport - yet there was no-one around to kick these off.  Putting two and two together, this must have been when the disk failed - and I have not got around to setting up the autosupport!

So first things first lets dust off the instructions to setup autosupport:

First we check the transport, it is set to https but we wish to use smtp so we reset it:

options.autosupport.support.transport smtp

We then need to check it sends correctly so we send a test message:

options.autosupport.doit TEST

Bingo that is now working, so we repeat the commands on the other node.


Now, I login to the now.netapp.com website and see my system is not listed :(   So I raise a support call with Netapp to add it - hopefully this will be resolved tomorrow and the new disk can then be shipped.




Aggregates:

Our consultant just before Christmas said we should setup the system with one large aggr0 and not use aggr1 like we had on our old system as this wastes 3 disks - i.e. nearly 1TB of space per node (as we have 300GB disks).

Duh, we have already set this up with aggr1 on both new nodes, so lets first offline and destroy aggr1 on both nodes, using the http interface.

Now we need to zero the disks:

disk zero spares

This seems to finish instantly and not do much, so we move onto moving the spare disks into the aggr0:

aggr add aggr0 20

where 20 is derived from 24 disks in the shelf, minus 3 already used minus 1 for spare

This takes a while to run and says it is zeroing the disks as it goes, so I wonder if the disk zero spares command is actually required?!

This is then repeated on the other node.

After a few minutes the aggr0 is populated with the extra disks, although 1 of the nodes is showing an error with a lack of spare disks - not suprised seeing as 1 disk is failed.

So onwards to tomorrow whereby I can hopefully get the system onto Netapp support correctly and receive another disk!