Adding an additional node to an existing RAC cluster.
This document assumes the following information for the cluster:
| Hostname | Secondary Name | IP Address | Purpose | Location |
| racnode1-san | racnode1-san.laptop.com | 192.168.3.101 | Storage Network for the 1st node | hosts |
| racnode2-san | racnode2-san.laptop.com | 192.168.3.102 | Storage Network for the 2nd node | hosts |
| racstorage-san | racstorage-san.laptop.com | 192.168.3.103 | Storage Network for the Shared Storage | hosts |
| racnode1 | racnode1.laptop.com | 192.168.100.1 | Public ip address for 1st node | hosts,dns |
| racnode2 | racnode2.laptop.com | 192.168.100.2 | Public ip address for 2nd node | hosts,dns |
| racstorage | racstorage.laptop.com | 192.168.100.3 | Public ip address for storage | hosts,dns |
| racnode1-priv | racnode1-priv.laptop.com | 192.168.200.1 | Private interconnect for 1st node | hosts |
| racnode2-priv | racnode2-priv.laptop.com | 192.168.200.2 | Private interconnect for 2nd node | hosts |
| racnode1-vip | racnode1-vip.laptop.com | 192.168.100.21 | VIP address for 1st node | hosts,dns |
| racnode2-vip | racnode2-vip.laptop.com | 192.168.100.22 | VIP address for 2nd node | hosts,dns |
| rac-scan | rac-scan.laptop.com | 192.168.100.10 | 1st address for the scan | dns |
| rac-scan | rac-scan.laptop.com | 192.168.100.11 | 2nd address for the scan | dns |
We will be adding a third node, with the following network configuration:
| Secondary Name | IP Address | Purpose | Location |
| racnode3-san.laptop.com | 192.168.3.104 | Storage Network for the 3rd node | hosts |
| racnode3.laptop.com | 192.168.100.4 | Public ip address for 3rd node | hosts,dns |
| racnode3-priv.laptop.com | 192.168.200.4 | Private interconnect for 3rd node | hosts |
| racnode3-vip.laptop.com | 192.168.100.24 | VIP address for 3rd node | hosts,dns |
| rac-scan.laptop.com | 192.168.100.12 | 3rd address for the scan | dns |
For the purposes of this document I will be making the hosts as Vms under VMWare workstation, but these could equally be physical servers.
Create a new machine, by installing the operating system.
Configure the networking, for the public address, interconnect and storage networking, making sure that the interface names corresponding to the same networks as those on the other nodes.
Point the host at the correct dns server.
Populate the /etc/hosts file:
192.168.3.101 racnode1-san racnode1-san.laptop.com
192.168.3.102 racnode2-san racnode2-san.laptop.com
192.168.3.103 racstorage-san racstorage-san.laptop.com
192.168.3.104 racnode3-san racnode3-san.laptop.com
192.168.100.1 racnode1 racnode1.laptop.com
192.168.100.2 racnode2 racnode2.laptop.com
192.168.100.3 racstorage racstorage.laptop.com
192.168.100.4 racnode3 racnode3.laptop.com
192.168.200.1 racnode1-priv racnode1-priv.laptop.com
192.168.200.2 racnode2-priv racnode2-priv.laptop.com
192.168.200.4 racnode3-priv racnode3-priv.laptop.com
192.168.100.21 racnode1-vip racnode1-vip.laptop.com
192.168.100.22 racnode2-vip racnode2-vip.laptop.com
192.168.100.24 racnode3-vip racnode3-vip.laptop.com
And ensure the following is added to the DNS server:
| Record Key | Value |
| racnode3 | 192.168.100.4 |
| racnode3-vip | 192.168.100.24 |
| rac-scan | 192.168.100.12 |
And add the following entries to the /etc/hosts file on each of the existing nodes and the storage:
192.168.3.104 racnode3-san racnode3-san.laptop.com
192.168.100.4 racnode3 racnode3.laptop.com
192.168.200.4 racnode3-priv racnode3-priv.laptop.com
192.168.100.24 racnode3-vip racnode3-vip.laptop.com
Add the following lines to the /etc/security/limits.conf file on node 3:
oracle soft nofile 1024
oracle hard nofile 65536
oracle soft nproc 2047
oracle hard nproc 16384
oracle soft memlock 3145728
oracle hard memlock 3145728
oracle soft stack 10240
And the kernel parameters to /etc/sysctl.conf on node 3:
fs.aio-max-nr = 1048576
fs.file-max = 6815744
kernel.sem = 250 256000 100 1024
net.ipv4.ip_local_port_range = 9000 65500
net.core.rmem_default = 262144
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
net.core.wmem_max = 4194304
kernel.shmmax = 1073741824
kernel.shmmni = 4096
kernel.shmall = 2097152
And install the required additional packages:
#> yast -i sysstat
#> yast -i libcap1
Create the grid home directory:
#> mkdir /home/grid
#> chmod 777 /home/grid
And add the following lines to the /etc/fstab file to mount the shared storage:
racstorage-san:/storage/oracle_dbf_testdb /data/oradata/TESTDB nfs rw,bg,hard,rsize=32768,wsize=32768,vers=3,nointr,proto=tcp,suid,noac 0 0
racstorage-san:/storage/oracle_crs /oracle_crs nfs rw,bg,hard,rsize=32768,wsize=32768,vers=3,nointr,proto=tcp,suid,noac 0 0
Now create the oinstall, dba groups and the oracle account with the following Ids:
| Group | Group ID |
| dba | 1010 |
| oinstall | 1011 |
| User | UserId | Primary Group | Secondary Group |
| oracle | 1010 | 1011 (oinstall) | 1010 (dba) |
And create oracle's .bash_profile as the oracle user:
$> vi /home/oracle/.bash_profile
if [ $USER = “oracle” ]; then
if [ $SHELL = “/bin/ksh” ]; then
ulimit -p 16384
ulimit -n 65536
else
ulimit -u 16384 -n 65536
fi
umask 022
fi
Now make the structure for the shared drives:
#> cd /
#> mkdir /data
#> mkdir /data/oradata
#> mkdir /data/oradata/TESTDB
#> mkdir /oracle_crs
#> chown -R oracle:oinstall /data
#> chmod -R 755 /data
#> chown -R oracle:oinstall /oracle_crs
#> chmod -R 755 /oracle_crs
Now on the storage node, modify the /etc/exports file to include our third node, the entries should now look like:
/storage/oracle_crs racnode1-san(rw,no_root_squash) racnode2-san(rw,no_root_squash) racnode3-san(rw,no_root_squash)
/storage/oracle_dbf_testdb racnode1-san(rw,no_root_squash) racnode2-san(rw,no_root_squash) racnode3-san(rw,no_root_squash)
Make the change permanent:
#> exportfs -a
Now reboot the 3rd node:
#> reboot
Make sure our volumes are mounted with the following command:
#> df -k
We now need to set up the ssh connectivity from the third node to the existing two nodes, and vice versa:
node3_oracle$> ssh-keygen -t rsa
Copy the contents of the /home/oracle/.ssh/id_rsa.pub file on node 3 into the /home/oracle/.ssh/authorized_keys file on both node 1 and node 2.
Copy the /home/oracle/.ssh/id_rsa.pub file to /home/oracle/.ssh/authorized_keys on node 3, then add to /home/oracle/.ssh/authorized_keys file on node 3, the contents of the id_rsa.pub files from nodes 1 and 2 into this file
We now need to connect to each connection via ssh so that we can manually answer the question that appears, therefore do the following, answering yes to all the prompts that appear:
on node 3:
ssh oracle@racnode1
ssh oracle@racnode1-priv
ssh oracle@racnode1-san
ssh oracle@racnode2
ssh oracle@racnode2-priv
ssh oracle@racnode2-san
ssh oracle@racnode3
ssh oracle@racnode3-priv
ssh oracle@racnode3-san
And on the existing two nodes:
ssh oracle@racnode3
ssh oracle@racnode3-priv
ssh oracle@racnode3-san
We now need to configure ntp on our third node, to get its time from the storage node.
perform the following:
#> yast
Navigate to Network Services, NTP Configuration
Change the Start NTP Daemon to Now and On Boot
Click on Delete
Confirm the deletion
Click on Add
Ensure Server is selected, click on Next
Enter 192.168.100.3 in the Address box
Click on OK
Click on OK
Click on Quit
Then edit the /etc/sysconfig/ntp file, changing the line:
NNTP_OPTIONS=”-g -u ntp:ntp”
to
NNTP_OPTIONS=”-x”
And restart the ntp service:
#> service ntp restart
We should now be in a position to add the node, so lets run the cluster verification utility from one of the existing nodes, to make sure everything is correct:
node1_oracle$> /tmp/oracle/grid/runcluvfy.sh stage -pre nodeadd -n racnode3
Make sure this returns a successful message.
We can now add our node into Oracle Clusterware, so from the existing node 1, run:
node1_oracle$> /home/grid/app/11.2.0/grid/oui/bin/addNode.sh “CLUSTER_NEW_NODES={racnode3}” “CLUSTER_NEW_VIRTUAL_HOSTNAMES={racnode3-vip}” -silent
We now need to run the root scripts on node3:
node3#> /home/oracle/app/oraInventory/orainstRoot.sh
node3#> /home/grid/app/11.2.0/grid/root.sh
Make sure the node is added, by using the cluster verification tool on node 1:
node1_oracle$> /tmp/oracle/grid/runcluvfy.sh stage -post nodeadd -n racnode3
This should return as successful.
We now need to extend the Oracle Home to the third node, so from the first node:
node1_oracle$> /home/oracle/app/oracle/product/11.2.0/dbhome_1/oui/bin/addNode.sh -silent “CLUSTER_NEW_NODES={racnode3}”
We now need to run the root scripts on node3:
node3#> /home/oracle/app/oracle/product/11.2.0/dbhome_1/root.sh
We now need to add the instance to the node. So we run dbca on node 1:
oracle_node1$> export ORACLE_HOME=/home/oracle/app/oracle/product/11.2.0/dbhome_1
oracle_node1$> export PATH=$PATH:$ORACLE_HOME/bin
oracle_node1$> dbca &
Select Oracle RAC database
click on Next
Select Instance Management
click on Next
Select Add an Instance
click on Next
Make sure TESTDB is selected, enter sys for the username and enter the password
click on Next
click on Next
make sure the TESTDB3 is shown along with racnode3
click on Next
click on Finish
click on OK
Now use the cluster verification tool to confirm the addition:
oracle_node1$> export ORACLE_HOME=/home/oracle/app/oracle/product/11.2.0/dbhome_1
oracle_node1$> /tmp/oracle/grid/runcluvfy.sh comp admprv -o db_config -d $ORACLE_HOME -n racnode1,racnode2,racnode3
You should now have completed the addition of a third node into your cluster.
What a nice blog classical poetry
ReplyDelete