« January 2007 | Main | March 2007 »

February 23, 2007

Implementing Parallel Concurrent Processing

Implementing Parallel Concurrent Processing

Parallel Concurrent Processing is the ability to simultaneously run your concurrent managers of various nodes. This results in efficient processing power and also allows the ability of failover. That is in the even of one of your concurrent manager node becomes un available the standard managers will fail over to the other node.

You can implement PCP either in a normal implementation of Oracle Applications or by implementing RAC, however to exploit the feature of PCP completely it is best that you implement RAC to ensure that you have a oncurrent managers available even if your DB node comes down.

In my current post i will write about implementing PCP on a non- RAC environment.

Rapidinstall does not allow you to configure two concurrent processing nodes at the time of install for which I will I have to use rapid clone to create another concurrent processing node for me.

In the current example i have a single node installation of Oracle Applications (11.5.10.2) running on Node A.

The brief overview of steps involved in this implementation of PCP are

  • Use Rapidclone to clone and create another node.
  • Use adaddnode and register the new node in the database.
  • Use Oracle Application Manager to turn PCP on.
  • Define and Verfiy the Concurrent Manager seetings and Workshifts.
  • Test Failover.


Use Rapidclone to clone and create another node

Running Preclone on current node
#su - applsam
$cd samappl
$. ./APPSORA.env
$cd $COMMON_TOP/admin/scripts/SAM_SAMLXA
$perl adpreclone.pl appsTier

Disable security option
In Oracle Applications 11.5.10 the SQL*NET restricted security option is enabled by default. As a result your new node Node_B  will not be able to connect to the database. You must either disable this security option or the better way is to add Node_B is the list of trusted nodes through OAM.
Oracle Applications Manager -> Security ->Disable Restricted Sectrity Option.

After this you must run Autoconfig on the database Tier and restart your listener.
# su - orasam
$ cd $ORACLE_HOME/appsutil/scripts/SAM_NOde_A
$ ./adautocfg.sh

Restart your listener
$ ./addlnctl.sh stop sam
$ ./addlnctl.sh start sam

Preparing the new node (Node B) to copy the application tier file system.
On Node B make the following dir structure
# cd /u02
# mkdir sam
# chmod -R 777 sam

Copy the files across to Node B
Shutdown on Application and Database services on Node A
tar -cf - applsam | ssh Node_B tar -xf - -C /u02/sam/

Post Clone on Node B
Now you must run your post clone (adcfgclone.pl) on Node B and specify that it is your admin and concurrent processing node
# su - applsam
$ cd samcomn/clone/bin
$ perl adcfgclone.pl appsTier

Use adaddnode and register the new node in the database
After sourcing your environment file you must run the adaddnode.pl script on node A
#su - applsam
$ cd $COMMON_TOP/clone/bin
$ perl adaddnode.pl

The script will prompt you for your APPS user and password and do the required new node entries to the database FND tables.

Use Oracle Application Manager to turn PCP on
Once all the configured services have been started on both the nodes log on to your Oracle Applications Manager and turn the value for the APPLDCP variable to ON. Please make sure to do this change on the application tier context files on both the nodes.

Autoconfig
Shutdown the application tier services of both the nodes and run autoconfig on both the nodes.
After autoconfig completes successfully restart the application tier services on both the nodes.

Verfiy TNSNAMES.ora
Verify that the tnsnames.ora of your application tier has the FNDFS entries of both the nodes
FNDFS_SAMLXA=

        (DESCRIPTION=
                (ADDRESS=(PROTOCOL=tcp)(HOST=SAMLXA.appsdbablog.com)(PORT=1676))
            (CONNECT_DATA=
                (SID=FNDFS)
            )
        )
FNDFS_SAMLXA.appsdbablog.com=
        (DESCRIPTION=
                (ADDRESS=(PROTOCOL=tcp)(HOST=SAMLXA.appsdbablog.com)(PORT=1676))
            (CONNECT_DATA=
                (SID=FNDFS)
            )
        )
FNDFS_SAMLXB=
        (DESCRIPTION=
                (ADDRESS=(PROTOCOL=tcp)(HOST=SAMLXB.appsdbablog.com)(PORT=1676))
            (CONNECT_DATA=
                (SID=FNDFS)
            )
        )
FNDFS_SAMLXB.appsdbablog.com=
        (DESCRIPTION=
                (ADDRESS=(PROTOCOL=tcp)(HOST=SAMLXB.appsdbablog.com)(PORT=1676))
            (CONNECT_DATA=
                (SID=FNDFS)
            )
        )

Also verify that the failover entry exits in the tnsnames.ora of the application tier on both the nodes
SAM_BALANCE=
        (DESCRIPTION=
            (LOAD_BALANCE=YES)
            (FAILOVER=YES)
            (ADDRESS_LIST=
                (ADDRESS=(PROTOCOL=tcp)(HOST=SAMLXA.appsdbablog.com)(PORT=1571))
            )
            (CONNECT_DATA=
                (SID=SAMP)
            )
        )

Define and Verify your concurrent Manager definitions
Define the Primary and Secondary nodes for your Standard Manager and Internal Concurrent Manager
Log in to Oracle Applications as the SYSADMIN user and navigate to Concurrent Manager Administer screen.



Also define your Primary and Secondary nodes for your Internal Monitor : Node A and Internal Monitor : Node B. The primary Node for your Internal Monitor Node A will be Node A and the secondary will be Node B and vice versa for the Internal monitor Node B.


Testing Fail Over
You can test the failover process by shutting down node B in which case the the standard manager will failover to the secondary node which is Node B. Once the node comes back alive the concurrent managers will again failback.

Since PCP here is configured in a non RAC environment you cannot have an instance failover here, that is if your DB Node is down the concurrent managers will not failover.

February 21, 2007

Using Distributed AD

Using Distributed AD

In my last post i talked about implementing a Shared Application Tier Filesystem. On of the important advantages of this is that ability to drastically reduce your patching down time by using the concept of Distributed AD.My current post talks about using distributed AD.The main concept behind distributed AD is that you can run part of your workers on one node and  distribute the remaining among other nodes which share the same application tier filesystem for applying the same patch. This results in an improved performance and helps in reducing your downtime related to patching.

Two important pre requisites for using Distributed AD are

  • You must have AD Minipack H (patch 2673262)
  • You must have implement either a shared application tier filesystem or a shared APPL_TOP

Assuming i have implemented a Shared Application Tier Filesystem between Node A and Node B i could apply the ATG RUP 4 Patch using distributed AD in the following way.

On Node A i start the adpatch session after putting my system in maintenance mode. To start adpatch in a distributed mode you can use the localworkers and workers parameter.

On Node A

su - applsam
$adpatch localworkers=4 workers=8
                     Copyright (c) 2002 Oracle Corporation
                       Redwood Shores, California, USA
                        Oracle Applications AutoPatch
                                 Version 11.5.0
NOTE: You may not use this utility for custom development
unless you have written permission from Oracle Corporation.
Attention: AutoPatch no longer checks for unapplied pre-requisite patches.
You must use OAM Patch Wizard for this feature. Alternatively, you can
review the README for pre-requisite information.
Your default directory is '/s01/apps/samappl'.
Is this the correct APPL_TOP [Yes] ?
AutoPatch records your AutoPatch session in a text file
you specify.  Enter your AutoPatch log file name or press [Return]
to accept the default file name shown in brackets.
Filename [adpatch.log] : sampt04.log
Num Workers = 8.
You can be notified by email if a failure occurs.
Do you wish to activate this feature [No] ?
Please enter the batchsize [1000] :
Please enter the name of the Oracle Applications System that this
APPL_TOP belongs to.
The Applications System name must be unique across all Oracle
Applications Systems at your site, must be from 1 to 30 characters
long, may only contain alphanumeric and underscore characters,
and must start with a letter.
Sample Applications System names are: "prod", "test", "demo" and
Development_2".
Applications System Name [SAM] : SAM *
NOTE: If you do not currently have certain types of files installed
in this APPL_TOP, you may not be able to perform certain tasks.
Example 1: If you don't have files used for installing or upgrading
the database installed in this area, you cannot install or upgrade
the database from this APPL_TOP.
Example 2: If you don't have forms files installed in this area, you cannot
generate them or run them from this APPL_TOP.
Example 3: If you don't have concurrent program files installed in this area,
you cannot relink concurrent programs or generate reports from this APPL_TOP.
Do you currently have files used for installing or upgrading the database
installed in this APPL_TOP [YES] ? YES *
Do you currently have Java and HTML files for HTML-based functionality
installed in this APPL_TOP [YES] ? YES *
Do you currently have Oracle Applications forms files installed
in this APPL_TOP [YES] ? YES *
Do you currently have concurrent program files installed
in this APPL_TOP [YES] ? YES *
Please enter the name Oracle Applications will use to identify this APPL_TOP.
The APPL_TOP name you select must be unique within an Oracle Applications
System, must be from 1 to 30 characters long, may only contain
alphanumeric and underscore characters, and must start with a letter.
Sample APPL_TOP Names are: "prod_all", "demo3_forms2", and "forms1".
APPL_TOP Name [samlx01] : samlx01 *
You are about to apply a patch to the installation of Oracle Applications
in your ORACLE database 'SAM'
using ORACLE executables in '/s01/apps/samora/8.0.6'.
Is this the correct database [Yes] ?
AutoPatch needs the password for your 'SYSTEM' ORACLE schema
in order to determine your installation configuration.
Enter the password for your 'SYSTEM' ORACLE schema:
The ORACLE username specified below for Application Object Library
uniquely identifies your existing product group: APPLSYS
Enter the ORACLE password of Application Object Library [APPS] :
AutoPatch is verifying your username/password.
The status of various features in this run of AutoPatch is:
                                           <-Feature version in->
Feature                          Active?   APPLTOP    Data model    Flags
------------------------------   -------   --------   -----------   -----------
CHECKFILE                        Yes       1          1             Y N N Y N Y
PREREQ                           Yes       6          6             Y N N Y N Y
CONCURRENT_SESSIONS              No        2          2             Y Y N Y Y N
PATCH_TIMING                     Yes       2          2             Y N N Y N Y
PATCH_HIST_IN_DB                 Yes       6          6             Y N N Y N Y
SCHEMA_SWAP                      Yes       1          1             Y N N Y Y Y
Identifier for the current session is 1781
Reading product information from file...
Reading language and territory information from file...
Reading language information from applUS.txt ...
Reading database to see what industry is currently installed.
Reading FND_LANGUAGES to see what is currently installed.
Currently, the following language is installed:
Code   Language                                Status
----   --------------------------------------- ---------
US     American English                        Base
Your base language will be AMERICAN.
Setting up module information.
Reading database for information about the modules.
Saving module information.
Reading database for information about the products.
Reading database for information about how products depend on each other.
Reading topfile.txt ...
Saving product information.
AD code level : [11i.AD.I.4]
Trying to obtain a lock...
  Attempting to instantiate the current-view snapshot...
  Was already instantiated. So no instantiation done this time.
     **************** S T A R T   O F   U P L O A D ****************
Start date: Wed Feb 21 2007 01:20:10
0 "left over" javaupdates.txt files uploaded to DB: Wed Feb 21 2007 01:20:10
0 patches uploaded from the ADPSV format patch history files: Wed Feb 21 2007 01:20:10
Uploading information about files copied during the previous runs ...
0 "left over" filescopied_<session_id>.txt files uploaded to DB: Wed Feb 21 2007 01:20:10     ****************** E N D   O F   U P L O A D ******************
End date: Wed Feb 21 2007 01:20:10
Enter the directory where your Oracle Applications patch has been unloaded
The default directory is [/s01/apps/4676589] :
Please enter the name of your AutoPatch driver file : u4676589.drv

On Node B
You can start adctrl on node b in a distributed mode by using the distributed parameter. It will prompt for the number of works you need to run on this node at which you can specify the range

$su - applsam
$adctrl distributed=y
                     Copyright (c) 2002 Oracle Corporation
                        Redwood Shores, California, USA
                                 AD Controller
                                 Version 11.5.0
NOTE: You may not use this utility for custom development
      unless you have written permission from Oracle Corporation.
Attention: AD Controller no longer checks for unapplied pre-requisite patches.
You must use OAM Patch Wizard for this feature. Alternatively, you can
review the README for pre-requisite information.
Your default directory is '/s01/apps/samappl'.
Is this the correct APPL_TOP [Yes] ?
AD Controller records your AD Controller session in a text file
you specify.  Enter your AD Controller log file name or press [Return]
to accept the default file name shown in brackets.
Filename [adctrl.log] :adcrtls03.log
************* Start of AD Controller session *************
AD Controller version: 11.5.0
AD Controller started at: Wed Feb 21 2007 12:49:12
APPL_TOP is set to /s01/apps/samappl
Reading product information from file...
Reading language and territory information from file...
Reading language information from applUS.txt ...
Enter the ORACLE username of Application Object Library [APPLSYS] :
Enter the ORACLE password of Application Object Library [APPS] :
Feature                          Active?   APPLTOP    Data model    Flags
------------------------------   -------   --------   -----------   -----------
CHECKFILE                        Yes       1          1             Y N N Y N Y
PREREQ                           Yes       6          6             Y N N Y N Y
CONCURRENT_SESSIONS              No        2          2             Y Y N Y Y N
PATCH_TIMING                     Yes       2          2             Y N N Y N Y
PATCH_HIST_IN_DB                 Yes       6          6             Y N N Y N Y
SCHEMA_SWAP                      Yes       1          1             Y N N Y Y Y
Enter contiguous range of workers to run on this machine
Enter the worker range:5-8
Waiting...

The workers will be in waiting state till all pre requisite steps are finished by the adpatch session on the first node.

Managing the workers
You can see the status of all your works both local and remote using adctrl from any of the nodes.
        Control
Worker  Code      Context                 Filename              Status
------  --------  ----------------------  --------------------  --------------
     1  Run       AutoPatch R115          alalrt.odf            Running
     2  Run       AutoPatch R115          afflex.odf            Running
     3  Run       AutoPatch R115          afcmgr.odf            Running
     4  Run       AutoPatch R115          ecepoo.odf            Running
     5  Run       AutoPatch R115          afmisc.odf            Running
     6  Run       AutoPatch R115          afform.odf            Running
     7  Run       AutoPatch R115          ecxbase.odf           Running
     8  Run       AutoPatch R115          afatch.odf            Running

All the options of adcrtl hold good in a distributed AD environment.

Restarting a Failed worker
You can restart a failed worker from any of the nodes irrespective of the fact where the worker was actually running
Worker  Code      Context                 Filename              Status
------  --------  ----------------------  --------------------  --------------
     1  Run       AutoPatch R115                                Wait
     2  Run       AutoPatch R115          afcmgr.odf            FAILED
     3  Run       AutoPatch R115                                Wait
     4  Run       AutoPatch R115                                Wait
     5  Run       AutoPatch R115                                Wait
     6  Run       AutoPatch R115                                Wait
     7  Run       AutoPatch R115                                Wait
     8  Run       AutoPatch R115                                Wait

In the above case worker 2 has failed which has been running on node BThis can be restarted from either of the nodes after fixing the problem, using the option 2 from the adcrtl menu 'Tell worker
to restart a failed job'

February 08, 2007

Implementing Shared Application Tier Filesystem Using NFS

Implementing Shared Application Tier Filesystem Using NFS

A few posts ago I had discussed about the Shared APPL_TOPs and Shared Application Tier Filesystem feature in Oracle Application. While the ability to shared APPL_TOPs has been there for sometime, the feature of sharing the entire Application Tier Filesystem was introduced with 11.5.10.The difference between both being that while the shared APPL_TOP allowed sharing only of the application toip files among the nodes. In the Shared Application Tier filesystem the complete application tier file system along with the tech stacks are also shared among the nodes.

To implement a shared application tier filesystem we need top have a shared disk resource which is accessible to all the nodes which consist of Oracle Applications architecture. This shared disk can be in the form of a shared disk or an NFS mounted shared disk.In the current post I will blog about the steps involved in implementing a shared application tier file system using NFS on Redhat Linux.

System Configuration
Node_A (To be used for web server and forms server)
Node_B (To be used for db tier and concurrent manager and admin node)

Objective
To implement a shared application tier file system using an NFS mount and having a full install on Node_B and sharing the existing with Node_A.

Create OS Users
Note that is is necessary to have the same UID and GID for both the OS users on Node_A and Node_B.Also the user names must be the same so as not to have any file permission problems.

Create OS Users on Node_A
# useradd -g dba -d /home/applsam applsam

Create OS Users on Node_B
# useradd -g dba -d /home/orasam orasam
# useradd -g dba -d /home/applsam applsam

Create Directory Structure
I used the following directory structure for the installation, you may use your own and make sure that it is shared accordingly.

On Node_B
mkdir /s01/apps (to hold the application tier filesystem)
mkdir /s01/data (to hold the database tier filesystem)

Setup NFS Mount
Setup NFS mount to share the /s01/apps filesystem of Node_B  to Node_A. You must however NFS mount with the nolock option to avoid issues with autoconfig and adpatch.

On Node_B
Edit the exports file to add in the shared filesystem details
#vi /etc/exports
/s01/apps (rw) Node_A
Restart the NFS Service on Node_B
# service nfs restart
Shutting down NFS mountd:                                  [  OK  ]
Shutting down NFS daemon:                                  [  OK  ]
Shutting down NFS quotas:                                  [  OK  ]
Shutting down NFS services:                                [  OK  ]
Starting NFS services:                                     [  OK  ]
Starting NFS quotas:                                       [  OK  ]
Starting NFS daemon:                                       [  OK  ]
Starting NFS mountd:                                       [  OK  ]

On Node_A
Create a directory to hold the mount and do an NFS mount of the filesystem
mkdir /s01/apps
mount -t nfs Node_B:/s01/apps /s01/apps -o nolock

Starting the Installation

On Node_B
The installation is started on Node_B with the root user. Before that make sure you set up the path for your oracle inventory

Setup orainventory location
vi /etc/oraInst.loc
inventory_loc=/s01/oraInventory
inst_group=dba

During the rapidwiz installation screen you must choose a multi node install option and check the Share Application Tier Filesystem Option.



Choose the nodes where you wish to do a complete install and the nodes where you want to share the existing installation.



After post installation checks are complete you can proceed to the next node.
On Node_A
On Node_A since i have only one OS user (applsam) i start the installation form that user and source the config.txt file created as part of the installation done on Node_B.



Since Node_A is going to share the entire application tier filesystem from Node_B no installation will be required as such. However we run rapidwiz on Node_A to configure it to perform context value management and startup the configured services for that node.