NNMI - TeMIP - ATNI Integration

NNMi & TeMIP Integration:

On NNMi install these.

Note : Please refer temip6_ATNI6_InstallConfig.pdf for prerequisites & further information.

TeMIP (Telecom Management Information Platform)

1.       Install  TeMIP Utility package

rpm -ivh TeMIP-pkgutils-1.1-8.x86_64.rpm

Note : Install Java

[root@IPFmsImsProd29 TeMIP-TNT-V60]# pwd

/NNMiTeMIPIntegration/TeMIP-TNT-V60/JDK15

[root@IPFmsImsProd29 TeMIP-TNT-V60]# rpm -ivh jdk-1_5_0_15-linux-amd64.rpm

Preparing...                ########################################### [100%]

   1:jdk                    ########################################### [100%]

Export the path : export JAVA_HOME=/usr/java/jdk1.5.0_15/

2.       Install the TeMIP Adapter subset (TNTADAPTER), execute the following command with root privileges in the KIT directory:

pkgadd -R /opt/OV/ -a $PWD/adminFile -d $PWD TNTADAPTERV600L

[root@IT]# pkgadd -R /opt/OV  -a  $PWD/adminFile  -d  $PWD  TNTADAPTERV600L NOTE:  <KIT LOCATION or  $PWD> = /root/Desktop/NNMiTeMIPIntegration/TeMIP-TNT-V60/KIT/TNTV60L_RevA/TeMIP-V60L-UNIX/TeMIP-TNT/KIT Check the output result at last as: . . . Configure product using temip_adapter_configure Installation of HP OpenView TeMIP NNM Toolkit Adapter                  (TNTADAPTERV600L) was successful. To check the installed package :                [root@ipnmsimstnd46 KIT]# pkginfo -R /opt/OV/                  application TNTADAPTERV600L                HP OpenView TeMIP NNM Toolkit Adapter 1.       Install the Customization Toolkit subset (TNTCUSTTK), execute the following command with root privileges in the KIT directory: pkgadd -R /opt/OV/ -a <Kit_location>/adminFile -d <Kit_location>      TNTCUSTTK<Version> @ IDEA [root@IT]# pkgadd -R /opt/OV/ -a  $PWD/adminFile -d $PWD TNTCUSTTKV600L NOTE: <KIT LOCATION or  $PWD> = /root/Desktop/NNMiTeMIPIntegration/TeMIP-TNT-V60/KIT/TNTV60L_RevA/TeMIP-V60L-UNIX/TeMIP-TNT/KIT Check the output result at last as: . . . Installation of HP OpenView TeMIP NNM Toolkit - Customization Toolkit (TNTCUSTTKV600L) was successful. 2.       Install the Third Party Products subset (TNTTPP) execute the following command with root privileges in the KIT directory:       #pkgadd -R /opt/OV -a <kit_location>/adminFile -d <kit_location> TNTTPP<Version>        [root@IT]# pkgadd -R /opt/OV/ -a $PWD/adminFile -d $PWD TNTTPPV600L NOTE : <KIT LOCATION or  $PWD>  = /root/Desktop/NNMiTeMIPIntegration/TeMIP-TNT-V60/KIT/TNTV60L_RevA/TeMIP-V60L-UNIX/TeMIP-TNT/KIT/ Check the output result at last as: . . . Installation of HP OpenView TeMIP NNM Toolkit - Third party products (TNTTPPV600L) was successful. 3. Install the Customization Toolkit subset (TNTCUSTTK), execute the following command with root privileges in the KIT directory: pkgadd -R /opt/OV/ -a <Kit_location>/adminFile -d <Kit_location> TNTCUSTTK<Version>  [root@IT]# pkgadd -R /opt/OV/ -a $PWD/adminFile -d $PWD TNTCUSTTKV600L NOTE: <KIT LOCATION or $PWD> = /root/Desktop/NNMiTeMIPIntegration/TeMIP-TNT-V60/KIT/TNTV60L_RevA/TeMIP-V60L-UNIX/TeMIP-TNT/KIT Check the output result at last as:  . . . Installation of HP OpenView TeMIP NNM Toolkit - Customization Toolkit (TNTCUSTTKV600L) was successful. 4. Install the Third Party Products subset (TNTTPP) execute the following command with root privileges in the KIT directory: #pkgadd -R /opt/OV -a <kit_location>/adminFile -d <kit_location> TNTTPP<Version> [root@IT]# pkgadd -R /opt/OV/ -a $PWD/adminFile -d $PWD TNTTPPV600L NOTE : <KIT LOCATION or $PWD> = /root/Desktop/NNMiTeMIPIntegration/TeMIP-TNT-V60/KIT/TNTV60L_RevA/TeMIP-V60L-UNIX/TeMIP-TNT/KIT/ Check the output result at last as:  . . . Installation of HP OpenView TeMIP NNM Toolkit - Third party products (TNTTPPV600L) was successful. To check the installation of all the above 3 packages use the below command:  [root@ipnmsimstnd46 KIT]# pkginfo -R /opt/OV application TNTADAPTERV600L            HP OpenView TeMIP NNM Toolkit Adapter application TNTCUSTTKV600L              HP OpenView TeMIP NNM Toolkit - Customization Toolkit application TNTTPPV600L                       HP OpenView TeMIP NNM Toolkit - Third party products 5. Apply the below 2 patches ( Mandatory ) else integration fails. [ Not mentioned in Document or Sorry if I was unable to get this thing in document  ] TEMIPTNTLIN_00018.tar TEMIPTNTLIN_00019.tar  In the patch folder TEMIPTNTLIN_00019 find the file “TEMIPTNTLIN_00019.text” and edit below file as per the instructions to add custom. /var/opt/OV/TNT/adapter/conf/ tnt_configuration.properties $PWD = /root/Desktop/NNMiTeMIPIntegration/TeMIP_Patches/TEMIPTNTLIN_00019/TEMIPTNTLIN_00019 pkgadd -R /opt/OV -a $PWD/adminFile -d $PWD Check the patch installation by using the below command [root@IT]# pkginfo -R /opt/OV Output should include these two along with those 3 packages: application TEMIPTNTLIN-00018 TNTPATCH07605V600L:ComputedVariable in AddText application TEMIPTNTLIN-00019 TNTPATCH07351V600L:Support new CIA pattern 6. Configure the TeMIP Adapter (Mandatory) a. Go To adapter directory : /opt/OV/TNT/adapter/bin Be sure that the adapter services are stopped. To check the adapter status use the below command: temip_adapter_show Stop the adapter service by using the command:       temip_adapter -stop b. Create the user in NNMi with guest role. c. Configure TeMIP Adapter by using the command /opt/OV/TNT/adapter/bin/temip_adapter_configure -h // -h for help Note :        Add this line in ~/.bash_profile before configuring temip adapter. export JAVA_HOME=/usr/java/jdk1.5.0_15/ and run . ~/.bash_profile [root@in]# ./temip_adapter_configure -F -n <NNMi FQDN> -i 80 -u <system> -p <system-pwd> -g temip -w temip WHERE: 80 > NNMi Port System,  userwith Web Service Client role. Temip > Username and password created for integration with guest access Start the TeMIP adapter services using the below command” [root@ipnmsimstnd46 bin]# temip_adapter –start On successful configuration the status should be like below and [root@bin]# temip_adapter_show TeMIP Adapter processes: PID     UID     APPLICATION ---     ----    ------------- 4684 root TeMIP Adapter (Servlet Engine) The Adapter is initialized and ready  In the NNMi Custom Attributes should be updated on every node END

Veritas Cluster Debugging Tips

Veritas Cluster Debugging Tips

Initial Notes

Veritas cluster server is a high availability server. This means that processes switch between servers when a server fails. All database processes are run through this server - and as such, this needs to run smoothly. Note that the oracle process should only actually be running on the server which is active. On monitoring tools, the procs light for whichever box is secondary should be yellow, because oracle is not running. Yet, the cluster is running on both systems.

Cluster Not Up -- HELP

• The normal debugging of steps includes: checking on status, restarting if no faults, checking licenses, clearing faults if needed, and checking logs. To find out Current Status:
  
  • /opt/VRTSvcs/bin/hastatus -summary
    This will give the general status of each machine and processes
    
    /opt/VRTSvcs/bin/hares -display
    This gives much more details - down to the resource level. If hastatus fails on both machines (it returns that the cluster is not up or returns nothing), try to start the cluster
    
    • /opt/VRTSvcs/bin/hastart
      
      /opt/VRTSvcs/bin/hastatus -summary
      will tell you if processes started properly. It will NOT start processes on a FAULTED system.
  Starting Single System NOT Faulted
  
  • If the system is NOT FAULTED and only one system is up, the cluster probably needs to have gabconfig manually started. Do this by running:
    • /sbin/gabconfig -c -x /opt/VRTSvcs/bin/hastart
      /opt/VRTSvcs/bin/hastatus -summary
    If the system is faulted, check licenses and clear the faults as described next.
  To check licenses:
  
  • vxlicense -p Make sure all licenses are current - and NOT expired! If they are expired, that is your problem. Call VERITAS to get temporary licenses.
    There is a BUG with veritas licences. Veritas will not run if there are ANY expired licenses -- even if you have the valid ones you need. To get veritas to run, you will need to MOVE the expired licenses. [Note: you will minimally need VXFS, VxVM and RAID licenses to NOT be expired from what I understand.]
    
    • vxlicense -p
      Note the NUMBER after the license (ie: Feature name: DATABASE_EDITION [100]) cd /etc/vx/elm
      mkdir old
      mv lic.number old [do this for all expired licenses]
      vxlicense -p [Make sure there are no expired licenses AND your good licenses are there]
      hastart
      If still fails, call veritas for temp licenses. Otherwise, be certain to do the same on your second machine.
  To clear FAULTS:
  
  • hares -display
    For each resource that is faulted run:
    • hares -clear resource-name -sys faulted-system
    
    If all of these clear, then run hastatus -summary and make sure that these are clear. If some don't clear you MAY be able to clear them on the group level. Only do this as last resort:
    • hagrp -disableresources groupname
      hagrp -flush group -sys sysname
      hagrp -enableresources groupname
    
    To get a group to go online:
    • hagrp -online group -sys desired-system
    If it did NOT clear, did you check licenses?
  Bringing up Machines when fault will NOT clear:
  
  • System has the following EXACT status:
    

    gedb002# hastatus -summary
    
    -- SYSTEM STATE
    -- System               State                Frozen
    
    A  gedb001              RUNNING              0
    A  gedb002              RUNNING              0
    
    -- GROUP STATE
    -- Group           System               Probed     AutoDisabled    State         
    
    B  oragrp          gedb001              Y          N               OFFLINE       
    B  oragrp          gedb002              Y          N               OFFLINE       
    
    gedb002#  hares -display | grep  ONLINE
    nic-qfe3  State           gedb001   ONLINE
    nic-qfe3  State           gedb002   ONLINE
    
    gedb002# vxdg list
    NAME         STATE           ID
    rootdg       enabled  957265489.1025.gedb002
    
    gedb001# vxdg list
    NAME         STATE           ID
    rootdg       enabled  957266358.1025.gedb001
    
    

    Recovery Commands:
    
    • hastop -all
      on one machine hastart
      wait a few minutes
      on other machine hastart
  Reviewing Log Files:
  If you are still having troubles, look at the logs in /var/VRTSvcs/log. Look at the most recent ones for debugging purposes (ls -ltr). Here is a short description of the logs in /var/VRTSvcs/log:
  
  • hashadow-log_A: hashadow checks to see if the ha cluster daemon (had) is up and restarts it if needed. This is the log of that process. engine.log_A: primary log, usually what you will be reading for debugging
    Oracle_A: oracle process log (related to cluster only)
    Sqlnet_A: sqlnet process log (related to cluster only)
    IP_A: related to shared IP
    Volume_A: related to Volume manager
    Mount_A: related to mounting actual filesystes (filesystem)
    DiskGroup_A: related to Volume Manager/Cluster Server
    NIC_A: related to actual network device
  By looking at the most recent logs, you can know what failed last (or most recently). You can also tell what did NOT run which may be jut as much of a clue. Of course, if none of this helps, open a call with veritas tech support.
  Calling Tech Support:
  If you have tried the previously described debugging methods, call Veritas tech support: 800-634-4747. Your company needs to have a Veritas support contract.

Restarting Services:

If a system is gracefully shutdown and it was running oracle or other high availability services, it will NOT transfer them. It only transfers services when the system crashes or has an error.

• hastart
  
  hastatus -summary
  will tell you if processes started properly. It will NOT start processes on a FAULTED system. If the system is faulted, clear the faults as described above.

Doing Maintenance on DBs:

BEFORE working on DB

• Run hastop -all -force

AFTER working on Dbs:

• You MUST bring up oracle on same machine Once Oracle is up, run:
  
  • hastart on the same machine as you started the work on (the first on system with oracle running)
    wait 3-5 minutes
    then run hastart on the other system
  
  If you need the instance to run on the other system, you can run: hagrp -switch oragrp -to othersystem

Shutting down db machines:

If you shutdown the machine that is running veritas cluster, it will NOT start on the other machine. It only fails over if the machine crashes. You need to manually switch the services if you shutdown the machine. To switch processes:

• Find out groups to transfer over
  hagrp -display
  Switch over each group
  hagrp -switch group-to-move -to new-system
  
  Then shutdown machine as desired. When rebooted will start cluster daemon automatically.

Doing Maintenance on Admin Network:

If the admin network is brought down (that the veritas cluster uses), veritas WILL fault both machines AND bring down oracle (nicely). You will need to do the following to recover:

• hastop -all
  On ONE machine: hastart
  wait 5 minutes
  On other machine: hastart

Manual start/stop WITHOUT veritas cluster:

THIS IS ONLY USED WHEN THERE ARE DB FAILURES

If possible, use the section on DB Maintenance. Only use this if system fails on coming up AND you KNOW that it is due to a db configuration error. If you manually startup filesystems/oracle -- manually shut them down and restart using hastart when done.

To startup:

Make sure ONLY rootdg volume group is active on BOTH NODEs. This is EXTREMELY important as if it is active on both nodes corruption occurs. [ie. oradg or xxoradg is NOT present]

• vxdg list
  hastatus (stop on both as you are faulted on both machines )
  hastop -all (if either was active make sure you are truly shutdown!)

Once you have confirmed that the oracle datagroup is not active, on ONE machine do the following:

• vxdg import oradg [this may be xxoradg where xx is the client 2 char code]vxvol -g oradg startall
  mount -F vxfs /dev/vx/dsk/oradg/name /mountpoint [Find volumes and mount points in /etc/VRTSvcs/conf/config/main.cf]
  Let DBAs do their stuff

To shutdown:

• umount /mountpoint [foreach mountpoint]vxdg deport oradg
  vxvol -g oradg stopall
  clear faults; start cluster as described above

HP OM Commands

HP OM Commands

############################################

To Start the OMU management server process
ostart –c

Ovc -start

To Stop the OMU management server process
ovstop –c

Ovc -kill

To check the OMU server status
ovstatus –c

Opcagt -status

To install agent on Unix
opc_inst -s

To install agent on Windows
cscript opc_inst.vbs -s

To Unistall agent
opc_inst -r

To check the agent status
opcagt status [local agent status]
opcragt [ remote agent status from management server ]

To Distribute Monitor, Action, Commands & Templates
opcragt -distrib -monitor -action -commands -templates -force

To add a node to node bank
opcnode -add_node node_name="" node_label="" net_type="NETWORK_IP" mach_type=" IP

Test opcmsg
/opt/OV/bin/OpC/opcmsg s=critical o=Test a=Test msg_text="This server is now being monitored"

To start the heartbeat polling of managed node
opchbp -start

To List Certificates:
ovcm -listpending -l
opccsa -list_pending_cr

To grant a certificate
ovcm -grant

To List and check status of certificates
ovcert -list
ovcert -status

To remove certificated
ovcert -remove

To create a new certificate request
ovcert -certreq

To list the templates/policies
opctemplate -l or ovpolicy -l

To enable/disable a template
opctemplate -e
opctemplate -d

To download histroy messages
opchistdwn -older 0s -file

To acknowledge messaged
opcackmsg -u
opcackmsgs -g -n

Manual Certificate Deployment
From management server
a) opccsacm -issue -name -file /tmp/a -pass
From agent
b) ovcert -importcert -file /tmp/a -pass

To get the remote coreid of the agent:
bbcutil -ping http://

To change the coreid of a node in the management server
opcnode -chg_id node_name= id=

To list the core id set in the node bank
opcnode -list_id node_list=

To map the node in the node bank to the certificate request
opccsa –map_node =

To create a certificate request from the managed node
ovcert -certreq

To create a new core id from the managed node
ovcoreid -create -force

To List the set of configurations from agent
ovconfget

To change the agent configuration
ovconfchg -ns -set

To list the groups in node group bank
opcnode -list_groups|grep Name|awk '{print $3}'

To change the communication type
opcnode -chg_commtype node_list= net_type=NETWORK_IP comm_type=COMM_BBC

To change the node type
opcnode -chg_nodetype node_list= node_type= [eg.. MESSAGE_ALLOWED]

To List nodes enrolled in OVO
opcnode -list_nodes|grep Name|awk '{print $3}'

To view coda collections
codautil -dumpds value

To check DB SPI connection
/var/opt/OV/bin/OpC/monitor/dbspicao -p -v -d or /var/opt/OV/bin/OpC/cmds/dbspiadm conncheck -nw

To switch off monitoring for DB instance
dbspicol OFF

Integrating DBSPI with OVPA [unix] :
/var/opt/OV/bin/OpC/cmds/dbspi_mw_int ( For enabling Graphing )
/var/opt/OV/bin/OpC/cmds/dbspi_mw_int -osm .. -udm ( For reporter )

To list open sessions of JAVA and Motif GUIs
checkuiwww

To check OM management server queue files
/opt/OV/contrib/OpC/opcqschk
/opt/OV/contrib/OpC/opcqchk

To create trap template from trapd.conf
ovtrap2opc /etc/opt/OV/share/conf/C/trapd.conf

To check the syntax of flexible management templates
opcmomchk

To list the services
opcservice -list

To assign a service to opc user
opcservcie -assign


Oracle start up scripts
/sbin/init.d/ovoracle start/stop
############################################

UNIX Agent recycle :
1. Kill the agent process using : ovc -kill
2. Remove all the files under "/var/opt/OV/log/OpC/"
3. Start the agent process using : ovc -start

############################################

opcdeploy/ovdeploy:
opcdeploy -cmd "" -node
opcdeploy -deploy -file -node -targetdir -trd absolute

############################################
SQL Tweaks

To get unassigned nodes to NG in OVO
select nn.node_name from opc_nodes n, opc_node_names nn
where nn.node_id = n.node_id
and not exists (select 1 from opc_nodes_in_group nig where nig.node_id = nn.node_id);

Show Assignments on Node
SELECT OPC_NODE_NAMES.NODE_NAME, OPC_SOURCE_TEMPL.TEMPLATE_NAME
FROM OPC_NODE_CONFIG, OPC_NODE_NAMES, OPC_SOURCE_TEMPL
WHERE OPC_NODE_CONFIG.NODE_ID = OPC_NODE_NAMES.NODE_ID AND
OPC_NODE_CONFIG.TEMPLATE_ID = OPC_SOURCE_TEMPL.TEMPLATE_ID AND
OPC_NODE_NAMES.NODE_NAME LIKE 'ENTER NODE NAME HERE';

Show Template Assignment
SELECT OPC_NODE_NAMES.NODE_NAME, OPC_SOURCE_TEMPL.TEMPLATE_NAME
FROM OPC_NODE_CONFIG, OPC_NODE_NAMES, OPC_SOURCE_TEMPL
WHERE OPC_NODE_CONFIG.NODE_ID = OPC_NODE_NAMES.NODE_ID AND
OPC_NODE_CONFIG.TEMPLATE_ID = OPC_SOURCE_TEMPL.TEMPLATE_ID AND
OPC_SOURCE_TEMPL.TEMPLATE_NAME LIKE 'ENTER TEMPLATENAME HERE';

############################################
Set environment variables in .profile

. /opt/OV/bin/ov.envvars.sh
PATH=$PATH:/opt/OV/bin
export PATH

############################################

/opt/OV/contrib/OpC/opcsystst
-a Do everything below
-h Display the hostname
-u Display the uname info
-i Display all configured network interfaces
-l Display the local IP address
-o Display the OpC mgmt server
-n Resolve and display the IP addresses for all displayed node
names and resolve and display the node names for all
displayed IP addresses
-r Display all registered RPC servers

############################################

/opt/OV/bin/OpC/install/opctranm -jobfile

whereas can have for example the following entries:

nodename.domain.com
/tmp/file1 /tmp/file2

=> this will transfer file1 from the management server /tmp directory to /tmp/file2 on the agent nodename.domain.com

nodename.domain.com
!ls

=> this will execute an ls on the unix agent node coco and return the result

nodename.domain.com
!ifconfig /all

############################################

You can use the SQL scripts in the below dir to get reports.

/etc/opt/OV/share/conf/OpC/mgmt_sv/reports/C/


For ex: call_sqlplus.sh sel_nodes --> will give the node details

############################################
OVPA
perfstat -p , -e , -v

/opt/perf/bin/utility -xc /var/opt/perf/alarmdef --> to chk syntax of alarmdef file
/opt/perf/bin/utility -xc /var/opt/perf/parm --> chk parm file

############################################

Directories

/var/opt/OV/share/tmp/OpC/distrib template Distribution directory
/etc/opt/OV/share/conf/OpC/mgmt_sv Flexible management templates
/var/opt/OV/share/conf/SPISvcDisc/ Service Discovery xml
/var/opt/OV/log/SPISvcDisc/discovery_error.log Discovery Error Log

Agent Policies
/var/opt/OV/datafiles/policies/configsettings/ --> nodeinfo
/var/opt/OV/datafiles/policies/le/ --> logfile templates
/var/opt/OV/datafiles/policies/mgrconf/ --> MoM templates
/var/opt/OV/datafiles/policies/monitor/ --> monitor templates
/var/opt/OV/datafiles/policies/msgi/ --> opcmsg templates
/var/opt/OV/datafiles/policies/trapi/ --> SNMP trap templs

############################################
Unix utils

Tar and Untar:
tar -cvf dir
tar -xvf a.tar

Mounting iso image to HPUX 11.31

mkdir /isoimg
lvcreate -n ISOLV -L 3096 /dev/vg00
dd if=isoimage of=/dev/vg00/rISOLV bs=8192
mount /dev/vg00/ISOLV /isoimg


using pfs mount

/usr/sbin/pfs_mountd &
/usr/sbin/pfsd &
pfs_mount -o xlat=unix /iso_image.iso /ISO_CDROM/

To change kernel parameters with command line
kctune -K -h = [Ex : kctune -K -h maxuprc=3687 ]

To Check the quality pack installed on HPUX
/usr/sbin/swlist | grep -e QPK -e GOLD -e BUNDLE11i

swinstall -s -x autoreboot=false \
-x patch_match_target=true