SOFTWARE RELEASE DOCUMENT ---------------------------------------------------------------------------- Product Name Integrity XC System NonStop Clusters for SCO UnixWare Product Number T5649D10IPM10 Part Number 139508 Softdoc Date 7 August 1998 ---------------------------------------------------------------------------- Release Summary This release of NonStop Clusters® for SCO® UnixWare® is an update to the D10 release of the Tandem® single system image (SSI) UNIX® clustering technology. This release includes features such as: * Cluster-wide (SSI): o Access to storage devices o Access to other system resources o Swap/paging service o System V IPC, devices, pipes o STREAMS access o Parallel networking o Cluster time synchronization o Distributed processes * Journaled file system * Disk mirroring using the Compaq SMART-2DH controller * File system failover * Application restart and failover Product Table of Contents This document contains descriptions of the following product components. Comp T Number Product Component Comp T Number Product Component T5649D10IPM10 NonStop Clusters Superseded IPMs This release supersedes the D10 release of NonStop Clusters. Requisite IPMs None Required Hardware None Required Firmware Version TPN 131790 A03-03 of the Tandem ServerNet(TM) SAN Switch. Required Software T5649D10 NonStop Clusters for SCO UnixWare operating system. T5649D10IPM10 NonStop Clusters for SCO UnixWare operating system. SCO UnixWare 2.1.2 (supplied with the original factory shipment) Required Disk Space Minimum 2 GB Manuals The manuals associated with this product are defined in the following table. Part Number Manual Title 140802 Integrity XC NonStop Clusters for SCO UnixWare System Administrator's Guide 140803 Integrity XC Site Preparation and Hardware Installation Migration Considerations None Installation Considerations Refer to this softdoc for D10 IPM10 installation instructions, and to the "Known Problems" section of this softdoc for installation considerations. General Notes 1. An on-line version of this softdoc will be updated as new information is available about the product. The current softdoc for this product is viewable on the World Wide Web and is accessed from a link on the softdoc index page: https://www.austx.tandem.com/softdocs/ These pages are password-protected. At the prompts, enter "Tandem" for the userID and "!ntegr1ty" for the password. The server is case-sensitive, so note the uppercase T in the userID and the exclamation point and number 1 in the password. Trademarks Tandem, Integrity, NonStop, NonStop-UX, and the Tandem logo are trademarks or registered trademarks of Tandem Computers Incorporated in the United States and/or other countries. Oracle is a registered trademark, and Oracle 7 is a trademark of Oracle Corporation. Compaq, ProLiant, and the Compaq logo are trademarks or registered trademarks of Compaq Computers Incorporated in the United States and/or other countries. SCO, The Santa Cruz Operation, UnixWare, and the SCO logo are trademarks or registered trademarks of Santa Cruz Operation Incorporated in the United States and/or other countries. INFORMIX is a registered trademark and ONLINE Dynamic Server is a trademark of Informix Software Incorporated. UNIX is a registered trademark in the United States and other countries, licensed exclusively through X/Open Company Limited. NFS is a registered trademark of Sun Microsystems, Inc. Veritas, VxFS, and VxVA are registered trademarks of Veritas Software Corporation. UniSoft is a registered trademark of UniSoft Corporation. All other brand and product names are trademarks or registered trademarks of their respective companies. Copyright Notice Copyright (c) 1998, Tandem Computers Incorporated. All rights reserved. The computer program listings, specifications and documentation herein are the property of Tandem Computers Incorporated or a third party supplier and shall not be reproduced, copied, disclosed, or used in whole or in part for any reason without the prior express written permission of Tandem Computers Incorporated. ---------------------------------------------------------------------------- Product Component: NonStop Clusters - T5649D10IPM10 ---------------------------------------------------------------------------- Description This is the D10IPM10 release of the NonStop Clusters for SCO UnixWare operating system for Compaq ProLiant Intel based PCs of the single-system image, distributed operating system. Contents The following packages belong to this component: Package Version Package Description D10IPM10 D10IPM10 NonStop Clusters software nscman D10IPM10 NonStop Clusters manpages D10IPM10doc D10IPM10 NonStop Clusters software release document Notes 1. Notes 1 and 2 of this document refers to installing D10IPM10 on an existing cluster installed with D10. If installing on a ProLiant 1600R/400 cluster, please see the installation instructions in Note 3. Before updating to D10IPM10, verify that all stand slices are mounted. At the prompt, enter the command "ptn -v". This will show all nodes that are currently up. For all of these nodes, the mount utility needs to show the /stand/ slice as mounted and the files in the directory need to be accessible. At the prompt, enter the command "mount". Examine the output that looks similar to the following: /stand/1 on /dev/dsk/n1c0b0t0d0sa read/write on Sun Apr 5 10:37:15 1998 /stand/2 on /dev/dsk/n2c0b0t0d0sa read/write on Sun Apr 5 10:21:08 1998 /stand/3 on /dev/dsk/n3c0b0t0d0sa read/write on Sun Apr 5 09:33:10 1998 If all the nodes are mounted, proceed with the update. Otherwise, mount the missing file systems and continue the update. Before performing the non-destructive installation of NonStop Clusters, back up the following files in the format .D10IPM10. o /etc/tnc/load_level/migrate_commands o /etc/tnc/load_level/rexec_commands o /etc/tnc/load_level/parameters o /etc/conf/mtune.d/tnc o /etc/conf/mtune.d/cfs o /etc/conf/mtune.d/tnc_dnet o /etc/conf/mtune.d/tnc_dswap o /etc/conf/mtune.d/tnc_nerd o /etc/conf/mtune.d/tncglue o /etc/conf/mtune.d/inet o /etc/conf/mtune.d/sysdump o /etc/conf/mtune.d/ipc o /etc/conf/sassign.d/kernel o /etc/conf/mtune.d/proc o /etc/conf/mtune.d/kdb_util o /etc/conf/init.d/kernel o /etc/vfstab o /stand/[1-6]/boot Example: cp /etc/tnc/load_level/migrate_commands /etc/tnc/load_level/migrate_commands.D10IPM10 Install the NonStop Clusters Package over D10 using the procedure in the Upgrading the Package section in the jewel box document and the procedure in Note 2 below. Before rebooting: Do not perform the procedure in the Examine the Logs and Resolve Files of the jewel box document. Instead, for each of the above file names, resolve any conflicts between it and the saved file. When finished, remove the saved file. 2. To install D10IPM10 you must have previously installed the D10 release of NonStop Clusters. The installation procedure is: cd / pkgadd -d cdrom1 -q D10IPM10 Since this package is an update to the D10 release, there are no additional questions that are asked. Once the package add is complete, you must manually finish the installation by merging in any local changes made to files that are also updated by the IPM. For D10IPM10, there are two such files in the"/etc/conf/mtune.d" directory. If you have made no changes to these kernel tunable files you can simply remove the files "kernel.0" and "tnc_dswap.0" (note the filenames end in zero). Otherwise, edit the "kernel/tnc_dswap" files to incorporate your local changes. Refer to the regular D10 installation instructions for details on how to handle these CMF (Customer Modifiable Files) class files. shutdown -y -g0 -i6 # regular shutdown/reboot sequence To remove the IPM: cd / pkgrm D10IPM10 # wait for package to be removed shutdown -y -g0 -i6 # regular shutdown/reboot 3. Install procedures for a ProLiant 1600R/400 Cluster. To Install the Root Node, execute the following steps: 1. Install UnixWare completely. 2. Confirm all ServerNet SAN cables are connected correctly to the ports of the nodes (and switches, if present). 3. Boot the root node and run the unix kernel. Login as the root user on the root node. NOTE: You must be in single-user mode, or run level 1, 2, or 3. To verify your current run level, use the "who -r" command. 4. Confirm that "/stand" is mounted and is accessible: # ls /stand (should show the boot file and the unix kernel) 5. Insert the NonStop Clusters D10 CD into the CD ROM drive, then enter the following command: # pkgadd -d cdrom1 -q ncu 6. Enter the NonStop Clusters serial number and activation key when prompted for it. 7. Enter "1" for the root node's number when prompted for it. 8. Enter "yes" at the "Enable Backup Failover Server" prompt. The pkgadd command will take about 10 to 20 minutes to complete. 9. After the message: *** IMPORTANT NOTICE *** Insert NonStop Clusters D10IPM10 CD into the CD ROM drive, then enter the following command: # pkgadd -d cdrom1 -q D10IPM10 This will take 10 to 20 minutes. 10. After the message: *** IMPORTANT NOTICE *** Edit the "/etc/hosts" file in the root file system: Example /etc/hosts: Include the names and IP addresses for the nodes in the cluster. ## Internet host table 127.0.0.1 localhost ## IP addresses and network node names ## root node 168.85.100.11 system1 ## dependent nodes 168.85.100.12 system2 168.85.100.13 system3 168.85.100.14 system4 168.85.100.15 system5 168.85.100.16 system6 11. For RAID configurations, ensure that the following boot parameter is set correctly in the boot files on both the primary and backup root nodes rootdisk=cCbBtTdD Where: C= controller number of root disk. If you have your root disk on a RAID controller, type the command "/etc/scsi/pdiconfig -l" to determine the controller number for the RAID (ida). B= bus number (0 for RAID) T= target (0 typically) D= Logical disk (0) 12. For RAID systems ONLY, check for separate "/usr", "/var", "/home", and "/home2" file systems. Examine the "/etc/vfstab" file. If "/usr", "/var", "/home", or "/home2" is present in the vfstab file, edit the "/etc/vfstab" file and add "n< nodenum >" in front of cCbBtT for each occurrence of "/usr", "/var", "/home", and "/home2". If the "n" is already present, do not modify the line. (nodenum is the node number of the root node). Edit the "/etc/tnc/fdevtab" file according to the man page for fdevtab(4). 13. Check Version + Determine which version log file to examine. # ls /var/sadm/install/logs There should be a file named D10. + Check the /var/sadm/install/logs/D10 file by searching for the string "Save & Overwrite". (e.g., grep "Save & Overwrite" /var/sadm/install/logs/D10) + You may receive a list containing one or both of the following types of lines: Save & Overwrite: ... < file_x > saved in < file_y >.< version > Save & Overwrite: ... < file_x > moved to < file_y >.< version > NOTE: < version > = 0 when installing over base UnixWare; otherwise < version > = < version of NSC being installed over >. + If there were any "special" changes done to < file_x > prior to this install and the changes are still needed, then merge those changes from < file_y >.< version > back into < file_x >. + Remove EACH < file_y >.< version > file listed. WARNING: All of the < file_y >.< version > files MUST be removed; otherwise the idbuild will fail. # rm < file_y >.< version > + Check for < filename >.0 files in /etc/conf/mtune.d. # ls /etc/conf/mtune.d If there are < filename >.0 files listed, then remove them. # rm < filename >.0 (That is a zero after the dot) 14. Rebuild the kernel using the command: # /etc/conf/bin/idbuild -B Wait for kernel to re-build. If there are NO errors, then continue; otherwise seek assistance. 15. Reboot the root node: # cd / # /etc/shutdown -i6 -g0 -y Wait for shutdown. 16. Boot the root node and run the NonStop Clusters unix kernel (unix_mp). Login as the root user on the root node To Install a Dependent Node, execute the following steps: 1. On the ROOT NODE, format a 1.44 MB, 3.5" HD diskette. Insert a NON-write-protected diskette into the drive. # format /dev/rdsk/f03ht 2. On the ROOT NODE, make the NonStop Clusters Dependent Node Boot diskette. # cd /etc/tnc/nodeup # ./make_floppy diskette1 Press < Enter > when prompted. This will take about 20 minutes. 3. Remove the diskette from the root node and write protect it. 4. Insert the NonStop Clusters Dependent Node Boot diskette into the next node to be installed and boot the node. 5. Select the default settings for each prompt except "nodenum" and "nodename". For the "nodenum", enter the appropriate NUMBER for the node in the cluster that is being installed. For the "nodename", enter the appropriate NAME for the node in the cluster that is being installed. 6. After the node that is being installed boots up to the "Console Login" prompt, remove the diskette from the node. 7. On the node being installed, login as the root user and choose the appropriate terminal setup from the following (Choose ONLY one of the following). + FOR LOCAL CONSOLES # export TERM=at386 + FOR TELNET SESSIONS: On a workstation (i.e., not the cluster): # xhost + # telnet < cluster > Login as root # export DISPLAY=< your_workstation >:0 # export LD_LIBRARY_PATH=/usr/X/lib # /usr/X/bin/xterm Use the xterm window created for the following steps. 8. On the node being installed, enter the following # onnode /etc/tnc/nodeup/lcudcfg < nodenum > (Where nodenum is the number for the node that is being installed.) 9. Select the default settings for each prompt except "node name" and "IP Address". For "node name", enter the appropriate NETWORK NAME for the node that is being installed. For "IP Address", enter the appropriate IP ADDRESS for the node being installed. 10. At the "Hard Disk setup" screen, select "Disk 1". 11. Select "Automatic configuration". 12. Back at the "Hard Disk setup" screen, select "Apply changes to hard disk(s)". 13. For node 2, select "Make this a Backup Failover" 14. Select "Apply changes and exit". 15. When you receive the command prompt again, check the boot file of the node being installed. Edit the "/stand/< nodenum >/boot" file and look for the following: Check the value for rootdisk + For the primary and secondary nodes, rootdisk=cCbBtTdD where C, B, T, and D are appropriate numbers for the rootdisk. + For the other nodes, rootdisk=c0b0t0d0 Check the value for clms + For RAID clusters, clms=1,2 16. If you used telnet sessions to install the dependent node, then exit out of all of them. 17. On the ROOT NODE, enter the following: # sync # nodedown < nodenum > (Where nodenum is the number for the node that is being installed.) 18. The dependent node should reboot and rejoin the cluster. NOTE: If you let the dependent node rejoin the cluster, then "/stand/< nodenum >" may not automatically remount itself. This will be fixed by the last step in this dependent node installation. 19. Repeat the "Install a Dependent Node" instructions (starting at step 4) for each dependent node in the cluster. 20. After all nodes in the cluster have been installed, reboot the entire cluster. # cd / # /etc/shutdown -i6 -y -g0 4. In general, heavy, write intensive System V IPC (Inter Process Communication) should be done on a single node for performance reasons. Because of this, all server processes of a single instance of the INFORMIX-OnLine Dynamic Server or Oracle7 Server should be pinned to a single NonStop Clusters node. This can be done by using the "onnode -p " command, which pins a process and all its children to a single node. 5. One of the defects fixed by this IPM (PTS #38022 - keepalive process ended up with large number of non-contiguous one page segments. Keepalive is leaking memory while continuously calling mmap()/munmap().) resolves a situation where munmap() is failing due to a coding error in keepalive or munmap() is not functioning correctly. The net result is keepalive continually increases the size of its virtual address space possibly causing a system outage. An estimate has been calculated for this defect to consume about 24 MB of storage a day. This is now corrected and keepalive no longer acquires this space. 6. Real time applications are not supported in this release. They may cause the cluster to hang. 7. Multiple, simultaneous nodedown occurrences may cause unexpected behavior in the cluster, including node panics. This should be avoided. In general, take nodes out of the cluster one at a time. 8. During the UnixWare 2.1.2 installation, it is very important to execute the following steps (also described in the NonStop Clusters for SCO UnixWare System Administrator's Guide), which must be done before installing NonStop Clusters. Prior to rebooting your machine after the UnixWare 2.1.2 install, execute the following: o # /etc/conf/bin/idtune -m HDATLIM 0x7FFFFFFF o # /etc/conf/bin/idtune -m SDATLIM 0x7FFFFFFF o # /etc/conf/bin/idtune -m SVMMLIM 0x7FFFFFFF o # /etc/conf/bin/idtune -m HVMMLIM 0x7FFFFFFF o #/etc/conf/bin/idbuild The idbuild will ensure the kernel tunables will take effect upon reboot. After verifying the kernel tunables are correct, reboot the system. 9. Back up the following files. If these files inadvertently disappear or are corrupted and you have not backed them up, a system reinstall may be required to recover them. o /etc/device.tab o /var/sadm/install/contents o /etc/saf/_sactab o /etc/security/ddb/ddb_dsfmap 10. The following INFORMIX OnPerf menu options are not supported in this release: o physical processor metrics o physical processor tool This is a result of differences between UnixWare and NonStop Clusters. The current workaround is to use the sar utility. Also, the OnPerf utility may give the following error: Couldn't open data file /var/adm/metreg.data This error is also a result of the differences between UnixWare and NonStop Clusters. The current workaround is to start OnPerf with standard error messages routed to "/dev/null". No fix is currently planned for the preceding problems. OnPerf will report proper performance information on INFORMIX metrics. To obtain machine specific performance information, use the sar utility. 11. The signal name, SIGNODE, is now defined in "sys/signal.h" as SIGCLUSTER. Please see the signal(5) man page for more information. 12. NFS cluster hard mounts are not supported in this release. The default for NFS mounts is cluster soft and NFS hard. To add interruptibility to the NFS default, insert "-o intr" to the mount command line. If the mounting node goes down, it will be necessary to manually unmount and remount the NFS file system before access can continue. All processes actively using the file system will receive errors upon next access. NFS mounts are only supported on the primary node. To mount an NFS file system, determine the primary node by entering the command "where /". Then, enter the command "onnode -p mount", where is the value returned by the where command. The following defect applies to NFS mounts: Incorrect EBUSY when unmounting NFS file systems (PTS 33780). Symptom: If a process was accessing the NFS file system when the node failed, unmount will return EBUSY. Workaround: Use the "-o force" option with the unmount command to unmount the file system before remounting it. 13. To ensure proper operation of your cluster, make sure each node in the cluster has at least as much swap space as there is physical memory. 14. Two tunable parameters that are associated with system panic dump generation are still applicable for the D10IPM10 release. SYSDUMP_SELECTIVE: Flag value that determines if the system will dump all pages (0), or selectively dump only pages mapped in the kernel (1). The default value is 1: to generate a selective dump, occupying much less space, rarely losing information. SYSDUMP_POLL: Flag value that determines if the system will dump relying upon the disk interrupt (0), or poll for I/O completion (1). The default value is 1: to poll with disk interrupt disabled, in case the interrupt is shared with other devices which might stop the dump. 15. This release includes the Cluster Virtual IP Address (CVIP) concept. In the IP world, an IP address corresponds not to a host, but to an interface card attached to a particular subnet. That is, a host (or NonStop Cluster) with several interfaces to a network also has several IP addresses. An association 5-tuple contains embedded IP addresses. The Internet protocol suite does not have the ability to rename a connection or endpoint once its name is established; the same identifying 5-tuple {protocol, laddr, lport, faddr, fport} must be used for the life of the connection or endpoint. Therefore, in order to recover from netserver node failure and allow another netserver to support an existing connection, use an IP address in associations not tied to any particular physical network interface. That IP address is called the cluster virtual IP address (CVIP). Connection associations using the CVIP stay the same regardless of which physical IP address (PIP) is used to exchange packets with the external host. Particular network server nodes have PIPs. The CVIP must be on the same subnet as the PIPs that will service the CVIP. The CVIP should be registered in the Internet Domain Name Service (DNS) database for hostname-to-address translation and reverse translation. The PIPs may optionally be registered with the DNS for network management purposes. Note the use of a PIP address to contact the cluster does not guarantee that resulting processes will be spawned on the netserver node on which the PIP address is configured. Configuring CVIP and the requirements needed: CVIP requires that an address is assigned to the cluster that is on the same subnet as all of the physical interfaces that may be used to support traffic to the CVIP. The CVIP and all supporting interfaces must be on the same network with the same subnetmask. Currently, there must only be one CVIP per cluster. When configuring CVIP, the name of the cluster must be set to the CVIP name; otherwise RPC, and other applications, might not work after failover. The nodename parameter in the "/stand//boot" file must be the name for the physical interface for each node. 1. Add the name used to access the CVIP and its address to the "/etc/hosts" file. 2. Add a CVIP line to the "/etc/confnet.d/inet/interface" file. The line in the interface file should contain the following: cvip:0::/dev/tnccvip:netmask {your netmask}:add_cvip: where is the name for the CVIP that is in /etc/hosts file {your netmask} is the netmask for your network. It can be copied from the netmask value of the physical network, which should already have an entry in the interface file. 3. Set the name of the cluster to the CVIP name. setuname -n 4. A cluster reboot is required for the CVIP to become active. NOTE: If you change any network information with the "menu" program, a cluster reboot is required to restore correct network function and enable the new configuration. 16. Software patches, or PTFs, from SCO for UnixWare are not supported with NonStop Clusters. NonStop Clusters must be installed over UnixWare 2.1.2. No SCO PTFs are to be installed before or after NonStop Clusters is installed. 17. The Site Preparation and Hardware Installation manual describes a Japanese PDU that may not be available at D10IPM10. A North American low-voltage PDU will be substituted until the Japanese PDU is available for shipment. The following known problems have workarounds: 1. Boot loop resolution needed on base SCO UnixWare installation (PTS 35084). Symptom: System attempts to rebuild and reboot continuously after system installation. Workaround: The following method can be used to resolve this situation should it occur. The following message will repeatedly appear on the console screen immediately after a system installation or reconfiguration: The UNIX Operating System kernel will now be rebuilt to incorporate recent configuration changes. Press the ENTER key when ready or the ESC key to stop. You should press the ESC key. At the prompt, enter the command "# resmgr". Examine the output to find the following entries: 19 cpqsc 0 5 4 10 7000 70ff 40101000 401010ff - - 4 0x1000000F 0 - 1 21 ida 2 5 4 5 8000 80ff c8000 cbfff - - 4 0xE11AE10 - - 2 22 ide 2 5 3 14 1f0 1ff 10000 10001 -1 - 0 - - - - NOTE: The key numbers (the first field) may not match the preceding example. The third field (unit number) is conflicting. The unit numbers are incorrect and must be manually changed to their correct values as shown below: KEY UNIT UNIT#. cpqsc 0 ... ida 2 ... ide 1... The first field is the "KEY". Using the appropriate KEY, modify the appropriate unit number entry to match the preceding values. In this example, key is 22 and ide is changed from unit 2 to unit 1. resmgr -k 22 -p "UNIT" -v "1" Commit the configuration changes by entering the command "/etc/conf/bin/idconfupdate -f". Reboot the system and allow it to rebuild the UNIX operating environment. 2. Unable to copy wild card files from a mounted file system on a CM-1300(PTS 34195). Symptom: When trying to copy several files using the wild card symbol, the files do not copy correctly or it seems to "hang" and not copy anything. This only occurs when using ksh while running on a file system NFS mounted from a CM-1300. It works correctly on all other platforms. Workaround: Do not attempt file locking (for example, use csh) when accessing files NFS mounted from a CM-1300. 3. keepalive has no interface for "shutdown" scripts. (PTS 36470). Symptom: Currently, keepalive's only interface to tell a process to do a controlled shutdown is via a SIGTERM. This does not work well with processes that should be shut down another way,(e.g., lpsched). Workaround: Unregister the process with keepalive. Then use whatever shutdown procedure is appropriate for the process. 4. /etc/rc2.d/S51nis file missing domain name when created by "pkgadd nis" from SCO CDROM (PTS 36525). Symptom: Domain name will not have a valid entry. Workaround: In the file "/etc/rc2.d/S51nis", change the line: domain to: domain where is the internet domain name for the machine. 5. Installing NIS after NonStop Clusters overwrites "/etc/init.d/nis". This results in the NIS daemons no longer being registered with keepalive (PTS 36767). Symptom: NonStop Clusters does not install NIS by default. When pkgadd is used to install NIS from the SCO 2.1.2 CD, it replaces the Tandem version of the file "/etc/init.d/nis". As a result, all the changes that are required for spawndaemon are lost and NIS does not failover. Workaround: 1. Copy the file "/etc/init.d/nis" to "/tmp/nis" before installing NIS. 2. Install NIS from the SCO CD on each cluster. Do not configure NIS when it is installed, as that will be overwritten by step 3. 3. Copy the file saved in "/tmp/nis" to "/etc/init.d/nis" to each cluster. 4. Run "ypinit" with the correct option to configure each cluster. 6. Token deadlock in forced unmount (PTS 37167). Symptom: Under rare conditions, if you do a "umount -o force" on a running file system, it can hang indefinitely. Workaround: Do not forcibly unmount running file systems. 7. force unmount of NFS FS with yet-to-be-noticed downed server loops forever in cfs_hide_cnodes (PTS 37238). Symptom: When trying to unmount a file system using "umount -o force", a downed server may hang. Workaround: Do not force an unmount with a downed NFS server. 8. Closes done from process exit on downed hardmounts are non-interruptible (PTS 38167). Symptom: Process with files on a downed hardmount cannot be killed. Workaround: Remount or forcibly unmount the file system. 9. Unshare of downed file system hangs system shutdown (PTS 38283). Symptom: Shutdown hangs on a system that has a downed, shared, hardmount file system. Workaround: Use "umount -o force" to bring the file system down. 10. NonStop Clusters OS: unexpected defaults while running: console keyboard and mouse lockup on UnixWare 2.1.1 upgrade. HISPANET. (PTS 38293). Symptom: On some platforms, after adding the UnixWare 2.1.1 update package (update211), the local keyboard and mouse will not respond after rebooting the new kernel. This defect corrects itself after installing the UnixWare 2.1.2 package. Workaround: The user can telnet into the system, continue to pkgadd the update212 package, and reboot the system. Alternatively, if networking is not setup or otherwise unavailable, it may be possible to boot a 2.1.0 kernel (saved as unix.old during the 2.1.1 upgrade) and perform the 2.1.2 pkgadd using the old kernel. 11. new_rem_swapfile() can deadlock with tnc_anon_resv_adjust() over registrar locks (PTS 38390). Symptom: If the cluster has been running under a significant memory load, the root node appears "hung". Workaround: Ensure there is always sufficient swap space on each node to satisfy the peak per-node memory load. 12. UX:init: ERROR: Command is respawning too rapidly. Check for possible errors. id: co "/usr/lib/saf/ttymon -g -v -p "Console Login: " -d /dev/console.3 -l console" (PTS 38407). Symptom: After adding the D10IPM10 package, the following message may appear on the primary node's console prior to rebooting the node: UX:init: ERROR: Command is respawning too rapidly. Check for possible errors. id: kp "/sbin/keepalive -t 10 -P 1 -S 2" Workaround: If this message should occur, remove any core files in the / (root) directory which may have been created during the time the preceding messages was displayed. The message itself can be ignored. 13. cnet Adapter Check causes the whole cluster to hang (PTS 38666). Symptom: When installing Oracle, mounting the Oracle distribution CD on one node and the Oracle install scripts are executed from a different node, the following error may occur: WARNING: cnet: unit 0: Adapter Check interrupt 100002! Interface 0 in slot 16 has failed. Attempting to restart. When this happens, the cluster may hang. Workaround: Ensure the install scripts are on the same node as the mounted Oracle CD. For example, if the Oracle CD is in the CD ROM drive on node 2, run the Oracle install script on node 2 using the following command: onnode -p 2 ./orainst /c This may happen with other applications that access the CDROM drive heavily. New Features The following new features are supported with the NonStop Clusters D10IPM10 release: * Database recovery (dbms_guard) now supports Oracle recovery on node failure in addition to support for INFORMIX. * ProLiant 1600R/400 support. * 18 GB disk support. * Async I/O support. New ServerNet Information 1. PL1600R/400 and the ServerNet PCI Adapter SHIP 1.5 Rev E Due to incompatibilities between the PCI bus interface chip on the ServerNet PCI Adapter SHIP 1.5 Rev C and the PCI bus on the PL1600R/400 servers, a new version of the ServerNet PCI Adapter (SPA) has been released. The new SPA, SHIP 1.5 Rev E, must be used with the PL1600R/400 servers. This new card is also backwardly compatible with other qualified Integrity® XC platforms. A SHIP 1.5 Rev C SPA will lock-up the PL1600R/400 server with a minimum amount of ServerNet System Area Network (SAN) traffic. This is a hard lock-up that requires a power-cycle of the server to clear. During system boot and initialization, a new message has been added to the ServerNet PCI Adapter Driver that prints an identification string to report the version of the SPA detected during initialization. The text string will appear as follows: [SNET] Identified ServerNet PCI Adapter: SHIP 1.5 Rev X where "X" will be either "E" or "C". 2. ServerNet SAN Link Exception Detection and Handling Link exception detection and handling has been added to the ServerNet PCI Adapter Driver (SPAD). This means that the ServerNet SAN connections on the ServerNet PCI Adapter (SPA) are actively monitored for link exceptions at the physical layer. This will detect cable and port failures, such as an unattached cable or failed ports. As a result, warning messages concerning ServerNet SAN connections will now be seen that have not been reported previously. For example: WARNING: [SNET] (node 2 (0xf0080)) Multiple link exceptions detected on path Y. WARNING: [SNET] (node 2 (0xf0080)) Disabling path Y until the condition is corrected. WARNING: [SNET] (node 2 (0xf0080)) Verify that path Y is cabled properly. This is most apparent for 2-node directly connected clusters (i.e., a 2-node cluster that does not use a ServerNet SAN Switch). Until both nodes have booted and are joined as a cluster, the first node booted will report link exceptions for both the X and Y paths. This is because the SPA in the other node is not responding to link level "heartbeat" type messages. It will begin responding to the messages once the node boots and initializes its SPA, which occurs during the boot of the NonStop Cluster. Watch for a console message indicating that the link exception condition has been corrected and the path has been re-enabled. For example: NOTICE: [SNET] Link exception condition on path Y has been resolved. NOTICE: [SNET] Re-enabling path Y. For 2-node clusters, if you do not see this message within a few minutes of both nodes being booted and joined as a cluster, check the ServerNet SAN cabling for the node reporting the exceptions. Occasionally during boot of a ServerNet SAN Switch connected cluster, a node will report link exceptions during boot. Once the node is booted and running, the link exception condition is reported to be resolved. This is due to noise on the card/cable/port during system boot. It is harmless and can be ignored. As long as the messages indicating the condition has been corrected are displayed, the connections are fine. If you do not see the condition corrected and re-enabling the path messages within a few minutes, check the ServerNet SAN cabling for the node reporting the exceptions. 3. ServerNet SAN Off-Line Diagnostics ServerNet SAN off-line diagnostics gives the administrator a way to verify that the _local_ ServerNet PCI Adapter (SPA) is functioning properly. ServerNet SAN off-line diagnostics can be run at boot time prior to the node joining the cluster. ServerNet SAN off-line diagnostics consists of a series of tests that exercise various components of the SPA to validate that it can function properly. The off-line diagnostics do not validate proper cabling or non-local ServerNet SAN end-points. o When to run ServerNet SAN off-line diagnostics. ServerNet SAN off-line diagnostics should be run when a node does not join the cluster properly and it appears as if it is a local problem (i.e., other nodes can join, but this one cannot). The diagnostics should also be run whenever the local SPA is suspected of being the source of a node failure. o Overview of tests. The following menu appears when ServerNet SAN off-line diagnostics are invoked. The tests available are: D = Display SAIL registers. P = PCI Bridge (PLX) register test. S = SAIL register test. I = Interrupt test. L = Internal loopback test. Choose a test (or q to quit): D: "Display SAIL registers" dumps the current value of the SAIL registers. P: "PCI Bridge (PLX) register test" verifies that the PCI interface chip on the SPA is functioning properly. S: "SAIL register test" verifies that the SAIL ASIC on the SPA is functioning properly. I: "Interrupt test" verifies that the interrupt mechanisms of the ASIC and SPA are functioning properly. L: "Internal loopback test" verifies that the block transfer engine (BTE) is functioning properly. This option has the following sub-menu: SAIL internal loopback test: 1 = Read own, write other. 2 = Write other, read own. 3 = Quit. The first two options allow the user to select which BTE to write or read from. Either option will validate the BTEs. Both options are provided for completeness. Option 3 will exit the sub-menu and return to the main menu. All of the tests with the exception of "Display SAIL registers" run in a loop until the operator hits a key. Allow each test to run at least ten passes (if not more) before stopping the test. If any of the tests fail, the SPA should be replaced. NOTE: On 2-node direct connect clusters, link exception detection and reporting can be affected by the diagnostics. That is, the other node in the cluster (if booted and running NonStop Clusters) may report not being able to communicate with the node under test during portions of the off-line diagnostic tests and then report that the link exception condition is resolved or vice versa. This is due to the fact that link exception detection takes place at the physical level. If the SPA on the other end of the cable becomes active (even for local loopback tests), it is seen by the other SPA. This will not happen on a ServerNet SAN Switch connected cluster. In addition, on a 2-node direct connect cluster, it is possible for noise from the SPA under test to occasionally be reported as a bad packet on the other node. This will not happen on a ServerNet SAN Switch connected cluster. o How to start ServerNet SAN off-line diagnostics. Off-line diagnostics are started by setting a boot argument. In order to set boot arguments, you need to be at the boot prompt. To accomplish this, the automatic boot sequence has to be interrupted. The boot sequence can be interrupted by pressing the space bar immediately after the "SCO Unixware" logo "splash screen" is displayed. The OS boot sequence watches for keyboard activity during the display of the splash screen and if it detects any, will drop to the boot prompt. The boot prompt is the "[boot]#" symbol. At the boot prompt, enter the following and press the key: snetdiags=yes Then type "go" and press the key again. This will cause the node to start booting. Instead of booting and joining the cluster as usual, the node will stop at the ServerNet SAN off-line diagnostics menu as shown below. The tests available are: D = Display SAIL registers. P = PCI Bridge (PLX) register test. S = SAIL register test. I = Interrupt test. L = Internal loopback test. Choose a test (or q to quit): Selecting a menu option will execute the specified off-line diagnostic test. o Exiting the tests. Press the "q" key to quit the ServerNet SAN off-line diagnostics. The SPA will be reset and the boot sequence will proceed as normal. If the SPA is functioning properly, the node should join the cluster. If it does not (but all the above tests passed without failures), then the problem is either in the software, cabling, or the ServerNet SAN Switch. However, if all the tests passed, the local SPA is valid and not the source of the problem. 4. ServerNet SAN Fault Reporting Every message from the ServerNet SAN PCI Adapter Driver (SPAD) will be reported via cmn_err(D3). cmn_err(D3) takes a severity indication and a text string as input. The severity indicators are: "CE_CONT", "CE_NOTE", "CE_WARN", and "CE_PANIC". These indicators are converted to text strings in the message displayed to the console and logged to the system log. Respectively, they are converted to: "", "NOTICE", "WARNING", and "PANIC". o CE_CONT will be used to continue a previous message or to display an informative message not associated with a fault. o CE_NOTE will be used for messages that indicate the recovery from a fault situation or for messages that do not necessarily require action, but may interest the system administrator. o CE_WARN will be used to report fault conditions of which the system administrator needs to be aware. CE_WARN messages indicate a serious problem that may warrant action on behalf of the system administrator. o CE_PANIC will be used to report a catastrophic failure in which the SPAD is in such a state that it cannot continue to operate. Every message from the SPAD will be prepended with the following string: "[SNET]". The "[SNET]" string will be between the severity indicator and the actual fault string. For example: WARNING: [SNET] The SAIL on the SHIP card has frozen. Following is a list of the possible messages that the SPAD can report. Along with each message is a brief description of what the message indicates. The messages are classified based on their cmn_err(D3) severity. Examples of the messages are shown. That is, a message such as the one that reports the ServerNet SAN ID for the node can be: [SNET] SHIP SNet ID: 0xF0040 [SNET] SHIP SNet ID: 0xF0080 [SNET] SHIP SNet ID: 0xF00C0 [SNET] SHIP SNet ID: 0xF0100 [SNET] SHIP SNet ID: 0xF0140 [SNET] SHIP SNet ID: 0xF0180 for nodes 1 through 6 respectively. However, for this document, only one example is shown. Memory addresses, status words, etc., will be shown as 0x00000000. Any place that "path" is mentioned, valid values are "X", "Y", 0 or 1. Any place that SPA revisions is mentioned, valid values are "C" or "E". 1. CE_CONT - [SNET] Message [SNET] SHIP SNet ID: 0xF0040 This is an informative message that is displayed during initialization. The node's ServerNet SAN ID is printed. [SNET] AVT entry 0x00000000 @ 0x00000000: I/O Address 0x00000000, Type = Data [SNET] AVT entry 0x00000000 @ 0x00000000: I/O Address = 0x00000000, Type = Interrupt The above two lines are two separate error cases that will be followed by additional information from the AVT entry associated with the problem. Usually this "dump" of information will be accompanied by some other error message indicating the problem. Information from the AVT entry specified by one of the above two lines will be printed out to help diagnose the source of the problem. [SNET] Dump of Exception Packet @ 0x00000000: The above line will be followed by additional information from the packet in question. Usually this packet "dump" will be accompanied by some other error message indicating the problem. The exception packet is dumped to help diagnose what caused the problem. This message will usually be seen in conjunction time-outs on ServerNet SAN requests. 2. CE_NOTE - NOTICE: [SNET] Message [SNET] Switching to path Y This message indicates that the ServerNet SAN driver cannot communicate over the current path, so it is attempting to switch to the other path to determine if it can communicate over it. [SNET] Successfully recovered from frozen SAIL. This error message indicates that the SAIL ASIC stopped responding due to an internal self-check problem (usually from being held off of the PCI bus too long). This message indicates that the condition was detected and that the ASIC is being reset and processing will resume. [SNET] Link exception condition on path X has been resolved. [SNET] Re-enabling path X. Indicates that a link exception condition on a path has been resolved and that link exception detection and processing has been re-enabled for that path. [SNET] Identified ServerNet PCI Adapter: SHIP 1.5 Rev E This informative message is displayed during system initialization to report the revision of the ServerNet SAN PCI Adapter that was identified by the ServerNet SAN PCI Adapter Driver (SPAD). [SNET] Barrier failed on path:X snetID:0xF0040 curpath:X [SNET] Barrier succeeded on path:Y snetID:0xF0080 curpath:X These messages are displayed when a new node attempts to join a cluster. The message indicates whether the new node was able to communicate with the target node over the given path (X/Y). If a path is cabled, then a success message is expected. If the path is not cabled (e.g., X side cabled and Y side not cabled), then a failure message is expected for that path. 3. CE_WARN - WARNING: [SNET] Message [SNET] ServerNet: NAK on ServerNet Barrier request (status=0x00000000) This message indicates that a NAK was received as the result of a barrier request that did not successfully complete. The path is down for some reason (e.g., not connected, incorrectly cabled, etc). [SNET] 00000000: Timeout on ServerNet request: cpu0, path=X, This message indicates that a ServerNet SAN driver operation trying to communicate with another node in the cluster has timed-out. This could indicate that the path is down or that there is a problem with the driver and/or hardware. If a path is down, it will be reported as such. [SNET] 00000000: Timeout on both paths This message indicates that the ServerNet SAN driver has attempted to communicate over both paths and neither path is functional. [SNET] F0040: (ServerNet) Halt received on X port [SNET] F0040: (ServerNet) Halt received on Y port This message indicates that a ServerNet SAN HALT command was received on the specified port (X/Y). This stops the transmitter/receiver on that port. [SNET] F0080: (ServerNet) rcvd spurious packet acknowledge, src=0x00000000 This message indicates that an unexpected packet acknowledgment arrived. Usually this message can be linked with a ServerNet SAN time-out message. The acknowledgment from the packet that was timed-out arrives. Since the packet was timed-out, an acknowledgment is not expected. [SNET] F00C0: ServerNet transceiver error: status=0x00000000 This message indicates that a ServerNet SAN error has occurred. The error is not an expected or normal error; therefore, it is classified and reported as a transceiver error. This usually indicates a hardware problem. [SNET] The SAIL on the SHIP card has frozen (IIF_SELF_CHECK). This error message indicates that the SAIL ASIC stopped responding. This indicates an unrecoverable condition. This message will be followed by a dump of the SAIL ASIC registers and a PANIC. [SNET] The SAIL on the SHIP card has frozen. This error message indicates that the SAIL ASIC stopped responding due to an internal self-check problem (usually from being held off of the PCI bus too long). This self-check condition is recoverable. The ASIC will be reset and processing will resume. [SNET] bte status immediately after SC status = 00000000 sc_to = 00000000 int_mode = 1 This error message, if seen, will follow a SAIL ASIC self-check. It indicates that after the SAIL ASIC self-check recovery procedure completed, the BTE status register contained an incorrect value and has to be re-initialized. [SNET] F0040: (ServerNet) rcvd invalid interrupt packet , src=0x00000000 [SNET] F0080: (ServerNet) rcvd packet w/ invalid address, addr=0x00000000, src=0x00000000 [SNET] F00C0: (ServerNet) rcvd packet w/ invalid path, path=1,src=0x00000000 [SNET] F0100: (ServerNet) rcvd packet w/ invalid source, addr=0x00000000, src=0x00000000 [SNET] F0140: (ServerNet) rcvd packet w/ invalid permissions, addr=0x00000000, src=0x00000000 The above messages indicate that a ServerNet SAN interrupt packet was received and the packet itself or information in the packet is invalid. This can be an indication of problems with the ServerNet SAN driver and/or hardware. [SNET] (F0080) Multiple link exceptions detected on path Y. [SNET] (F0080) Disabling path Y until the condition is corrected. [SNET] (F0080) Verify that path Y is cabled properly. This series of messages indicate that a burst of link exceptions has been detected on a ServerNet SAN path. Please refer to the link exception processing section for more information. [SNET] Path Y still disabled due to link exceptions. [SNET] Verify that path Y is cabled properly. [SNET] No more warnings regarding path Y being disabled will be printed until the condition is corrected. This series of messages indicates that a continuous burst of link exceptions was detected. As a result, link exception reporting is being turned off for this path until the condition is corrected. Please refer to the link exception processing section for more information. [SNET] (F0040) exception queue error. [SNET] (F0180) transmitter write response overflow [SNET] (F0140) receiver read request overflow. [SNET] (F00C0) interrupt queue overrun. These message indicate error conditions detected during interrupt processing. Queue overruns and transmitter/receiver overflows indicate a potential loss of a response due to buffer space exhaustion. [SNET] (F0080) low condition on external LSERR input detected. [SNET] (F0080) illegal burst on the i960 bus detected. [SNET] (F0100) invalid register access on the i960 bus detected. [SNET] (F0040) error pulse on the external PCHK input detected. [SNET] (F0180) data parity error detected. [SNET] (F00C0) address parity error detected. [SNET] (F0140) multiple address or data parity errors have occurred. These messages indicate error conditions detected during interrupt processing. These errors are all related to access errors in the SAIL. [SNET] bte_error: BTE error stack overflow. [SNET] bte_error: Invalid BTE command (no pdev). [SNET] cf_formatname: called with invalid pdev struct [SNET] cf_formatname: called with null name pointer [SNET] F0040: (ServerNet) interrupt queue full, src=0x00000000 [SNET] F0040: Unmapped interrupt received These messages indicate BTE errors. 4. CE_PANIC - PANIC: [SNET] Message [SNET] Unsupported MITE-based ServerNet PCI Adapter found. [SNET] ship_init: Unsupported MITE-based ServerNet PCI Adapter detected. These messages indicate that during the discovery and initialization of the ServerNet SAN PCI Adapter, a MITE-based adapter was found. MITE-based adapters are not supported by NonStop Clusters. [SNET] ServerNet PCI Adapter not found using vendor ID 10E4, device IDs 0000 0003. This message indicates that during the discovery and initialization of the ServerNet SAN PCI Adapter, no ServerNet SAN PCI Adapter was found. [SNET] ship_init: Unsupported revision of the SAIL ASIC detected. CIN=0x00000000 This message indicates that a ServerNet SAN PCI Adapter was found, but the SAIL ASIC on it is not a recognized revision. Currently the driver recognizes revisions A and B of the SAIL ASIC; however, Rev B is the only revision supported by NonStop Clusters. [SNET] ship_init: Unknown revision of the SAIL ASIC detected. CIN=0x00000000 The SAIL ASIC detected is not revision A or B. [SNET] ship_init: Unknown revision of the ServerNet PCI Adapter found. Rev ID=0x0 This message indicates that during the discovery and initialization of the ServerNet SAN PCI Adapter (SPA), an unknown revision of the SPA was detected. The SPA driver currently recognizes the MITE-based SPA, and revisions A, B, C, and E of the SAIL-based SHIP 1.5 SPA. NonStop Clusters only supports the SHIP 1.5 Rev C and SHIP 1.5 Rev E SPAs. [SNET] avt_init: unable to allocate virtual mem for AVT This message indicates a shortage of memory. [SNET] bte_memreserve: insufficient physically contiguous memory [SNET] intr_memreserve: insufficient physically contiguous memory [SNET] initSail: Insufficient memory for Interrupt State Block dump. The above error messages would be displayed during initialization indicating a lack of needed memory. Check memory utilization and distribution in the kernel tunables. [SNET] The PLX_ABORT_ACTIVE bit is set (shipintr()) This message indicates that the PLX aborted a PCI operation. This means that a previous SHIP read or write operation has been aborted. Unfortunately, which operation is not known, so it cannot be retried. Ignoring the aborted operation could lead to unreported data loss or corruption. [SNET] SAIL frozen; see SAIL state printed above. This message will be preceded by a message indicating that the SAIL is frozen and a dump of the SAIL registers. This message indicates that the SAIL ASIC has stopped responding in such a way that it is not recoverable. [SNET] SAIL frozen; unrecoverable snet self check - source = 00000000 This message indicates that the SAIL ASIC stopped responding in such a way that it is not recoverable. This error message will be preceded by a dump of the SAIL registers. [SNET] SHIP_INTCAUSE_QIP7 already set This interrupt line is being used to help flush the SAIL register values to memory. If it is already set when it is about to be used, a serious problem has occurred elsewhere. [SNET] (F0080) internal SAIL logic error detected. This message indicates that an internal problem with the SAIL ASIC was detected. This implies that the hardware has failed. [SNET] intr_init: spawn_daemon_thread failed. This message indicates that an attempt to spawn a kernel daemon thread failed. Since this daemon is vital to the proper functionality of the driver, if it fails to start, we abort the initialization sequence. [SNET] avt_define_q: invalid interrupt queue size: 0000 [SNET] Invalid parameters on ServerNet Request [SNET] Invalid status on ServerNet Request: 0x00000000 [SNET] bte_error: Invalid BTE command descriptor. [SNET] intr_init: qintr_map failed [SNET] ioint: invalid ioaddr 0x00000000 [SNET] ship_init: physmap failed. [SNET] PHYS_TO_VIRT: invalid address 0x00000000 [SNET] allocSNdev: out of sndev table space [SNET] snetConfig: bad cmd 5 [SNET] snetOpen: invalid mode 0x00000000 These messages indicate bte errors. Procedures for RAID Configuration and Recovery 1. This procedure describes failover disk configuration guidelines for NonStop Clusters. Disk configuration includes adding disks, array configuration, and file system configuration. Adding disks Disks can be added to the disk cabinets at any time, before or after system installation. Use the following guidelines when adding disks. Disks must be added in pairs; place the same number of disks in both disk cabinets. Each cabinet should have disks in corresponding slots. Fill the disk cabinets from left to right. The first disk is disk 0. Array Configuration Array configuration must be accomplished before a new disk can be used for slices and file systems; and before the first system installation can be preformed. Array configuration cannot be performed on a running system. It is accomplished using stand-alone software booted on the primary root node. First, the cluster must be shut down and the root node must be booted from the SMARSTART CD or the system partition (F10 prompt during boot) in order to run the Compaq Array Configuration Utility. More information about disk arrays and disk array configuration can be found in the Compaq SMART-2DH Array Controller Reference Guide. This publication is included in the delivery of every SMART-2DH Array controller. There is a variety of logically equivalent configuration options that can be used. However, adherence to these guidelines will reduce the likelihood of problems due to unusual or untested configurations. RAID configuration has two steps: creation of disk arrays and creation of logical disks. Disk Arrays A disk array is comprised of two associated physical disks. One disk in each array should come from port 1, and the other disk should come from port 2. Disks are numbered from left to right facing the front of the cabinet, starting with disk 0. Each array should contain two disks with the same disk number. If you have added new disks to a previously installed system, create a new array with each pair of new disks. Supported Array configuration: Array A: SCSI port 1 disk 0 SCSI port 2 disk 0 Array B: SCSI port 1 disk 1 SCSI port 2 disk 1 etc. up to Array G: SCSI port 1 disk 6 SCSI port 2 disk 6 Create Logical Disks For each logical disk, select RAID 1 mirroring. If this is a new system, create a 2 GB logical disk in array A (logical disk 0). This will be the system disk. Create another logical disk in array A that contains the remainder of the disk space in the array. Create one logical disk in each of the remaining disk arrays. Each logical disk should contain all the space in the array. If you have added new disks to a previously installed system, simply create one logical disk in each of the new disk arrays. Disk Device Pathnames Disk device pathnames for the logical disks are created automatically when the system is booted or when the installation is complete. Each logical disk appears to be a separate disk device. Pathnames are of the following form: /dev/[r]dsk/nNcCbBtTdDsS where: N - Node number. When the failover disks are attached to node 1, the value of N is 1. If node 2 is booted as root or a failover has occurred, the value of N is 2. C - Controller number. The controller number of the RAID controller is usually 2. You can determine the controller number of the RAID controller using the command "/etc/scsi/pdiconfig -l". B - Bus number. The bus number of RAID disk pathnames is currently always 0. T - Target This is the logical disk number. Each logical disk is treated as a SCSI target. D - Logical Unit Number This is currently always 0 for RAID disk pathnames. S - Slice. By default, 16 slices are created for each logical disk regardless of whether the slice has actually been created. File System Configuration System installation creates the root file system on logical disk 0. It is usually not necessary to make any further modifications to the system disk. For each additional logical disk, it is necessary to create slices. Use the following guidelines. A maximum of 13 file systems can be created on each logical disk. This limit is enforced by the disksetup utility. The number of file systems to create on a each logical disk depends on the size of the logical disk and the size of the file systems. The size of a file system must not exceed 2 GB. The diskadd command does not currently support node encoded disk pathnames. Avoid using diskadd. Use the fdisk command to create a UNIX partition on each logical disk. Use the disksetup command to create new file systems. This command will prompt you for all the information necessary. It is necessary to add entries to "/etc/tnc/fdevtab" for the failover file systems. This file contains commented out entries that can be used as examples. Also, refer to the man page. 2. Procedure for Creating a Backup of the root file system on a RAID system. This procedure explains how to create a backup of the disk that contains the root file system located on the RAID disk and is mirrored using the standard RAID setup for NonStop Clusters. The backup will only be necessary in the event of a catastrophic failure where both mirrors are lost due to root file system corruption or hardware damage. This procedure will also work only for RAID 1 configurations. For this procedure, you will need a spare disk of the same size as the disk containing the root file system that is being mirrored in the RAID cabinets. This procedure should be performed after NonStop Clusters has been installed and all disks and file systems have been configured. Any configuration change that alters files in the root file system made after the backup has been created will be lost. If configuration changes are made after the backup creation, the modified files should be backed up to tape so that they can be restored after the base root file system is restored. As a general rule, keep a written record of all disk arrays and logical disks that have been created on the RAID controller. The array and logical disk configuration can be seen using the SmartStart disk array configuration utility. In particular, the sizes of each of the logical disks in the arrays need to be recorded. In the event of RAID controller hardware failure and replacement, the correct logical disk configuration can be recreated on the new controllers using the SmartStart array configuration utility. Creating the backup: 1. After the cluster is completely configured, shut down all dependent nodes in the cluster, leaving only the root node up. Quiesce the system as much as possible by not running any user applications and only running the processes that are normally started by the system. The object of this step is to minimize I/O to the root file system prior to the removal of one of the disks containing the root file system. 2. Remove the disk at SCSI ID 0 from one of the two RAID cabinets. Keep a written record of which port (cabinet) the disk was removed from so that it can be replaced in the same position if recovery is necessary. Keep this disk in a remote secure place. It will contain your backup. 3. Place your spare disk in the empty slot. The system will automatically begin to remirror the replaced drive. While the remirroring is being done, the top LED on the drive will be continuously blinking. When the remirror is complete, the same LED will remain on steady, like it is on the same drive in the other cabinet. The time it takes to remirror will depend on the disk size and the priority of the recovery that was specified at array configuration time in SmartStart. The priority is usually set to low as a default. For a 4.3 GB disk, the remirror time is generally about 15 minutes. 4. While the disk is being remirrored, you can reboot the dependent nodes in the cluster and restart all applications. Normal operation of the cluster can resume as soon as the replacement disk is inserted and the mirroring starts. 3. Procedure for Using a Backup of the root file system on RAID to Restore System Operation. This procedure is used in the event that both RAID mirrors were damaged due to root file system corruption or total hardware failure of both drives. It should not be used in the event of a single disk failure. If a single disk fails, a new disk should be inserted while the system is running and the remirroring will be done automatically. The first step in this procedure assumes that you have lost both RAID controllers as well as both RAID cabinets. When a RAID controller is replaced, the array and logical disk configurations need to be recreated using the SmartStart array configuration utility. If the controllers have not been replaced, the first step is unnecessary and you can begin the disk recovery with step 2. If the controllers were damaged and were replaced, you will need two new disks of the same size as the root backup disk. If just the disks and not the controllers were damaged, you will only need one new disk. In both cases, you will need the backup disk from the vault that was previously created. This procedure must be followed exactly as written, or the recovery will not be successful and the system will need to be reinstalled. This procedure also assumes that SmartStart is installed on the internal disk and that the internal disk also contains the boot file system for NonStop Clusters. In other words, this procedure assumes that the internal disk for the root node has already been recovered using the recovery procedure defined for the internal disk. Using the backup disk from the vault: 1. If your controllers were damaged and replaced in addition to your disks, you will need to use the SmartStart array configuration utility to reconfigure your arrays and logical disks. Prior to using the SmartStart utility, make sure that you place two previously unused (new) disks in the first slot of each of the RAID cabinets. Do NOT place the backup disk (vault disk) in at this time, as the SmartStart array configuration utility will destroy data on the disks when the arrays and the logical disks are configured. You must set up the arrays and logical disks to be identical to those that were defined on the controllers that were destroyed. Also, you will need to run the SmartStart array configuration utility on both nodes that contain the new RAID controllers before failover will work. The logical disk configuration is stored on the controllers, and if the controllers are replaced, they need to be reinitialized with SmartStart. See the SmartStart documentation for help in using the SmartStart array configuration utility. 2. If the F1 prompt has been disabled previously with SmartStart advanced options, it needs to be reenabled at this time using SmartStart. 3. Bring down all nodes in the cluster and power off all nodes. Also power off both RAID cabinets. 4. Place the backup disk (vault disk) into the same RAID port (cabinet) where it was when the backup was created. Place a previously unused disk in the first slot of the other RAID cabinet. 5. Power on both RAID cabinets, and then power on the root node of the cluster. 6. During POST, the initialization of the RAID controller will display a message that states that the disks have been replaced. It will then stop and wait for you to press F1 to continue or press F10 for system utilities. When it is stopped at this prompt, remove the disk that does NOT contain the backup from the proper cabinet. Then hit the F1 key. The root node should boot at this time and should use the remaining backup root disk for the root file system. 7. Once NonStop Clusters is booted, place the disk that was removed in the previous step back into the RAID cabinet. The system should automatically begin to remirror the root disk. 8. At this time you can reboot all dependent nodes in the cluster and resume operations, or continue with other recovery operations as necessary. 9. When mirroring is complete, make a new backup disk using the "Procedure for Creating a Backup of the root file system on a RAID system". Problems Corrected 1. LCCrq05473: onnode 2 audit set fails in AGETUSR because "no such process" exists. (PTS 31774, No TPR, Severity 2) 2. During shutdown, tnc_async_ref_t reference count went negative. (PTS 34518, No TPR, Severity 2) 3. Panic from dnet_nfs_forward during virtual socket test suite execution. (PTS 34647, No TPR, Severity 2) 4. ics_internal_traffic_listen() terminates ICS_FETCHCDATA_REQUEST with ELCU_OK when there is an error. (PTS 34808, No TPR, Severity 3) 5. Semaphore grab causes spawndaemon to exec daemons too slowly. (PTS 35373, No TPR, Severity 2) 6. A panic seen while running truss on keepalive. (PTS 35700, No TPR, Severity 3) 7. Trussing init while booting multiple dependent nodes causes incorrect behavior in nfsbiod. (PTS 35991, No TPR, Severity 3) 8. keepalive should be started on transitioning from run-level S or 1 to run-level 2. (PTS 36464, No TPR, Severity 2) 9. drv_getparm(D3D) for PPID on cross node opens returns wrong or unexpected process PID. (PTS 36511, No TPR, Severity 2) 10. Executing "date -a 1.0" causes system date to be reset to a different day after the cluster is rebooted. (PTS 36729, No TPR, Severity 2) 11. When node 2 is booted as primary node, the system time is set incorrectly. (PTS 36740, No TPR, Severity 2) 12. arp replies of published entries advertise incorrect hardware address. (PTS 36796, No TPR, Severity 2) 13. utmpx PIDs are truncated when you log in to a dependent node console after cluster boot. (PTS 36938, No TPR, Severity 2) 14. After failover, init s from /dev/console.3 causes /dev/console.3 to produce unpredictable behavior. (PTS 36942, No TPR, Severity 2) 15. If root file system failover fails because vxfs cannot remount it, the cluster hangs. (PTS 37048, No TPR, Severity 2) 16. keepalive's named pipe has read/write permissions allowed for all users. (PTS 37106, No TPR, Severity 3) 17. Warning messages in tncglue cause slow remote net msg processing. (PTS 37116, No TPR, Severity 2) 18. Ran out of memory while trying to mount the root file system for the first time. (PTS 37163, No TPR, Severity 2) 19. PANIC: trap in cspec_svrfrlock() caused by PTS35478. (PTS 37169, No TPR, Severity 2) 20. nodedown for node 5 fails to complete on node 6, resulting in node 6 hang on reboot. (PTS 37248, No TPR, Severity 3) 21. Panic releasing prnodes. (PTS 37252, No TPR, Severity 2) 22. Need to purge "register_wait" from daemon configuration files and user documentation. (PTS 37549, No TPR, Severity 1) 23. Keepalive primary should be watched by init and keepalive shadow should be deleted from the architecture. (PTS 37561, No TPR, Severity 2) 24. Panic in bcopy() because of a bad length field in an LWP audit buffer. Lucent Technologies. (PTS 37566, No TPR, Severity 3) 25. TNC dswap uses too many anon_table entries, can't add more swap if full. (PTS 37938, No TPR, Severity 2) 26. Panic with cms_log: error = 5 caused by the cmssvr remount failed. (PTS 37978, No TPR, Severity 2) 27. Keepalive process ended up with large number of non-contiguous 1 page segments. Keepalive is leaking memory while continuously calling mmap()/munmap(). (PTS 38022, No TPR, Severity 3) 28. Init failover can hang running /sbin/initprivs. (PTS 38044, No TPR, Severity 2) 29. icfstok_giveback() must account for down hardmounts. (PTS 38164, No TPR, Severity 2) 30. EXNFS client: PERF: cfs_attrcache_inval() flushes and invalidates data page. (PTS 38180, No TPR, Severity 2) 31. Bad pointer in gethostent when using libnsl with libthread. SEMA. (PTS 38333, No TPR, Severity 2) 32. Critical memory pad not maintained. (PTS 38377, No TPR, Severity 2) 33. onall command should also be in /sbin for use during failover recovery. (PTS 38389, No TPR, Severity 2) 34. chard option not recognised in /etc/vfstab. (PTS 38432, No TPR, Severity 2) 35. Fix for PTS 38222 broke locking in TCP. (PTS 38440, No TPR, Severity 3) 36. HANFS server test caused assertion (int)resv >= 0, file svrcfs.c, line 545. (PTS 38450, No TPR, Severity 2) 37. Root filesystem failover failed resulting in a panic. (PTS 38462, No TPR, Severity 2) 38. cflush() holds off cnode creation. (PTS 38532, No TPR, Severity 2) 39. Writing to a MANDLOCK file in CHARD mount will hang process. (PTS 38537, No TPR, Severity 3) 40. TCP performance degradation for small packets. (PTS 38659, No TPR, Severity 3) 41. Rexec/migrate can panic during failover when parent's vproc isn't found. (PTS 38674, No TPR, Severity 2) 42. PERF: starting ss7 stack on node 1 followed by node 3 thrashes the cluster. (PTS 38751, No TPR, Severity 3) Known Problems Remaining 1. Stale RPC delivered after a nodedown. (PTS 36062, No TPR, Severity 2) 2. TCP hangs on inp_rwlck while running VSTS test. (PTS 36081, No TPR, Severity 2) Currently under investigation. 3. pdimkdev makes devices with controller number 65534. (PTS 36203, No TPR, Severity 2) 4. Spurious "too many nodes" reboot from licensing support code. (PTS 36204, No TPR, Severity 3) 5. Races in forced unmount need to be examined. (PTS 36343, No TPR, Severity 2) Some issues are present where forced unmount of a file system can panic. Most likely, NFS would have problems. 6. select() supposed to time out in 5 seconds but does not return for over 1 minute. (PTS 36401, No TPR, Severity 2) 7. Stale cshandle after failover. (PTS 36457, No TPR, Severity 2) 8. /tmp, /var/tmp, /usr, and /var should all be hard mounted separate file systems. (PTS 36659, No TPR, Severity 2) 9. System will not boot when 1 RAID cabinet is powered off. (PTS 36770, No TPR, Severity 3) It is best to have both RAID cabinets powered on when rebooting the system. 10. pkgadd of tcpip_nis.so fails. i.VUF script needs work? (PTS 36904, No TPR, Severity 2) This defect occurs during the pkgadd of the NonStop Clusters package. This occurs when tcpip_nis.so is included in the NonStop Clusters prototype file and the NIS package is not installed. Currently, tcpip_nis.so is not included in the NonStop Clusters package. 11. Panic in tcp_respond(). (PTS 37117, No TPR, Severity 2) Currently under investigation. 12. cli_ipcname_getid() has failover race; CMS has a similar problem. (PTS 37157, No TPR, Severity 2) Currently under investigation. 13. ics_msg_ack_done() kernel panic after a node goes down. (PTS 37170, No TPR, Severity 3) 14. memfs may report out of space due to the way dswap usage is calculated. (PTS 37246, No TPR, Severity 2) There must be at least as much total swap in the cluster as there is total RAM in the cluster. Otherwise, it is possible that shared memory segments cannot be created and memfs file systems may report out-of-space. 15. Substantial memory leak found in ics/llics after performing nodedowns. (PTS 37262, No TPR, Severity 2) 16. PANIC in cfs_blkrsrv_svrnodedown(). (PTS 37292, No TPR, Severity 2) Currently under investigation. 17. Verify that if idbuild fails on shutdown that idcpunix does not propagate kernel. (PTS 37408, No TPR, Severity 2) 18. panic in prreadobjdir after failover. (PTS 37528, No TPR, Severity 2) Under certain rare conditions during failover, the node taking over root/clms/init responsibility panics with the following: 00: PANIC: kernel-mode address fault Crash dump analysis will indicate the faulting thread is in the function prreadobjectdir(). Currently, there is no known workaround. 19. initprivs/filepriv are not distributed. (PTS 37533, No TPR, Severity 2) 20. interface direct routes are handled incorrectly. (PTS 37534, No TPR, Severity 2) 21. NFS client needs to present itself as CVIP for all client messages. (PTS 37642, No TPR, Severity 2) 22. VPROC nodedown hung trying to remove pgrp from nonexistent session. (PTS 37711, No TPR, Severity 3) This is a very rare hang seen when a node goes down. This is unfortunately fatal to the cluster in that nodedown processing wedges and never completes. Currently, the only recovery is to reboot the cluster. 23. Hang during nodedown processing after failover - semconv() spinning on lock. (PTS 37721, No TPR, Severity 3) 24. /etc/dfs/sharetab seems to get corrupted, causes mount access denial. (PTS 37785, No TPR, Severity 2) This defect is reporting that the "/etc/dfs/sharetab" file has become corrupted. It would be a good practice for the administrator of the cluster to keep a copy of the "/etc/dfs/sharetab" file in case of future corruption. 25. Kernel must reboot automatically for unattended operation. (PTS 37795, No TPR, Severity 2) 26. XDM error log fills /tmp. (PTS 37797, No TPR, Severity 2) 27. PANIC: kernel-mode address fault on user address 0x100. (PTS 37918, No TPR, Severity 3) 28. DEADLOCK: CMS locking must be fixed to prevent nodedown deadlock. (PTS 37930, No TPR, Severity 2) Possible deadlocks exist in the CMS code during nodedown. There is nothing to distinguish these from other possible deadlocks in nodedown. 29. UX:nfs mount: ERROR: get_addr: (host cluster20, program 100003, version 2) is not found on any transport. (PTS 37966, No TPR, Severity 2) Currently under investigation. 30. Need to fix sorting of routes in rtrequest(). (PTS 37985, No TPR, Severity 2) 31. NFS code is cfs_rb_dolocks() creates a bogus cnode. (PTS 37990, No TPR, Severity 2) Currently under investigation. 32. Diskadd command is broken. (PTS 37993, No TPR, Severity 2) Instead of using the command diskadd, use the commands fdisk and disksetup. These commands do exactly the same function as diskadd. Both fdisk and disksetup are explained in the man pages. 33. EXNFS: A dirty page from CFS as NFS client causes cfs_data_downgrade loop. (PTS 38002, No TPR, Severity 2) This defect will be seen as "hung" processes. Currently, wait for the NFS server to come back up. The processes will no longer be "hung". Even if NFS is mounted with "intr", it is rare that the processes will not be interruptible. 34. Failover clusters need CVIP interfaces. (PTS 38016, No TPR, Severity 2) 35. HANFS: Backup locks not rebuilt when backup lock node joins cluster. (PTS 38025, No TPR, Severity 2) The user will not be able to detect this defect. This occurs if node 1 fails after node 2 has failed and rejoined the cluster, and if file locks are being backed up. 36. PANIC: Range tokens do not handle 2 GB or higher values. (PTS 38177, No TPR, Severity 2) This occurs because no single file or disk partition can be larger that 2 GB. Do not use the "dd" command to copy a disk partition larger than 2 GB. 37. NonStop Clusters OS: abending while running: open(2) causes cluster crash. (PTS 38247, TPR # 980626-1633-16640, Severity 2) 38. EXNFS client: ASSERTION CAUGHT: i < MAXIDS, file: cfs_hm.c, line: 1167. (PTS 38249, No TPR, Severity 3) To prevent this defect, do not mount client NFS mounts "chard". 39. ASSERTION CAUGHT: !swappedin, file: rwsleep.c, line: 1257. (PTS 38253, No TPR, Severity 2) This defect is corrected in the source base, but is not part of the D10 IPM10 release. 40. EX NFS client: Node discovery of NFS mount with down NFS server can loop. (PTS 38359, No TPR, Severity 2) The user will see processes get "hung" non-interruptible. This might even affect non-NFS processes. Currently, get the NFS server available to the cluster again. 41. EX NFS server: sharing of chard mount on non-lockd node will ASSERT on remote lockf. (PTS 38435, No TPR, Severity 2) 42. EX NFS server: lockd can be hung on unavailable chard cluster mount. (PTS 38436, No TPR, Severity 3) 43. EX NFS client/server: statd restart is broken. (PTS 38465, No TPR, Severity 2) 44. Double failure of root server and root backup cache node deadlocks root failover. (PTS 38485, No TPR, Severity 2) ---------------------------------------------------------------------------- End of Softdoc