Patch Name: PHKL_15883 Patch Description: s700 10.01 mmap/munmap, no ACLs on JFS cumulative patch Creation Date: 98/07/07 Post Date: 98/07/13 Hardware Platforms - OS Releases: s700: 10.01 Products: N/A Filesets: OS-Core.CORE-KRN Automatic Reboot?: Yes Status: General Release Critical: Yes PHKL_15883: HANG PHKL_15283: PANIC PHKL_7242: OTHER Performance bad due to excess protection id faults Path Name: /hp-ux_patches/s700/10.X/PHKL_15883 Symptoms: PHKL_15883: Processes may hang doing mmap(2) due to a lock-order violation; the offending process' stack trace will look something like this: _swtch+0x138 _mp_b_sema_sleep+0xe0 vnode_vas_lock+0x78 freereg+0x194 smmap_common+0x5d0 smmap+0x38 syscall+0x1a4 $syscallrtn+0x0 PHKL_15283: This defect causes a panic in a multi processor system when two processes are doing mmap/munmap on portions of the same file using a sliding window. The user will see the system panic with "panic: rmfree: overlap" message. The stack trace will be as shown below: panic+0x10 rmfree+0x268 quaddealloc34+0x30 hdl_detach+0x108 detachreg+0x3c do_munmap+0x190 do_munmap+0x84 foreach_pregion+0xec munmap+0x74 syscall+0x1a4 A different panic could occur due to an unrelated race condition when mmap/munmap is called. This second panic is a result of data page fault. The stack trace in this case will be as shown below: panic+0x3C report_trap_or_int_and_panic+0x8C trap+0xC18 $RDB_trap_patch+0x20 smmap+0x8F0 syscall+0x1A4 PHKL_7242: MP T500 running 10.01 or 10.10 with Informix database exhibits poor performance and high levels of protection id traps after reboot. Problem exhibits itself after stopping/restarting application until system rebooted. /usr was a JFS VxFS file system. Defect Description: PHKL_15883: Under certain conditions, the system could violate internal lock-order rules while mmap(2)ing; the fix was to give up and re-acquire the region lock to avoid this. PHKL_15283: Both the problems (panics) discussed earlier occur in a multiprocessor system when two processes are doing map/munmap on portions of the same file using a sliding window. Panic-1 -------- The panic is caused by a race condition in hdl_mmf_attach() in hdl_policy.c in the machine directory. The race condition is in case-3 of "overlapping" pregions. In this case, the new process' pregion starts within and extends past the existing process pregion. In the original implementation, we were releasing the region lock to call mapvnode(). At that time, the new pregion is not yet placed on the region's pregion-list and hence opens a window for a race condition. With the fix, the pregion is placed on the region's pregion-list before releasing the region lock, thereby eliminating the race condition. Panic-2 -------- The second panic is caused by a race condition in the error path of smmap() in vm_map.c in the sys directory. In a code segment following the label "bad", in the case where vnode is associated with the region, we release the region lock to be able to call dectext(), we copy the file descriptor from the vas, acquire the file-table lock and then check that the file descriptor is not zero before proceeding further. In the meantime, the file descriptor could have been closed and hence dereferencing it would cause a data page fault. To avoid this race condition, we need to postpone calling dectext(), which is what the fix does. PHKL_7242: We only allow a public mapping if we are mapping shared (and thus read-only for now) and if the file's modes meet our tests (no ACL, and mode at least r-xr-xr-x). In one case, /usr (where the shared libraries reside) on their system turns out to be a JFS file system. JFS does not know about ACL and does not manipulate the va_acl bit in 'struct vattr', when VOP_GETATTR() is called. HFS on the other hand will set va_acl to 1 if that file has ACL and 0, if not. Since, smmap() does not initialize the structure (on the stack) and calls VOP_GETATTR() and then looks at the va_acl bit, it can either be a 1 or a 0, randomly. It is not correct for VM subsystem to assume all file systems support ACL and will correct set/reset the bit in the vattr structure as there could be third party file system (eg: JFS) that do not support it. When the libc.1 is first mmapp'ed, the smmap() code finds that va_acl is set (wrongly), it decides not to do public mapping (the region does not get the RHDL_MMAP_PUBLIC set) and so the protection id is set to the second quadrant space id of that process, instead of the public protection id of 0 Since, the informix (oninit) processes have a lot of shared memory segments, each of which has an unique protection id, and constantly accesses the shared memory region and execute library code, the protection id (pid3 & pid4) is getting thrashed in the pid registers. The instruction protection fault code is getting thrashed in the pid registers. The instruction protection fault code (IMEM_PROT) does not make use of the cache (kept per processor) and always uses the trap (long) path to update the protection registers. Hence, you see about 30-40% of the time in trap code. The following fix will ensure that the shared library text segments always get the public protection id. This will cut down the number of protection faults seen in informix. SR: 1653235176 4701319335 4701391730 Patch Files: /usr/conf/lib/libhp-ux.a(hdl_policy.o) /usr/conf/lib/libhp-ux.a(vm_mmap.o) what(1) Output: /usr/conf/lib/libhp-ux.a(hdl_policy.o): hdl_policy.c $Date: 98/05/15 16:23:46 $ $Revision: 1 .9.72.16 $ PATCH_10.01 (PHKL_15283) /usr/conf/lib/libhp-ux.a(vm_mmap.o): vm_mmap.c $Date: 98/07/07 16:28:35 $ $Revision : 1.12.72.36 $ PATCH_10.01 (PHKL_15883) cksum(1) Output: 1732646012 11128 /usr/conf/lib/libhp-ux.a(hdl_policy.o) 288777593 19456 /usr/conf/lib/libhp-ux.a(vm_mmap.o) Patch Conflicts: None Patch Dependencies: None Hardware Dependencies: None Other Dependencies: None Supersedes: PHKL_7242 PHKL_15283 Equivalent Patches: PHKL_15884: s800: 10.01 PHKL_15885: s700: 10.10 PHKL_15886: s800: 10.10 Patch Package Size: 90 KBytes Installation Instructions: Please review all instructions and the Hewlett-Packard SupportLine User Guide or your Hewlett-Packard support terms and conditions for precautions, scope of license, restrictions, and, limitation of liability and warranties, before installing this patch. ------------------------------------------------------------ 1. Back up your system before installing a patch. 2. Login as root. 3. Copy the patch to the /tmp directory. 4. Move to the /tmp directory and unshar the patch: cd /tmp sh PHKL_15883 5a. For a standalone system, run swinstall to install the patch: swinstall -x autoreboot=true -x match_target=true \ -s /tmp/PHKL_15883.depot 5b. For a homogeneous NFS Diskless cluster run swcluster on the server to install the patch on the server and the clients: swcluster -i -b This will invoke swcluster in the interactive mode and force all clients to be shut down. WARNING: All cluster clients must be shut down prior to the patch installation. Installing the patch while the clients are booted is unsupported and can lead to serious problems. The swcluster command will invoke an swinstall session in which you must specify: alternate root path - default is /export/shared_root/OS_700 source depot path - /tmp/PHKL_15883.depot To complete the installation, select the patch by choosing "Actions -> Match What Target Has" and then "Actions -> Install" from the Menubar. 5c. For a heterogeneous NFS Diskless cluster: - run swinstall on the server as in step 5a to install the patch on the cluster server. - run swcluster on the server as in step 5b to install the patch on the cluster clients. By default swinstall will archive the original software in /var/adm/sw/patch/PHKL_15883. If you do not wish to retain a copy of the original software, you can create an empty file named /var/adm/sw/patch/PATCH_NOSAVE. Warning: If this file exists when a patch is installed, the patch cannot be deinstalled. Please be careful when using this feature. It is recommended that you move the PHKL_15883.text file to /var/adm/sw/patch for future reference. To put this patch on a magnetic tape and install from the tape drive, use the command: dd if=/tmp/PHKL_15883.depot of=/dev/rmt/0m bs=2k Special Installation Instructions: None