HP-UX: Asynchronous i/o

PURPOSE
-------

The purpose of this document is to discuss the implementation of
asynchronous i/o (aio) on HP-UX, specifically to enable aio for the Oracle
RDBMS Server.


SCOPE & APPLICATION
-------------------

This note is intended for DBAs and Support Personnel.


1.0 Introduction
================
On HP-UX, when the Oracle datafiles reside on a filesystem, then the DBWR
process(es) make synchronous writes to these datafiles. This means that each
write must return with a 'succesful completion' before the next write is
issued. This serial operation can lead to a i/o bottleneck. There are two ways
to counteract this:
  a. configure multiple DBWR processes
  b. use asynchronous i/o

Before deciding on one of these two options, it should be noted that on HP-UX,
aio is *only* possible on a raw device. Put in another way, aio *cannot* be used
on a filesystem.

Multiple DBWRs can be used on a filesystem.

It is not recommended to use both multiple DBWRs and aio.

2.0 Configuring asynchronous i/o
================================
aio requires configuration of both Oracle and HP-UX

2.1 HP-UX Configuration
-----------------------
a. create the /dev/async character device
   % /sbin/mknod /dev/async c 101 0x0
   % chown oracle:dba /dev/async
   % chmod 660 /dev/async

   If the device does not exist, then ENODEV (Err #19) is returned when the device
   is accessed:
   Async driver not configured  : errno=19

   If the permissions are incorrect, then EACCES (Err #13) is returned:
   Async driver not configured  : errno=13

b. configure the async driver in the kernel
   Using SAM
     -> Kernel Configuration
        -> Drivers
           -> the driver is called 'asyncdsk'
   Generate new kernel
   Reboot

c. set max_async_ports
   Using SAM
     -> Kernel Configuration
        -> Configurable Parameters
           -> max_async_ports

   max_async_ports limits the maximum number of processes that can concurrently
   use /dev/async. Set this parameter to the sum of
     'processes' from init.ora + number of background processes

   The background processes started at instance startup will open /dev/async
   twice.

   If max_async_ports is reached, subsequent processes will use synchronous i/o.

 
d. set aio_max_ops
   Using SAM
     -> Kernel Configuration
        -> Configurable Parameters
           -> aio_max_ops

   aio_max_ops limits the maximum number of asynchronous i/o operations that
   can be queued at any time. Set this parameter to the default value (2048),
   and monitor over time (use glance).

e. if HP-UX patch PHKL_22126 (or one that supersedes it) is installed, then
   see Section 3.0

2.2 Oracle Configuration
------------------------
The following init.ora parameters are required.

a. pre 7.3.0
   no init.ora parameters required
b. 7.3.X
   use_async_io = TRUE
c. 8.X
   disk_asynch_io = TRUE    

3.0 Implications of HP-UX Patch PHKL_22126
==========================================

This patch is called:
PHKL_22126: s700_800 11.00 VM,async,hyperfabric

The HP-UX patch description says:
The effective user ID of the process calling async driver,
typically called by a process for database applications such
as Oracle, must be a superuser or the user must
be a member of a group that has the MLOCK privilege.

In essence, aio will not work for Oracle, unless the dba group has the MLOCK
privilege.

To check if a group has the MLOCK privilege, execute:
% /usr/bin/getprivgrp <group-name>

If the dba group is not listed, then set it:
% /usr/sbin/setprivgrp <group-name> MLOCK

Note, the next reboot will clear this privilege. To automate this at startup,
create /etc/privgroup with the entry
dba MLOCK

This can be tested with
% /usr/sbin/setprivgrp -f /etc/privgroup
See 'man 1m setprivgrp' for more information.

If the MLOCK privilege is not granted to the dba group, then instance startup
will exhibit the following:

a. prior to 8.1.7, each background and shadow process will dump a trace file
   with the following:

   Ioctl ASYNC_CONFIG error, errno = 1

   errno 1 is EPERM (Not super-user).

   Such trace files may be found in
      $ORACLE_HOME/rdbms/log
      background_dump_dest
      user_dump_dest

   Additionally, a tusc of instance startup will show the following for
   each background process:
   [16044] open("/dev/async", O_RDWR, 01760) ................ = 14
   ...
   [16044] ioctl(14, 0x80106101, 0x7f7f51b0) ................ ERR#1 EPERM
 
   Similar output will be given for a tusc of a shadow process.

   The instance does start, but i/o is synchronous

b. in 8.1.7, the instance will not start, and will error with:
   SVRMGR> startup
   ORA-00445: background process "PMON" did not start after 120 seconds

   See Note:133007.1 for the alert on this.

c. in 9.0.1, the behaviour is similar to pre-8.1.7, in that trace files with
   the following are dumped:

   Ioctl ASYNC_CONFIG error, errno = 1

   Additionally, the following is also seen:

   Ioctl ASYNC_ADDSEG error, errno = 1

   The instance does start, but i/o is synchronous.

4.0 Checking that Asynchronous i/o is being used
================================================
Async i/o is being used if both the following are true:
  a. /dev/async is open by DBWR
  b. the relevant init.ora partameter is set (see Section 2.2)
  c. the datafiles are on raw devices

To check if /dev/async is open by DBWR, do one of:
    i. % fuser /dev/async
   ii. % lsof -p <dbwr pid>
         lsof can be downloaded from ftp://vic.cc.purdue.edu/pub/tools/unix/lsof
  iii. use glance/gpm to check files opened by DBWR

NB - Oracle server processes (background and foreground) will attempt to open
     /dev/async if the async driver is enabled in the HP-UX kernel, regardless
     of init.ora settings.

If (a), (b)and (c) are true, and i/o is still perceived to be synchronous, attach
to DBWR with tusc, and check that write calls to /dev/async are not returning an
error. The file descriptor for /dev/async will be needed, which can be retrieved
using lsof.

5.0 The fs_async kernel parameter
=================================
The kernel parameter fs_async can be set to allow asynchronous writes to file
systems. However, write calls can return without the data being physically
written to disk (the write sits in the UNIX buffer cache). The data in question
is file-system metadata such as free space lists, blocks and inodes.
A system crash would potentially lose this data, and leave the filesystem in an
inconsistent state, causing database corruption.

Oracle still opens files with the O_DSYNC flag (see 'tusc' snipet
below), which insists that writes are physically written:
open("/oracle/datafiles/system01.dbf", O_RDWR|O_LARGEFILE|O_DSYNC, 0) = 19

In summary, fs_async is ignored for datafiles(due to open() with O_DSYNC).
However, filesystem metadata may be lost, potentially causing datafile
corruption.

Oracle does not recommend setting fs_async to '1'.

Settings:
  fs_async=0 Do not use async writes to file systems
  fs_async=1 Do async writes to file systems