Kernel Parameter Setting

Unix kernel parameters can be the most mysterious part of a Unix operating system.  There are sizing parameters that set the sizes of tables or the amount of a resource that will be available for use.  The settings of these parameters are usually dictated by the nature of the application workload.  There are characterization parameters that set environmental values, such as the time zone and the daylight savings time policy switch.  These parameters are usually set incorrectly, unless your system lives in Colorado (MST).  There is also a timing parameter that sets the duration of a CPU time slice.  And finally, there are behavioral parameters that dictate the way in which the kernel performs its functions.

It is very unusual for a systems’ administrator to set any parameters other than the sizing parameters and the timezone parameter.  Most system tuning is done by adjusting the value of such parameters.  Occasionally, there is a need to alter the value of a behavioral parameter, such as create_fastlinks.  In such cases, the functionality provided or denied is of greater consequence than the performance impact of such a change.

Below, I will explain the nature of the most commonly used kernel parameters and the effects of changing their value in a typical environment.  Generic explanations of all kernel parameters can be found in the sam on-line help.  Bear in mind that all kernel parameters should be changed only after careful consideration and with a specific goal in mind.  The “shotgun” approach to kernel parameter modification is highly discouraged.  If it is not plainly obvious what a given kernel parameter’s value should be, I suggest that you research the matter until it is.

Ø       Sizing parameters - The following list of sizing parameters have two characteristics in common.  First, all of these parameters control or limit the amount of a certain system resource that is available.  Second, the specification of these parameters has a direct impact on memory consumption.  The higher the value of a parameter, the greater the memory consumption for the associated resource.

ü       bufpages, nbuf - These two parameters set the size of the fixed buffer cache.  The nbuf parameter sets the number of buffer headers and the bufpages parameter sets the number of 4KB pages used for buffer cache.  When both are set to 0, the kernel sets the buffer cache size to 10% of physical memory, unless the dbc_max_pct and dbc_min_pct parameters are set.  When only one parameter is set to a positive, non-zero value, that parameter sets the buffer cache size.  When both are set to a positive, non-zero value, the kernel negotiates a settlement between the two values at boot time.  The bufpages parameter takes precedence over nbuf; and both parameters take precedence over the dbc_max_pct and dbc_min_pct parameters when all are set.  The nbuf and bufpages parameters should both be set to 0.  Use of the dbc_max_pct and dbc_min_pct parameters to set the buffer cache size is preferred.

ü       dbc_max_pct, dbc_min_pct - These two parameters set the maximum and minimum percent of physical memory to be used for the dynamic buffer cache.  Their default values are 50 and 5.  This means that 5% of physical memory will be allocated for the buffer cache at boot time.  As files and all blocked devices are accessed, the buffer cache can grow to as much as 50% of all memory.  The dynamic buffer cache algorithm dictates that memory that is allocated dynamically for the buffer cache will be returned to the system when programs require the space.  I have observed that this does not occur as documented.  When these two parameters are different, it appears that the dbc_max_pct percent of memory is allocated permanently.  I suggest that these two parameters be set to 10, in order to fix the size of the buffer cache at 10% of physical memory.  On systems where the workload is file access intensive, these values should be increased.  However, in the case of Oracle and Sybase, this does not necessarily apply.  Many Oracle instances consist of table spaces that reside in raw logical volumes.  Raw IO does not use the buffer cache.  A large buffer cache is not necessary even in an Oracle instance that uses the file system since Oracle uses it’s own memory area, called SGA, as a cache.  Sybase exhibits the same behavior.

ü       maxdsiz, maxssiz, maxtsiz - When a program is dispatched to run, a virtual address space is allocated for each of three parts of a program: text or code, data and stack, and shared memory.  a fourth quadrant is allocated for the kernel.  Each of these spaces can be, theoretically, 1 GB in size.  These three kernel parameters set the actual limits on the data, shared and text segment size for each program.  On the surface, it may appear that increasing the value of these parameters to their theoretical limit would solve a multitude of problems.  However, one or two instances of a poorly behaved program could consume all available memory, and so produce a memory bottleneck.  In the case of an Oracle database server, there is no such problem.  In fact, the maxssiz parameter must be set to at least the size of the largest SGA that Oracle intends to allocate.  Other programs may require a large data space.  In such cases, the maxdsiz parameter should be increased.  The maxtsiz parameter rarely requires modification.  For the most part, only the application developer, or the reference manual, can say for sure what these values should be.  As a safety mechanism, these parameters should be set no higher than the applications actually demand when they are running correctly.

ü       maxfiles, maxfiles_lim - The maxfiles parameter sets the maximum number of concurrently open files any program can have.  That limit can be overridden, by using a system call, to raise the allowable limit up to maxfiles_lim.  This override can only be done by the program that is trying to open the files.  This parameter should be set according to the needs of the application running on the system.  A Sybase server system may need this parameter’s value increased if it’s database resides in the file system, since the dataserver process would attempt to open all the database files.

ü       maxuprc - The maxuprc parameter sets the maximum number of processes that a user can start concurrently.  The default value of 64 is usually acceptable for most applications.  However, when an application has many modules, more than 64, and they are all started by a user that has been configured specifically for the application, the value of this parameter would need to be increased.  The value of this parameter should not be increased without good reason since this parameter acts as a throttle, or control point, for run away processes.

ü       maxusers - This parameter is actually a macro.  That means that the default value of other parameters is calculated from this value.  Specifically, the nclist, nfile, ninode and nproc default values are based on the value of maxusers.  Further, the default value of ncallout is based on the value of nproc.  I suggest that the value of maxusers not be changed, and that the values of its five dependent parameters be set explicitly.  This will exert a greater degree of control over your operating system than by using the maxusers parameter.

ü       nfile, nflocks, ninode - These three parameters set the maximum number of concurrently open files, concurrent locks and in core inodes system wide.  These parameters are stored as integers in the kernel, and at boot time, the kernel builds the file table, the lock table and the inode table to the sizes specified by the values of nfile, nflocks and ninode respectively.  These table entries are relatively small, so over sizing these tables has a small impact on memory consumption.

ü       nproc - The nproc parameter sets the maximum number of concurrent processes system wide.  This value is stored as an integer in the kernel and, at boot time, the kernel builds the process table to the size specified by the value of nproc.  The value of this parameter should be set only as high as needed.  The size of a process table entry is greater than 1 KB.  An excessively high value for nproc will cause an excessively large process table to be built in memory, thereby reducing the available memory for processes and data.  Tuning this parameter can be done over a period of time by periodically monitoring the size of the process table and it’s occupancy.  Use the command: “sar -v 1”.  If you notice that the process table is consistently under utilized, less than 75%, even during peak usage times, you could consider lowering it’s value.  If the value of nproc is too low, some of the application processes will not run.  This is not a performance problem.  This is a functionality problem.

ü       msgmap, msgmax, msgmnb, msgmni, msgseg, msgssz, msgtql - These parameters control the resources associated with the message queue IPC mechanism.  The default values of these parameters are acceptable for  the basic operating system and tools, such as GlancePlus and the C++ compiler.  Other applications, including CA Unicenter, Oracle, Sybase, Holos, SAS, etc., have their own requirements for message resources.  Each application running on a system must specify its message resource needs.  If the installation guide for a given application or tool does not explicitly specify values for message resource parameters, then it probably does not require them.  When there are multiple applications running on a system, there are two things to keep in mind.  First, for parameters that specify the maximum quantity of a given resource, accumulate (add up) those requirements across all applications and set that parameter appropriately.  Second, for parameters that specify a maximum value or size, find the largest value across all applications and set that parameter appropriately.

ü       semmap, semmni, semmns, semaem, semmnu, semume, semvmx - These parameters control the resources associated with the semaphore IPC mechanism.  The explanation above regarding message queue related parameters applies to these parameters as well.

ü       shmmax, shmmni, shmseg - These parameters control the resources associated with the shared memory IPC mechanism. The explanation above regarding message queue related parameters applies to these parameters as well.

Ø       Characterization parameters - The system clock in Unix is kept as an integer count of seconds since the Epoch, January 1, 1970 at 12:00 AM.  The timezone and dst parameters simply control the manner in which the time is displayed.

ü       timezone - The timezone parameter should be set to the number of minutes west of Coordinated Universal Time, or Greenwich Mean Time, for the location of the system.  The correct value for Eastern Standard Time is 300.  The default value is 420, for Mountain Standard Time.

ü       dst - The dst parameter should be set to 1, indicating that this timezone does use the Daylight Savings Time policy.

Ø       Behavioral parameters:

ü       create_fastlinks - When a symbolic link is created in the file system, a file is created that contains nothing more than the name of the linked file.  Access to the linked file requires an open, read and close of the link, followed by an open of the linked file.  When the create_fastlinks parameter is set to 1, the file system will store the name of the linked file in the inode of the link.  Thus access to symbolically linked files increases dramatically.  In systems whose applications depend heavily on symbolic links, this parameter should be set to 1.  The default value is zero.  So far, I have found no down side risk associated with setting this value to 1 in all systems.

ü       default_disk_ir - Most disks today have some amount of cache embedded in them.  Sometimes, this cache is only 1 KB in size.  When a system writes to a disk device, the data is first transferred to the cache, then the disk drive transfers the data from it’s own cache to the disk medium.  When this second transfer is complete, the disk issues an IO completion to the system, and life goes on.  When Immediate Reporting has been enabled for a certain disk, the IO completion is issued after the first transfer.  While this behavior allows an IO intensive application to run very fast, there is a risk to the data integrity in the event of a system or power failure.  The system thinks the IO has completed, but the disk may not have actually written any or all of the data before failing.  The default_disk_ir parameter sets the default value or state of this feature.  The default value of this parameter is 0.  Don’t change it.  There is no performance benefit to be gained by enabling Immediate Reporting for EMC arrays.  The EMC generates a true IO completion when a write transfer is complete to its cache.

ü       fs_async - The fs_async parameter specifies how writes to a file system are performed: synchronously or asynchronously.  Synchronous writes, the default, cause the files system to write data in a very specify sequence.  This sequential operation helps preserve the integrity of the data.  Asynchronous writes allow the file system to post the updates at it’s own pace and order.  Such behavior allows for much faster file system IO, with an increased risk to the file system and data integrity.  In the event of a system or power failure, data and file system structures may be out of sync.  The risk to the data is not worth the performance gain.

Ø       Timing parameter

ü       timeslice - This parameter sets the maximum amount of time that the CPU will spend executing a process before it does a context switch to another process.  I have not yet encountered a situation in which altering the timeslice was a good idea.  I strongly suggest that this parameter never be altered from its default value of 10.