When Memory_target Is Set and Swap Size Is Not Big (Doc ID 2356025.1)

Kkjcre1p: unable to spawn jobq slave process Happened Intermittently When Memory_target Is Set and Swap Size Is Not Big (Doc ID 2356025.1)

To Bottom

---

In this Document

Symptoms

Cause

Solution

References

---

 

APPLIES TO:

Oracle Database - Enterprise Edition - Version 11.2.0.1 to 12.2.0.1 [Release 11.2 to 12.2]
Information in this document applies to any platform.

SYMPTOMS

Database is running, and the processes which previously connected to database are working fine; however new sessions cannot be created with below symptom:

SQL*Plus: Release 11.2.0.4.0 Production on Wed Sep 28 02:24:24 2016
Copyright (c) 1982, 2013, Oracle. All rights reserved.
Connected to an idle instance.                          <============ sqlplus shows that it connects to an idle instance
SQL> show parameter cpu
ORA-01034: ORACLE not available

Alert log shows that "Process J000 died, see its trace file" and "kkjcre1p: unable to spawn jobq slave process" at bad time frame.

The problem happened intermittently. Memory_target is being used.

Strace output shows that a mmap() OS system call failed, the mmap() is trying to "map"/"claim" it will use memory as big as SGA_MAX_SIZE/MEMORY_TARGET size, and OS returned error ENOMEM (Cannot allocate memory).

[pid 16253] 15:38:39.787948 shmat(262149, 0, 0) = ?
[pid 16253] 15:38:39.787980 shmdt(0x7fd939905000) = 0
[pid 16253] 15:38:39.788011 mmap(0x66000000, 17079205888, PROT_READ|PROT_WRITE, MAP_SHARED|MAP_FIXED|MAP_ANONYMOUS, 0, 0) = -1 ENOMEM (Cannot allocate memory) <===== map 15GB memory
[pid 16253] 15:38:39.788044 munmap(0x60000000, 33554432) = 0
[pid 16253] 15:38:39.788072 munmap(0x62000000, 33554432) = 0
[pid 16253] 15:38:39.788098 munmap(0x64000000, 33554432) = 0

However the free memory is more than 4GB.

CAUSE

Swap size is under configured:

When memory_target is configured, server process will try to mmap a memory as big as sga_max_size/memory_target when it is created. (It is not to allocate so much memory, but only to "claim" it could use so much memory in the future, given sga_max_size/memory_target is set). If free memory + free swap is smaller than sga_max_size/memory_target, operating system will fail with ENOMEM (Cannot allocate memory). So if swap size is under configured, the issue will be likely happened.

SOLUTION

Add more swap size (eg, 0.75 x memory size, ideally swap size should at least be equal to memory size)