Linux2.2 修改LinuxThreads线程限制

This document describes how to modify the per-process file descriptor limit and task limit of the Linux version 2.2 kernel to increase the number of sockets and threads that a Java server application can create. In particular, it describes the socket and thread requirements of the VOLANO^® chat server, but similar requirements are likely to exist in any server application written for the Java platform. Note that the current Linux kernel version is 2.6, so this information is unlikely to apply to any recent Linux distribution.

File descriptors and tasks

When running on the Linux 2.2 kernel, each process is limited by default to 1,024 file descriptors. Since the VolanoChat server requires one file descriptor for each person chatting through a VolanoChat applet, you are limited by default to a maximum of 1,024 simultaneous users when using the Linux 2.2 kernel.

Some Java virtual machines, such as the IBM Developer Kit for Linux, Java Technology Edition, use native threads on Linux which require one task for each Java thread. Since VolanoChat requires two Java threads for each connection, you may need to increase the maximum number of tasks per user for such virtual machines. The default maximum number of tasks in a Linux 2.2 system is 512, with a maximum of 256 tasks per user. So by default, a VolanoChat server running under IBM JDK 1.1.6 for Linux can support a maximum of 128 people chatting at the same time.

You can increase the limits testing the the number of file descriptors and the number of tasks in the Linux 2.2 kernel by following the steps below. These steps were performed on a Red Hat Linux 6.0 system using Linux kernel 2.2.5-15. The instructions may be different for your Linux system.

To increase the maximum file descriptors per process to 4,096 (from 1,024) and the maximum tasks per user to 4,090 (from 256) on Linux 2.2, do the following steps.

1. Change the following C header files.

   In /usr/include/linux/tasks.h, change:

   NR_TASKS            512          to 4090
   MAX_TASKS_PER_USER  (NR_TASKS/2) to NR_TASKS
   

   In /usr/include/linux/limits.h, change:

   NR_OPEN             1024 to 4096
   OPEN_MAX             256 to 4096
   

   In /usr/include/linux/posix_types.h, change:

   __FD_SETSIZE        1024 to 4096
   

   In /usr/include/bits/types.h, change:

   __FD_SETSIZE        1024 to 4096
   

2. To allow users to increase their file descriptor limits, change the following configuration files:

   In /etc/security/limits.conf, add the lines:

   *       soft    nofile  1024
   *       hard    nofile  4096
   

   In /etc/pam.d/login, add:

   session required /lib/security/pam_limits.so
   

3. To increase the system-wide file descriptor limit, add the following three lines to the /etc/rc.d/rc.local startup script:

   # Increase system-wide file descriptor limit.
   echo  8192 > /proc/sys/fs/file-max
   echo 24576 > /proc/sys/fs/inode-max
   

4. Change to the directory /usr/src/linux and set up the kernel sources with:

   make mrproper
   

5. Make any other kernel configuration changes, such as setting the processor family type, by entering the command below. Save your changes and exit the configuration panel when you are done.

   make xconfig
   

6. Prepare for the kernel build by entering:

   make dep
   make clean
   

7. Rebuild your kernel with the commands:

   make bzImage
   make modules
   

8. Backup your old modules directory by entering (for example):

   mv /lib/modules/2.2.5-15 /lib/modules/2.2.5-15.old
   

9. Create a new set of kernel modules with the command:

   make modules_install
   

10. Backup your old kernel by entering (for example):

    mv /boot/vmlinuz /boot/vmlinuz.old
    

11. Copy over your new kernel image with:

    cp /usr/src/linux/arch/i386/boot/bzImage /boot/vmlinuz
    

12. Edit the Linux loader configuration in /etc/lilo.conf so that your new kernel is the default and your old kernel is still available from the boot prompt, as in the example shown below:

    boot=/dev/hda
    map=/boot/map
    install=/boot/boot.b
    prompt
    timeout=50
    image=/boot/vmlinuz
            label=linux
            root=/dev/hda1
            read-only
    image=/boot/vmlinuz.old
            label=linux.old
            root=/dev/hda1
            read-only
    

13. Generate the new Linux loader configuration by running:

    /sbin/lilo
    

14. Shutdown and reboot with your new kernel by entering:

    shutdown -r now
    

15. After rebooting, verify that the changes were effective as shown in the sections below.

LinuxThreads 0.7 limits

Even after you rebuild your Linux 2.2 kernel to increase the maximum number of threads per Linux user from 256 to 4,090, there are still definitions in the thread library itself which limit the number of threads per process to less than 1,000. The VolanoChat server requires two threads for each person chatting through a VolanoChat applet, so the LinuxThreads implementation creates a limit of 500 people to a single VolanoChat server.

You can increase this limit to 8,192 threads with the following steps. Even though the thread library will allow 8,192 threads per process after these changes, you'll still be limited by the Linux 2.2 kernel to a maximum of 4,090 threads for the Linux user account running the VolanoChat server after the changes in the previous section. The steps below are shown for a Red Hat Linux 6.0 system, so they may be different for your Linux distribution.

1. Install the Glibc 2.1.1 source package from /mnt/cdrom/SRPMS/glibc-2.1.1-6.src.rpm on the Red Hat Linux CD-ROM #2.

2. Unpack the source archive /usr/src/redhat/SOURCES/glibc-990416.tar.gz so that you can modify two of the files.

3. Change the following C header files.

   In /usr/src/redhat/SOURCES/glibc/linuxthreads/internals.h, change the size of the thread stack reserve from 2 megabytes down to 256 kilobytes (with a page size of 4,096 bytes):

   STACK_SIZE  (2 * 1024 * 1024) -> (64 * PAGE_SIZE)
   

   In sysdeps/unix/sysv/linux/bits/local_lim.h under /usr/src/redhat/SOURCES/glibc/linuxthreads, change the Posix thread implementation limit from 1,024 per process to 8,192 per process:

   PTHREAD_THREADS_MAX  1024 -> 8192
   

4. Rebuild the source archive, /usr/src/redhat/SOURCES/glibc-990416.tar.gz, to include your changes, replacing the original copy.

5. Change to the RPM specification directory, /usr/src/redhat/SPECS, and enter the command:

   rpm -ba glibc-2.1.spec
   

6. After a few hours, the new LinuxThreads library will be found as libpthread.so under:

   /usr/src/redhat/BUILD/glibc/build-i386-linux/linuxthread
   

7. Copy the new thread library into your own lib subdirectory:

   cd /usr/src/redhat/BUILD/glibc/build-i386-linux
   cp linuxthread/libpthread.so ~/lib
   cd ~/lib
   ln -s libpthread.so libpthread.so.0
   

8. Define the environment variable LD_LIBRARY_PATH to include the subdirectory containing the new thread library libpthread.so.

9. Verify the new thread limits as instructed below.

Verifying the new file descriptor limit

Increase your new limits to their maximum by entering "ulimit -n 4096" for the Bash shell or "unlimit" for the C shell. You can then check whether your changes were effective by running the following C program:

#include <sys/types.h>
#include <fcntl.h>
#include <stdio.h>

void main(void) {
    int k = 3;
    while (dup(0) != -1)
        k++;
    printf("Opens = %d Fdsize = %d\n", k, sizeof(fd_set) * 8);
}

Both numbers printed should give your new file descriptor limit.

Verifying the new thread limit

You can use the following program to verify your new thread limits in the modified Linux 2.2 kernel and in the modified LinuxThreads library.

#include <stdio.h>
#include <unistd.h>
#include <pthread.h>

#define MAX_THREADS 10000
int i;

void run(void) {
    char c;
    if (i < 10)
        printf("Address of c = %u KB\n", (unsigned int) &c / 1024);
    sleep(60 * 60);
}

int main(int argc, char *argv[]) {
    int rc = 0;
    pthread_t thread[MAX_THREADS];
    printf("Creating threads ...\n");
    for (i = 0; i < MAX_THREADS && rc == 0; i++) {
        rc = pthread_create(&(thread[i]), NULL, (void *) &run, NULL);
        if (rc == 0) {
            pthread_detach(thread[i]);
            if ((i + 1) % 1000 == 0)
                printf("%i threads so far ...\n", i + 1);
        }
        else
            printf("Failed with return code %i creating thread %i.\n",
                rc, i + 1);
    }
    exit(0);
}

Compile the program with:

gcc -lpthread -o count count.c

Running with the original LinuxThreads library will print:

blue:/home/john> count
Creating threads ...
Address of c = 3137535 KB
Address of c = 3135487 KB
Address of c = 3133439 KB
Address of c = 3131391 KB
Address of c = 3129343 KB
Address of c = 3127295 KB
Address of c = 3125247 KB
Address of c = 3123199 KB
Address of c = 3121151 KB
1000 threads so far ...
Failed with return code 11 creating thread 1023.

while running with the modified LinuxThreads library will print:

blue:/home/john> count
Creating threads ...
Address of c = 3144703 KB
Address of c = 3144447 KB
Address of c = 3144191 KB
Address of c = 3143935 KB
Address of c = 3143679 KB
Address of c = 3143423 KB
Address of c = 3143167 KB
Address of c = 3142911 KB
Address of c = 3142655 KB
1000 threads so far ...
2000 threads so far ...
3000 threads so far ...
4000 threads so far ...

You may need to enter Ctrl-C to exit the program.

Notice that the thread stack addresses are spaced apart by 256 kilobytes instead of the earlier reserve of 2 megabytes per stack. Also notice that the program can create threads right up to the Linux 2.2 kernel limit of 4,090 for the system rather than the earlier LinuxThreads limit of 1,024 per process.

 

http://www.volano.org/articles/linux-2.2-sockets-threads/