Installing FreeBSD 6 for Internet Server

Chatchawan Wongsiriprasert

<[e-mail]>
          

Copyright © 2005 Chatchawan Wongsiriprasert

$Id: article.sgml,v 1.2 2006/05/27 05:44:30 cws Exp $

FreeBSD is a registered trademark of the FreeBSD Foundation.

Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in this document, and the FreeBSD Project was aware of the trademark claim, the designations have been followed by the “™” or the “®” symbol.

---

Table of Contents

1. Overview

2. Installing FreeBSD 6

3. Install Application & Web Service

4. Install Mail Service

5. Setup User WWW Site

6. Server Maintainance

1. Overview

This document is a guidline for install an FreeBSD for Internet hosting. My company ,MiracleNet Group, is a web base software solution provider. Sometime we need to setup a server to host the solution for our customer which is my responsibility.

This guildline was start from notes I has been taken when I install those servers. I assume that the reader has some experience on FreeBSD and has already read the FreeBSD Handbook.

The requirement for this Internet server are:

• It must be an e-mail server with virus and spam filter. The customer must be allow to add/delete an e-mail without the need to contact us.

• It must support POP3/IMAP4/POP3S/IMAP4S,webmail , and e-mail relay for our customers.

• It must host our customers web sites. Each customer must not be able to access files of other customers.

• The customer must not be able to login on this server , except for upload and download the web pages.

---

2. Installing FreeBSD 6

First of all, please read my suggestion on Partition Layout because it is the only thing you can not change after install FreeBSD. Then, you can proceed to install FreeBSD as indicated in FreeBSD Handbook. You can download ISO images for i386 or amd64 from FreeBSD.org or mirror sites. Only the first disk ,6.0-xxx-xxx-disc1.iso, is required.

---

2.1. Partition Layout

Before start the installation process. You must make up your mind about the partition layout of the hard disk because it is only the thing you can not change after install the system

Assume that you have single disk of moderate size (32GB or up) my suggestion for partition layout are:

Table 1. Partition Layout for 32GB

Partition Filesystem Size Description

a	/	256 MB	FreeBSD handbook suggest 100 MB to this partition but for a 32GB-up disk set it 256 MB may be better.
b	N/A	4-8 GB	This according to FreeBSD handbook that suggest 2-3 x RAM. Upgrade RAM is easy ,just put the new RAM module but add swap space is mean add new disk which may not be applicable in 1U RACK.Anyways, with 32GB hard disk or i386, 8 GB may be too much.
d	/var	2-4 GB	Server need a lot of space on /var for logging and house keeping. Some software use /var to store temporary data by default.
e	/tmp	1-2 GB	Many software and user scripts assume that /tmp is world writable. Put this directory on it own partition will prevent a runaways user process to eat up all the space on more critical partition such as / or /var.
f	/usr	5-10 GB	We need this partition to store source/ports tree and do the system building. 5 GB is fine but with large hard disk (72 GB) , 10 GB will not hurt you.
g	/home	Rest of disk	This partition will store all user data or anything that you don't want to touch when reinstall the system. Moreover, you may want to set quota on this slice.

---

2.2. Upgrade FreeBSD source and ports

After install FreeBSD 6 and ports tree from CD. You need to upgrade your system to the lastest patch to protect your system against various types of attack.

You need a pre-build program (or package in FreeBSD) to upgrade your system. The package is net/cvsup-without-gui which can be used to upgrade both source and ports tree.

For example,as the time of this writing version of 6 is 6.0 and assume that the platform is i386. The command to download and install cvsup are:

    # fetch ftp://ftp.freebsd.org/pub/FreeBSD/releases/i386/6.0-RELEASE/packages/net/cvsup-without-gui-16.1h_2.tbz
    cvsup-without-gui-16.1h_2.tbz                 100% of  754 kB   37 kBps 00m00s
    # pkg_add cvsup-without-gui-16.1h_2.tbz

Edit cvs-supfile to upgrade lastest update of FreeBSD 6.0. This is my cvs-supfile:

    #For complete list of cvsupd see CVSup Sites on FreeBSD handbook.
    *default host=cvsup12.freebsd.org
    *default base=/usr
    *default prefix=/usr
    *default release=cvs
    *default delete use-rel-suffix
    *default tag=RELENG_6_0
    
    *default compress
    src-all
    ports-all tag=.
   

Run cvsup ,It will take a while to fetch both src and ports tree.

    # /usr/local/bin/cvsup -L2 cvs-supfile
   

See Using CVSup section on FreeBSD handbook for more detail about using cvsup.

---

2.3. Rebuild FreeBSD

Edit your /etc/make.conf (copy the default from /usr/src/share/examples/etc/make.conf). At least change the CPUTYPE to match your machine. See gcc 3.4.4 manual for detail of each CPUTYPE. FreeBSD building system may not be able to use all CPUTYPE in gcc manual. If your CPUTYPE cause a build error, try the other one. This is the example of my make.conf:

CPUTYPE?= p4         #Use ?= not = to allow FreeBSD build process to override this value
#CPUTYPE?= k8       #For Athlon64 on i386
#CPUTYPE?= athlon64 #For Athlon64 on AMD64

Modify your kernel configuration. You should read Configuring the FreeBSD Kernel and /usr/src/sys/i386/conf/NOTES or /usr/src/sys/amd64/conf/NOTES for each options of the kernel. This is my kernel configuration for i386/AMD64 on my Althon64 test machine:

machine         i386
#machine        amd64
cpu             I686_CPU
#cpu            HAMMER

#options        SMP         # Symmetric MultiProcessor Kernel

ident           GAIA-I386
#ident          GAIA-AMD64

#Adjust memory limit for 4G RAM for i386
options         KVA_PAGES=384               #1.5 G for kernels
options         MAXDSIZ=(1536UL*1024*1024)  #1.5 G for data
options         MAXSSIZ=(128UL*1024*1024)   #128M for stack
                                            #Leave 896KB for code segment
options         DFLDSIZ=(1536UL*1024*1024)  #Set default data size to 1.5G

options         SCHED_4BSD
options         PREEMPTION              # Enable kernel thread preemption
options         INET                    # InterNETworking
options         FFS                     # Berkeley Fast Filesystem
options         SOFTUPDATES             # Enable FFS soft updates support
options         UFS_ACL                 # Support for access control lists
options         UFS_DIRHASH             # Improve performance on big directories
options         MD_ROOT                 # MD is a potential root device
options         CD9660                  # ISO 9660 Filesystem
options         PROCFS                  # Process filesystem (requires PSEUDOFS)
options         PSEUDOFS                # Pseudo-filesystem framework
options         GEOM_GPT                # GUID Partition Tables.
options         COMPAT_43               # Compatible with BSD 4.3 [KEEP THIS!]
#options        COMPAT_IA32             # Compatible with i386 binaries
options         COMPAT_FREEBSD4         # Compatible with FreeBSD4
options         COMPAT_FREEBSD5         # Compatible with FreeBSD5
options         SCSI_DELAY=5000         # Delay (in ms) before probing SCSI
options         KTRACE                  # ktrace(1) support
options         SYSVSHM                 # SYSV-style shared memory
options         SYSVMSG                 # SYSV-style message queues
options         SYSVSEM                 # SYSV-style semaphores
options         _KPOSIX_PRIORITY_SCHEDULING # POSIX P1003_1B real-time extensions
options         KBD_INSTALL_CDEV        # install a CDEV entry in /dev

options         ADAPTIVE_GIANT          # Giant mutex is adaptive.

#Kernel Options for PostgreSQL with large shared memory (312.5M)
options         SYSVSHM                 #SYSV-style shared memory
options         SYSVMSG                 #SYSV-style message queues
options         SYSVSEM                 #SYSV-style semaphores
options         SHMMAXPGS=80000
options         SHMSEG=256
options         SHMMNI=256
options         SEMMNI=256
options         SEMMNS=512
options         SEMMNU=256
options         SEMMAP=256
#PostgreSQL use a alot of shared memory - default is 200
options         PMAP_SHPGPERPROC=512

#Firewall & NAT & DummyNet, may be needed in jail setup
options         IPFIREWALL
options         IPDIVERT
options         IPFIREWALL_DEFAULT_TO_ACCEPT
options         IPFIREWALL_VERBOSE
options         IPFIREWALL_VERBOSE_LIMIT=100
options         DUMMYNET
options         IPFIREWALL_FORWARD

options         QUOTA

device          apic                    # I/O APIC
device          pci
# Floppy drives
device          fdc

# ATA and ATAPI devices
device          ata
device          atadisk         # ATA disk drives
device          ataraid         # ATA RAID drives
device          atapicd         # ATAPI CDROM drives
device          atapifd         # ATAPI floppy drives
device          atapist         # ATAPI tape drives
options         ATA_STATIC_ID   # Static device numbering

# SCSI peripherals
device          scbus           # SCSI bus (required for SCSI)
device          ch              # SCSI media changers
device          da              # Direct Access (disks)
device          sa              # Sequential Access (tape etc)
device          cd              # CD
device          pass            # Passthrough device (direct SCSI access)
device          ses             # SCSI Environmental Services (and SAF-TE)

# atkbdc0 controls both the keyboard and the PS/2 mouse
device          atkbdc          # AT keyboard controller
device          atkbd           # AT keyboard
device          psm             # PS/2 mouse

device          vga             # VGA video card driver
device          splash          # Splash screen and screen saver support

# syscons is the default console driver, resembling an SCO console
device          sc
device          agp             # support several AGP chipsets

# Power management support (see NOTES for more options)
device          apm
# Add suspend/resume support for the i8254.
device          pmtimer

# Serial (COM) ports
device          sio             # 8250, 16[45]50 based serial ports
# Parallel port
device          ppc
device          ppbus           # Parallel port bus (required)
device          lpt             # Printer
device          plip            # TCP/IP over parallel
device          ppi             # Parallel port interface device
#device         vpo             # Requires scbus and da

# PCI Ethernet NICs that use the common MII bus controller code.
# NOTE: Be sure to keep the 'device miibus' line in order to use these NICs!
device          miibus          # MII bus support
device          sk              # SysKonnect SK-984x & SK-982x gigabit Ethernet

# Pseudo devices.
device          loop            # Network loopback
device          random          # Entropy device
device          ether           # Ethernet support
device          sl              # Kernel SLIP
device          ppp             # Kernel PPP
device          tun             # Packet tunnel.
device          pty             # Pseudo-ttys (telnet etc)
device          md              # Memory "disks"
device          gif             # IPv6 and IPv4 tunneling
device          faith           # IPv6-to-IPv4 relaying (translation)
device          io
device          mem

# The `bpf' device enables the Berkeley Packet Filter.
# Be aware of the administrative consequences of enabling this!
# Note that 'bpf' is required for DHCP.
device          bpf             # Berkeley packet filter

# USB support
device          uhci            # UHCI PCI->USB interface
device          ohci            # OHCI PCI->USB interface
device          ehci            # EHCI PCI->USB interface (USB 2.0)
device          usb             # USB Bus (required)
#device         udbp            # USB Double Bulk Pipe devices
device          ugen            # Generic
device          uhid            # "Human Interface Devices"
device          ukbd            # Keyboard
device          ulpt            # Printer
device          umass           # Disks/Mass storage - Requires scbus and da
device          ums             # Mouse
device          uscanner        # Scanners

Rebuild your world and kernel as told in the handbok.

# cd / 
# mergemaster -pai 
# cd /usr/src 
# make -j2 buildworld  -- For dual CPU use -j4 
# make -j2 buildkernel KERNCONF=XXX 
# make installkernel KERNCONF=XXX 
# cd / 
# mergemaster -ai 
-- clear temproot 
# cd /var/tmp/temproot 
# chflags noschg var/empty 
# find . -type l -delete 
# find . -empty -delete 
-- check the leftover files, replace or delete as you please  
# cd /var/tmp 
# rm -rf temproot

If you have the console access

 # shutdown now

If you can only has a ssh access,close as many daemons as you can except sshd and daemons spawn by kernel. This method should work for patch level upgrade (6.0 to 6.0p1), may work for minor version upgrade (6.0 to 6.1) and unlikely to work for major version upgrade (4.x to 5.x).

# cd /usr/src
# make installworld

Before reboot, Set your System Configuration because some setting will be in effect only after reboot. Set them first save you another reboot. If everything is fine, it is the time to reboot your server with shutdown -r now

---

2.4. Set System Configuration

There are 4 system configuration files you may need to modify.

1. /etc/rc.conf, check that you have these 3 lines

   sshd_enable="YES"
   sendmail_enable="NONE"
   syslogd_flags="-ss"
   
   firewall_enable="YES"
   firewall_type="/etc/ipfw.rules"
   
   #If your ISP has a reliable DNS service you can use its service,
   #otherwise it better to rely on ourself.
   #Don't forget to run : cd /etc/namedb/ && ./make-localhost
   named_enable="YES"
   
   quota_enable="YES"
   #It is a time consume job, better run it later after we got access to the system
   check_quotas="NO"
   #Don't forget to run : quotacheck -a after next reboot to create a quota file
   

   
   

2. /etc/sysctl.conf

   security.bsd.see_other_uids=0
   kern.coredump=0
   net.inet.icmp.drop_redirect=1
   net.inet.tcp.blackhole=2
   net.inet.udp.blackhole=1
   net.inet.ip.rtexpire=2
   net.inet.ip.rtminexpire=2
   kern.ipc.somaxconn=512
   

   
   

3. /boot/loader.conf

   autoboot_delay="3"
   kern.ipc.maxsockets=81920
   kern.ipc.maxsockbuf=1048576
   

   
   

4. /etc/hosts

   You should swap the first 2 lines to make sure that you will get IPv4 (127.0.0.1) address for localhost instead of IPv6 (::1) because some program does not support IPv6.

   127.0.0.1               localhost localhost.my.domain
   ::1                     localhost localhost.my.domain
   #Our IP is 10.0.0.34 and our name is gaia.net0.intranet
   10.0.0.34               gaia gaia.net0.intranet
   

   
   

5. /etc/ssh/sshd_config

   #Assume that our IP is 10.0.0.34
   ListenAddress 10.0.0.34:22
   # Change to yes to enable built-in password authentication.
   # SecureCRT need this option 
   PasswordAuthentication yes
   # If UseDNS is "yes" and your resolver is not work (i.e DNS server is down), 
   # you can not log in.
   UseDNS no
   
   #Allow only admin to login from anywhere
   AllowUsers cws@*
   
   Subsystem   sftp    /usr/libexec/sftp-server
   

   
   

6. /etc/fstab

   /dev/ad6s1g  /home  ufs  rw,userquota,groupquota   2     2
   

   
   

7. /var/named/etc/namedb/named.conf

   listen-on       { 127.0.0.1; };
   allow-recursion { 127.0.0.1; };
   

   
   

8. /etc/resolv.conf

   nameserver 127.0.0.1
   

   
   

9. /etc/ipfw.rules

   #more rules later
   
   add 65535 allow ip from any to any
   

   
   

It is also a good idea to change /etc/motd to something that look more legal such as

* * * * * * * * * * * * * W A R N I N G * * * * * * * * * * * * * * *
THIS SYSTEM IS RESTRICTED TO AUTHORIZED USERS FOR AUTHORIZED USE ONLY.
UNAUTHORIZED ACCESS IS STRICTLY PROHIBITED AND MAY BE PUNISHABLE UNDER
THE COMPUTER FRAUD AND ABUSE ACT OF 1986 OR OTHER APPLICABLE LAWS.
IF NOT AUTHORIZED TO ACCESS THIS SYSTEM, DISCONNECT NOW. BY CONTINUING,
YOU CONSENT TO YOUR KEYSTROKES AND DATA CONTENT BEING MONITORED. ALL
PERSONS ARE HEREBY NOTIFIED THAT THE USE OF THIS SYSTEM CONSTITUTES
CONSENT TO MONITORING AND AUDITING. THE ADMINISTRATORS ALSO RESERVE THE
RIGHT TO CANCEL OR LOCK YOUR ACCOUNT AT ANY GIVEN TIME. ALL TERMS
DESCRIBED ABOVE ARE SUBJECT TO CHANGE WITHOUT ANY GIVEN NOTICE. IF YOU
DO NOT AGREE TO THESE TERMS LOGOUT NOW!
* * * * * * * * * * * * * W A R N I N G * * * * * * * * * * * * * * *

which I copied from a web site somewhere.

---

3. Install Application & Web Service

This is a time to install program from ports tree. It is possible that the previous installation process may already install some ports on your system. Use pkg_delete to remove each installed ports except net/cvsup-without-gui because building this port require a lot of programs that will be never used elsewhere.

---

3.1. System Utilities

The system utilities I always install on my server are:

Table 2. System Utilities

Port Description Note

lang/perl5.8	Mandatory port.
shells/bash	Shell for users who can login to this server.
security/portaudit	Checks installed ports against a list of security vulnerabilities.
sysutils/portupgrade	FreeBSD ports/packages administration and management tool suite.	Don't check BDB4 box.
security/bcwipe	BCWipe securely erases data from magnetic and solid-state memory.
net/rsync	A network file distribution/synchronization utility.
security/sudo	Allow others to run commands as root.
sysutils/lsof	Lists information about open files (similar to fstat(1) ).
misc/compat4x	Compatible module for application that compiled for FreeBSD 4	Add compat4x_enable="YES" to /etc/rc.conf to enable FreeBSD 4 compatible.
misc/compat5x	Compatible module for application that compiled for FreeBSD 5	Add compat5x_enable="YES" to /etc/rc.conf to enable FreeBSD 5 compatible.

---

3.2. Install Databases

Table 3. System Utilities

Port Description Note

database/mysql41-server	We use mysql to store administrative data.	Append the line WITH_XCHARSET=all to /etc/make.conf before install the port. This will add many international languages (such as Thai) support (search/sort) in MySQL.
databases/postgresql81-server	We use postgresql to store data for the application.
databases/phpmyadmin	Tool to manipulate MySQL.	Install this after you install WWW server. Select all options except MYSQLI.
databases/phppgadmin	Tool to manipulate PostgreSQL.	Install this after you install WWW server.
databases/p5-DBD-mysql	MySQL driver for the Perl5 Database Interface (DBI).	Some of the perl scripts need MySQL access.

---

3.2.1. Config MySQL server

I place my database in /home/mysql , so my /etc/rc.conf for mysql are:

mysql_enable="YES"
mysql_dbdir="/home/mysql"
mysql_args="--bind-address=127.0.0.1"

If you want to access mysql from another machine, remove the third line. Before start mysql, you may need to set my.cnf to change mysql options:

# mkdir /home/mysql 
# mkdir /home/mysql/tmp 
# cp /usr/local/share/mysql/my-medium.cnf /home/mysql/my.cnf 
# chown -R mysql:mysql /home/mysql

I always set mysqld tmpdir to /home/mysql/tmp unless I have a very large /tmp on another disk. Sometime mysql use a lot of tmpdir when you run a complex query. Read mysql manual for more detail.

[mysqld]
...
max_allowed_packet = 4M
...
#log-bin
skip-innodb
tmpdir          = /home/mysql/tmp

#For development machine, you may need slow query log 
#to track a badly write SQL.
long_query_time = 10
log_slow_queries = /home/mysql/slow-query.log

...

Don't forget to set MySQL root password

# /usr/local/etc/rc.d/mysql-server.sh start
# mysql -u root
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 2 to server version: 4.1.14

Type 'help;' or '\h' for help. Type '\c' to clear the buffer.

mysql> SET PASSWORD FOR root@localhost=PASSWORD('xxx');
Query OK, 0 rows affected (0.02 sec)

mysql>

---

3.2.2. Config PostgreSQL server

Like mysql , I place postgresql databases in /home/pgsql. My /etc/rc.conf for postgresql are:

postgresql_enable="YES"
postgresql_data="/home/pgsql/data"

Use vipw to change home directory of pgsql user to /home/pgsql.

-- rsync preserve symbolic link while cp is not
# rsync -a -v /usr/local/pgsql /home/
# su -m pgsql
# initdb /home/pgsql/data

You must edit /home/pgsal/data/pg_hba.conf before start postgresql

# TYPE  DATABASE    USER        CIDR-ADDRESS          METHOD
# "local" is for Unix domain socket connections only
local   all         pgsql                             ident sameuser
local   all         all                               md5
# IPv4 local connections:
host    all         all         127.0.0.1/32          md5
# IPv6 local connections:
host    all         all         ::1/128               md5

Read PostgreSQL manual and Tuning PostgreSQL for performance for more details.

This is the change I made for my server:

shared_buffers = 30000 # min 16, at least max_connections*2, 8KB each
work_mem = 32768               # min 64, size in KB

max_fsm_pages = 40000          # min max_fsm_relations*16, 6 bytes each
max_fsm_relations = 1000       # min 100, ~50 bytes each

wal_buffers = 32       # min 4, 8KB each
checkpoint_segments = 8        # in logfile segments, min 1, 16MB each
effective_cache_size = 4000    # typically 8KB each

#logging
log_destination = 'stderr' 
redirect_stderr = on
log_directory = 'pg_log'
log_filename = 'postgresql-%Y-%m-%d_%H%M%S.log'
log_rotation_age =1440
log_rotation_size = 10240
#slow query log -- enable for developer to check slow query
#log_min_duration_statement = 10
#log_line_prefix = '%t [%u:%d] '

By default PostgreSQL root is pgsql or any system user that own the database files. You should create another database adminstrator account to allow postgresql user such as sa to act as database adminstrator.

# su pgsql
# psql template1
Welcome to psql 8.0.4, the PostgreSQL interactive terminal.

Type:  \copyright for distribution terms
       \h for help with SQL commands
       \? for help with psql commands
       \g or terminate with semicolon to execute query
       \q to quit

template1=# CREATE USER sa WITH PASSWORD 'xxxx' CREATEDB CREATEUSER
CREATE USER
template1=#

---

3.2.3. Config WWW tools

After install WWW service, you may want to install database/phpmyadmin and database/phppgadmin to manage your databases. You must access these packages via HTTPS only because both require you to enter the database user and password on the webpage.

# cd /home/www/public_ssl
# ln -s /usr/local/www/phpMyAdmin
# ln -s /usr/local/www/data/phpPgAdmin

Copy /usr/local/www/phpMyAdmin/libraries/config.default.php to /usr/local/www/phpMyAdmin/config.inc.php and change the following lines to use http authentication:

$cfg['Servers'][0]['host']          = 'localhost';
$cfg['Servers'][0]['connect_type']  = 'socket';
$cfg['Servers'][0]['auth_type']     = 'http';

---

3.3. Install WWW Server

Table 4. Ports for WWW Service

Port Description Note

www/apache13-modssl	A www server of a choice.	Append the line WITH_APACHE_MODDEFLATE=yes to /etc/make.conf install mod_deflate.
lang/php4	Our main development language.	select OPENSSL box.
lang/php4-extension	A "meta-port" to install PHP extensions.	Append the line WITHOUT_X11=yes to /etc/make.conf before install the port. This will prevent any reference to X11 which include XBM support in GD.
devel/ZendOptimizer	An optimizer for PHP code.	It free but closed source. May cause a core-dump with some php extenstions. Unfortunely, the current version of ZendOptimizer (2.5.10) is not support FreeBSD AMD64. If you really want to run it you may need to enable 32bit support in the kernel and run a 32-bit version of Apache/PHP -- see Setup User WWW Site for more detail.
www/awstat	Free real-time logfile analyzer to get advanced web statistics.
net/p5-Geo-IP	Gets country name by IP or hostname.

---

3.3.1. Config Apache

I usually move apache's document root from /usr/local/www/data to /home/www/public_html for HTTP service and /home/www/public_ssl for HTTPS service. Another change I usually made to /usr/local/etc/apache/httpd.conf is remove the univeral listen line Port 80 or Listen 80 to more specified listen Listen xxx.xxx.xx.xx:80 because I need to run another apache in a jail(8). I also change a log format and logfile name.Here is a result of the command diff -u /usr/local/etc/apache/httpd.conf-dist /usr/local/etc/apache/httpd.conf and also the complete version of httpd.conf. Don't foget to create a folder to store your log file. For example:

# mkdir /var/log/httpd

If you have a lot of virtual hosts on the server, it is more preferable to move the virtual host configuration to another file and use apache Include directive to include that configuration to httpd.conf.

To enable mod_deflate, you must add the line

AddModule mod_deflate.c
#The following lines can be put in .htaccess if you want
#to enable deflate per directory
<IfModule mod_deflate.c>
DeflateEnable On
DeflateMinLength 3000
DeflateCompLevel 1
DeflateProxied Off
DeflateHTTP 1.0
DeflateDisableRange "MSIE 4."
DeflateTypes text/plain text/html
</IfModule>

to httpd.conf

The last concern for httpd.conf is to remove unused modules. Read Apache modules Manual to see which module is not need for your server. Or, just remove them all ,then add the module one by one untils your site work as you want.

If you run HTTPS service, you may need to create a valid SSL certificate. There is a good doucument about Client Authentication with SSL at The FreeBSD Diary.

# sh /etc/periodic/weekly/310.locate
# locate CA.pl
# /usr/src/crypto/openssl/apps/CA.pl -newreq
Generating a 1024 bit RSA private key
.......................................++++++
...........++++++
writing new private key to 'newreq.pem'
Enter PEM pass phrase:xxxxxx
Verifying - Enter PEM pass phrase:xxxxxx
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:TH
State or Province Name (full name) [Some-State]:Bangkok
Locality Name (eg, city) []:Phayathai
Organization Name (eg, company) [Internet Widgits Pty Ltd]:MiracleNet Group Co., Ltd.
Organizational Unit Name (eg, section) []:Hosting Service
Common Name (eg, YOUR name) []:gaia.net0.intranet
Email Address []:[e-mail]

Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:
An optional company name []:MiracleNet Group Co., Ltd.
Request (and private key) is in newreq.pem
# openssl rsa < newreq.pem > newkey.pem
Enter pass phrase:xxxxxx
writing RSA key

Send your newreq.pem to Certificate Authority for real server or sign it yourself for the test one.

If you want to sign the certificate yourself. You must create yourown Certificate Authority first (assume that we will put the CA in /home/admin/CA,then sign the certificate:

# mkdir -p /home/admin/CA
# cd /home/admin/CA
# /usr/src/crypto/openssl/apps/CA.pl -newca
CA certificate filename (or enter to create)
<ENTER>
Making CA certificate ...
Generating a 1024 bit RSA private key
........................++++++
........++++++
writing new private key to './demoCA/private/cakey.pem'
Enter PEM pass phrase:xxxxx2
Verifying - Enter PEM pass phrase:xxxxx2
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:TH
State or Province Name (full name) [Some-State]:Bangkok
Locality Name (eg, city) []:Phayathai
Organization Name (eg, company) [Internet Widgits Pty Ltd]:Miraclenet Group Co.,  Ltd.
Organizational Unit Name (eg, section) []:Hosting Service
Common Name (eg, YOUR name) []:miraclenet.co.th
Email Address []:[e-mail]
# cp /home/admin/CA/newreq.pem .
# /usr/src/crypto/openssl/apps/CA.pl -sign
Using configuration from /etc/ssl/openssl.cnf
Enter pass phrase for ./demoCA/private/cakey.pem:xxxxx2
Check that the request matches the signature
Signature ok
...
Certificate is to be certified until Nov 29 02:13:01 2006 GMT (365 days)
Sign the certificate? [y/n]:y

1 out of 1 certificate requests certified, commit? [y/n]y
Write out database with 1 new entries
Data Base Updated
Signed certificate is in newcert.pem

Anyways, Use this self signed certificate will generate the warning message from the browser becase it don't known your Certificate Authority. To get rid of this warning, you must make the browser know your CA which can be done For firefox and opera, just copy the file demoCA/cacert.pem to the client machine,then, import it to your browser( Preferences/Advanced/Manage Certificates/Authories/Import or just put it on your web page and allow user to download and install the certificate ). For IE, change the file extension to .crt and import it with Internet Options/Contents/Publishers/Trusted Root Certification Authorities/Import.

After that, copy the signed request and key to /usr/local/etc/apache and modify your httpd.conf accordingly.

# cp newcert.pem /usr/local/etc/apache/ssl.crt/gaia.crt
# cp newkey.pem /usr/local/etc/apache/ssl.key/gaia.key
# cd /usr/local/etc/apache/ssl.crt/
# make
-- Don't forget to edit SSLCertificateFile and SSLCertificateKeyFile 
-- in httpd.conf to point to new crt and key

Don't forget to add the line apache_enable="YES" to /etc/rc.conf to enable apache service.

---

3.3.2. Config PHP

I need to patch PHP to make serialize command run faster , see the bug report "Slow serialize on FreeBSD". To apply the patch, just download patch-ph_smart_str.h and copy the patch to ports/lang/php4/files before build the php4 port. Anyways, this patch never made it ways through php porject or FreeBSD port tree, use it with your own risk.

If you want the OPENSSL support on PHP, don't forget to add OPENSSL option when build PHP. The OPENSSL can not work when compiled as an extension.

You may need install PHP extensions only install the required extension. The less extension installed, the less problem from PHP. The extensions normally installed on my server are BCMATH, BZ2, CTYPE, CURL, GD, IMAP, MBSTRING, MCRYPT, MHASH, MYSQL , OVERLOAD, PCRE , PDF, PGSQL, POSIX , SESSION , SOCKETS, SYSVSEM,SYSVSHM,SYSVMSG,TOKENIZER,XML and ZLIB.

Don't forget to add

<IfModule mod_php4.c>
    AddType application/x-httpd-php .php
    AddType application/x-httpd-php-source .phps
</IfModule>

to /usr/local/etc/apache/httpd.conf to automatic run php when user access .php file.

There are some dependency mismatch on FreeBSD 6.0/6.1 that cause apache start after compat5x which prevent ZendOptimizer from starting when you reboot the system. Run /sbin/rcorder to check for this problem

# rcorder /etc/rc.d/* /usr/local/etc/rc.d/*
...
/etc/rc.d/yppasswdd
/usr/local/etc/rc.d/apache.sh
/etc/rc.d/LOGIN
/usr/local/etc/rc.d/rsyncd.sh
/usr/local/etc/rc.d/mysql-server.sh
/usr/local/etc/rc.d/010.pgsql.sh
/usr/local/etc/rc.d/000.pkgtools.sh
/usr/local/etc/rc.d/000.compat5x.sh
/usr/local/etc/rc.d/000.compat4x.sh
...

If you see that above result, you have this problem. Edit /usr/local/etc/rc.d/apache.sh to force compat5x to start before apache.

gaia# diff -u apache.sh.org apache.sh
--- apache.sh.org       Sat May 20 14:04:48 2006
+++ apache.sh   Sat May 20 14:04:56 2006
@@ -2,7 +2,7 @@
 # $FreeBSD: ports/www/apache13-modssl/files/rcng.sh,v 1.5 2006/02/20 20:47:46 dougb Exp $

 # PROVIDE: apache
-# REQUIRE: DAEMON
+# REQUIRE: DAEMON compat5x
 # BEFORE: LOGIN
 # KEYWORD: shutdown

Rerun /sbin/rcorder to recheck that apache start after compat5x.

# rcorder /etc/rc.d/* /usr/local/etc/rc.d/*
...
/etc/rc.d/yppasswdd
/usr/local/etc/rc.d/000.compat5x.sh
/usr/local/etc/rc.d/apache.sh
/etc/rc.d/LOGIN
...

Some parameters in /usr/local/etc/php.ini may need to be consider such as:

output_buffering = On
zlib.output_compression = On
register_argc_argv = Off
magic_quotes_gpc = Off  #When On, It cause more problems because we don't know 
                        #the quote come from user input or from this option.
[Zend]
zend_optimizer.optimization_level=15
zend_extension_manager.optimizer="/usr/local/lib/php/20020429/Optimizer"
zend_extension_manager.optimizer_ts="/usr/local/lib/php/20020429/Optimizer_TS"
zend_extension="/usr/local/lib/php/20020429/ZendExtensionManager.so"
zend_extension_ts="/usr/local/lib/php/20020429/ZendExtensionManager_TS.so"

---

3.3.3. Config Web Statistic

Due to the volumn of log messages, we does not use syslogd to keep apache access/error log. So, we wrote rotatelog.pl to rotate logs file every midnight to prevent them grow too large. You need to put the rotatelog.pl in your crontab to run it every midnight.

Next step is to setup awstat. awstats require a configuration file which should to be placed in /usr/local/etc/awstats. There are small changes I made on /usr/local/www/awstats/cgi-bin/awstats.model.conf to create my configuration file.

awstats.gaia.conf

LogFile="bunzip2 -dc /var/log/httpd/access.log.0.bz2 |"
SiteDomain="gaia.net0.intranet"
HostAliases="localhost 127.0.0.1"
DNSLookup=0   
DirData="/home/www/public_html/stats/data"
DirCgi="/stats/cgi-bin"
DirIcons="/stats/icons"
LoadPlugin="geoip GEOIP_STANDARD /usr/local/share/GeoIP/GeoIP.dat"
UseFramesWhenCGI=0
LogFormat=1

The GeoIP database also need to update. The database version that I use is a free GeoLite Country Database which update once a month. Put this geoip_update.sh shell script in your crontab update the database.

To allow user to view the statistic, don't forget to setup a URL for awstats and setup authenticate:

# mkdir -p /home/www/apache
# htpasswd -cm /home/www/apache/passwd stats
# mkdir -p /home/www/public_html/stats/data
# cd /home/www/public_html/stats
# ln -s /usr/local/www/awstats/cgi-bin
# ln -s /usr/local/www/awstats/classes
# ln -s /usr/local/www/awstats/css
# ln -s /usr/local/www/awstats/icons
# ln -s /usr/local/www/awstats/js

Create /home/www/public_html/stats/index.php:

<?
header("Location:/stats/cgi-bin/awstats.pl?config=gaia");
?>

Create /home/www/public_html/stats/.htaccess:

AuthType Basic
AuthName "Gaia Access Statistic"
AuthUserFile /home/www/apache/passwd
Require user stats
Options FollowSymLinks

Create /home/www/public_html/stats/cgi-bin/.htaccess to run perl script with mod_perl:

Options ExecCGI
AddHandler cgi-script .pl

Don't forget to create a crontab entries for rotate access log ,update statistic and update GeoIP database.

#crontab -e
0 0 * * * /home/admin/bin/rotatelog.pl
10 0 * * *  /home/www/public_html/stats/cgi-bin/awstats.pl -config=gaia -update
0 0  2 * * /home/admin/bin/geoip_update.sh

---

4. Install Mail Service

Mail service (SMTP/POP/IMAP) is one of the function for this server. It much support virtual mailboxs for our customers. The server must act as a mail relay for the customer and spam and virus filter are a must have features. The following table show list of ports I use to implement mail service on this server:

Table 5. Mail Service

Port Description Note

security/courier-authlib	Courier authentication library base.	Select AUTH_MYSQL and AUTH_USERDB when build the port.
security/cyrus-sasl2	RFC 2222 SASL	Add the following lines to /etc/make.conf WITH_AUTHDAEMON= yes WITHOUT_OTP= yes WITHOUT_NTLM= yes WITHOUT_GSSAPI= yes WITH_MYSQL41=yes #If you use mysql41-server to remove unused authentication method.
mail/postfix	More secure than mail/sendmail and easier to extend than mail/qmail.	Select SASL2,TLS and MySQL. Answer y to every post installation questions.
security/amavisd-new	Our spam and virus filter.	Remove all options (MILTER is set by default - uncheck it).
mail/dspam	Bayesian spam filter.	Append the following lines to /etc/make.conf: DSPAM_OWNER=vscan DSPAM_GROUP=vscan DSPAM_HOME_OWNER=vscan DSPAM_HOME_GROUP=vscan Use default options when building the port.
security/clamav	a GPL anti-virus toolkit for UNIX. New version of dspam install clamav by default. You may not need to install this port manually.	No options is need. If you don't like clamav, see /usr/local/etc/amavisd.conf for another virus scanner supported by amavisd-new.
mail/courier-imap	Our POP3, POP3S, IMAP4 and IMAP4S server.	Select OPENSSL, TRASHQUOTA and AUTH_MYSQL. Unselect IPV6 unless you need it.
mail/squirrelmail	Greate web mail for small and medium size mail server.	go to /usr/port and run make search key=webmail to see another webmail in ports tree.

The mail server that I create is not the hight-performane one. On moderate hardware (Althon64 2800 with 1GB RAM and SATA disk) it can process about 3 mails a second (180 mails per minute) which is enought for small or medium company. So , if you a looking for the hight-performance mail server , this setup may not for you.

---

4.1. Prepare Mail System Database

We store our customer e-mail accounts on MySQL database to make it easier to manipulate and increase look up speed. Most of the information on this section come from Martin List-Petersen's ISP Mailserver Solution Howto.

CREATE DATABASE maildb;
USE maildb;

CREATE TABLE `alias` (
  `email` varchar(255) NOT NULL default '',    
  `destination` varchar(255) NOT NULL default '',
  `customer_id` varchar(16) NOT NULL default '',
  PRIMARY KEY `email` (`email`),
  KEY `customer_id` (`customer_id`)
) ENGINE=MyISAM;

Table 6. alias

email	The originally email-address. The email can be [e-mail] for single email or @example.com for all user in that domain.
destination	The destination email-address for the email.
customer_id	System customer id to check record owner. If the id is removed from the system all record with customer_id will be deleted.

CREATE TABLE `transport` (
  `domain` varchar(255) NOT NULL default '',
  `transport` varchar(128) NOT NULL default '',
  `customer_id` varchar(16) NOT NULL default '',
  PRIMARY KEY (`domain`),
  KEY `customer_id` (`customer_id`)
) ENGINE=MyISAM;

Table 7. transport

domain	Domain name of interest.
transport	Postfix transport type can be local: for local domain, virtual: for virtaul domain and smtp:another.mail.server if you need to forward mail for the domain to another server.
customer_id	System customer id to check record owner. If the id is removed from the system all record with customer_id will be deleted.

CREATE TABLE `user` (
  `email` varchar(128) NOT NULL default '',
  `passwd` varchar(128) NOT NULL default '$1$X$XXX',
  `name` varchar(128) NOT NULL default '',
  `uid` int(6) NOT NULL default '65534',    
  `gid` int(6) NOT NULL default '65534',
  `home` varchar(255) NOT NULL default '',  
  `maildir` varchar(255) NOT NULL default '',
  `allow_login` enum('Y','N') NOT NULL default 'Y', 
  `allow_receive` enum('Y','N') NOT NULL default 'Y', 
  `customer_id` varchar(16) NOT NULL default '',
  PRIMARY KEY  (`email`),
  KEY `customer_id` (`customer_id`)
) ENGINE=MyISAM;

Table 8. user

email	User email address ([e-mail]).
passwd	Encrypted password. Use /usr/local/sbin/userdbpw to create an encrypted password.
name	The users name. This is only for record keeping and it is not use by the mail system.
uid/gid	FreeBSD user id/group id of the mailbox owner.
home	The users home path. This is only for record keeping and it is not use by the mail system.
maildir	Path to user mailbox , for example "/home/vhost/user_x/mail/domain.com/user/". Don't remove the trailing slash or else postfix will deliver your mail to a mailspool format instead of a maildir.
allow_login	If it is 'N' user is not allow to access POP3/IMAP4 and SASL.
allow_receive	If it is 'N' , the user email is closed from receiving mail.
customer_id	System customer id to check record owner. If the id is removed from the system all record with customer_id will be deleted.

We need 3 MySQL user accounts with difference privileges .

1. maildb -- Owner of the database can do everything to the database.

2. maildb_auth -- Can read every fields in user table. Use by courier-authlib.

3. maildb_smtp -- Can read every fields on the database except passwd. Use by postfix. The user/password of this account must store in a word readable file in /usr/local/etc/postfix ,therefore, give minimum access to this account.

GRANT USAGE ON maildb.* TO 'maildb'@'localhost' IDENTIFIED BY '*********';
GRANT ALL PRIVILEGES ON `maildb` . * TO 'maildb'@'localhost' WITH GRANT OPTION;
GRANT USAGE ON maildb. * TO 'maildb_auth'@'localhost' IDENTIFIED BY '*********';
GRANT SELECT ON `maildb`.`user` TO 'maildb_auth'@'localhost';
GRANT USAGE ON maildb. * TO 'maildb_smtp'@'localhost' IDENTIFIED BY '*********';
GRANT SELECT ON `maildb`.`alias` TO 'maildb_smtp'@'localhost';
GRANT SELECT ON `maildb`.`transport` TO 'maildb_smtp'@'localhost';
GRANT SELECT (
`email` , `name` , `uid` , `gid` , `home` , `maildir` , `allow_login` , `allow_receive` , `customer_id`
) ON `maildb`.`user`
TO 'maildb_smtp'@'localhost';

---

4.2. Config amavisd-new and dspam

First append the lines to /etc/rc.conf to enable the service.

#Amavis/ClamAV/SpamAssasin
clamav_clamd_enable="YES"
clamav_freshclam_enable="YES"
amavisd_enable="YES"

Add clamav user to vscan group to enable clamd to access the amavisd filtering mail.

# vi /etc/group
spamd:*:58:
vscan:*:110:clamav
clamav:*:106:

To run dspam from amavisd-new you need to make some change to installed dspam.

# chmod u-s,a+rx /usr/local/bin/dspam
# cd /var/amavis
# ln -s /var/db/dspam

Then edit /usr/local/etc/amavisd.conf as show below:

$mydomain = 'gaia.net0.intranet';   # a convenient default for other settings
$dspam = 'dspam';              # Allow dspam

#Don't forget to uncomment 'ClamAV-clamd' to enable clamav

#If you want to accept .zip and .bz2, remove the comment on
#[ qr'^\.(Z|gz|bz2)$'           => 0 ] and
#[ qr'^\.(zip|rar|arc|arj|zoo)$'=> 0 ] 

#Discard all filtered mail -- don't notify sender
$final_virus_destiny      = D_DISCARD;
$final_banned_destiny     = D_DISCARD;
$final_spam_destiny       = D_DISCARD;
$final_bad_header_destiny = D_DISCARD;

$recipient_delimiter = '-';

#If someting go wrong enable the following options and take a look at
#/var/log/maillog and your mailheader
#$log_level = 5;
#$sa_tag_level_deflt = 0;

You can see the result of the command diff -u amavisd.conf-dist amavisd.conf on my server here.

To setup dspam, you must cread a dspam user and database on MySQL. Give that user full access to the database and run the script in /usr/local/share/examples/dspam/mysql/mysql_objects-4.1.sql.

# mysql -u root -p
Enter password:
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 144 to server version: 4.1.14

Type 'help;' or '\h' for help. Type '\c' to clear the buffer.

mysql> CREATE DATABASE dspam;
Query OK, 1 row affected (0.00 sec)
mysql> GRANT USAGE ON dspam.* TO 'dspam'@'localhost' IDENTIFIED BY '*********';
Query OK, 1 row affected (0.00 sec)
mysql> GRANT ALL PRIVILEGES ON `dspam` . * TO 'dspam'@'localhost' WITH GRANT OPTION;
Query OK, 1 row affected (0.00 sec)
mysql> USE dspam;
Database changed
mysql> \. /usr/local/share/examples/dspam/mysql/mysql_objects-4.1.sql

Edit /usr/local/etc/dspam.conf to add MySQL user and password:

StorageDriver /usr/local/lib/libmysql_drv.so
MySQLServer     /tmp/mysql.sock
#MySQLPort
MySQLUser               dspam
MySQLPass               xxxxxx
MySQLDb                 dspam
#MySQLCompress          true

#For Relearn false negative and false positive
MySQLUIDInSignature     on
Preference "signatureLocation=headers"

#We work with amavisd-new
IgnoreHeader X-Spam-Status
IgnoreHeader X-Spam-Scanned
IgnoreHeader X-Virus-Scanner-Result

#Add the following line and take a look at /var/log/dspam/dspam.debug 
#if something don't work as expected
#Debug vscan

dspam will not activate util it see about 2,000 spam/nospam mails, so you must wait for this threashold to be reach.

---

4.3. Config courier-authlib and cyrus-sasl2

Edit /usr/local/etc/authlib/authmysqlrc:

MYSQL_SERVER          localhost
MYSQL_USERNAME        maildb_auth
MYSQL_PASSWORD        xxxxx
MYSQL_SOCKET          /tmp/mysql.sock
MYSQL_DATABASE        maildb
MYSQL_USER_TABLE      user
MYSQL_CRYPT_PWFIELD   passwd
MYSQL_LOGIN_FIELD     email
MYSQL_MAILDIR_FIELD   maildir
MYSQL_WHERE_CLAUSE    allow_login='Y'

Edit /usr/local/etc/authlib/authdaemonrc:

authmodulelist="authpam authmysql"

Don't forget to add the line courier_authdaemond_enable="YES" to /etc/rc.conf.

For cyrus-sasl2, create the file /usr/local/lib/sasl2/smtpd.conf with content:

pwcheck_method: authdaemond
authdaemond_path: /var/run/authdaemond/socket

and change permission of /var/run/authdaemond to allow other to access the directory.

# chmod o+x /var/run/authdaemond

---

4.4. Config postfix

4.4.1. /etc/rc.conf

Edit the file to run postfix as mail service instead of the built-in sendmail:

#Postfix
postfix_enable="YES"
sendmail_enable="NONE"
sendmail_flags="-bd"
sendmail_outbound_enable="NO"
sendmail_submit_enable="NO"
sendmail_msp_queue_enable="NO"
daily_clean_hoststat_enable="NO"
daily_status_mail_rejects_enable="NO"
daily_status_include_submit_mailq="NO"
daily_submit_queuerun="NO"

---

4.4.2. /usr/local/etc/rc.d/postfix.sh

Older postfix port does not come with FreeBSD startup script. If ther is no /usr/local/etc/rc.d/postfix.sh use the following one:

#!/bin/sh
# PROVIDE: postfix
# REQUIRE: NETWORKING SERVERS
# BEFORE: DAEMON
# AFTER: mysql-server
# KEYWORD: shutdown

. /etc/rc.subr

name="postfix"
rcvar=`set_rcvar`

load_rc_config ${name}

: ${postfix_enable="NO"}

command=/usr/local/sbin/postfix
pidfile=/var/spool/${name}/pid/master.pid
start_cmd="postfix_cmd start"
stop_cmd="postfix_cmd stop"
restart_cmd="postfix_cmd stop && postfix_cmd start"
echo ${pidfile}
postfix_cmd () {
    case $1 in
    start)
        echo "Starting ${name}."
        ${command} start
        ;;
    stop)
        echo "Stopping ${name}."
        ${command} stop
        ;;
    esac
}

run_rc_command "$1"

---

4.4.3. /usr/local/etc/postfix/master.cf

We need to run amavisd and let postfix smtpd use it. First remove the standard smtpd service line at the begining of the file

smtp      inet  n       -       n       -       -       smtpd

and appened the following lines to start smtpd with amavis filter. It this configuration, we don't filter the outgoing mail (127.0.0.1:smtp). Assume that the server IP is 10.0.0.34

smtp-amavis unix -  -   n -   2  lmtp
    -o lmtp_data_done_timeout=1200
    -o lmtp_send_xforward_command=yes

127.0.0.1:smtp      inet  n       -       n       -       -       smtpd
        -o content_