Solaris Source Insight: PCI system implementation - Part 1

PCI implementation in Solaris/OpenSolaris

Author: [e-mail]
Date: October 28th, 2009

On a typical x86 platfomr, the PCI system implementation in Solaris/OpenSolaris includes the following modules.

[root@blu-nhm-ep:~]modinfo | grep PCI
15 fffffffffba46ce0 bfb0 - 1 pci_autoconfig (PCI BIOS interface)
37 fffffffffbacd3f0 ce28 183 1 npe (Host to PCIe nexus driver)
38 fffffffffbad95c8 5f50 - 1 pcihp (PCI nexus hotplug support)
40 fffffffffbae14f0 bb00 - 1 pcie (PCIE: PCI framework)
89 fffffffff7bff000 4c68 184 1 pcieb (PCIe to PCI nexus driver)
90 fffffffff7999000 1d68 84 1 pci_pci (PCI to PCI bridge nexus driver)

1. misc/pci_autoconfig
Pci_autoconfig module is the interface between PCI system and the BIOS. It "Determines the PCI configuration mechanism recommended by the BIOS". It was implemented in the following files.

156 PCI_AUTOCONFIG_OBJS += pci_autoconfig.o pci_boot.o pcie_nvidia.o \
157 pci_memlist.o pci_resource.o

The module itself only registers a bus probe function to the kernel.

intel/io/pci/pci_autoconfig.c
62 int
63 _init(void)
64 {
65 int err;
66
67 if ((err = mod_install(&modlinkage)) != 0)
68 return (err);
69
70 impl_bus_add_probe(pci_enumerate);
71 return (0);
72 }

i86pc/os/ddi_impl.c
2514 void
2515 impl_setup_ddi(void)
2516 {
... ...
2569 |_______/* do bus dependent probes. */
2570 |_______impl_bus_initialprobe();
2571 }

The first round of bus probe was called in impl_setup_ddi() during boot process. Bus reprober was called later in

i86pc/os/ddi_impl.c
157 /*
158 * Configure the hardware on the system.
159 * Called before the rootfs is mounted
160 */
161 void
162 configure(void)
163 {
... ...
203 |_______/* reprogram devices not set up by firmware (BIOS) */
204 |_______impl_bus_reprobe();

The bus probes in the linked list are executed in Last In First Service mode. If you have any dependency between the probes, you need to take care about this.

Let's pull the topic back by diving into the implementation of pci bus prober.

intel/io/pci/pci_autoconfig.c
94 /*
95 * This function is invoked twice: first time, with reprogram=0 to
96 * set up the PCI portion of the device tree. The second time is
97 * for reprogramming devices not set up by the BIOS.
98 */
99 void
100 pci_enumerate(int reprogram)
101 {
102 |_______extern void add_pci_fixes(void);
103 |_______extern void undo_pci_fixes(void);
104
105 |_______add_pci_fixes();
106
107 |_______if (reprogram) {
108 |_______|_______pci_reprogram();
109 |_______|_______undo_pci_fixes();
110 |_______|_______return;
111 |_______}
112
113 |_______/* setup device tree */
114 |_______pci_setup_tree();
115 |_______undo_pci_fixes();
116 }

add_pci_fixes()/undo_pci_fixes()
================================
Fix some hardware specific issues before enumerating or reprogramming the pci system, and undo the fix afterward. Currently, only fix for AMD-8111 LPC device is in the list.

pci_setup_tree()
================
This function was called to enumerate the pci system handed over by BIOS. Solaris kernel maintains a global variable in i86pc/os/pci_cfgspace.c

int pci_bios_maxbus;

It records the maximum number of pci bus on this hardware platform. During boot process, when mlsetup() got called,

i86pc/os/mlsetup.c
97 /*
98 * Setup routine called right before main(). Interposing this function
99 * before main() allows us to call it in a machine-independent fashion.
100 */
101 void
102 mlsetup(struct regs *rp)
103 {
... ...
162 |_______/*
163 |_______ * lgrp_init() and possibly cpuid_pass1() need PCI config
164 |_______ * space access
165 |_______ */
166 #if defined(__xpv)
167 |_______if (DOMAIN_IS_INITDOMAIN(xen_info))
168 |_______|_______pci_cfgspace_init();
169 #else
170 |_______pci_cfgspace_init();
171 #endif

pci_cfgspace_init() will prepare the pci system for configuration space access. The call stack is:

i86pc/os/pci_cfgspace.c
85 void
86 pci_cfgspace_init(void)
87 {
88 |_______mutex_init(&pcicfg_mutex, NULL, MUTEX_SPIN,
89 |_______ (ddi_iblock_cookie_t)ipltospl(15));
90 |_______mutex_init(&pcicfg_chipset_mutex, NULL, MUTEX_SPIN,
91 |_______ (ddi_iblock_cookie_t)ipltospl(15));
92 |_______if (!pci_check()) {
93 |_______|_______mutex_destroy(&pcicfg_mutex);
94 |_______|_______mutex_destroy(&pcicfg_chipset_mutex);
95 |_______}
96 }

i86pc/os/pci_cfgspace.c
98 /*
99 * This code determines if this system supports PCI and which
100 * type of configuration access method is used
101 */
102
103 static int
104 pci_check(void)
105 {
... ...
142 |_______pci_bios_cfg_type = pci_check_bios();
... ...

i86pc/os/pci_cfgspace.c
202 static int
203 pci_check_bios(void)
204 {
... ...
228 |_______pci_bios_mech = (ax & 0x3);
229 |_______pci_bios_vers = regs.ebx.word.bx;
230 |_______pci_bios_maxbus = (regs.ecx.word.cx & 0xff);

pci_bios_maxbus is retrieved from bios here! The logic described here are compiled into CORE_OBJ, and will be called before main() during system boot up.

To be continue ...