Solaris Source Insight: PCI bus driver moduls - npe Part 1

Mon Nov 9 16:17:38 CST 2009

We have seen how PCI bus probe enuerate and configure the PCI system. Let's go ahead with the details of PCI bus drivers. The PCI leaf device driver is function related, we will never touch the details of the fucntionality of a specific device.

[root@blu-nhm-ep:~]modinfo | grep PCI
15 fffffffffba46ce0 bfb0 - 1 pci_autoconfig (PCI BIOS interface)
37 fffffffffbacd3f0 ce28 183 1 npe (Host to PCIe nexus driver)
38 fffffffffbad95c8 5f50 - 1 pcihp (PCI nexus hotplug support)
40 fffffffffbae14f0 bb00 - 1 pcie (PCIE: PCI framework)
89 fffffffff7bff000 4c68 184 1 pcieb (PCIe to PCI nexus driver)
90 fffffffff7999000 1d68 84 1 pci_pci (PCI to PCI bridge nexus driver)


npe - Host to PCIe nexus driver
===============================

It's a bus driver registered with the following data.

[i86pc/io/pciex/npe.c]
149 struct dev_ops npe_ops = {
150 |_______DEVO_REV,|______|_______/* devo_rev */
151 |_______0,|_____|_______|_______/* refcnt */
152 |_______npe_info,|______|_______/* info */
153 |_______nulldev,|_______|_______/* identify */
154 |_______nulldev,|_______|_______/* probe */
155 |_______npe_attach,|____|_______/* attach */
156 |_______npe_detach,|____|_______/* detach */
157 |_______nulldev,|_______|_______/* reset */
158 |_______&npe_cb_ops,|___|_______/* driver operations */
159 |_______&npe_bus_ops,|__|_______/* bus operations */
160 |_______NULL,|__|_______|_______/* power */
161 |_______ddi_quiesce_not_needed,||_______/* quiesce */
162 };

Look at attach function firstly.

---- Question: ----
How to access configure space of a PCI device in drivers?

---- Answer: ----
You can use the standard DDI interfaces to access the configure space. The interfaces include at least the following functions.

ddi_regs_map_setup()
ddi_regs_map_free()
ddi_getX()

Here is a sample:

[i86pc/io/pciex/npe_misc.c]
330 void
331 npe_enable_htmsi_children(dev_info_t *dip)
332 {
333 |_______dev_info_t *cdip = ddi_get_child(dip);
334 |_______ddi_acc_handle_t cfg_hdl;
335
336 |_______for (; cdip != NULL; cdip = ddi_get_next_sibling(cdip)) {
337 |_______|_______if (pci_config_setup(cdip, &cfg_hdl) != DDI_SUCCESS) {
338 |_______|_______|_______cmn_err(CE_NOTE, "!npe_enable_htmsi_children: "
339 |_______|_______|_______ "pci_config_setup failed for %s",
340 |_______|_______|_______ ddi_node_name(cdip));
341 |_______|_______}
342
343 |_______|_______(void) npe_enable_htmsi(cfg_hdl);
344 |_______|_______pci_config_teardown(&cfg_hdl);
345 |_______}
346 }

In this function, pci_config_setup() is called firstly to set up the pci configure space mapping.

323 |_______reg = pci_config_get16(cfg_hdl, ptr + PCI_CAP_ID_REGS_OFF);
324 |_______reg |= PCI_HTCAP_MSIMAP_ENABLE;
325
326 |_______pci_config_put16(cfg_hdl, ptr + PCI_CAP_ID_REGS_OFF, reg);

in npe_enable_htmsi(), pci configure space access functions, such as pci_config_get16() are able to be called.

For more deepness, how does the code work?

[common/os/sunpci.c]
35 int
36 pci_config_setup(dev_info_t *dip, ddi_acc_handle_t *handle)
37 {
38 |_______caddr_t|cfgaddr;
39 |_______ddi_device_acc_attr_t attr;
40
41 |_______attr.devacc_attr_version = DDI_DEVICE_ATTR_V0;
42 |_______attr.devacc_attr_endian_flags = DDI_STRUCTURE_LE_ACC;
43 |_______attr.devacc_attr_dataorder = DDI_STRICTORDER_ACC;
44
45 |_______/* Check for fault management capabilities */
46 |_______if (DDI_FM_ACC_ERR_CAP(ddi_fm_capable(dip))) {
47 |_______|_______attr.devacc_attr_version = DDI_DEVICE_ATTR_V1;
48 |_______|_______attr.devacc_attr_access = DDI_FLAGERR_ACC;
49 |_______}
50
51 |_______return (ddi_regs_map_setup(dip, 0, &cfgaddr, 0, 0, &attr, handle));
52 }

In pci_config_setup(), ddi_regs_map_setup() is called to set up a mapping for a register address space. If you read the man page for this fucntion, you will find the parameter description.

int ddi_regs_map_setup(dev_info_t *dip, uint_t rnumber, caddr_t *addrp,
offset_t offset, offset_t len, ddi_device_acc_attr_t *accattrp,
ddi_acc_handle_t *handlep);

PARAMETERS
dip Pointer to the device's dev_info structure.

rnumber Index number to the register address space set.

addrp A platform-dependent value that, when added to
an offset that is less than or equal to the len
parameter (see below), is used for the dev_addr
argument to the ddi_get, ddi_mem_get, and
ddi_io_get/put routines.

offset Offset into the register address space.

len Length to be mapped. If both len and offset are 0, the
entire space is mapped.

accattrp Pointer to a device access attribute structure
of this mapping (see ddi_device_acc_attr(9S)).

handlep Pointer to a data access handle.

So line 51 is to set up mapping of the whole first register address space with the "attr" attribution. If you check add_reg_props() which was called during the first pass of pci enumeration,

[intel/io/pci/pci_boot.c]
2297 /*
2298 * Add the "reg" and "assigned-addresses" property
2299 */
2300 static int
2301 add_reg_props(dev_info_t *dip, uchar_t bus, uchar_t dev, uchar_t func,
2302 int config_op, int pciide)

You will find the first register set was encoded with bus, device and function numbers.

[intel/io/pci/pci_boot.c]
2327 |_______devloc = (uint_t)bus << 16 | (uint_t)dev << 11 | (uint_t)func << 8;
2328 |_______regs[0].pci_phys_hi = devloc;
2329 |_______nreg = 1;|______/* rest of regs[0] is all zero */

So, how ddi_regs_map_setup() manages to setup the mapping? It's simple as just call the bus driver's bus operation.

[common/os/sunddi.c]
7612 |_______result = ddi_map(dip, &mr, offset, len, addrp);

[common/os/sunddi.c]
134 int
135 ddi_map(dev_info_t *dp, ddi_map_req_t *mp, off_t offset,
136 off_t len, caddr_t *addrp)
137 {
138 |_______dev_info_t *pdip;
139
140 |_______ASSERT(dp);
141 |_______pdip = (dev_info_t *)DEVI(dp)->devi_parent;
142 |_______return ((DEVI(pdip)->devi_ops->devo_bus_ops->bus_map)(pdip,
143 |_______ dp, mp, offset, len, addrp));
144 }

Oh, PCI bus operations are defined in the nexus driver such as npe. Let's check how npe handles it.

In summary, npe bus map function support three sorts of mappings: IO, Mem and configure space. If the request is for IO or memory, npe_bus_map() relies on its parent's (rootnex) bus map operation. If it's a configure space request, it depends on whether the device support MMIO of configure space.

[i86pc/io/pciex/npe_misc.c]
285 /*
286 * Checks to see if MMCFG is supported.
287 * Returns: TRUE if MMCFG is supported, FALSE if not.
288 *
289 * If a device is attached to a parent whose "dev_type" is "pciex",
290 * the device will support MMCFG access. Otherwise, use legacy IOCFG access.
291 *
292 * Enable Legacy PCI config space access for AMD K8 north bridges.
293 *|_____Host bridge: AMD HyperTransport Technology Configuration
294 *|_____Host bridge: AMD Address Map
295 *|_____Host bridge: AMD DRAM Controller
296 *|_____Host bridge: AMD Miscellaneous Control
297 * These devices do not support MMCFG access.
298 */
299 boolean_t
300 npe_is_mmcfg_supported(dev_info_t *dip)
301 {
302 |_______int vendor_id, device_id;
303
304 |_______vendor_id = ddi_prop_get_int(DDI_DEV_T_ANY, dip, DDI_PROP_DONTPASS,
305 |_______ "vendor-id", -1);
306 |_______device_id = ddi_prop_get_int(DDI_DEV_T_ANY, dip, DDI_PROP_DONTPASS,
307 |_______ "device-id", -1);
308
309 |_______return !(npe_child_is_pci(dip) ||
310 |_______ IS_BAD_AMD_NTBRIDGE(vendor_id, device_id));
311 }

If it has MMIO supported configure space, it will be treated as a memory mapping request. Otherwise, the lagacy methods are used.

[i86pc/io/pciex/npe.c]
386 /*
387 * Configure the access handle for standard configuration space
388 * access (see pci_fm_acc_setup for code that initializes the
389 * access-function pointers).
390 */
391 static int
392 npe_setup_std_pcicfg_acc(dev_info_t *rdip, ddi_map_req_t *mp,
393 ddi_acc_hdl_t *hp, off_t offset, off_t len)
394 {
395 |_______int ret;
396
397 |_______if ((ret = pci_fm_acc_setup(hp, offset, len)) ==
398 |_______ DDI_SUCCESS) {
399 |_______|_______if (DDI_FM_ACC_ERR_CAP(ddi_fm_capable(rdip)) &&
400 |_______|_______ mp->map_handlep->ah_acc.devacc_attr_access
401 |_______|_______ != DDI_DEFAULT_ACC) {
402 |_______|_______|_______ndi_fmc_insert(rdip, ACC_HANDLE,
403 |_______|_______|_______ (void *)mp->map_handlep, NULL);
404 |_______|_______}
405 |_______}
406 |_______return (ret);
407 }

in pci_fm_acc_setup(), the get/put function pointers are assigned. It always set cautious mechanism for config space gets, while set those for put according to the attributes specified in devacc_attr_access of ddi_device_acc_attr structure.

[man ddi_device_acc_attr]
152 The values defined for devacc_attr_access are:
153
154 DDI_DEFAULT_ACC If an I/O fault occurs, the system will
155 take the default action, which might be
156 to panic.
157
158
159 DDI_FLAGERR_ACC Using this value indicates that the
160 driver is hardened: able to cope with
161 the incorrect results of I/O operations
162 that might result from an I/O fault. The
163 value also indicates that the driver
164 will use ddi_fm_acc_err_get(9F) to check
165 access handles for faults on a regular
166 basis.
167
168 If possible, the system should not panic
169 on such an I/O fault, but should instead
170 mark the I/O handle through which the
171 access was made as having faulted.
172
173 This value is advisory: it tells the
174 system that the driver can continue in
175 the face of I/O faults. The value does
176 not guarantee that the system will not
177 panic, as that depends on the nature of
178 the fault and the capabilities of the
179 system. It is quite legitimate for an
180 implementation to ignore this flag and
181 panic anyway.
182
183
184 DDI_CAUTIOUS_ACC This value indicates that an I/O fault
185 is anticipated and should be handled as
186 gracefully as possible. For example, the
187 framework should not print a console
188 message.


[i86pc/io/pci/pci_common.c]
1151 |_______/*
1152 |_______ * always use cautious mechanism for config space gets
1153 |_______ */
1154 |_______ap->ahi_get8 = i_ddi_caut_get8;
1155 |_______ap->ahi_get16 = i_ddi_caut_get16;
1156 |_______ap->ahi_get32 = i_ddi_caut_get32;
1157 |_______ap->ahi_get64 = i_ddi_caut_get64;
1158 |_______ap->ahi_rep_get8 = i_ddi_caut_rep_get8;
1159 |_______ap->ahi_rep_get16 = i_ddi_caut_rep_get16;
1160 |_______ap->ahi_rep_get32 = i_ddi_caut_rep_get32;
1161 |_______ap->ahi_rep_get64 = i_ddi_caut_rep_get64;
1162 |_______if (hp->ah_acc.devacc_attr_access == DDI_CAUTIOUS_ACC) {
1163 |_______|_______ap->ahi_put8 = i_ddi_caut_put8;
1164 |_______|_______ap->ahi_put16 = i_ddi_caut_put16;
1165 |_______|_______ap->ahi_put32 = i_ddi_caut_put32;
1166 |_______|_______ap->ahi_put64 = i_ddi_caut_put64;
1167 |_______|_______ap->ahi_rep_put8 = i_ddi_caut_rep_put8;
1168 |_______|_______ap->ahi_rep_put16 = i_ddi_caut_rep_put16;
1169 |_______|_______ap->ahi_rep_put32 = i_ddi_caut_rep_put32;
1170 |_______|_______ap->ahi_rep_put64 = i_ddi_caut_rep_put64;
1171 |_______} else {
1172 |_______|_______ap->ahi_put8 = pci_config_wr8;
1173 |_______|_______ap->ahi_put16 = pci_config_wr16;
1174 |_______|_______ap->ahi_put32 = pci_config_wr32;
1175 |_______|_______ap->ahi_put64 = pci_config_wr64;
1176 |_______|_______ap->ahi_rep_put8 = pci_config_rep_wr8;
1177 |_______|_______ap->ahi_rep_put16 = pci_config_rep_wr16;
1178 |_______|_______ap->ahi_rep_put32 = pci_config_rep_wr32;
1179 |_______|_______ap->ahi_rep_put64 = pci_config_rep_wr64;
1180 |_______}

For cautious mechanism, the configure space access request will be handled as a bus_ctl request.

[common/os/sunddi.c]
682 /*
683 * Request bus_ctl parent to handle a bus_ctl request
684 *
685 * (The sparc version is in sparc_ddi.s)
686 */
687 int
688 ddi_ctlops(dev_info_t *d, dev_info_t *r, ddi_ctl_enum_t op, void *a, void *v)
689 {
690 |_______int (*fp)();
691
692 |_______if (!d || !r)
693 |_______|_______return (DDI_FAILURE);
694
695 |_______if ((d = (dev_info_t *)DEVI(d)->devi_bus_ctl) == NULL)
696 |_______|_______return (DDI_FAILURE);
697
698 |_______fp = DEVI(d)->devi_ops->devo_bus_ops->bus_ctl;
699 |_______return ((*fp)(d, r, op, a, v));
700 }

In npe_bus_ctl(), the folowing path deals with pci configure space put/get.

[i86pc/io/pciex/npe.c]
775 |_______case DDI_CTLOPS_PEEK:
776 |_______case DDI_CTLOPS_POKE:
777 |_______|_______return (pci_common_peekpoke(dip, rdip, ctlop, arg, result));

At last, the request is handled with one of the following functions.

pci_getb_func()
pci_getw_func()
pci_getl_func()
pci_putb_func()
pci_putw_func()
pci_putl_func()

These interfaces are defined in ./i86pc/os/pci_cfgspace.c.

[./i86pc/os/pci_cfgspace.c]
56 /*
57 * These function pointers lead to the actual implementation routines
58 * for configuration space access. Normally they lead to either the
59 * pci_mech1_* or pci_mech2_* routines, but they can also lead to
60 * routines that work around chipset bugs.
61 */
62 uint8_t (*pci_getb_func)(int bus, int dev, int func, int reg);
63 uint16_t (*pci_getw_func)(int bus, int dev, int func, int reg);
64 uint32_t (*pci_getl_func)(int bus, int dev, int func, int reg);
65 void (*pci_putb_func)(int bus, int dev, int func, int reg, uint8_t val);
66 void (*pci_putw_func)(int bus, int dev, int func, int reg, uint16_t val);
67 void (*pci_putl_func)(int bus, int dev, int func, int reg, uint32_t val);

These gobal pointers were initialized at pci_check().

[./i86pc/os/pci_cfgspace.c]
98 /*
99 * This code determines if this system supports PCI and which
100 * type of configuration access method is used
101 */
102
103 static int
104 pci_check(void)
105 {

which is called in mlsetup() at boot time. mlsetup() contains routines called right before main(). Interposing this function before main() allows us to call it in a machine-independent fashion.

Okay, back to the topic. For non-cautious mechanism, it will directly call pci configure space access methods.

pci_getb_func()
pci_getw_func()
pci_getl_func()
pci_putb_func()
pci_putw_func()
pci_putl_func()

That means, in current kernel there is no difference between the different configure access attribute in "devacc_attr_access".

---- End of Q/A ----

To be continued ... ....