Solaris Source Insight: PCI system implementation - Part 4

continue with pci_reprogram()

(b) Fix-up unit-address assignments for peer bus, and create a devcache for this assignment
The unit-address is stored in the first integer of "reg" property. "unit-address" is assigned from 0 and increased for each root bus in the system. "Non-root bus" is equal to "peer bus" here.

(c) root-bus resource discovery
c.1 find resources associated with this root bus
The resources for a root bus can be enumerated in these ways: (i) call ACPI "_CRS" method to find the current resource settings for the device (root bus); (ii) search them in PCI Hot-Plug Resource Table starting at 0xF0000 if that exists; (iii) search them in MP spec table if that exists. The implementation relies on the methods in order and returns if one is successful. Resources include IO range, memory range, prefetchable memory range and bus number range.

The resources are stored in pci_bus_res[] array:

intel/io/pci/pci_boot.c
1406 |_______/* scan BIOS structures */
1407 |_______pci_bus_res[bus].pmem_avail = find_bus_res(bus, PREFETCH_TYPE);
1408 |_______pci_bus_res[bus].mem_avail = find_bus_res(bus, MEM_TYPE);
1409 |_______pci_bus_res[bus].io_avail = find_bus_res(bus, IO_TYPE);
1410 |_______pci_bus_res[bus].bus_avail = find_bus_res(bus, BUSRANGE_TYPE);

c.2 extend pci_bus_res[bus].sub_bus according to bus ranges in pci_bus_res[bus].bus_avail

c.3 code comments
intel/io/pci/pci_boot.c
1430 |_______|_______/*
1431 |_______|_______ * Special treatment of bus 0:
1432 |_______|_______ * If no IO/MEM resource from ACPI/MPSPEC/HRT, copy
1433 |_______|_______ * pcimem from boot and make I/O space the entire range
1434 |_______|_______ * starting at 0x100.
1435 |_______|_______ */

c.4 code comments
intel/io/pci/pci_boot.c
1444 |_______/*
1445 |_______ * Create 'ranges' property here before any resources are
1446 |_______ * removed from the resource lists
1447 |_______ */

Question:
We saw many ACPI-related funcation calls on the way. It was assumed that root bus device node has been built a 1-1 map with PCI root complex object in ACPI name space. Peaple can use acpica_get_handle() to retrieve the ACPI handle from the dip and acpica_get_devinfo() to retrive dip vice vasa. But when did this relationship built?

Answer:
The system does maintain the mapping of a node in device tree and an object in ACPI name space, if they present the same physical devices. For a device node, "acpinamespace" property is created, which is the string of the path of an ACPI object in ACPI name space; while, in acpi object struct, there is data related to device node dip. Global variable "d2a_done" marks if the mapping between them has been built.

101 static int d2a_done = 0;

If not, scan_d2a_map() will be called to establish the map.

[intel/io/acpica/osl.c]
1811 |_______if (!d2a_done)
1812 |_______|_______scan_d2a_map();

End of the answer...

(d) Remove used PCI and ISA resources from bus resource map
[intel/io/pci/pci_boot.c]
1321 |_______|_______memlist_remove_list(&pci_bus_res[bus].io_avail,
1322 |_______|_______ pci_bus_res[bus].io_used);
1323 |_______|_______memlist_remove_list(&pci_bus_res[bus].mem_avail,
1324 |_______|_______ pci_bus_res[bus].mem_used);
1325 |_______|_______memlist_remove_list(&pci_bus_res[bus].pmem_avail,
1326 |_______|_______ pci_bus_res[bus].pmem_used);
1327 |_______|_______memlist_remove_list(&pci_bus_res[bus].mem_avail,
1328 |_______|_______ pci_bus_res[bus].pmem_used);
1329 |_______|_______memlist_remove_list(&pci_bus_res[bus].pmem_avail,
1330 |_______|_______ pci_bus_res[bus].mem_used);
1331
1332 |_______|_______memlist_remove_list(&pci_bus_res[bus].io_avail,
1333 |_______|_______ isa_res.io_used);
1334 |_______|_______memlist_remove_list(&pci_bus_res[bus].mem_avail,
1335 |_______|_______ isa_res.mem_used);

(e) Reserve <1M space
[intel/io/pci/pci_boot.c]
1337 |_______|_______/*
1338 |_______|_______ * 3. Exclude <1M address range here in case below reserved
1339 |_______|_______ * ranges for BIOS data area, ROM area etc are wrongly reported
1340 |_______|_______ * in ACPI resource producer entries for PCI root bus.
1341 |_______|_______ * |____00000000 - 000003FF|____RAM
1342 |_______|_______ * |____00000400 - 000004FF|____BIOS data area
1343 |_______|_______ * |____00000500 - 0009FFFF|____RAM
1344 |_______|_______ * |____000A0000 - 000BFFFF|____VGA RAM
1345 |_______|_______ * |____000C0000 - 000FFFFF|____ROM area
1346 |_______|_______ */

(f) add bus-range property for root/peer bus nodes
[intel/io/pci/pci_boot.c]
1355 |_______/* add bus-range property for root/peer bus nodes */
1356 |_______for (i = 0; i <= pci_bios_maxbus; i++) {
1357 |_______|_______/* create bus-range property on root/peer buses */
1358 |_______|_______if (pci_bus_res[i].par_bus == (uchar_t)-1)
1359 |_______|_______|_______add_bus_range_prop(i);
1360
1361 |_______|_______/* setup bus range resource on each bus */
1362 |_______|_______setup_bus_res(i);
1363 |_______}

(g) get user option "pci-reprog" and set pci_reconfig according to the value.

(h) Remove the resources which are already used by devices under a subtractive bridge from the bus's resources lists

619 /*
620 * Remove the resources which are already used by devices under a subtractive
621 * bridge from the bus's resources lists, because they're not available, and
622 * shouldn't be allocated to other buses. This is necessary because tracking
623 * resources for subtractive bridges is not complete. (Subtractive bridges only
624 * track some of their claimed resources, not "the rest of the address space" as
625 * they should, so that allocation to peer non-subtractive PPBs is easier. We
626 * need a fully-capable global resource allocator).
627 */

What's PCI subtractive decoding?

This description of a PCI bus transaction assumes that both initiating and target devices are PCI devices. However, even today most PCs require an ISA or other expansion bus in order to be able to install legacy peripherals. This is achieved using a PCI to expansion bus bridge. In this configuration, the PCI bus is the primary bus, and the legacy bus is the secondary bus.
If an initiator begins a transaction for a device that is on a secondary expansion bus, no PCI device will acknowledge that it is the target. One of two things could happen next. The PCI to expansion bus bridge could claim the transaction on behalf of its own peripherals; however, this would require that the bridge be programmed with the addresses of all the devices on the other side of it. This is a possibility in the case of MCA, EISA and plug-and-play ISA boards. However, ordinary ISA boards are not plug-and-play so a PCI to ISA bridge can have no knowledge of what memory and I/O addresses are on the ISA bus.
The method normally used to handle transactions destined for an ISA expansion bus is to use a process of subtractive decoding, or "if nobody else wants it, it must be for me." The expansion bus bridge claims the transaction if it is for a memory address in the first 16MB of address space or an I/O port address in the first 64KB, and no PCI device has claimed the transaction within a set delay period. The delay period depends on the speed of the PCI device address decoders, which can take from one to three clock cycles to respond with an acknowledgement.
The speed, and hence the length of the delay, is determined during the power on configuration process. The presence of even one slow device will require a delay of four bus clock cycles for every ISA bus transfer. This will degrade the performance of peripherals on the ISA expansion bus.

Citing from "How the PCI Bus Works"

(i) reprogram the non-subtractive PPB, root PCI bus will not reconfigured

intel/io/pci/pci_boot.c
865 /*
866 * Assign valid resources to unconfigured pci(e) bridges. We are trying
867 * to reprogram the bridge when its
868 * |____|_______i) SECBUS == SUBBUS|__||
869 * |____|_______ii) IOBASE > IOLIM|____||
870 * |____|_______iii) MEMBASE > MEMLIM
871 * This must be done after one full pass through the PCI tree to collect
872 * all BIOS-configured resources, so that we know what resources are
873 * free and available to assign to the unconfigured PPBs.
874 */
875 static void
876 fix_ppb_res(uchar_t secbus, boolean_t prog_sub)
877 {

Questions:
How to retrive PCI BDF (bus, device, function) numbers from a PCI dev_info?

Answer:
PCI BDF numbers are encoded in the first integer of "reg" property.

[intel/io/pci/pci_boot.c]
2327 |_______devloc = (uint_t)bus << 16 | (uint_t)dev << 11 | (uint_t)func << 8;
2328 |_______regs[0].pci_phys_hi = devloc;

[intel/io/pci/pci_boot.c]
903 |_______rv = ddi_prop_lookup_int_array(DDI_DEV_T_ANY, dip, DDI_PROP_DONTPASS,
904 |_______ "reg", &regp, &reglen);
905 |_______if (rv != DDI_PROP_SUCCESS || reglen == 0)
906 |_______|_______return;
907 |_______physhi = regp[0];
908 |_______ddi_prop_free(regp);
909
910 |_______func = (uchar_t)PCI_REG_FUNC_G(physhi);
911 |_______dev = (uchar_t)PCI_REG_DEV_G(physhi);
912 |_______bus = (uchar_t)PCI_REG_BUS_G(physhi);

This is how acpica_get_bdf() implemented in

[intel/io/acpica/osl.c]
1774 /*
1775 * Return bus/dev/fn for PCI dip (note: not the parent "pci" node).
1776 */
1777 int
1778 acpica_get_bdf(dev_info_t *dip, int *bus, int *device, int *func)
1779 {

But, don't trust this function if you are not sure the passed dip is really for a PCI device. For example, if you pass a device node under "isa", acpica_get_bdf() will return a bdf of (0,0,0).

End of answer ...

The following things have been done in fix_ppb_res().
(i) If we have a Cardbus bridge, but no bus space, Try to find and allocate a bus-range starting at subbus+1 from the parent of the PPB.
(ii) Calculate the io size and mem size. Let the code say what's the algorithm.

968 |_______/*
969 |_______ * Calculate required IO size and alignment
970 |_______ * If bus io_size is zero, we are going to assign 512 bytes per bus,
971 |_______ * otherwise, we'll choose the maximum value of such calculation and
972 |_______ * bus io_size. The size needs to be 4K aligned.
973 |_______ *
974 |_______ * We calculate alignment as the largest power of two less than the
975 |_______ * the sum of all children's IO size requirements, because this will
976 |_______ * align to the size of the largest child request within that size
977 |_______ * (which is always a power of two).
978 |_______ */
979 |_______io_size = (subbus - secbus + 1) * 0x200;
980 |_______if (io_size < pci_bus_res[secbus].io_size)
981 |_______|_______io_size = pci_bus_res[secbus].io_size;
982 |_______io_size = P2ROUNDUP(io_size, PPB_IO_ALIGNMENT);
983 |_______io_align = io_size;
984 |_______P2LE(io_align);
985
986 |_______/*
987 |_______ * Calculate required MEM size and alignment
988 |_______ * If bus mem_size is zero, we are going to assign 1M bytes per bus,
989 |_______ * otherwise, we'll choose the maximum value of such calculation and
990 |_______ * bus mem_size. The size needs to be 1M aligned.
991 |_______ *
992 |_______ * For the alignment, refer to the I/O comment above.
993 |_______ */
994 |_______mem_size = (subbus - secbus + 1) * PPB_MEM_ALIGNMENT;
995 |_______if (mem_size < pci_bus_res[secbus].mem_size) {
996 |_______|_______mem_size = pci_bus_res[secbus].mem_size;
997 |_______|_______mem_size = P2ROUNDUP(mem_size, PPB_MEM_ALIGNMENT);
998 |_______}
999 |_______mem_align = mem_size;
1000 |_______P2LE(mem_align);

(iii) do the real reprogram job for unconfigured PPB.

(j) Reprogram the subtractive PPB
At this time, all its siblings should have got their resources already.

(k) Configure the devices/functions attached to this bus.
enumerate_bus_devs(i, CONFIG_NEW);

The actual reprogram logic is in add_reg_props()

1509 |_______|_______|_______if (entry->reprogram ||
1510 |_______|_______|_______ pci_bus_res[bus].io_reprogram ||
1511 |_______|_______|_______ pci_bus_res[bus].mem_reprogram) {
1512 |_______|_______|_______|_______/* reprogram device(s) */
1513 |_______|_______|_______|_______(void) add_reg_props(entry->dip, bus,
1514 |_______|_______|_______|_______ entry->dev, entry->func, CONFIG_NEW, 0);
1515 |_______|_______|_______}

(l) All dev programmed, so we can create available prop

Summary, pci_reprogram() configures all the PCI devices/functions which were not configured by BIOS.

Question:
I only saw PCI resource reprogramed, is there any reconfiguration for bus numbering?

Answer:
Yes, it's possible that pci_reprogram() could renumber PCI buses by modifying the secondary and suboridinary bus numbers in registers of a PPB (PCI-to-PCI bridges).


934 |_______/*
935 |_______ * If we have a Cardbus bridge, but no bus space
936 |_______ */
937 |_______if (pci_bus_res[secbus].num_cbb != 0 &&
938 |_______ pci_bus_res[secbus].bus_avail == NULL) {
939 |_______|_______uchar_t range;
940
941 |_______|_______/* normally there are 2 buses under a cardbus bridge */
942 |_______|_______range = pci_bus_res[secbus].num_cbb * 2;
943
944 |_______|_______/*
945 |_______|_______ * Try to find and allocate a bus-range starting at subbus+1
946 |_______|_______ * from the parent of the PPB.
947 |_______|_______ */
948 |_______|_______for (; range != 0; range--) {
949 |_______|_______|_______if (memlist_find_with_startaddr(
950 |_______|_______|_______ &pci_bus_res[parbus].bus_avail,
951 |_______|_______|_______ subbus + 1, range, 1) != NULL)
952 |_______|_______|_______|_______break; /* find bus range resource at parent */
953 |_______|_______}
954 |_______|_______if (range != 0) {
955 |_______|_______|_______memlist_insert(&pci_bus_res[secbus].bus_avail,
956 |_______|_______|_______ subbus + 1, range);
957 |_______|_______|_______subbus = subbus + range;
958 |_______|_______|_______pci_bus_res[secbus].sub_bus = subbus;
959 |_______|_______|_______pci_putb(bus, dev, func, PCI_BCNF_SUBBUS, subbus);
960 |_______|_______|_______add_bus_range_prop(secbus);
961
962 |_______|_______|_______cmn_err(CE_NOTE, "!reprogram bus-range on ppb"
963 |_______|_______|_______ "[%x/%x/%x]: %x ~ %x\n", bus, dev, func,
964 |_______|_______|_______ secbus, subbus);
965 |_______|_______}
966 |_______}