PCI EP driver support MSI doorbell from host
From: | Frank Li <Frank.Li-AT-nxp.com> | |
To: | maz-AT-kernel.org, tglx-AT-linutronix.de, robh+dt-AT-kernel.org, krzysztof.kozlowski+dt-AT-linaro.org, shawnguo-AT-kernel.org, s.hauer-AT-pengutronix.de, kw-AT-linux.com, bhelgaas-AT-google.com | |
Subject: | [PATCH v4 0/4] PCI EP driver support MSI doorbell from host | |
Date: | Fri, 12 Aug 2022 16:52:38 -0500 | |
Message-ID: | <20220812215242.2255824-1-Frank.Li@nxp.com> | |
Cc: | kernel-AT-vger.kernel.org, devicetree-AT-vger.kernel.org, linux-arm-kernel-AT-lists.infradead.org, linux-pci-AT-vger.kernel.org, peng.fan-AT-nxp.com, aisheng.dong-AT-nxp.com, jdmason-AT-kudzu.us, kernel-AT-pengutronix.de, festevam-AT-gmail.com, linux-imx-AT-nxp.com, kishon-AT-ti.com, lorenzo.pieralisi-AT-arm.com, ntb-AT-lists.linux.dev, lznuaa-AT-gmail.com | |
Archive-link: | Article |
┌───────┐ ┌──────────┐ │ │ │ │ ┌─────────────┐ │ │ │ PCI Host │ │ MSI │◄┐ │ │ │ │ │ Controller │ │ │ │ │ │ └─────────────┘ └─┼───────┼──────────┼─Bar0 │ │ PCI │ │ Bar1 │ │ Func │ │ Bar2 │ │ │ │ Bar3 │ │ │ │ Bar4 │ │ ├─────────►│ │ └───────┘ └──────────┘ Many PCI controllers provided Endpoint functions. Generally PCI endpoint is hardware, which is not running a rich OS, like linux. But Linux also supports endpoint functions. PCI Host write BAR<n> space like write to memory. The EP side can't know memory changed by the Host driver. PCI Spec has not defined a standard method to do that. Only define MSI(x) to let EP notified RC status change. The basic idea is to trigger an IRQ when PCI RC writes to a memory address. That's what MSI controller provided. EP drivers just need to request a platform MSI interrupt, struct MSI_msg *msg will pass down a memory address and data. EP driver will map such memory address to one of PCI BAR<n>. Host just writes such an address to trigger EP side IRQ. If system have gic-its, only need update PCI EP side driver. But i.MX have not chip support gic-its yet. So we have to use MU to simulate a MSI controller. Although only 4 MSI IRQs are simulated, it matched vntb(pci-epf-vntb) network requirement. After enable MSI, ping delay reduce < 1ms from ~8ms IRQchip: imx mu worked as MSI controller: let imx mu worked as MSI controllers. Although IP is not design as MSI controller, we still can use it if limited IRQ number to 4. pcie: endpoint: pci-epf-vntb: add endpoint MSI support Based on ntb-next branch. https://github.com/jonmason/ntb/commits/ntb-next Using MSI as door bell registers This patch is totally independent on previous on. It can be applied to ntb-next seperately. i.MX EP function driver is upstreaming by Richard Zhu. Some dts change missed at this patches. below is reference dts change --- a/arch/arm64/boot/dts/freescale/imx8-ss-hsio.dtsi +++ b/arch/arm64/boot/dts/freescale/imx8-ss-hsio.dtsi @@ -160,5 +160,6 @@ pcieb_ep: pcie_ep@5f010000 { num-ib-windows = <6>; num-ob-windows = <6>; status = "disabled"; + MSI-parent = <&lsio_mu12>; }; --- a/arch/arm64/boot/dts/freescale/imx8-ss-lsio.dtsi +++ b/arch/arm64/boot/dts/freescale/imx8-ss-lsio.dtsi @@ -172,6 +172,19 @@ lsio_mu6: mailbox@5d210000 { status = "disabled"; }; + lsio_mu12: mailbox@5d270000 { + compatible = "fsl,imx6sx-mu-MSI"; + msi-controller; + interrupt-controller; + reg = <0x5d270000 0x10000>, /* A side */ + <0x5d300000 0x10000>; /* B side */ + reg-names = "a", "b"; + interrupts = <GIC_SPI 191 IRQ_TYPE_LEVEL_HIGH>; + power-domains = <&pd IMX_SC_R_MU_12A>, + <&pd IMX_SC_R_MU_12B>; + power-domain-names = "a", "b"; + }; + Change Log - Change from v3 to v4 Fixed dt-binding document according to Krzysztof Kozlowski's feedback Fixed irqchip-imx-mu-worked-as-msi-controller according to Marc Zyngier's comments. There are still two important points, which I am not sure. 1. clean irq_set_affinity after platform_msi_create_irq_domain. Some function, like platform_msi_write_msg() is static. so I have to set MSI_FLAG_USE_DEF_CHIP_OPS flags, which will set irq_set_affinity to default one. 2. about comments > + msi_data->msi_domain = platform_msi_create_irq_domain( > + of_node_to_fwnode(msi_data->pdev->dev.of_node), > + &imx_mu_msi_domain_info, > + msi_data->parent); "And you don't get an error due to the fact that you use the same fwnode for both domains without overriding the domain bus token?" I did not understand yet. Fixed static check warning, reported by Dan Carpenter pcie: endpoint: pci-epf-vntb: add endpoint MSI support - Change from v2 to v3 Fixed dt-binding docment check failure Fixed typo a cover letter. Change according Bjorn's comments at patch pcie: endpoint: pci-epf-vntb: add endpoint MSI support - from V1 to V2 Fixed fsl,mu-msi.yaml's problem Fixed irq-imx-mu-msi.c problem according Marc Zyngier's feeback Added a new patch to allow pass down .pm by IRQCHIP_PLATFORM_DRIVER_END -- 2.35.1