diff mbox

[dpdk-dev] pci/uio: enable prefetchable resources mapping

Message ID 1517447902-4166-1-git-send-email-changpeng.liu@intel.com (mailing list archive)
State Rejected, archived
Delegated to: Thomas Monjalon
Headers

Checks

Context Check Description
ci/checkpatch success coding style OK
ci/Intel-compilation success Compilation OK

Commit Message

Liu, Changpeng Feb. 1, 2018, 1:18 a.m. UTC 
  For PCI prefetchable resources, Linux will create a
write combined file as well, the library will try
to map resourceX_wc file first, if the file does
not exist, then it will map resourceX as usual.

Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
---
 drivers/bus/pci/linux/pci_uio.c | 19 ++++++++++++++-----
 1 file changed, 14 insertions(+), 5 deletions(-)

Comments

Bruce Richardson Feb. 1, 2018, 9:59 a.m. UTC | #1 
On Thu, Feb 01, 2018 at 09:18:22AM +0800, Changpeng Liu wrote:
> For PCI prefetchable resources, Linux will create a
> write combined file as well, the library will try
> to map resourceX_wc file first, if the file does
> not exist, then it will map resourceX as usual.
> 
> Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
> ---
>  drivers/bus/pci/linux/pci_uio.c | 19 ++++++++++++++-----
>  1 file changed, 14 insertions(+), 5 deletions(-)
>

Hi,

Given the lack of ordering guarantees with write-combined memory, I
would have thought that this is very risky to do without a complete set
of changes inside the PMDs to add in the necessary memory barriers to
ensure ordering of operations to the BARs.  Therefore, instead of
mapping one file or another, I think the change should be made to map
*both* in DPDK if available. Then each driver can chose whether to write
a given device register using uncacheable memory type or write-combining
memory type + any appropriate barriers.

For example, with many NICs the initialization of the device involves
many register writes in a pretty defined order, so wc operations are
probably to suitable as performance is not a concern. However, for data
path operations, a driver may chose to use wc memory for the occasional
device writes there, for performance reasons.

Regards,
/Bruce
Andrew Rybchenko Feb. 1, 2018, 10:08 a.m. UTC | #2 
On 02/01/2018 12:59 PM, Bruce Richardson wrote:
> On Thu, Feb 01, 2018 at 09:18:22AM +0800, Changpeng Liu wrote:
>> For PCI prefetchable resources, Linux will create a
>> write combined file as well, the library will try
>> to map resourceX_wc file first, if the file does
>> not exist, then it will map resourceX as usual.
>>
>> Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
>> ---
>>   drivers/bus/pci/linux/pci_uio.c | 19 ++++++++++++++-----
>>   1 file changed, 14 insertions(+), 5 deletions(-)
>>
> Hi,
>
> Given the lack of ordering guarantees with write-combined memory, I
> would have thought that this is very risky to do without a complete set
> of changes inside the PMDs to add in the necessary memory barriers to
> ensure ordering of operations to the BARs.  Therefore, instead of
> mapping one file or another, I think the change should be made to map
> *both* in DPDK if available. Then each driver can chose whether to write
> a given device register using uncacheable memory type or write-combining
> memory type + any appropriate barriers.
>
> For example, with many NICs the initialization of the device involves
> many register writes in a pretty defined order, so wc operations are
> probably to suitable as performance is not a concern. However, for data
> path operations, a driver may chose to use wc memory for the occasional
> device writes there, for performance reasons.

+1

I think so too that it would be useful to have both mappings available and
allow driver to choose which one to use.
diff mbox

Patch

diff --git a/drivers/bus/pci/linux/pci_uio.c b/drivers/bus/pci/linux/pci_uio.c
index d423e4b..2957a5f 100644
--- a/drivers/bus/pci/linux/pci_uio.c
+++ b/drivers/bus/pci/linux/pci_uio.c
@@ -293,7 +293,7 @@ 
 
 	/* update devname for mmap  */
 	snprintf(devname, sizeof(devname),
-			"%s/" PCI_PRI_FMT "/resource%d",
+			"%s/" PCI_PRI_FMT "/resource%d_wc",
 			rte_pci_get_sysfs_path(),
 			loc->domain, loc->bus, loc->devid,
 			loc->function, res_idx);
@@ -307,13 +307,22 @@ 
 	}
 
 	/*
-	 * open resource file, to mmap it
+	 * open prefetchable resource file first, try to mmap it
 	 */
 	fd = open(devname, O_RDWR);
 	if (fd < 0) {
-		RTE_LOG(ERR, EAL, "Cannot open %s: %s\n",
-				devname, strerror(errno));
-		goto error;
+		snprintf(devname, sizeof(devname),
+				"%s/" PCI_PRI_FMT "/resource%d",
+				rte_pci_get_sysfs_path(),
+				loc->domain, loc->bus, loc->devid,
+				loc->function, res_idx);
+		/* then try to map resource file */
+		fd = open(devname, O_RDWR);
+		if (fd < 0) {
+			RTE_LOG(ERR, EAL, "Cannot open %s: %s\n",
+					devname, strerror(errno));
+			goto error;
+		}
 	}
 
 	/* try mapping somewhere close to the end of hugepages */