Re: [SPDK] memory region requirement for spdk_mem_register

-----Original Message-----
From: SPDK [mailto:spdk-bounces@lists.01.org] On Behalf Of Feng Li
Sent: Thursday, June 27, 2019 10:07 PM
To: spdk(a)lists.01.org
Subject: [SPDK] memory region requirement for spdk_mem_register
Hi,
I was trying to share DMA memory between two processes with SPDK in this way
but failed:
* Process A creates a memory region ( 2M aligned pinned memory from
anonymous mmap with MAP_HUGETLB)
* Process B then registers the above memory region from A, so that SPDK can
use it as DMA memory for nvme driver.
My current approach is (if there are better ways please let me know~):
1. Use a kernel module to store the starting physical address of this region
shared from process A.
2. process B then also uses the kernel module to map this range of physical
region to its virtual space (phys_addr->virt_addr_2) (using
remap_pfn_range)
3. process B uses spdk_mem_register to register virt_addr_2.

Why not just create files on a hugetlbfs filesystem? That's what DPDK does, and then
any process can mmap them. You don't need your own kernel module then.

My questions are:
i. The remap_pfn_range in step 2 only gives me 4K mappings(but they are
actually backed by hugepages allocated in process A). Does this violate "The
memory region must map to pinned huge pages (2MB or greater"
requirement in spdk_mem_register? (since the mappings are in 4K). This
thread(https://lists.01.org/pipermail/spdk/2018-July/002143.html) might be
related.

SPDK's memory maps only go down to 2MB granularity today, mostly for space reasons.
But if you have 2MB chunks registered in 4K space, just register a 2MB aligned chunk and
it will be fine.

ii. Currently the registration failed with -14 in process B. I traced into spdk and
found out that the VFIO_IOMMU_MAP_DMA ioctrol failed in
vtophys_iommu_map_dma (note that I had vfio enabled). Am i missing some
steps (like in https://lists.01.org/pipermail/spdk/2018-May/001884.html)? Or the
failure might just be caused by the 4k mapping i had?

I'm not familiar enough with the ways in which VFIO_IOMMU_MAP_DMA can fail - you'd
have to dig into that driver I think to find what specifically -14 means.