Not sure I follow the question at all... For normal read/write operation, whether DMA is used or not is entirely up to the subsystem that is hosting the IO. Let's say it's a read of a file on an nvme device, yes DMA is certainly used, as PIO would be very slow. But io_uring doesn't know or care, it simply asks for "please put data from location X into memory Y". For normal buffered IO, Y is a page cache location, kernel memory. For O_DIRECT IO, Y is a userspace address, so the device generally DMA's directly into the userspace page.

Thanks for your answer @axboe . So am I understanding correctly all I can do from the userspace side with the ioruing api is specify O_DIRECT and let the kernel do the inference on whether DMA is possible or not? I thought there were mechanisms in the hand of a programmer that you can use to initiate DMA transfers directly. I'm just trying to figure out if I would need to step down into the kernel somehow to enable DMA or ioruing does that for me. Also maybe I should mention that I'm very interested in the RDMA capabilities of iouring as well. If you say that ioruing does not care about those intrinsics, who does? Will the kernel make sure that RDMA is used whenever possible (e.g. given a supported network card, bus etc..)? This would be the best case for me. But maybe RDMA is a whole different can of worms?

To give some context: I would like to give our database the capability to retrieve data from a storage backend regardless if the storage backend is a local disk or another machine entirely. If I could use ioruing for both (DMA disk reads, RDMA reads from other network nodes) that would significantly simplify everything for me.

I'm still interested in this for our application. We really would like to enable RDMA for exchange of big binary data over ethernet(RoCE)/infiniband, but it seems that at least for InfiniBand one needs to use a specialized library (libibverbs) to induce the connection. From the surface it looks thought like something the liburing or its kernel-side implementation might be able to abstract that away, so my question is: does it?

@MartyMcFlyInTheSky the kernel doesn't do any RDMA by itself. You need a library and special hardware with the capabilities and the driver to match it. Then, you have to set it up from userspace entirely.

Since RDMA requires a supporting network protocol (bare TCP doesn't work), io_uring can't do it. io_uring replicates the base kernel networking interface. The closest you can get is io_uring_prep_send_zc(), but that's still mimic-ing a normal send(), so it won't automatically be put in the user memory of the receiving end.