Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly
Sign up
Appearance settings

Getting UDP recv_multishot to work with buffer rings #1432

Answered by axboe
mbr0wn asked this question in Q&A
Discussion options

I am trying to set up a multishot recv() call on a UDP socket. However, while the single-shot version works, the multi-shot version always fails. I have a hunch it's with how I'm setting up the buffer ring, but I can't see the problem. All comparisons to similar code snippets make it seem like this is correct (I tried to boil it down as far as I could):

#include <arpa/inet.h>
#include <liburing.h>
#include <stdio.h>
#include <sys/socket.h>
#include <sys/types.h>
#include <unistd.h>
#include <stdlib.h>
#include <string.h>
#define BUF_BGID 0
#define N_BUFS 128
#define PAGESIZE 4096
#define BUFSIZE 8192
#define PORT 49000
int main(int argc, char *argv[])
{
 struct sockaddr_in addr;
 int sock_fd;
 char* bufs; // All bufs go here
 int i;
 struct io_uring ring;
 struct io_uring_buf_ring* br = NULL;
 struct io_uring_sqe *sqe;
 struct io_uring_cqe* cqe;
 /* Set up socket */
 addr.sin_family = AF_INET;
 addr.sin_port = htons(PORT);
 inet_aton("0.0.0.0", &addr.sin_addr);
 /* Create UDP socket */
 sock_fd = socket(AF_INET, SOCK_DGRAM, IPPROTO_UDP);
 bind(sock_fd, (struct sockaddr*)&addr, sizeof(addr));
 /* Create uring */
 if (io_uring_queue_init(N_BUFS, &ring, 0) < 0) {
 fprintf(stderr, "Could not init ring.\n");
 return 1;
 }
 /* Create buffers and buffer ring */
 if (posix_memalign((void**)&bufs, PAGESIZE, BUFSIZE * N_BUFS)) {
 fprintf(stderr, "Could not allocate memory.\n");
 return 1;
 }
 memset((void*)bufs, 0, BUFSIZE * N_BUFS);
 int err = 0;
 br = io_uring_setup_buf_ring(&ring, N_BUFS, BUF_BGID, 0, &err);
 if (!br) {
 fprintf(stderr, "Could not create buffer ring.\n");
 }
 io_uring_buf_ring_init(br);
 for (i = 0; i < N_BUFS; i++) {
 io_uring_buf_ring_add(br, bufs + i * BUFSIZE, BUFSIZE, i,
 io_uring_buf_ring_mask(N_BUFS), i);
 }
 io_uring_buf_ring_advance(br, N_BUFS);
 /* Set up recv SQE */
 sqe = io_uring_get_sqe(&ring);
 io_uring_sqe_set_flags(sqe, IOSQE_BUFFER_SELECT);
 sqe->buf_group = BUF_BGID;
 io_uring_prep_recv_multishot(sqe, sock_fd, NULL, 0, 0); /* This doesn't work! */
 /*io_uring_prep_recv(sqe, sock_fd, bufs, BUFSIZE, 0);*/ /* This works! */
 if (io_uring_submit(&ring) < 0) {
 fprintf(stderr, "Failed to submit SQE.\n");
 return 1;
 }
 if (io_uring_wait_cqe(&ring, &cqe) < 0) {
 fprintf(stderr, "Could not get CQE.\n");
 return 1;
 }
 if (cqe->res < 0) {
 fprintf(stderr, "Bad CQE result: %s\n", strerror(-cqe->res));
 return 1;
 }
 fprintf(stderr, "Buffer index: %d\n", cqe->flags >> IORING_CQE_BUFFER_SHIFT);
 /* Now do handling and cleanup. */
 return 0;
}

This can be compiled and executed without modifications (gcc recv_multishot.c -luring).

With the multishot version, the return value cqe->res of the read call is always -EINVAL, but the single-shot version works, making me think it's not a problem on the socket, but with the provided buffers. Any suggestions where I'm going wrong?

Notes:

  • Tested with Linux 6.8, liburing version 2.5 (on Ubuntu 24.04).
  • I don't think this is an rlimit issue, I see the same behaviour as root
  • I tried replacing the buffer ring with the older API (io_uring_prep_provide_buffers() and friends), but I get the exact same behaviour
  • I've seen the recv-multishot test case, but nothing stands out to me as obviously different
You must be logged in to vote

Here's what I ran, only change is looping for CQEs and doing the prep side in a more solid fashion. The latter may make a difference on your old liburing.

#include <arpa/inet.h>
#include <liburing.h>
#include <stdio.h>
#include <sys/socket.h>
#include <sys/types.h>
#include <unistd.h>
#include <stdlib.h>
#include <string.h>
#define BUF_BGID 0
#define N_BUFS 128
#define PAGESIZE 4096
#define BUFSIZE 8192
#define PORT 49000
int main(int argc, char *argv[])
{
 struct sockaddr_in addr;
 int sock_fd;
 char* bufs; // All bufs go here
 int i;
 struct io_uring ring;
 struct io_uring_buf_ring* br = NULL;
 struct io_uring_sqe *sqe;
 struct io_uring_cqe* cqe;
 ...

Replies: 3 comments 2 replies

Comment options

You probably want to upgrade your liburing, that's ancient. Not sure what the problem is, I ran it here with a newer kernel (should not matter) and with current liburing, and it works fine for me.

General note - do io_uring_prep_foo() first, before fiddling with the SQE. The sqe prep may overwrite some bits you already set.

Here's running a slightly modified version of yours which loops getting CQEs until the mshot terminates:

axboe@m2max-kvm ~> ./mshot-rep
Buffer index: 0, res 8192
Buffer index: 1, res 8192
Buffer index: 2, res 8192
Buffer index: 3, res 8192
Buffer index: 4, res 8192
Buffer index: 5, res 8192
Buffer index: 6, res 8192
Buffer index: 7, res 8192
Buffer index: 8, res 8192
Buffer index: 9, res 8192
Buffer index: 10, res 8192
Buffer index: 11, res 8192
Buffer index: 12, res 8192
Buffer index: 13, res 8192
Buffer index: 14, res 8192
Buffer index: 15, res 8192
Buffer index: 16, res 8192
Buffer index: 17, res 8192
Buffer index: 18, res 8192
Buffer index: 19, res 8192
Buffer index: 20, res 8192
Buffer index: 21, res 8192
Buffer index: 22, res 8192
Buffer index: 23, res 8192
Buffer index: 24, res 8192
Buffer index: 25, res 8192
Buffer index: 26, res 8192
Buffer index: 27, res 8192
Buffer index: 28, res 8192
Buffer index: 29, res 8192
Buffer index: 30, res 8192
Buffer index: 31, res 8192
Buffer index: 32, res 8192
Buffer index: 33, res 8192
Buffer index: 34, res 8192
Buffer index: 35, res 8192
Buffer index: 36, res 8192
Buffer index: 37, res 8192
Buffer index: 38, res 8192
Buffer index: 39, res 8192
Buffer index: 40, res 8192
Buffer index: 41, res 8192
Buffer index: 42, res 8192
Buffer index: 43, res 8192
Buffer index: 44, res 8192
Buffer index: 45, res 8192
Buffer index: 46, res 8192
Buffer index: 47, res 8192
Buffer index: 48, res 8192
Buffer index: 49, res 8192
Buffer index: 50, res 8192
Buffer index: 51, res 8192
Buffer index: 52, res 8192
Buffer index: 53, res 8192
Buffer index: 54, res 8192
Buffer index: 55, res 8192
Buffer index: 56, res 8192
Buffer index: 57, res 8192
Buffer index: 58, res 8192
Buffer index: 59, res 8192
Buffer index: 60, res 8192
Buffer index: 61, res 8192
Buffer index: 62, res 8192
Buffer index: 63, res 8192
Buffer index: 64, res 8192
Buffer index: 65, res 8192
Buffer index: 66, res 8192
Buffer index: 67, res 8192
Buffer index: 68, res 8192
Buffer index: 69, res 8192
Buffer index: 70, res 8192
Buffer index: 71, res 8192
Buffer index: 72, res 8192
Buffer index: 73, res 8192
Buffer index: 74, res 8192
Buffer index: 75, res 8192
Buffer index: 76, res 8192
Buffer index: 77, res 8192
Buffer index: 78, res 8192
Buffer index: 79, res 8192
Buffer index: 80, res 8192
Buffer index: 81, res 8192
Buffer index: 82, res 8192
Buffer index: 83, res 8192
Buffer index: 84, res 8192
Buffer index: 85, res 8192
Buffer index: 86, res 8192
Buffer index: 87, res 8192
Buffer index: 88, res 8192
Buffer index: 89, res 8192
Buffer index: 90, res 8192
Buffer index: 91, res 8192
Buffer index: 92, res 8192
Buffer index: 93, res 8192
Buffer index: 94, res 8192
Buffer index: 95, res 8192
Buffer index: 96, res 8192
Buffer index: 97, res 8192
Buffer index: 98, res 8192
Buffer index: 99, res 8192
Buffer index: 100, res 8192
Buffer index: 101, res 8192
Buffer index: 102, res 8192
Buffer index: 103, res 8192
Buffer index: 104, res 8192
Buffer index: 105, res 8192
Buffer index: 106, res 8192
Buffer index: 107, res 8192
Buffer index: 108, res 8192
Buffer index: 109, res 8192
Buffer index: 110, res 8192
Buffer index: 111, res 8192
Buffer index: 112, res 8192
Buffer index: 113, res 8192
Buffer index: 114, res 8192
Buffer index: 115, res 8192
Buffer index: 116, res 8192
Buffer index: 117, res 8192
Buffer index: 118, res 8192
Buffer index: 119, res 8192
Buffer index: 120, res 8192
Buffer index: 121, res 8192
Buffer index: 122, res 8192
Buffer index: 123, res 8192
Buffer index: 124, res 8192
Buffer index: 125, res 8192
Buffer index: 126, res 8192
Buffer index: 127, res 8192
Bad CQE result: No buffer space available
You must be logged in to vote
0 replies
Comment options

Here's what I ran, only change is looping for CQEs and doing the prep side in a more solid fashion. The latter may make a difference on your old liburing.

#include <arpa/inet.h>
#include <liburing.h>
#include <stdio.h>
#include <sys/socket.h>
#include <sys/types.h>
#include <unistd.h>
#include <stdlib.h>
#include <string.h>
#define BUF_BGID 0
#define N_BUFS 128
#define PAGESIZE 4096
#define BUFSIZE 8192
#define PORT 49000
int main(int argc, char *argv[])
{
 struct sockaddr_in addr;
 int sock_fd;
 char* bufs; // All bufs go here
 int i;
 struct io_uring ring;
 struct io_uring_buf_ring* br = NULL;
 struct io_uring_sqe *sqe;
 struct io_uring_cqe* cqe;
 /* Set up socket */
 addr.sin_family = AF_INET;
 addr.sin_port = htons(PORT);
 inet_aton("0.0.0.0", &addr.sin_addr);
 /* Create UDP socket */
 sock_fd = socket(AF_INET, SOCK_DGRAM, IPPROTO_UDP);
 bind(sock_fd, (struct sockaddr*)&addr, sizeof(addr));
 /* Create uring */
 if (io_uring_queue_init(N_BUFS, &ring, 0) < 0) {
 fprintf(stderr, "Could not init ring.\n");
 return 1;
 }
 /* Create buffers and buffer ring */
 if (posix_memalign((void**)&bufs, PAGESIZE, BUFSIZE * N_BUFS)) {
 fprintf(stderr, "Could not allocate memory.\n");
 return 1;
 }
 memset((void*)bufs, 0, BUFSIZE * N_BUFS);
 int err = 0;
 br = io_uring_setup_buf_ring(&ring, N_BUFS, BUF_BGID, 0, &err);
 if (!br) {
 fprintf(stderr, "Could not create buffer ring.\n");
 }
 io_uring_buf_ring_init(br);
 for (i = 0; i < N_BUFS; i++) {
 io_uring_buf_ring_add(br, bufs + i * BUFSIZE, BUFSIZE, i,
 io_uring_buf_ring_mask(N_BUFS), i);
 }
 io_uring_buf_ring_advance(br, N_BUFS);
 /* Set up recv SQE */
 sqe = io_uring_get_sqe(&ring);
 io_uring_prep_recv_multishot(sqe, sock_fd, NULL, 0, 0); /* This doesn't work! */
 io_uring_sqe_set_flags(sqe, IOSQE_BUFFER_SELECT);
 sqe->buf_group = BUF_BGID;
 if (io_uring_submit(&ring) < 0) {
 fprintf(stderr, "Failed to submit SQE.\n");
 return 1;
 }
 do {
 if (io_uring_wait_cqe(&ring, &cqe) < 0) {
 fprintf(stderr, "Could not get CQE.\n");
 return 1;
 }
 if (cqe->res < 0) {
 fprintf(stderr, "Bad CQE result: %s\n", strerror(-cqe->res));
 return 1;
 }
 fprintf(stderr, "Buffer index: %d, res %d\n", cqe->flags >> IORING_CQE_BUFFER_SHIFT, cqe->res);
 if (!(cqe->flags & IORING_CQE_F_MORE)) {
	 fprintf(stderr, "No more CQEs\n");
	 break;
	}
	io_uring_cqe_seen(&ring, cqe);
 } while (1);
 /* Now do handling and cleanup. */
 return 0;
}
You must be logged in to vote
2 replies
Comment options

Many thanks! Even on the old liburing, this works. Amazing. Thanks for the tip to prep before flagging.

(Note that I'm not attached to old software, or Ubuntu 24.04, but this is a commonly used distro so I'm happy if I can develop code that lots of people can run out of the box).
Comment options

Yeah I know you or others don't run unsupported kernels out of love for them, it's more a jab at distros like Ubuntu! But they are moving towards doing the right thing, at least.
Answer selected by mbr0wn
Comment options

As another general note, the 6.8 kernel is some ubuntu special which isn't part of stable and doesn't receive updates from me. It may have bugs, I don't know, as they don't use the stable series of kernels that I support. Probably not your issue here (I suspect old liburing), but worth mentioning as a general caution that I do not support these types of kernels, as it's simply not possible for me to do.
You must be logged in to vote
0 replies
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Category
Q&A
Labels
None yet
2 participants

AltStyle によって変換されたページ (->オリジナル) /