How to manipulate packets flow in a TCP proxy

Question

I'm testing on my laptop a C++ TCP proxy listening on a fixed port given to command line; the proxy works with the usual bind and listen and loops on incoming connections with accept, forking for each accepted connection. GET and CONNECT requests are managed.

Now I'm planning to deliver the packets not as they come, but following a priority: some connections, therefore some packets, have higher priority and must be delivered before others. The priority is determined by looking at Host field in the first packet sent by client, the one containing GET or CONNECT request.

Packets are inserted and removed in four deque, one for each level of priority: packets are inserted in the matching deques according to their priorities and are removed from higher deques (higher priorities) first.

Elements stored in deque, representing a single packet, are custom struct:

typedef struct infoPacket {
 // client and server socket descriptors for send() and recv() 
 int clientFd;
 int serverFd;
 // 0 = GET, 1 = CONNECT
 int request;
 // 0 = DIRECTION_UPLOAD, 1 = DIRECTION_DOWNLOAD
 int direction;
 int priority;
 // actual data
 std::string payload;
 int payloadSize;
 in_addr_t serveraddr;
 std::string host;
 std::chrono::time_point<std::chrono::system_clock> start;
} infoPkt;

The matter is I have a design issue: while a normal proxy would handle incoming connections by forking and letting the child process to deal with the packets flow, now I have to manually receive a packet from both sides of connections, check packet's priority, put it in the matching deque while I remove a packet from a deque, see what connection it belongs to and send it to its destination...for all the packets of all the connections.

If I keep on forking to deal each connection, there must be some kind of IPC since the main process is where the deques containg the packets are stored (packets will be inserted from children and removed to be used by children) while children processes is where the connections are active (packets will be sent to deque when received and to their destinations when removed by deque).

I may not deal this with forking, and keep a list of socket descriptors pair for each connection, an inserting thread will select or poll to check available reading socket file descriptors and insert packets in deque, while a removing thread will take packets from deque and send them to their destination. But, if I remember well, the pair is not a proper way to identify a connection.

deque mechanism of priority is done: what confuses me is the bunch of sockets pairs and how to properly recv from and send to them.

Any advice on how to deal with this?

Working on Ubuntu x86_64.

Answer 1

... the pair is not a proper way to identify a connection

it's fine: so long as you can figure out when receiving on some socket, which other socket you should forward to, you have everything you need.

Since sockets are represented by integers, it's easy to keep a map<int,int> or unordered_map<int,int> storing both source->dest and dest->source for easy lookups.

In practice, you'll accept some incoming connection, giving a single (source) socket, then create the related outgoing connection, giving another (destination) socket, and then you have a pair of sockets. Add them both to your lookup(s), and to your select/poll fd sets for reading, and you're good to go.

If you're keeping this single-threaded, make sure to use non-blocking I/O on both sockets. You do need to keep track of unfinished writes, but your priority queues will help there anyway.

• elmazzun: I realized that only outcoming packets will be prioritized (the ones removed from the priority queues), since there is no way to prioritize incoming packets taken by select readings, but that's how it should work...I guess.

Answer 2

You might consider letting the network stack handle most of this for you.

You can use setsockopt (SO_PRIORITY ,...) to set priorities on the outgoing sockets. The protocol stack will then order the packets based on the priority given to each socket.

This should let you maintain the existing architecture with little or no change since you don't have to collect the packets back in one place to prioritize them.