How to emulate queue for CGI script?

Question

In my CGI script i make long (up to 10 seconds) request to another server, parsing results and show response to my user (via AJAX). But another server owner ask me to perform each request no more than 1 request per 10 seconds so:

• i need to save each request of my user;
• every ten seconds i can make only one request to another server;

First i think about Cron which will open simple text file (queue file), read first line and send it as a request to another server. After that it will save result in another file (where i'll cache all results). So my CGI will first check cache file and try to find result in it and after that (if result is not found) it will save task in the queue file (for the Cron).

But Cron run only once per minute so my user must wait for so long time...

So how i can do this via CGI?

May be:

1. After checking the cache file CGI will estimate time to complete request (by reading current queue file) and send this estimation time to the HTML (where i can got this time and make another request after this time via AJAX).

2. After that it will save request to the queue file and fork. The forked process will wait untill it's request will be on the top of the queue and will make request to another server.

3. After that it will save result in the cache file. What you think?

May be some module already written for such tasks?

Answer 1

One option is to create a local daemon/service (Linux/Windows) that handles sending all requests to the remote server. Your web service can talk to this daemon instead of the remote service using the same protocol, except on a private port/socket. The daemon can accept requests from the web server/application and every ten seconds, if there is a pending request it can send it on to the remote server, and when there is a response, it can forward it back to the incoming request socket. You can think of this daemon as a proxy server that simply adds a queueing functionality. Note that the daemon doesn't actually have to parse either the incoming request or returning results; it just forwards the bits on to the destination in each case. It only has to implement the queueing and networking functionality.