Using queues to decouple functions/avoid direct calling?

Question

Kind of a functional programming newbie question here:

I've been reading the transcripts of some of Rich Hickey's talks, and in several of his more well-known ones, he recommends using queues as an alternative to having functions call one another. (E.g. in Design, Composition and Performance and in Simple Made Easy.)

I don't quite understand this, in a number of respects:

1. Is he talking about putting data in a queue and then having each function use it? So instead of function A calling function B to carry out its own computation, we just have function B slap its output on a queue and then have function A grab it? Or, alternatively, are we talking about putting functions on a queue and then successively applying them to data (surely not, because that would involve massive mutation, right? And also multiplication of queues for multiple-arity functions, or like trees or something?)

2. How does that make things simpler? My intuition would be that this strategy would create more complexity, because the queue would be a kind of state, and then you have to worry "what if some other function snuck in and put some data on top of the queue?"

One answer to an implementation question on SO suggests that the idea is creating a bunch of different queues. So each function puts its output in its own queue(??). But that also confuses me, because if you're running a function once, then why does it need a queue for its output when you could just take that output and slap a name on it as a (var, atom, entry in a big hash table, whatever). By contrast, if a function is running multiple times, and you stick its output onto a queue, then you've inflicted state on yourself again, and you have to worry about the order in which everything is called, downstream functions get less pure, etc.

Clearly I'm not understanding the point here. Can someone explain a little?

• Robert Harvey: I didn't see a single reference to a queue in you're first link, though I'll grant you that the post is incredibly long and I might have missed it. Seems like it is more a talk about artistry than it is about programming.
• Robert Harvey: Queues are briefly mentioned twice in the second article, but not expounded upon. In any case, the idea of using a messaging queue to communicate between applications or modules has been around for awhile. It seems unlikely that you would do this in a single application unless you were creating a processing pipeline or state engine.
• Paul Gowder: It's in the paragraph under the slide headed "Take apart time/order/flow" ("You can break systems apart, so there's less direct calling. You can use queues to do that.")
• Ext3h: Not worth making that a full answer, but he's really just giving a fuzzy description for the concept of job pools and event driven programming. So you pack a function call into a generic Job object, push that into a queue, and have one or more worker threads work on that queue. The Job then dispatches more Jobs into the queue upon completion. Return values are replaced by callbacks in that concept. It's a nightmare to debug and verify as you lack a call stack, and efficient and flexible for the very same reason.
• Paul Gowder: Thanks. Maybe my real problem is that I don't understanding messaging! (Heh, time to go learn smalltalk? :-) )

Answer 1

It's more of a design exercise than a general recommendation. You aren't usually going to put a queue between all your direct function calls. That would be ridiculous. However, if you don't design your functions as if a queue might be inserted between any of the direct function calls, you cannot justifiably claim you have written reusable and composable code. That's the point Rich Hickey is making.

This is a major reason behind the success of Apache Spark, for example. You write code that looks like it's making direct function calls on local collections, and the framework translates that code into passing messages on queues between cluster nodes. The kind of simple, composable, reusable coding style Rich Hickey advocates makes that possible.

• Robert Harvey: But isn't that just a change in the method binding process? At the end of the day, a function call is just a function call, right? What happens after that depends on what the function does. So it seems less about making function calls than it is about how the underlying infrastructure is designed.
• Robert Harvey: To put it another way, what change would you make to your function calls to make them "queue-friendly?"
• Karl Bielefeldt: Remember, the code on the other end of the queue doesn't necessarily have access to the same memory and IO. Queue-friendly functions are free of side effects and expect inputs and produce outputs that are immutable and easily serializable. That's not so easy a test to meet on many codebases.
• Robert Harvey: Ah, so "functional programming friendly" then. Kinda makes sense, since it is Rich Hickey discussing it.

Answer 2

One thing to note is that functional programming allows you to connect functions to each other indirectly through mediator objects that take care of procuring arguments to feed into the functions and flexibly routing their results to recipients that want their results. So suppose you have some straightforward direct calling code that looks like this example, in Haskell:

myThing :: A -> B -> C
myThing a b = f a (g a b)

Well, using Haskell's Applicative class and its <$> and <*> operators we can mechanically rewrite that code to this:

myThing :: Applicative f => f A -> f B -> f C
myThing a b = f <$> a <*> (g <$> a <*> b)

...where now myThing is no longer directly calling f and g, but rather connecting them through some mediators of type f. So for example, f could be some Stream type provided by a library that provides an interface to a queueing system, in which case we'd have this type:

myThing :: Stream A -> Stream B -> Stream C
myThing a b = f <$> a <*> (g <$> a <*> b)

Systems like this do exist. In fact, you can look at Java 8 streams as a version of this paradigm. You get code like this:

List<Integer> transactionsIds = 
 transactions.parallelStream()
 .filter(t -> t.getType() == Transaction.GROCERY)
 .sorted(comparing(Transaction::getValue).reversed())
 .map(Transaction::getId)
 .collect(toList());

Here you are using the following functions:

• t -> t.getType() == Transaction.GROCERY
• comparing(Transaction::getValue).reversed()
• Transaction::getId
• toList()

...and instead of having them call each other directly, you're using the Stream system to mediate between them. This code example isn't calling the Transaction::getId function directly—the Stream is calling it with the transactions that survived the earlier filter. You can think of the Stream as a very minimal sort of queue that couples functions indirectly and routes values between them.