In my previous blog post I wrote about why curl’s bitsets and some people asked to explain a bit more how they work and why I like them so much. So here is a introduction for someone unfamiliar with them.

Revisiting the Problem

Many applications have a few main types of data they need to deal with, but many instances of them. Let’s just call them “Things”. A “thing” has several properties the application needs to manage and often it needs to operate on “all things with prop1 == x”. The question is how you design this for working efficiently on thousands of “Things”.

In my last blog post I wrote about why linked lists are often not a good idea. In this post I want to present how we replaced some of curl’s lists with an array and bitsets.

The Problems

curl has two main APIs to perform internet transfers: the easy and the multi set of functions. If you want to do just a single transfer at a time, the easy interface is the right one for you. If you need to do several transfers in parallel or want connection reuse, the multi interface is best.

The always funny Alberta Tech has a video “Linked Lists Aren’t Real” which I stumbled upon again recently. That touched on a topic I long wanted to write about, so here we go (made possible by the curl summer of bliss).

The Trouble with Linked Lists

As an introductory topic for computer science beginners epic journey into data structures they are easy to grasp. You have an anchor (The List) and next pointers (The Elements) which you can chain together. The anchor points to the first element, each element points to the next. NULL or nil or None (whatever your language has as an absent value) terminate the list.

During the curl summer of bliss I found some breathing room again to look at a favourite topic of mine: performance.

Below you see how recent curl versions behave when downloading 20 thousand times the same 10KB resource from a local Apache httpd via HTTP/2. In the top part you see the speed in requests per second (higher is better) and in the lower part the memory in use during this is shown (lower is better).

(Disclaimer: I speak from my maintainer perspective. I have no deep insights into deployment and operations. My very personal opinions here.)

Also called coordinated vulnerability disclosure states on wikipedia that:

a vulnerability disclosure model in which a vulnerability or an issue is disclosed to the public only after the responsible parties have been allowed sufficient time to patch or remedy the vulnerability or issue.

which sounds very sensible. The intention clearly is to keep the bad guys in the dark until the good guys had time to patch everything.