In this new series, I want to shine some light onto specific parts of Monal’s internals. It’s dedicated to programmers or people curious about how Monal works internally. If you want to give some feedback, feel free to send an email to thilo@monal-im.org

Other articles in this series:

• Monal Internals - Handlers framework
• Monal Internals - XML Query Language

Serializable Promise Framework

XMPP as a protocol is, as most protocols are, inherently asynchronous. In Monal we therefore use the popular PromiseKit library to let the UI know when a XMPP action finished or failed.

But this has a huge drawback: PromiseKit-based promises aren’t serializable, so we can’t respond to events that got handled by the Notification Service App Extension while the app was suspended. The serializable promise framework is a new framework in Monal, which exactly fills that gap.

Overall Process

For instance, imagine you want to remove the avatar of a group chat in Monal, while on a slow, unreliable network.

1. You submit the request, and the loading screen appears over the view (0:03)
2. Because the network is slow, the loading screen persists for a long time
3. You get frustrated and switch apps, sending Monal to the background (0:06)
4. After 30s in the background, Monal gets suspended (0:36)
5. The server fails to remove the avatar and sends an error to Monal
6. The response is handled by Monal’s app extension in the background (0:51)
7. You switch apps back to Monal (0:57)

Without the promise framework, the app would have continued to show the loading screen indefinitely, requiring you to fully close and open the app.

However, with the promise framework, when you switch back to the app, the loading screen disappears and correctly shows the error returned by the server:

The promise framework allows any such interaction, where the UI has to update in response from the server, to be handled in a general way.

Background

We will briefly describe the important parts below. But these links provide more detailed context and are recommended reading:

• The app decomposition slides explain the split between the main app and app extension.
• The handler framework blog post explores some of the consequences of this split.

Main app and app extension

In Monal there are two separate processes: the "main app" (MonalAppDelegate) and "app extension" (NotificationService), which is sometimes abbreviated to "appex".

Since they do not share memory, they hand over to each other via the handler framework.

An xmpp stanza can be processed by either, depending on whether the app is running or not:

Diagram showing that, if the app is active or in the background, stanzas get processed by the main app, otherwise they get processed by the appex, triggered by Apple’s push servers if it is not already running

App states

Below is a slightly simplified diagram of the app’s lifecycle. A more detailed lifecycle from Apple’s perspective is available here.

Diagram showing the different states the app can be in. It is a visual depiction of the description below

• The app moves from "active" to "in background" when it is swiped away without being closed
  • It continues to run and process stanzas for ~30s in the background
• Once the 30s are complete, it moves to "suspended"
  • Apple saves the UI’s state
  • When the app is reopened, it returns to the same screen as before
  • Any stanzas that were processed in the meantime were processed by the appex

Appex states

Meanwhile, the appex has a more simple lifecycle - it is either running, or not running:

Diagram showing that the appex can be in two states: "running" or "not running". It moves to "running" state on receipt of a notification, then returns to "not running" state after 30s have passed

• Once the appex starts, it will connect to the server and run for 30s
• If the app is opened while the appex is running, the appex will return to "not running" state in deference to the app.

Serializable promise framework

Ok, now the background is out of the way, time to explain the promise framework!

While promises are very useful in SwiftUI, there is one problem: they only exist in that particular process, and cannot easily be passed between the app and appex.

This means that, if the promise was resolved in the appex, once the app is reopened (moves from suspended to active), it will not be consumed. This is because the appex has no way of tying the result back to the promise that updates the UI.

Since the handler framework is what allows the seamless handoff between the app and appex, we need a way to trigger consumption of a promise as a result of processing in the handler. This is precisely what the promise framework does!

High-level overview

We create a "serializable promise" class called MLPromise. This class stores a "UI promise" which is an AnyPromise from PromiseKit. Consumption of this AnyPromise is what ultimately causes the UI to update.

Whenever we want to bind a UI action to the result of a handler, we create an MLPromise and pass it as an argument to the handler. We then return the MLPromise’s AnyPromise to the UI.

Unlike the AnyPromise it contains, the MLPromise is (mostly) serializable. When the MLPromise is created, and whenever it is resolved, it persists itself to a new promises table in the database.

Meanwhile, whenever the app or appex is unfrozen, and the promise table is read into memory, the resolved arguments of any already existing promises are overwritten.

Note that this persisted version of the MLPromise is not complete - while the resolved argument can be persisted, the UI promise is not persisted at any stage.

This means that, while the resolved argument can be passed betwen the app and appex, the UI promise itself cannot. The consequence of this is that a promise cannot be consumed in the appex - it can only be resolved there, leaving consumption for when the app becomes active again:

This makes some sense, as the whole purpose of the AnyPromise is to update the UI as a result of some backend action. It only makes sense to update the UI from the process which manages the UI - the app itself.

As a result, whenever the app becomes active again, all outstanding MLPromises check the version retrieved from the DB if they have been resolved in the meantime, and if so, consume themselves. This is how the loading overlay gets removed in the above example.

Step-by-step overview

Let’s return to the initial scenario of removing the avatar of a group chat. The user chooses the new avatar and presses submit. Then, the following happens:

Diagram showing the flow of promises throughout the app, as explained in detail below

(1) UI code calls backend

The UI code responding to requests to remove the avatar calls the backend method to do this.

Once the backend function completes, it will return an AnyPromise, allowing the UI code to continue immediately:

showPromisingLoadingOverlay(overlay, headlineView:Text("Removing avatar..."), descriptionView:Text("")) {
 // this returns an AnyPromise used by the loading overly to hide itself once it gets resolved
 self.account.mucProcessor.publishAvatar(nil, forMuc: contact.contactJid)
}

(2) Core code creates promise

The core code creates an MLPromise:

-(AnyPromise*) publishAvatar:(UIImage* _Nullable) image forMuc:(NSString*) room
{
 MLPromise* promise = [MLPromise new];
 // ...

(3) Core code creates handler and (4) sends IQ

The core code creates a new handler, and passes the MLPromise to it as an argument. Recall that handlers serialize their arguments - this means that, even if the app is suspended and the handler is called in the appex, the MLPromise will be available.

[_account sendIq:vcard withHandler:$newHandlerWithInvalidation(self, handleAvatarPublishResult, handleAvatarPublishResultInvalidation, $ID(room), $ID(promise))];

(5) Promise returned to UI

Internally, the MLPromise has created an AnyPromise. This is a type provided by PromiseKit that can be returned directly to the UI, and that the SwiftUI code understands. It gets returned to the UI code.

return [promise toAnyPromise];

(6) XMPP server provides response

This happens on the server side and is not relevant to Monal.

However, while waiting for the response, the user puts the app into the background, and the app gets suspended.

(7) Handler resolves promise

Now, the response from the server activates the appex, since the app was suspended.

The handler is called with the response from the server. It resolves the promise (via either reject or fulfill) - passing that response to the promise.

$$instance_handler(handleAvatarPublishResult, account.mucProcessor, $$ID(xmpp*, account), $$ID(XMPPIQ*, iqNode), $$ID(MLPromise*, promise))
 if([iqNode check:@"/<type=error>"])
 {
 // ...
 [promise reject:error];
 // ...
 }
 // ...
 [promise fulfill:nil];
$$

(8) Promise waits until app is open

If the promise had been resolved from the app, it could have been consumed immediately. However, since it was resolved from the appex, the AnyPromise was not available to call at that time.

Therefore, the promise does not consume itself yet - it instead waits for the app to return to active state.

This waiting is performed by the following observer - whenever either the app or appex is unfrozen, the deserialize method is called:

-(instancetype) init
{
 // ...
 [[NSNotificationCenter defaultCenter] addObserver:self selector:@selector(deserialize) name:kMonalUnfrozen object:nil];
 // ...
}

deserialize calls attemptConsume on each run. attemptConsume checks if we are inside the appex (in which case we do not consume the promise), and only if we are inside the app does it consume the promise:

-(void) attemptConsume
{
 DDLogDebug(@"Intend to consume promise %@ with uuid %@ and argument %@", self, self.uuid, self.resolvedArgument);
 if([HelperTools isAppExtension])
 {
 DDLogDebug(@"Not consuming promise %@ with uuid %@ as we are in the app extension", self, self.uuid);
 return;
 }
 // ...
}

Note that both reject and fulfill also call attemptConsume after resolving the promise. This would let the promise be consumed immediately if we are inside the app instead of the appex.

(9) Promise consumes UI promise

Once we progress past the checks in attemptConsume, we finally consume the promise by calling the resolve callback tied to the PromiseKit AnyPromise returned earlier. This in turn prompts the UI to hide the loading overlay.

-(void) attemptConsume
{
 // ...
 PMKResolver resolve = _resolvers[self.uuid];
 // ...
 resolve(self.resolvedArgument);
 // ...
}

On December 9th I returned to the Trolley Barn to DJ again for my friend Valerie Young. This time around my goal was to have a better mix of high energy and lower energy "groovy" tunes, and I wanted to be better prepared for tying in endings at any time, no matter how many times through the caller requested.

Prep

While I still prepped by marking each time through the dance with whether it looped cleanly or not, this time I (mostly) marked it with a cue instead of a 64 beat loop. I also named the cue based on what other cues I could jump to cleanly from it. For example, if I were on the last time through the dance I would mark if it looped cleanly itself, but also that I could jump backwards 3 times to cue 6 (or whatever the case may be). This way if I’m nearing the end of the dance but the caller is running it a bit longer I can reset without a single 32 bar phrase starting to get repetitive. I can also plan ahead by seeing that cue 6 lets me jump back to cue 8 or 9 afterwards (let’s assume these are the last two cues), which means that if the caller immediately calls 2 more times through I know that I can cleanly jump back to 2 more times without incident. I also mark a handful of things in the music such as if this time through the dance is a big recognizable build up (which I might not want to repeat even if technically it sounds okay), or if I shouldn’t jump back before a cue because the energy is significantly lower and we don’t want to kill the energy that had already been built up on the dance floor. This made it much easier to always follow the callers instructions and not have to scramble to wrap up the song or ask if we can do 4 times through the dance instead of 2 or what not.

The Set

I started out the set with a slowed down mix of "Whelan’s", "Baghad Gus", and "Congress Reel" all by Wild Asparagus. This was a medley that I had planned for my previous set but hadn’t managed to use, but it worked quite well to "Whoever is Right is Right" by Jim Hemphill. The only iffy moment I had in this medley was in one of the transitions where I thought I executed it perfectly, only to have the tune I was introducing jump half a beat ahead, breaking the rhythm. I had left the Quantize option on (which locks the two beat grids together even if you’re a little bit off) and unfortunately the beat grid on the song in question was about half a beat off at the point I was doing the intro.

Lesson learned: pay more attention to what options you’ve got selected (and reset them between songs, this would be a problem several times during the evening) and in a song where the beat grid can’t be right through the entire song (ie. because of tempo changes), make sure it’s correct at the point you’re going to mix in.

For the second dance Valerie picked "Made Up Tonight" by Erik Hoffman. This dance is relatively easy and in some ways similar to the previous dance so I decided the dancers could handle a little bit higher tempo and energy level. I did a mix of "Bus Stop!" by The Free Raisins and "Rec Hall" by Kingfisher. This is another mix I had prepared for my previous set and one of the ones I was most excited about performing. Unfortunately I completely broke the transition by just forgetting to bring the volume up on Rec Hall, so I ended up killing the music entirely for a few beats before I realized what was happening and brought it back in. It sounded terrible, and this one the dancers couldn’t help but notice, but Valerie kept calling so they were on beat still when I finally realized what was happening.

Lesson learned: go ahead and beat match and mix in a few bars early, then do the transition by bringing the volume up instead of just hitting play, or just remember to have the volume up first.

At this point Valerie decided to take the evening in a different direction from the modern, improper and becket contras that the Trolley Barn normally asks callers to stick to. She called "The Steps of Waterloo", a traditional Sicilian circle, which I’ve never seen done at the Trolley Barn before. Since this one is both easy and a bit silly I used "Flying Ice Cream" by Giant Robot Dance which starts out very slow and groovy, then ramps up dramatically in the second half. Once the dance ramped up I mixed it with "The Randomainians" by The Great Bear Trio to make it last a little longer. This was the one mix and dance combo of the night that I was mostly extremely pleased with, when the tempo and energy ramped up half way through the song the dancers cheered and started frantically rushing to try and get to the basket swing in the limited time the (normally too fast) music allowed which was a lot of fun.

Lesson learned: prepare about twice as much music as you think you’ll need for the evening (to be fair, I already knew this, I was just struggling with the preparation for this set and didn’t end up with enough time to finish the other mixes I was considering).

Continuing Valerie’s theme of not doing modern contras she picked "The Hand Jive", a proper ceilidh, by Colin Towns next. Because this dance is a bit more "fun" and contains some patty-cake like hand games I decided to do a simple but fun mix of "Raccoon" by Wake Up Robin and Disco Snails by Vulfmon and Zachary Barker.

Unfortunately I hadn’t considered that, easy as the hand clapping is, it required Valerie to do a significant amount more calling than she normally would do and never drop out, so picking a song with lyrics was a bad idea. A few people still got a chuckle out of the "Disco Snails" cameo though.

Lesson learned: make sure to pay attention to how much the caller will have to talk and ask if they plan on dropping out before introducing something with lyrics.

We took our mid-dance waltz break at this point for which I played "Waiting for Landfall" by Julie Vallimont, "Indifference" by the String Beings, and "Norwegian Reinlender / Schottis From Idre" by Wild Asparagus. I’d hoped that some of the dancers would know the schottische for the second part of the medley, but mostly it just confused everybody who kept trying to waltz even though it was no longer a waltz.

Coming back we were running a bit late so I decided to drop the show-stopper for the evening, a mix of "Rainy Night" by The Dam Beavers, "Tween Spirit" by Giant Robot Dance, and "Smells Like Teen Spirit" (which, of course, the previous tune is a cover of) by Nirvana. This was paired with "Birminghams" by Gary Nelson.

I played the first two tunes as a normal medley, then when Valerie indicated that she was ready for three times left through the dance I mixed in the actual Nirvana version just for the ending. I was worried this crowd wouldn’t care for it, but when the tune first came in (as "Tween Spirit") a lot of the dancers started singing along, and when actual Nirvana came in at the end the energy really ramped up and there was a lot of cheering and stomping, so apparently even the younger members of the crowd were still excited about 90s grunge!

The only real problem with this dance is that I had one moment where I had a clean loop that I wanted to repeat but as I did so some of the dancers, knowing what came next in the song, started singing the next verse and then had to stop when the music wasn’t what they expected.

Lesson learned: when doing a pop song don’t do transitions that would break the progression of the song that people are already used to, even if they sound okay.

At this point I realized that I hadn’t prepared enough mixes for the dance and I was more or less out of material. For the next one Valerie asked to do a very short square dance, the "Cumberland Square Eight". Since it was going to be so short and timing didn’t matter I played "Heather’s Concussion" by The Great Bear Trio without mixing it into a medley.

This tune is itself very short (only three times through) and far too fast for a normal contra, so I slowed it down (though still keeping it too fast) and sped it up slowly through the dance to mess with the dancers. They looked like they had a good time!

This left us with just enough time for a no-walk-through contra. I had nothing else prepared but wanted to send us out on an easy-tempo dance after the various fast shenanigans from the rest of the night, but still have some good energy. I dug around in my library and ended up playing "Apple Blossom" by Great Bear. Since I didn’t have anything to mix this with I just played it through and looped once or twice near the end to keep the high-energy ending going. The dance Valerie picked was an easy ending dance that is unfortunately called "Unnamed Contra" and she didn’t know who wrote it.

We then closed down the evening (very late, much to the chagrin of the organizers) with "Mending" by "The Little Mercies".

Stuff I Learned

A recap of the things I need to take away from the evening:

• Be thoughtful about resetting all controls, levels, mixing, effects, etc. between tunes. Possibly write down the initial state for each song so that you can re-set the board at a glance.
• Beat match early then use the volume sliders for transitions, don’t just start playing immediately at the entry point (or just remember to turn up the volume as you approach the mix point).
• Consider how much the caller will have to talk in any given dance and how soon they’re dropping out before mixing in vocals.
• For pop songs, don’t violate the order of the verses or breaks that people expect, even if the final result sounds okay. It’s jarring for the dancers who are used to it being a certain way.

And finally: relax and enjoy it! While I spend the entire evening worrying about how the sound dropped out for a moment, or my transition was a beat off and I saw the dancers stumble and catch back up, or whatever other disaster might have befallen, no one remembers any of that. I got a very nice message from the organizer the next day telling me how excited everyone was and how many conversations she’d had with people telling her how much they enjoyed the music!

Other Links

• The set list on MusicBrainz
• Recordings of the evening can be found on the Internet Archive

The first time I DJed for a Contra Dance¹ was at Inman Park’s famous Trolley Barn. At the time I was DJing in the way other social dances are normally DJed: I had a laptop, I played a song, everyone danced. No fancy mixing, or effects: the most technical thing I did was loop 32 bar sections of music to stretch it out until the caller was ready to end the dance.

This time around, returning to the Trolley Barn, I wanted to see if my DJ skills had improved and try live mixing for the first time. Since I’ve never heard of anyone else DJing a traditional contra dance² I didn’t have a good idea of what I was doing but I worked with the caller, my friend Valerie Young, to plan a set that I thought the dancers would like. The set comprised mostly high energy contra dance bands who play traditional songs in less traditional ways: think the Gaslight Tinkers, Great Bear, and ContraForce. My goal is to be able to tell the caller, "just treat me like a band" so that I can play a medley of two or more songs (as the band would traditionally do), they can request a tempo, energy level or vibe, and signal me when they’d like to end the dance and I’ll get ready to wrap up the mix with a nice ending. I don’t think I quite achieved that goal with this set, but it was close and the dancers had a blast either way, even if occasionally some of them ended up out on the ends when the dance wrapped up due to me not being able to give the caller as many times through as she wanted and still make the ending sound clean.

Prep

I prepped the individual music in this set basically the same way I did last time: in Mixxx I marked each time through the dance (contra has a very specific form) with color-coded loops or hot cues depending on if they looped cleanly or not, and named the mix-in and mix-out points when I decided what song to pair them with.

A picture of the Mixxx user interface, specifically the deck region. The song "FIDDLE DISCO!" is loaded and several green loops, red and orange hot cues, etc. are selected with labels like "Skip to 4", "Bridge" or "Buildup"

I try to come up with enough mixes to fill the night, plus a few more that are both high and low energy to have in my back pocket in case the caller is just feeling a certain way that night. This way I hopefully always have something that I’ve practiced which works well for each dance in terms of where bouncy or smooth sections of the song fall. That said, I didn’t do a good job of matching songs to dances in every case this night (I would have liked songs with a harder hit on a few of the petronellas), but overall it worked well.

The Set

The night started out with the dance "Jefferson’s Sixpence" by Ann Fallon to which I played a mix of Gaslight Avenue and Reel du Nord , both by The Gaslight Tinkers. This went okay except that I let the track run a bit far before the caller was ready to stop and had to loop the same section a couple of times to make the mix keep working smoothly, which I felt got a bit boring after a handful of times through the dance. The dancers didn’t seem to notice or mind though and were enthusiastic about the start to the evening.

Please excuse the clipping on the recordings, one of the unfortunate problems with the evening (that we’ll come to later) is that the gain on the main board was up way too high and I didn’t have control over that, so it was clipping most of the night. More on that in the retrospective at the end of this post.

</audio>
<figcaption>Gaslight Avenue / Reel du Nord by the Gaslight Tinkers, DJ mix</figcaption>

The second song was a more traditional contra song, a mix of Highland by Wake Up Robin and Fleur de Mandragore by the Free Raisins to the dance "Made Up Tonight" by Erik Hoffman. Due to some technical difficulties with the sound pulled from the main board I didn’t wind up with a recording of this one (or most of the following songs), so let’s just pretend the mix went perfectly and everyone had a good time.

Next the caller picked "Trip to Elsan" by Joe Surdyk and Eric Schreiber. For this I branched out a bit, mixing the (more or less) traditional contra song, Basketball by Countercurrent with FIDDLE DISCO! by Elias Alexander. I had one minor issue with the mix towards the end of FIDDLE DISCO!, a loop that wasn’t really clean but was necessary for the number of times through the dance the caller wanted, but the dancers didn’t seem to notice and they went wild for the modern beats that the tune introduces! This was also the first time I had to re-mix a tune a bit on the fly besides just transitioning between two tunes since FIDDLE DISCO! is crooked and won’t work for contra dancing without (minor) adjustments.

</audio>
<figcaption>Basketball / FIDDLE DISCO! by Countercurrent and Elias Alexander, DJ Mix</figcaption>

The Tuesday night Trolley Barn dance is rather short, so we took a waltz break at this point. I played Here Comes Sunshine by George Paul at the callers request, and Les Yeux Noirs by Burçin since I knew that a number of people from the local Lindy Hop scene were in the crowd. The switch to swing made those dancers happy, even if it confused the waltzers.

Back in the main dance I did a mix of Lafferty’s and Camel Hump by Wild Asparagus to one of my personal favorite contra dances: Tika Tika Timing by Dean Snipes.

Afterwards came what I thought was going to be the second to last song of the evening (we’d been running dances a bit long and were short on time) so I pulled out the show stopper: a medley of Griffin Road and Trip to Moscow , both by Kingfisher, but I added a little contra joke by mixing Rasputin by Boney M in with Trip to Moscow, purely based on the name (but the combo also just worked really well). This one really brought the house down!

I didn’t get a recording of this one, sadly with all the times through, but here was the backup recording I made in advance in case my laptop died during the show or something. The transition sounds like I had a buffer underflow (or just bad timing) on this one, and the mixing is a bit iffy in places, but trust me, it was better live!

</audio>
<figcaption>Griffin Road / Trip to Moscow / Rasputin by Kingfisher and Boney M, DJ mix</figcaption>

At this point one of the organizers said she was having such a good enough time that she offered to let us run a bit over so we added "Vivian’s Catwalk" by Valerie herself to a mix of Muscles / Ride the Wheel by Buddy System and Garbage in A by Great Bear. This went well, except that I had layered the percussion from Garbage in A into Ride the Wheel so that when the transition happened it would have some continuity. Except that I somehow wound up off the beat and had to absolutely scramble to beat match; it sounded terrible, but somehow it didn’t throw Valerie off and once I finally got it beat matched we were still right on time.

For the final tune of the evening Valerie picked "Frock’s Rockin Frolick" by Will Mentor which works great as either a high or low energy dance depending on the caller. To compliment this I brought the energy down a bit with a mix of Brand by Potent Brew and El Bourbon Grande by ContraForce which brought the energy right back up to the high point by its rather loud ending!

Finally we closed the evening down with Ootpik Waltz by Wild Asparagus.

Stuff I Learned

• Either ask to do sound yourself, or don’t be afraid to ask the sound person if something sounds wrong (I didn’t want to be rude, so I didn’t ask, but they didn’t realize it was clipping all night and I’m sure they would have fixed it had I mentioned it!),
• I meant to both record my set on my laptop (no caller) but also record the main outputs from the board with a portable recorder (with caller), and ended up with only a few of the songs actually recorded. You have to actually remember to start recording...
• Prep endings better, if there are no clean loops near the end of the song, prepare a different ending or get better at covering the bad loop with effects or something. I had a few times where the caller asked for "3 more times" or "2 more times" and I had to say "uhhh, how does 1 or 3 sound?" or something along those lines. This was fine with my friend Valerie, but isn’t going to make you any other callers favorite band to work with.
• The jog wheels on the Traktor Kontrol S4 MK3 aren’t big enough. Do I do anything that technically means I need bigger jog wheels? No. Would I feel better about having them? Yes. Unfortunately the only controller that meets all my requirements and has bigger job wheels is the Rane Performer, and that’s way out of my price range (and will likely remain so forever), so maybe this is a "learn to get over it" type situation.
• Don’t just assume the waltz breaks don’t matter, they’re still part of your set and should be selected with care (the tempo was way too fast too, gotta pay more attention to the waltzes in general).

Other Links

• The event on MusicBrainz
• Some (non-live) practice recordings are available on Balfolk.social
• The few live recordings that we got can be found on the Internet Archive

Releases have happened recently that revolve around Poezio, a TUI (Terminal UI) client for XMPP, including Poezio itself, its backend XMPP library Slixmpp, and also the poezio and slixmpp plugins for OMEMO.

Many bug fixes and improvements

Mathieui has already made a proper release note for Slixmpp and I invite you to read it! It includes many bugfixes of course, and internal changes around async handling, that may reflect on some of the APIs you are using.

Poezio has also seen many improvements.

Internally, for one, our default branch has also been moved to "main", many type hints have been added, implicit casts (safeJID) have been removed, lots of event handlers and calls are now async, APIs from Slixmpp are being used instead of redoing our own, many refactoring, various performance improvements.

Pypy3 support was removed because it was causing many users to use the cffi module specifically implemented for pypy3 instead of the more performant C implementation. For those who are running from sources and not using the update script, don’t forget to run make to build the C module.

A license change has happened, and Poezio is now under GPLv3+! While I am not exactly in favour of intellectual property¹ , this is a straightforward lever we have against capitalism² . Poezio being a prime resource for Slixmpp examples, GPL code should reasonably ensure that the 4 freedoms reach end-users. In practice, this should allow for poezio-omemo to be merged into Poezio. I am now personally hoping for Slixmpp to change its license as well.

And other changes more visible to users! To name a few, quality of life improvements such as xmpp:...?join URIs handling in /join, impromptu rooms creation is now more reliable and creates rooms with shorter names, and tab names in the activity bar can be colored using Consistent Color Generation by setting autocolor_tab_names to True. Read more in the changelog.

Plugins have seen changes as well. A new untrackme plugin replaces the now deprecated remove_get_trackers. Link Mauve has also developed a sticker plugin (to send them), similar in essence to what Movim has been doing for ages. Rich presence (activity, gaming, mood and user tune) has been removed from Poezio core and moved in the user_extras plugin. And again many fixes.

Many of these fixes have been realized by mathieui, who is by far the biggest committer on the release, and in general probably the person with the best understanding of the project. Thanks also to louiz for providing the infrastructure all this time, and to eijebong, Ge0rG, Kaghav Gururajan, kaliko, Thomas Hrnciar, jonas’, and southerntofu for the many patches.

Archiving

Archive handling (MAM) was already in the previous release, but has been reworked and should now be more reliable.

When opening a tab, Poezio will fetch 2 screen pages worth of messages if it has no logs for this tab. Archives are automatically stored locally if configured (default), in which case they won’t be re-downloaded but read from the local copy directly the next time they’re requested.

To read older chat messages in a tab, just scroll up with PageUp and Poezio will fetch more automatically if it needs to.

This is configurable with options that have been introduced such as mam_sync or mam_sync_limit to enable/disable the use of MAM and how many messages to fetch at most. And use_log also configures the fact that archives are stored locally.

Some work around storing message IDs – that our log format doesn’t do – will be needed in the future to allow for easier message deduplication.

End-to-End Encryption

The Poezio E2EEPlugin API has been improved to accommodate changes in poezio-omemo, slixmpp-omemo and changes of the OMEMO backend library. Two plugins which are also seeing changes!

Heartbeats are now supported. Heartbeats are meta-messages which transfer only cryptographic key material (nothing else) and are used to strengthen OMEMO’s forward secrecy. This is particularly relevant on clients like Poezio that can stay running in the back for some time, receiving messages without replying.

Some other changes include colored fingerprints using the Consistent Color Generation document – such as specified in the current (0.8) OMEMO spec – and sending encrypted media (aesgcm URIs).

What hasn’t changed is that this plugin lacks a UI and trust management. Hopefully this should come soon, with a little motivation to do UI work.

What comes next

All in all, there aren’t (m)any revolutionary changes, but with these releases come many fixes for paper cuts that hopefully make users happier. This makes me think that even though Poezio is far from being perfect, there doesn’t seem to be many important things missing.

There are however changes that would require a lot of refactoring, such as a multi-account feature, or easier maintenance in general.

We have decided to start migrating Poezio to Rust, in part to be able to refactor the project more easily, and also because it’s a language we’ve come to appreciate over the years with experience in other projects, and more specifically with xmpp-rs, an XMPP library in Rust.

All of this will happen right after the release, and we invite interested people to join the effort!

P.S.: I am looking for poezio screenshots with various setups to display in public places, under a free license. Please send me your screenshots in relatively high quality at blog at bouah.net. And don’t forget to ask pixels appearing on the image for permission!

Monal is an XMPP instant messaging client for macOS and iOS which strives to be the go-to client for these platforms just like the app Conversations IM is for Android. XMPP in general is an open and standardized protocol for real time communication. Anyone can host their own server and communicate freely with each other, just like with email and just like email the used addresses are of the form "user@domain.tld". The user can use different apps and services, such as Monal, from a single but also multiple accounts. This serves a decentral and sovereign infrastructure and digital communication on the internet but also offers many potential for innovation. The chat client for iOS and macOS involves implementing various XEP standards (XMPP extension protocols, adding modern functionality to the XMPP-core and XMPP-im RFCs, see XMPP Extensions).

Designed for iOS and Mac

Things look and work the way you expect. iOS, iPadOS or macOS, there is a version of Monal for you.

Monal is developed under an open-source BSD license that serves the user, while not selling or tracking information for external parties (nor for anyone else). This app exists because it is key to ensure usability on all platforms and within the XMPP network with all its positives aspects when it comes to decentral communication and infrastructure.

Find the development on GitHub!

We are proud to present to you yaxim version 0.9.9 "FOSDEM 2020 Edition". Many things have changed under the hood (reliable messaging with MAM and Push, new UI with runtime permissions), and some exciting new features like even easier onboarding, service browsing and Matrix support. Taken together, yaxim now fulfills the Core IM and the Advanced Mobile profiles of the XMPP Compliance Suite 2020.

yaxim 0.9.9 screenshot

Material Design

Starting with this version, yaxim follows Google’s "Material Design" style. To comply with last year’s strictened Google Play publishing requirements, we had to replace the deprecated ActionBarSherlock library with Google’s own appcompat, which provides the Material style.

This also means that yaxim now requires at least an Android 4.0 device. As 4.0 was released in 2011, this only affects a single-digit number of devices. Users with a ten years old phone need to stay with older yaxim versions, which run on Android 2.3+.

Furthermore, on Android 6+ devices, the user will be asked to grant permission at the moment when they are actually needed (i.e. when sharing a file or taking a photo).

runtime permissions screenshot

On Android 8+, the new notification channels are used by yaxim. A new channel will be created for each contact with a custom ringtone. Once you receive a message from such a contact, you need to use the Android notification settings to change the ringtone, though.

Even Easier Easy Onboarding

yaxim 0.9 introduced Easy XMPP, using the purely client-side XEP-0379: Pre-Authenticated Roster Subscription, which required a server with active In-Band Registration.

The new XEP-0401: Easy User Onboarding allows you to invite new users to your server without being abused by spammers.

Here, you can see a poezio user on a prosody server creating an invitation that is used by yaxim to register and auto-add the inviter:

The invitation page in the example is making use of Google Play Install Referrers to let the newly installed yaxim know the inviter’s address, which has a privacy impact, and therefore is not rolled out to the official landing page yet.

Chat Rooms

There is a new view of your (bookmarked) rooms and a browser of public rooms powered by search.jabber.network.

room search screenshot

Your nickname ("display name") is now synchronized to the server using XEP-0172: User Nickname, and you can change it in the account settings.

Service Discovery and Matrix Support

The room browser can also be used to discover services by entering a valid XMPP address into the search field:

prosody.im search screenshot

prosody.im browser screenshot

prosody.im rooms screenshot

This is not limited to servers and rooms, you can also search for users, chat with them and add them to your contact list:

ge0rg search screenshot

While initially introduced as an April Fools’ Day joke, Matrix support (using the Bifröst bridge is now actually integrated into yaxim, using the official bridge on matrix.org, which has also been made ready for FOSDEM 2020.

Reliable Messaging

For users who are using yaxim in parallel to another client, the new support for XEP-0313: Message Archive Management (MAM) is good news. When connecting to the server, yaxim will now activate MAM and request all messages since the last synchronization. This will ensure that yaxim receives all messages which already were delivered to the other client.

Furthermore, when installed on devices with Google Play Services, yaxim will register for XEP-0357: Push Notifications via the push.yax.im server. This will ensure that the app is woken up from deep sleep or launched when somebody sends you a new message.

These important changes are also reflected in the app privacy policy.

Under the Hood Changes

The internal chat message database has been optimized by adding database indexes for all frequent operations, making yaxim much faster at loading chat windows with long histories.

Furthermore, yaxim was upgraded from the ancient Smack 3 to the Smack 4.3.x XMPP library.

Road to 1.0

This release brings significant changes, and we had hoped to be able to finalize even more to make a glamorous and exciting 1.0 release for the 10 years anniversary. However, the current code base brings some major improvements for reliability and usability and we did not want to hold them back even further.

More work is needed on the contacts view, to allow sorting by conversation age and to quickly search your contacts. Furthermore, the creation of rooms and inviting your friends into them need to be integrated.

MAM was long overdue, and for now only your private messages are queried for. Room history is still obtained using the legacy mechanism, meaning that sometimes, you might miss out on parts of room history.

The embedded image view does not have proper caching, and it will attempt to load any attachment, regardless its size and whether it can be displayed in yaxim. This needs to be restructured in a way that limits downloads to actual image files of a certain maximum size.

Another episode of the XMPP sprints series happened this weekend close to Stockholm in the Nacka prefecture, in a house we rented. Significant improvements to the sprint infra this time are sauna and crêpes!

We worked together on improving a new groupchat bookmarks specification, file transfer interoperability issues, and a future landing page for new XMPP users! As usual, every developer meetup comes up with its share of bug fixes, new ideas, and improvements.

Stockholm scenery

Groupchat Bookmarks (Bookmarks 2)

Last year, Dave Cridland and JC Brand submitted a new specification titled "Bookmarks 2 (This time it’s Serious)". This XEP didn’t get much attention in the community until this weekend.

As mentioned in a previous article, there are multiple specifications for bookmarks in XMPP, one using the Private XML storage, and another one using PEP as storage. Not so long after the Cambridge sprint last year, Daniel submitted a conversion XEP to facilitate client behaviour and thus user experience.

This new specification also uses PEP as storage but it brings a few improvements to the table. It splits updates to the bookmark storage into per entry operations instead of updating the whole storage at once. This allows for finer grained handling in clients and prevents some race conditions.

The XEP came up with its share of challenges that some of us attempted to fix in a pull request that has been submitted and is now awaiting feedback from the authors.

Bookmarks 2 is now implemented in at least 5 clients, (Conversations, Dino, Gajim, Movim, Renga, and some initial work in poezio), but will not be used as long as the feature is not advertised by the server. A new prosody module is also available for adventurous services operators.

New landing page

Roel and I worked on an idea that came up at the UX sprint in Brussels in January to have a landing page for new users. This page would recommend a specific server depending on different factors that would be gathered automatically for the most part (if not all). This is more or less similar to other portals like joinmastodon, or nextcloud sign-up process.

Building up the website isn’t the hardest part. What is hard is finding ways to convey to the user what "federation" or "public network" mean. Roel teaches in Interaction Design and was a great help over the weekend. We came up with a narrative for the project and a sketch for a sign-in flow.

The project is far from being over, this is only the tip of the iceberg. Lots of work needs to be done with the "stakeholders", that is mainly users and server operators.

To know what server to recommend to users we first need to get a list of servers we are confident about and willing to recommend. This would mandate discussing the issue with server operators to get feedback on a required "feature set" and policies. All this would then be fed into usability testing sessions for users to validate all of it. After that, we would need lots of promotion around it and that’s also going to take a significant amount of effort.

While I am excited about all this I don’t think diving in head first is a good strategy and I would rather take it slow.

And more

Pulkomando has been working on implementing IBR support in Renga, and reported with Link Mauve issues about server implementations that weren’t respecting the specification. The issue in prosody has been fixed and one has been opened for ejabberd.

Larma tackled issues with bot bridging where users of bridged networks are displayed as talking through the bot. This happens for example with matterbridge. This could be improved UI-wise but requires some groundwork and spawned discussions about the groupchat protocol in some specific cases.

Fiaxh spent some time improving the empty placeholder for no opened conversations in Dino. Here is a preview:

Dino no-conversations placeholder screen

Other people worked on Jingle File Transfer interoperability. They narrowed down the cause of a somewhat old issue in gajim, discovered an issue with the epoll backend in prosody, and another in dino.

You can contribute too!

I would like to take this opportunity to remind you that you can also contribute to sprints! If you are a developer, a translator, working on documentation, or in any other way contributing to an XMPP implementation, we encourage you to find 2-3 other people close to you and organize a sprint!

TODO: come up with a platform to show interest close to you.

The XMPP Standards Foundation (specifically the SCAM team) will be happy to help you get it all sorted, and also provide some budget for your event if necessary. Please do contact us!

yaxim enters the Matrix

Starting today, yaxim is switching its protocol foundation from the deprecated exchange of clumsy and inefficient XML streams to the modern and elegant combination of HTTP and JSON/REST, the Matrix protocol.

Protocol History and Comparison

The XMPP protocol celebrated its 20th birthday early this year. The Matrix followed two months later and is currently in the middle of its own celebration. Some fifteen years later, a small company decided to use the strong brand value of the Matrix name to reinvent XMPP with a modern facade.

Evil voices claim that MATRIX stands for Monolithic, Awefully Trendy Re-Implementation of XMPP, and there is some truth to this, if we compare the words of the respective founding fathers:

Jabber is a new project I recently started to create a complete open-source platform for Instant Messaging with transparent communication to other IM systems(ICQ, AIM, etc).

I think they missed the bit where Matrix is called Matrix because it bridges (matrixes) the existing networks (Slack, IRC, Telegram, Discord, XMPP, etc) in, rather than needing to convince everyone to join.

However, the Matrix protocol has outgrown Vector Ltd, there is a version (削除) 1.0 (削除ここまで) 0.4 specification, and even a Matrix.org Foundation.

This is much superior to XMPP, which is based on some arcane specifications maintained by a bunch of grey beards, plus a separate organisation for protocol extensions. In addition, Matrix supports working over 100 bits per second connections, while XMPP only gives you 75 bps.

The monolithic protocol is another huge advantage compared to many hundreds of optional extensions, and the rumors of Matrix fragmentation are a blatant lie.

Enter the Matrix

Therefore, the yaxim developers have decided to take the blue pill, to move forward, and to use the better and more modern and mobile friendly polling based HTTP scheme. Starting with the current beta release, you can enter Matrix chat rooms and talk to users on the Matrix.

The legacy XMPP protocol is remaining in the release for now, but will be removed in the near future to reduce the bloat of yaxim. You will be able to migrate your contacts to your new Matrix account by using the Bifröst bridge.

In parallel, we are working on switching yax.im from prosody to Synapse. The data transition is already completed, and we are only waiting for the data center provider to add 512GB of RAM to the machine before we can switch over.

We're pleased to announce the last release of ejabberd for 2014! Thanks to contributors, this release includes great improvements and opens road to 2015.

ejabberd Community 14.12 includes many bugfixes, and a few new features:

• New module mod_client_state implements XEP-0352: Client State Indication
• New module mod_fail2ban to ban IPs that show malicious signs
• New option store_empty_body in mod_offline

read more

Full announcement: ejabberd Community 14.05: the culmination of a year of change

ejabberd Community 14.05 has great new features, several improvements and many bugfixes over the previous 13.12 release:

ejabberd now includes support for:
- XEP-0198: Stream Management (EJAB-532)

read more

We are pleased to announce a new stable release of ejabberd, ejabberd Community 13.12.

It has several bugfixes over the previous 13.10 release, and a few new features:

• New OpenSSL ciphers option in c2s, s2s and s2s_out
• mod_roster: new access rule to restrict roster modificartion
• mod_pubsub: support for data migration from mnesia to odbc
• ejabberd_xmlrpc included

As usual, the release is tagged in the Git source code repository on github

read more

We are pleased to announce a new stable release of ejabberd, ejabberd Community 13.10.

It has some changes, several improvements and many bugfixes over the previous (not officially announced) 13.06. It is also the first official stable release of ejabberd Community after ejabberd 2.1.13. You are now pleased to use ejabberd community as reference for stable releases of ejabberd, from the master branch. ejabberd 2.1.x support is discontinued.

The most noticeable changes since 13.03-beta and 13.06 are:

read more

We are pleased to announce the bugfix release ejabberd 2.1.13.
It includes a few bugfixes over 2.1.12:

• Compilation: Detect correctly newer Darwin versions (EJAB-1594)
• Guide: ejabberd_service expects a shaper_rule, not a shaper
• MUC: Handle multiple < and > in mod_muc_log plaintext mode (EJAB-1640)
• MUC: Handle ~ control sequence in text of mod_muc_log (EJAB-1639)
• MUC: list_to_integer/2 only works in OTP R14 and newer
• Pubsub: access_createnode acl also applies to auto created nodes
• Web: Normalize HTTP path
  

read more

Xiph.org has just posted the second in its series of videos on digital media concepts and techniques. It’s packed with information and demonstrations, and you’re sure to learn a huge amount. As an added bonus, it’s hosted by Monty, the creator of Ogg Vorbis (and many other amazing things). You couldn’t ask for a more qualified teacher.

Watch below, or on Xiph.org.

There is also a detailed write up.

ejabberd 2.1.10, 3.0.0-alpha-5 and exmpp 0.9.9 have been released, after several months of development. They contain a few bugfixes.

ejabberd 2.1.10

These are the major bugfixes:

• Erlang/OTP compatibility
  • Support Erlang/OTP R15B regexp and drivers (EJAB-1521)
  • Fix modules update in R14B04 and higher
  • Fix modules update of stripped beams (EJAB-1520)
• XMPP Core

read more

ejabberd 2.1.9, ejabberd 3.0.0-alpha-4, and exmpp 0.9.8 have been released, after several months of development. They contain a lot of bugfixes, improvements and some new features.

ejabberd 2.1.9

This release includes a lot of bugfixes and improvements. This is just a short list of them:

• New SASL SCRAM-SHA-1 authentication mechanism (EJAB-1196)
• New option: resource_conflict (EJAB-650)

read more

ejabberd 2.1.6 has been released, after four months of development. It contains a lot of bugfixes, improvements and some new features.

This is a small list of changes:

• BOSH: Fix rare loop, support vhosts, allow module restart
• Config: Default configuration allows registrations only from localhost
• Config: Support to change loglevel per module at runtime
• Erlang/OTP: Fix compatibility from R10B-9 to R14B01
• ODBC: Compatibility with PostgreSQL 9.0
• Privacy lists: Fix to allow block by group and subscription again

read more

ejabberd gets 8 years old. But no party yet, Yozhik is bugfixing 2.1.6 and testing 3.0.0-alpha-2.

The source and more photographs of hedgehogs pets.

read more

After many months of planning, ejabberd and exmpp have been fully migrated to Git.

During the last 7 years, ejabberd source code was hosted at:

• CVS at Jabber.Ru
• CVS at JabberStudio.org
• SVN at ProcessOne
• Git preliminarly built with git-svn, at Github

Starting now, ejabberd source code is natively in Git, and hosted at:

• https://git.process-one.net/ejabberd
• http://github.com/processone/ejabberd

The minimal instructions to start using it are mentioned in:
http://www.process-one.net/en/ejabberd/downloads

read more

Yes, ejabberd is already 7 years old.

Let's celebrate with a timeline of ejabberd, Erlang/OTP, XMPP/Jabber protocol, and Tkabber:

If you find any mistake, please comment. I built the graph using EasyTimeLine.pl, if you want the datafile, please comment.

read more

To celebrate that ejabberd turns 6 years old, I've prepared a video that shows the history or ejabberd trunk SVN during those years: authors, acknowledgments, type of files, dates and releases. The video was built with code_swarm.

• RapidShare, click in "Free user" button, wait 50 seconds, then click in "Download" big button.
• Jabber.ru direct download

read more