long delays in mail notifications #2105

mbunkus commented 2025年08月25日 21:29:11 +02:00

Comment

Over the last couple of months mail notifications for issues, merge requests etc. have taken very long to arrive in my mailbox. Often it's around one hour; today it was more like 1.5 hours; I've also seen more than two hours. This makes interactive bug fixing rather tedious. If I ask a reporter for feedback I likely won't get the mail notification that they have provided the feedback for another three to four hours (1.5 to 2h until they get the notification of my request, another 1.5 to 2h until I get the notification of their feedback if they answer immediately).

This seems to be an issue inside the Codeberg infrastructure, and definitely not something between Codeberg's & my own mail server. Why do I think so? Two reasons:

1. The Received headers clearly show that the delay happens not when the email is generated, but during delivery within the Codeberg infrastructure.
2. I've looked through my mail server's Postfix log files for any connects from Codeberg today. The first contact was at 11:08 CEST (graylisted), another one at 11:09 CEST (graylisted again) & then a successful delivery at 11:10 CEST. However, the issue for which this was the notification was opened at 09:49 CEST already. No contact from Codeberg before 11:08 CEST.

The message I've just talked about is message ID <LINET-Services/linet-truenas-snmp/issues/1@codeberg.org>. The issue was opened on 2025年08月25日 09:49 CEST. The mail arrived on my server on 2025年08月25日 11:10 CEST, 1h20m later. Here are all Received headers from the raw email:

Received: from smtp.codeberg.org (smtp.codeberg.org [193.26.156.135])
 (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
 key-exchange X25519 server-signature ECDSA (secp384r1))
 (No client certificate requested)
 by liselle.bunkus.org (Postfix) with ESMTPS id 8B8A72403DE
 for <codeberg@bunkus.online>; 2025年8月25日 11:10:05 +0200 (CEST)
Received: from forgejo-prod-main.lxc.achtermann.local (achtermann.m.codeberg.org [217.197.91.189])
 by smtp.codeberg.org (Postfix) with ESMTPSA id 87C27160B33
 for <codeberg@bunkus.online>; 2025年8月25日 11:08:58 +0200 (CEST)
Received: by forgejo-prod-main.lxc.achtermann.local (Postfix, from userid 1001)
 id 0FBC3524FB6; 2025年8月25日 09:49:07 +0200 (CEST)

As usual the headers are in reverse chronological order, meaning:

1. The first server contacted was on 09:49 CEST, right after the issue was generated. Seems that generation of notifications works instantly.
2. The delivery to the next server (still within Codeberg's infrastructure) was at 11:10 CEST; this is obviously where the huge delay occurred
3. Delivery to my own mail server only took about a minute

I can provide further message IDs/dates & timestamps/log files if required.

### Comment Over the last couple of months mail notifications for issues, merge requests etc. have taken very long to arrive in my mailbox. Often it's around one hour; today it was more like 1.5 hours; I've also seen more than two hours. This makes interactive bug fixing rather tedious. If I ask a reporter for feedback I likely won't get the mail notification that they have provided the feedback for another three to four hours (1.5 to 2h until they get the notification of my request, another 1.5 to 2h until I get the notification of their feedback if they answer immediately). This seems to be an issue inside the Codeberg infrastructure, and definitely not something between Codeberg's & my own mail server. Why do I think so? Two reasons: 1. The `Received` headers clearly show that the delay happens not when the email is generated, but during delivery within the Codeberg infrastructure. 2. I've looked through my mail server's Postfix log files for any connects from Codeberg today. The first contact was at 11:08 CEST (graylisted), another one at 11:09 CEST (graylisted again) & then a successful delivery at 11:10 CEST. However, the issue for which this was the notification was opened at 09:49 CEST already. No contact from Codeberg before 11:08 CEST. The message I've just talked about is message ID `<LINET-Services/linet-truenas-snmp/issues/1@codeberg.org>`. The issue was opened on 2025年08月25日 09:49 CEST. The mail arrived on my server on 2025年08月25日 11:10 CEST, 1h20m later. Here are all `Received` headers from the raw email: ``` Received: from smtp.codeberg.org (smtp.codeberg.org [193.26.156.135]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1)) (No client certificate requested) by liselle.bunkus.org (Postfix) with ESMTPS id 8B8A72403DE for <codeberg@bunkus.online>; 2025年8月25日 11:10:05 +0200 (CEST) Received: from forgejo-prod-main.lxc.achtermann.local (achtermann.m.codeberg.org [217.197.91.189]) by smtp.codeberg.org (Postfix) with ESMTPSA id 87C27160B33 for <codeberg@bunkus.online>; 2025年8月25日 11:08:58 +0200 (CEST) Received: by forgejo-prod-main.lxc.achtermann.local (Postfix, from userid 1001) id 0FBC3524FB6; 2025年8月25日 09:49:07 +0200 (CEST) ``` As usual the headers are in reverse chronological order, meaning: 1. The first server contacted was on 09:49 CEST, right after the issue was generated. Seems that _generation_ of notifications works instantly. 2. The delivery to the next server (still within Codeberg's infrastructure) was at 11:10 CEST; this is obviously where the huge delay occurred 3. Delivery to my own mail server only took about a minute I can provide further message IDs/dates & timestamps/log files if required.

Gusted commented 2025年08月25日 21:49:14 +02:00

Hi,

Were these notifications from users who recently created an account? Such email notifications are currently delayed by about an hour, as they are more likely to be spam and can be removed from the queue if they turn out to be spam. This change was implemented after the spam wave earlier this year. We could look into reducing or dropping this delay depending on some heuristics, CC @fnetX

Hi, Were these notifications from users who recently created an account? Such email notifications are currently delayed by about an hour, as they are more likely to be spam and can be removed from the queue if they turn out to be spam. This change was implemented after the spam wave earlier this year. We could look into reducing or dropping this delay depending on some heuristics, CC @fnetX

mbunkus commented 2025年08月25日 22:05:32 +02:00

Thanks for the quick reply. That's quite likely, given that I haven't been on Codeberg that long — and that the mails today were from repositories that are less than two weeks old. The notification for your comment arrived pretty much immediately, which fits with your description.

Let me check the other notifications from today... Yep, both of the two users who opened issues today created their accounts today. Both are not spam, for what it's worth, but I get the argument.

I don't have a big problem with delaying the notification for the initial creation of the issue/MR, if I'm honest. What I dislike is that once I reply I cannot expect notifications to flow quickly, and that's mostly what irks me. If you can implement complex conditions for the anti-spam heuristics, maybe look into something like "if repo owner(s) reply to an issue/MR then don't delay OP's further replies to the same issue/MR anymore".

Thanks for the quick reply. That's quite likely, given that I haven't been on Codeberg that long — and that the mails today were from repositories that are less than two weeks old. The notification for your comment arrived pretty much immediately, which fits with your description. Let me check the other notifications from today... Yep, both of the two users who opened issues today created their accounts today. Both are not spam, for what it's worth, but I get the argument. I don't have a big problem with delaying the notification for the initial creation of the issue/MR, if I'm honest. What I dislike is that once I reply I cannot expect notifications to flow quickly, and that's mostly what irks me. If you can implement complex conditions for the anti-spam heuristics, maybe look into something like "if repo owner(s) reply to an issue/MR then don't delay OP's further replies to the same issue/MR anymore".

fnetX commented 2025年08月25日 23:15:24 +02:00

The current check is very basic and I see little potential to enhance it, unfortunately. It would require adding some checks into Forgejo itself. Which has been talked about for years now, but it's unlikely to make progress very soon.

The only heuristic I could think of is the volume of emails per user. But this would also break for repos where many people receive notification emails.

The current check is very basic and I see little potential to enhance it, unfortunately. It would require adding some checks into Forgejo itself. Which has been talked about for years now, but it's unlikely to make progress very soon. The only heuristic I could think of is the volume of emails per user. But this would also break for repos where many people receive notification emails.

mbunkus commented 2025年08月26日 00:04:40 +02:00

I understand. Thanks for the explanations. I'll close again as it's obvious working as intended.

I understand. Thanks for the explanations. I'll close again as it's obvious working as intended.

fnetX commented 2025年08月26日 17:42:08 +02:00

I appreciate your feedback and understand that it breaks your workflow. When we did the changes, we assumed that many people work with in-app notifications and that new users are a rare exception in the daily work of skilled developers who rely on email notifications only. For many users, email notifications are just a "reminder" that they missed something on Codeberg (when they don't sign in often), so a delay is acceptable.

However, it's valuable information that you rely on email notifications and that it slows down your conversation with new users. We did not consider this, but need to do this in the future.

I appreciate your feedback and understand that it breaks your workflow. When we did the changes, we assumed that many people work with in-app notifications and that new users are a rare exception in the daily work of skilled developers who rely on email notifications only. For many users, email notifications are just a "reminder" that they missed something on Codeberg (when they don't sign in often), so a delay is acceptable. However, it's valuable information that you rely on email notifications and that it slows down your conversation with new users. We did not consider this, but need to do this in the future.

mbunkus commented 2025年08月26日 18:14:26 +02:00

Just as an explanation of how I work & why I work that way: normally I'm working in my browser & my mail application for most of the day. Mail is simply one central place where notifications from different services (Codeberg, Github, various forums etc.) come together. Instead of having to check multiple tabs in my browser regularly for notifications I can simply check one application.

Back when I was on Gitlab there was at least a browser extension that queried Gitlab regularly & showed in-browser notifications for new stuff. That was working well, too. Unfortunately I haven't found a comparable extension for Forgejo/Gitea/Codeberg. That would alleviate the need to visit the Codeberg notification tab regularly.

As for the "a lot of new users" effect: yeah, I attribute that to two facts:

1. I'm relatively new on Codeberg, only moved here from Gitlab in March. Therefore many of my regular reporters have created accounts here as well as they hadn't been on Codeberg before.
2. My main project, MKVToolNix, has always had a rather high number of one-time contributors, both for issues & for merge requests. Those predominantly fall into the "new user" category, I guess.
3. My other projects, most of them CheckMK extensions, are rather new themselves, but judging from co-maintaining other extensions elsewhere I expect the amount of one-time contributors to be high there as well.

All this is just background information for consideration. As I said before, I understand the need for the implemented mechanisms.

Just as an explanation of how I work & why I work that way: normally I'm working in my browser & my mail application for most of the day. Mail is simply one central place where notifications from different services (Codeberg, Github, various forums etc.) come together. Instead of having to check multiple tabs in my browser regularly for notifications I can simply check one application. Back when I was on Gitlab there was at least a browser extension that queried Gitlab regularly & showed in-browser notifications for new stuff. That was working well, too. Unfortunately I haven't found a comparable extension for Forgejo/Gitea/Codeberg. That would alleviate the need to visit the Codeberg notification tab regularly. As for the "a lot of new users" effect: yeah, I attribute that to two facts: 1. I'm relatively new on Codeberg, only moved here from Gitlab in March. Therefore many of my regular reporters have created accounts here as well as they hadn't been on Codeberg before. 2. My main project, MKVToolNix, has always had a rather high number of one-time contributors, both for issues & for merge requests. Those predominantly fall into the "new user" category, I guess. 3. My other projects, most of them CheckMK extensions, are rather new themselves, but judging from co-maintaining other extensions elsewhere I expect the amount of one-time contributors to be high there as well. All this is just background information for consideration. As I said before, I understand the need for the implemented mechanisms.