Monthly Update for February 2025 #556

fnetX commented 2025年02月10日 21:57:44 +01:00

Please collect snippets and points of interest here.

I would appreciate if someone else could do the update. I see that I only manage to do it with more delay month-to-month and I suspect this is sign to take a break. I'm not good at repetitive tasks, unfortunately.

Please collect snippets and points of interest here. I would appreciate if someone else could do the update. I see that I only manage to do it with more delay month-to-month and I suspect this is sign to take a break. I'm not good at repetitive tasks, unfortunately.

earl-warren commented 2025年02月25日 10:02:10 +01:00

DDoS on code.forgejo.org

On 9 February, code.forgejo.org got hit by a DDoS. To this day, it is still unclear if it was malicious or an excessive number of requests from a crawler.

When code.forgejo.org timedout for the first time on that day, it was thought to be excessive crawling to be blocked, like a number of others in the past months. And indeed, such a crawler was quickly discovered by looking at the logs. It was not obeying robots.txt and did not identify itself with a proper User-Agent header. A small range of IP addresses was added to the existing block list.

When code.forgejo.org timedout again that same day, it was suspected that some IPs of the range were missed. However, after analyzing the logs, it appeared that the large number of requests did not originate from a small number of IP addressed: they were coming from thousands of them.

The k8s cluster hosting code.forgejo.org was setup at the end of 2024 and lacked good visibility on the incoming requests, which turned out to be a problem in this situation. While trying to figure this out, the DDoS slowed down code.forgejo.org so much that it stopped responding. The monitoring in place worked and rang the alarm of the devops team. However it recovered a few minutes later, as if the author of the DDoS was satisfied the service went down and left the scene.

After two days of general slowness and occasional disruptions enough logs were collected and analyzed to conclude that:

• The DDoS happened in waves, a few hours apart.
• A wave had from 10,000 to 50,000 unique IPs, each sending an average of two requests.
• Between 200,000 and 300,000 unique IPs sent request on a given day.

That was enough to try a simple mitigation:

• The IP ranges to which each IP belongs were collected (e.g 1.7.64.1).
• The top 500 IP ranges that contributed the most number of IPs to the DDoS were blocked.

This successfully stopped over 90% of the DDoS and code.forgejo.org recovered. However, blocking entire IP ranges, sometime millions of IPs at a time, was bound to also impact legitimate users. It was discovered on that occasion that the Golang proxies (keeping a copy of Go packages) are using a few different IP ranges and send requests from hundreds of different IPs. They were unblocked, as well as an individual contributor also blocked by accident.

A week later, on 18 February, the DDoS stopped as suddenly as it started. The number unique IPs hitting code.forgejo.org went from over 200,000 daily to 15,000 overnight. That concluded a DDoS which used around one million unique IPs to hit code.forgejo.org.

## DDoS on code.forgejo.org On [9 February](https://codeberg.org/forgejo/discussions/issues/297), code.forgejo.org got hit by a DDoS. To this day, it is still unclear if it was malicious or an excessive number of requests from a crawler. When code.forgejo.org timedout for the first time on that day, it was thought to be excessive crawling to be blocked, like a number of others in the past months. And indeed, such a crawler was quickly discovered by looking at the logs. It was not obeying `robots.txt` and did not identify itself with a proper `User-Agent` header. A small range of IP addresses was added to the existing block list. When code.forgejo.org timedout again that same day, it was suspected that some IPs of the range were missed. However, after analyzing the logs, it appeared that the large number of requests did not originate from a small number of IP addressed: they were coming from thousands of them. The k8s cluster hosting code.forgejo.org was setup at the end of 2024 and lacked good visibility on the incoming requests, which turned out to be a problem in this situation. While trying to figure this out, the DDoS slowed down code.forgejo.org so much that it stopped responding. The [monitoring in place](https://status.forgejo.ovh/) worked and rang the alarm of the [devops team](https://codeberg.org/forgejo/governance/src/branch/main/TEAMS.md#devops). However it recovered a few minutes later, as if the author of the DDoS was satisfied the service went down and left the scene. After two days of general slowness and occasional disruptions enough logs were collected and analyzed to conclude that: - The DDoS happened in waves, a few hours apart. - A wave had from 10,000 to 50,000 unique IPs, each sending an average of two requests. - Between 200,000 and 300,000 unique IPs sent request on a given day. That was enough to try a simple mitigation: - The IP ranges to which each IP belongs were collected (e.g [1.7.64.1](https://stat.ripe.net/widget/whois#w.resource=1.7.64.1)). - The top 500 IP ranges that contributed the most number of IPs to the DDoS were blocked. This successfully stopped over 90% of the DDoS and code.forgejo.org recovered. However, blocking entire IP ranges, sometime millions of IPs at a time, was bound to also impact legitimate users. It was discovered on that occasion that the Golang proxies (keeping a copy of Go packages) are using a few different IP ranges and send requests from hundreds of different IPs. They were unblocked, as well as an individual contributor also blocked by accident. A week later, on [18 February](https://codeberg.org/forgejo/discussions/issues/297#issuecomment-2842888), the DDoS stopped as suddenly as it started. The number unique IPs hitting code.forgejo.org went from over 200,000 daily to 15,000 overnight. That concluded a DDoS which used around one million unique IPs to hit code.forgejo.org.

earl-warren commented 2025年02月25日 10:02:30 +01:00

DDoS mitigation

The mitigation of this month DDoS was hand made. It was surprising to discover that Traefik, the reverse proxy used by the k8s cluster hosting code.forgejo.org, does not provide a method to compute and block the top most used IP ranges. It is a radical but effective measure, a practical way to keep the service afloat until the DDoS goes away.

It took a few days to figure out and implement this mitigation, using a number of shell snippets and learning how IP ranges are distributed over the net. In the spirit of saving time to deal with a similar DDoS, a implementation of this method was described to be part of Forgejo. But it does not belong in Forgejo: it should be implemented in a reverse proxy that sits in front and the idea was abandoned.

A blocker to such an implementation was however resolved: figuring out the list of IP ranges currently in use worldwide. In theory it could be extracted from the same database that is used by whois. But some of the Regional Internet Registries (RIRs) publish their database under restrictive usage terms. Instead, the Routing Information Base (RIB) used by BGP is analyzed weekly and around 500,000 IP ranges are compiled and published.

The next time a DDoS hits code.forgejo.org, the logs will be ready for forensic analysis. The latest database of IP ranges will be used for blocking some of them, keeping the DDoS contained until it goes away.

## DDoS mitigation The mitigation [of this month DDoS](https://codeberg.org/forgejo/discussions/issues/297) was hand made. It was surprising to discover that [Traefik](https://traefik.io/), the reverse proxy used by the k8s cluster hosting code.forgejo.org, does not provide a method to compute and block the top most used IP ranges. It is a radical but effective measure, a practical way to keep the service afloat until the DDoS goes away. It took a few days to figure out and implement this mitigation, using a number of shell snippets and learning how IP ranges are distributed over the net. In the spirit of saving time to deal with a similar DDoS, a [implementation of this method was described](https://codeberg.org/forgejo/forgejo/issues/6907) to be part of Forgejo. But it does not belong in Forgejo: it should be implemented in a reverse proxy that sits in front and the idea was abandoned. A blocker to such an implementation was however resolved: figuring out the list of IP ranges currently in use worldwide. In theory it could be extracted from the same database that is used by [whois](https://linux.die.net/man/1/whois). But some of the [Regional Internet Registries (RIRs)](https://www.arin.net/about/welcome/region/#regional-internet-registry-regions) publish their database under restrictive usage terms. Instead, the [Routing Information Base (RIB)](https://ris.ripe.net/docs/mrt/#name-and-location) used by BGP [is analyzed weekly](https://code.forgejo.org/forgejo/ipranges/src/commit/1220c093240966b421170850d2323ba78ccc1312/.forgejo/workflows/release.yml) and around 500,000 IP ranges are compiled and [published](https://code.forgejo.org/forgejo/ipranges). The next time a DDoS hits code.forgejo.org, the logs will be ready for forensic analysis. The [latest database of IP ranges](https://code.forgejo.org/forgejo/ipranges) will be used for blocking some of them, keeping the DDoS contained until it goes away.

0ko commented 2025年03月08日 15:15:30 +01:00

Some context for this writeup can be found in dev chatroom.

---

Localization

A new component forgejo-next is now in use. Unlike the main component, this one uses JSON for storing strings. It is easier to parse and free of many quirks the INI format has. Currently it only has a few strings, but it it suggested that new strings are now added to it. We'll also be moving old strings to it, to eventually get rid of the INI translations completely.

With JSON format Forgejo now also supports language-specific plurals, making translator experience more pleasant and UI more cohesive for languages where there's different number of plurals than two. This improvement will be available in Forgejo v11.

New translations have been started: Hebrew and Romanian.

Some context for this writeup can be found in dev chatroom. --- ### Localization A new component `forgejo-next` is now in use. Unlike the main component, this one uses JSON for storing strings. It is easier to parse and free of many quirks the INI format has. Currently it only has a few strings, but it it suggested that new strings are now added to it. We'll also be moving old strings to it, to eventually get rid of the INI translations completely. With JSON format Forgejo now also supports language-specific plurals, making translator experience more pleasant and UI more cohesive for languages where there's different number of plurals than two. This improvement will be available in Forgejo v11. New translations have been started: [Hebrew](https://translate.codeberg.org/projects/forgejo/-/he/) and [Romanian](https://translate.codeberg.org/projects/forgejo/-/ro/).

fnetX commented 2025年03月10日 00:00:59 +01:00

FOSDEM

As briefly mentioned in the last monthly update, some of the Forgejo developers have been to FOSDEM and met at the Codeberg+Forgejo stand. They met a lot of happy users, discussed ideas and code architecture and finally managed to meet each other in person.

There have been meetings with other projects that are considering to move to Forgejo or became curious about the project. There has been more in-depths meeting with the people at Fedora and talks about future collaboration. Two Forgejo developers have been invited to dinner by Fedora, thank you for that!

You can find a discussion issue with more impressions from FOSDEM.

(Apologies, I wanted to have something more verbose, but I currently don't find the time to write more and I also think that the discussion issue actually makes another writeup unnecessary)

## FOSDEM As briefly mentioned in the last monthly update, some of the Forgejo developers have been to FOSDEM and met at the Codeberg+Forgejo stand. They met a lot of happy users, discussed ideas and code architecture and finally managed to meet each other in person. There have been meetings with other projects that are considering to move to Forgejo or became curious about the project. There has been more in-depths meeting with the people at Fedora and talks about future collaboration. Two Forgejo developers have been invited to dinner by Fedora, thank you for that! You can find a [discussion issue with more impressions from FOSDEM](https://codeberg.org/forgejo/discussions/issues/290). *(Apologies, I wanted to have something more verbose, but I currently don't find the time to write more and I also think that the discussion issue actually makes another writeup unnecessary)*

fnetX commented 2025年03月10日 00:17:56 +01:00

Sustainability

NLnet, a dutch stichting devoted to the support of free/libre software development, agreed to extending a previous grant for Forgejo by a few months and repurpose the leftover budget. An amendment workplan was agreed on and several contributors started working on it.

The new tasks focus on improving the Forgejo documentation as well as bringing limited moderation capabilities (the option to report content to an instance's administrators). The architecture and database structure has been agreed on. The next tasks include an admin UI. If you are moderating communities via other software or otherwise have valuable insights that could help us design the UX workflows for admins, feel free to let us know in the corresponding design issue.

## Sustainability [NLnet](https://nlnet.nl/), a dutch stichting devoted to the support of free/libre software development, agreed to extending a previous grant for Forgejo by a few months and repurpose the leftover budget. An [amendment workplan](https://codeberg.org/forgejo/sustainability/src/commit/22c15e2ef92585259c4c0000286845f1a160caef/2022-12-01-nlnet/2025-02-07-extended-workplan.md) was agreed on and several contributors started working on it. The new tasks focus on improving the Forgejo documentation as well as bringing limited moderation capabilities (the option to report content to an instance's administrators). The [architecture and database structure has been agreed on](https://codeberg.org/forgejo/forgejo/pulls/6977). The next tasks include an admin UI. If you are moderating communities via other software or otherwise have valuable insights that could help us design the UX workflows for admins, feel free to let us know in the [corresponding design issue](https://codeberg.org/forgejo/design/issues/30).

fnetX commented 2025年03月10日 00:22:33 +01:00

Moving to Forgejo

During FOSDEM, we have learned about many projects that are considering or planning a migration to Forgejo, but are blocked due to various aspects. These range from missing features of various extent to the need of help setting up or planning the migration. To help keep in touch with interested projects, coordinating migration assistance and tracking necessary additions to Forgejo, a dedicated repository for migrations to Forgejo has been set up.

If you have a success story to share about your usage of Forgejo, or if you have projects that you'd like to see on Forgejo but that are not yet decided or blocked for some reason, feel free to submit to the tracker and let us know.

## Moving to Forgejo During FOSDEM, we have learned about many projects that are considering or planning a migration to Forgejo, but are blocked due to various aspects. These range from missing features of various extent to the need of help setting up or planning the migration. To help keep in touch with interested projects, coordinating migration assistance and tracking necessary additions to Forgejo, a [dedicated repository for migrations to Forgejo](https://codeberg.org/forgejo-contrib/moving-to-forgejo/) has been set up. If you have a success story to share about your usage of Forgejo, or if you have projects that you'd like to see on Forgejo but that are not yet decided or blocked for some reason, feel free to submit to the tracker and let us know.

fnetX commented 2025年03月10日 00:29:27 +01:00

Forgejo for Enterprise

There is high interest in using a simple and lightweight Git forge for commercial projects. During FOSDEM, we have been surprised to learn about quite some big companies ("off the recording") using Forgejo. However, many potential users are hesitating due to the perceived difficulty in finding commercial support for the software.

Forgejo's ecosystem has several advantages to enterprise customers. Contrary to competing software such as the fully proprietary GitHub and the open core GitLab and Gitea, there is no vendor lock in for Forgejo to a single vendor. There is an inventory of professional service providers that slowly evolve to an ecosystem where Forgejo users can freely choose from depending on their needs and satisfaction.

However, these advantages are not yet clearly used to attract customers, and open questions about the influence of business interest have to be discussed. A discussion issue about Forgejo for enterprise has been created for this purpose, tracking the status quo and potential improvements.

If you are using Forgejo for commercial purpose, it might help us a lot if you (ask if you can) publicly write about your case as a success story. Let us know in the issue.

## Forgejo for Enterprise There is high interest in using a simple and lightweight Git forge for commercial projects. During FOSDEM, we have been surprised to learn about quite some big companies ("off the recording") using Forgejo. However, many potential users are hesitating due to the perceived difficulty in finding commercial support for the software. Forgejo's ecosystem has several advantages to enterprise customers. Contrary to competing software such as the fully proprietary GitHub and the open core GitLab and [Gitea](https://codeberg.org/forgejo/discussions/issues/102), there is no vendor lock in for Forgejo to a single vendor. There is an inventory of [professional service providers](https://codeberg.org/forgejo/professional-services/issues) that slowly evolve to an ecosystem where Forgejo users can freely choose from depending on their needs and satisfaction. However, these advantages are not yet clearly used to attract customers, and open questions about the influence of business interest have to be discussed. A discussion issue about [Forgejo for enterprise](https://codeberg.org/forgejo/discussions/issues/308) has been created for this purpose, tracking the status quo and potential improvements. If you are using Forgejo for commercial purpose, it might help us a lot if you (ask if you can) publicly write about your case as a success story. Let us know in the issue.

fnetX commented 2025年03月10日 00:33:18 +01:00

I hoped to find the time to wrap things up before my current vacation. I did not, but now I posted most of what I consider important for February (from what I had in mind).

It would be great if someone else could use the snippets to finally wrap up the monthly letter. Otherwise, it will be around 16th of March for me.

I hoped to find the time to wrap things up before my current vacation. I did not, but now I posted most of what I consider important for February (from what I had in mind). It would be great if someone else could use the snippets to finally wrap up the monthly letter. Otherwise, it will be around 16th of March for me.

floss4good commented 2025年03月15日 01:30:50 +01:00

It would be great if someone else could use the snippets to finally wrap up the monthly letter.

I'm not good at writing, but putting together some snippets shouldn't be that hard, right?
My first try: #564

> It would be great if someone else could use the snippets to finally wrap up the monthly letter. I'm not good at writing, but putting together some snippets shouldn't be that hard, right? My first try: #564