Suggestion: new feature freeze until performance issues are fixed #908

aral commented 2023年01月23日 16:38:08 +01:00

Codeberg is basically unusable right now with a 2kb push taking over 2 minutes (see #887 (comment)).

I recommend a feature freeze so that all effort can be focused on not just fixing this but doing so with lots of headroom going forward.

I know it’s not popular but I’m also going to reiterate that new account sign-ups should be closed until what’s causing this is understood and rectified and until Codeberg is usable again. It won’t help to have more people experience Codeberg in its current state (quite on the contrary, it can hurt its reputation going forward).

Codeberg is basically unusable right now with a 2kb push taking over 2 minutes (see https://codeberg.org/Codeberg/Community/issues/887#issuecomment-779987). I recommend a feature freeze so that all effort can be focused on not just fixing this but doing so with lots of headroom going forward. I know it’s not popular but I’m also going to reiterate that new account sign-ups should be closed until what’s causing this is understood and rectified and until Codeberg is usable again. It won’t help to have more people experience Codeberg in its current state (quite on the contrary, it can hurt its reputation going forward).

Gusted commented 2023年01月23日 18:28:48 +01:00

I recommend a feature freeze so that all effort can be focused on not just fixing this but doing so with lots of headroom going forward.

The main infrastructure team is already putting all the effort into fixing this performance issue and the pages stability issues.

---

So to inform you about the current state of affairs:

We have concrete evidence that the slowdown is being caused in the queue code of Gitea/Forgejo, the code to push items into the queue (in this case, repository updates). It's a blocking call until the item is either in the "internal buffered queue" of the Go channel or if one of the workers is able to receive the item via the Go channel. We've noticed that the internal buffered queue is constantly full and that the workers are likely not fast enough to process the queue, this hasn't been a problem until recently (we do not have a clue why). We've already fine-tuned the queue settings, however this has only helped to reduce the waiting times.

We could fairly easily make the queue push operation a non-blocking call by doing the push in a new goroutine, but this will cause a side-effect that a git push may succeed while you don't see any result on the target repository or pull request.

There's an underlying issue as to why the workers have suddenly become slow, it has been quite a daunting task to diagnose this issue.

> I recommend a feature freeze so that all effort can be focused on not just fixing this but doing so with lots of headroom going forward. The main infrastructure team is already putting all the effort into fixing this performance issue and the pages stability issues. --- So to inform you about the current state of affairs: We have concrete evidence that the slowdown is being caused in the queue code of Gitea/Forgejo, the code to push items into the queue (in this case, repository updates). It's a blocking call until the item is either in the "internal buffered queue" of the Go channel or if one of the workers is able to receive the item via the Go channel. We've noticed that the internal buffered queue is constantly full and that the workers are likely not fast enough to process the queue, this hasn't been a problem until recently (we do not have a clue why). We've already fine-tuned the queue settings, however this has only helped to reduce the waiting times. We could fairly easily make the queue push operation a non-blocking call by doing the push in a new goroutine, but this will cause a side-effect that a `git push` may succeed while you don't see any result on the target repository or pull request. There's an underlying issue as to why the workers have suddenly become slow, it has been quite a daunting task to diagnose this issue.

circlebuilder commented 2023年01月23日 18:33:34 +01:00

Thank you for all the efforts that Forgejo team and Codeberg are putting in 💚

Thank you for all the efforts that Forgejo team and Codeberg are putting in 💚

NRK commented 2023年01月24日 07:12:05 +01:00

There's an underlying issue as to why the workers have suddenly become slow, it has been quite a daunting task to diagnose this issue.

In that case, IMHO closing signups temporarily makes sense until the team figures out the root cause. Because if it turns out that the current infra isn't sufficient to support the existing user-base, having more people sign up in the meanwhile would just make it worse.

this will cause a side-effect that a git push may succeed while you don't see any result on the target repository or pull request

Just FYI, this happens even as of now (I encountered it just yesterday) where it takes about 3~5mins (after the cli git push already completed) for a PR to "update" to the new (forced) pushes.

> There's an underlying issue as to why the workers have suddenly become slow, it has been quite a daunting task to diagnose this issue. In that case, IMHO closing signups temporarily makes sense until the team figures out the root cause. Because if it turns out that the current infra isn't sufficient to support the existing user-base, having more people sign up in the meanwhile would just make it worse. > this will cause a side-effect that a git push may succeed while you don't see any result on the target repository or pull request Just FYI, this happens even as of now (I encountered it just yesterday) where it takes about 3~5mins (*after* the cli `git push` already completed) for a PR to "update" to the new (forced) pushes.

JakobDev commented 2023年01月24日 10:49:02 +01:00

Closing sign ups would mean, that People can't report Bugs to Projects hosted on Codeberg unless they already have a Account

Closing sign ups would mean, that People can't report Bugs to Projects hosted on Codeberg unless they already have a Account

aral commented 2023年01月24日 16:04:27 +01:00

@JakobDev It might be an idea to add the ability to have Issue-only accounts for Forgejo, then, because the current situation is untenable. It just took me a minute to push 415 bytes.

Personally, I’m going to have to move my repositories to a different forge at least until these performance issues are sorted because I’m getting distracted every time and it’s ruining my flow.

@JakobDev It might be an idea to add the ability to have Issue-only accounts for Forgejo, then, because the current situation is untenable. It just took me a minute to push 415 bytes. Personally, I’m going to have to move my repositories to a different forge at least until these performance issues are sorted because I’m getting distracted every time and it’s ruining my flow.

Ghost commented 2023年01月27日 11:33:37 +01:00

I also want to suggest locking signups temporarily. A few of the people that I know have moved away from Codeberg as they deem it to be unreliable and buggy. The state that Codeberg is operating now is very less likely to seem as a reliable alternative to platforms like Github and Gitlab.

Closing sign ups would mean, that People can't report Bugs to Projects hosted on Codeberg unless they already have a Account

I may be wrong on this one, but I think there are a very few projects that use Codeberg exclusively as their main git forge that have their user base outside people using Codeberg. So, with proper communication, I don't think it will be much of an issue.
As for Codeberg itself, I think Codeberg has enough members for bugs to be noticed and reported.

I also want to suggest locking signups temporarily. A few of the people that I know have moved away from Codeberg as they deem it to be unreliable and buggy. The state that Codeberg is operating now is very less likely to seem as a reliable alternative to platforms like Github and Gitlab. > Closing sign ups would mean, that People can't report Bugs to Projects hosted on Codeberg unless they already have a Account I may be wrong on this one, but I think there are a very few projects that use Codeberg exclusively as their main git forge that have their user base outside people using Codeberg. So, with proper communication, I don't think it will be much of an issue. As for Codeberg itself, I think Codeberg has enough members for bugs to be noticed and reported.

n0toose commented 2023年04月19日 15:20:46 +02:00

Closing this issue because of inactivity and because the primary topic of this issue is referring to a situation that isn't present anymore, let me know if you believe that this shouldn't be the case.

Closing this issue because of inactivity and because the primary topic of this issue is referring to a situation that isn't present anymore, let me know if you believe that this shouldn't be the case.

fnetX commented 2023年09月21日 02:04:09 +02:00

@aral I am looking forward to your feedback on #1297.

@aral I am looking forward to your feedback on #1297.