Attempting to post comment in a specific issue results in HTTP 500 error #1092

MPeter commented 2023年06月30日 02:19:28 +02:00

When I try to post a comment to gitnex/GitNex#57, the server responds with a HTTP 500 error page, and the comment does not get saved.

<!-- NOTE: If you wish to report a security issue, please send an email to contact@codeberg.org (for Forgejo, see: https://forgejo.org/.well-known/security.txt) instead. Thank you. Welcome to the Codeberg Community Tracker. This is the place central for bug reports, feature requests and feedback. If you are not sure whether you should use trackers of other Codeberg Projects, then you should use this one. Please keep the following in mind: - Please check https://github.com/go-gitea/gitea/issues or https://codeberg.org/forgejo/forgejo to confirm that your issue hasn't already been reported. Codeberg uses Forgejo (referred to as the "upstream"), which is based on Gitea. - This is *not* a customer support hotline. We are volunteers with limited time and resources, and we consider you, the reporter, to be one as well. Please maintain a courteous and respectful tone, and provide as much information as possible, so that we can work together. - We may take a while with resolving your issue. Your work is still important, even if we cannot acknowledge it directly. Thank you for reporting your findings and giving feedback on Codeberg. ## FAQ ### What happens after I open an issue? Separate contributors will use your issue to communicate with each other. It will also be used to provide updates. There is a decent chance that we will ask you for additional feedback or information. ### sing_up: I noticed a typo in the URL of the registration page. This is an intentional, temporary, surprisingly effective change that helps us against spam. Nice catch! ### I want to contribute to Codeberg! That's awesome, thank you! Take a look at https://codeberg.org/Codeberg/Projects and this tracker, we have a variety of both technical and non-technical tasks that we need help with. --> When I try to post a comment to gitnex/GitNex#57, the server responds with a HTTP 500 error page, and the comment does not get saved.

fnetX commented 2023年06月30日 02:30:36 +02:00

Server logs:

2023年06月30日 00:29:53 ...rs/web/repo/issue.go:2851:NewComment() [E] [649e21fd-119] CreateIssueComment: Error 1213 (40001): Deadlock found when trying to get lock; try restarting transaction

Wow. This is probably related to forgejo/forgejo#220

Server logs: ~~~ 2023年06月30日 00:29:53 ...rs/web/repo/issue.go:2851:NewComment() [E] [649e21fd-119] CreateIssueComment: Error 1213 (40001): Deadlock found when trying to get lock; try restarting transaction ~~~ Wow. This is probably related to https://codeberg.org/forgejo/forgejo/issues/220

MPeter commented 2023年06月30日 02:37:44 +02:00

How far back do you keep logs? Would it be possible to check if others have also failed to comment on this issue, or in the repo?

How far back do you keep logs? Would it be possible to check if others have also failed to comment on this issue, or in the repo?

fnetX commented 2023年06月30日 02:42:03 +02:00

The problem appears to be serious, definitely. From the logs, it appears to happen in bursts, but my suspicion is that certain parts of the data structure are broken (as in: causing deadlocks), and people retry a few times, then give up.

I am currently trying to comment in the linked Forgejo thread, but it is ... well ... broken.

The problem appears to be serious, definitely. From the logs, it appears to happen in bursts, but my suspicion is that certain parts of the data structure are broken (as in: causing deadlocks), and people retry a few times, then give up. I am currently trying to comment in the linked Forgejo thread, but it is ... well ... broken.

MPeter commented 2023年06月30日 02:46:31 +02:00

I see :/

From the logs, it appears to happen in bursts

Does that mean that when it happens, it happens across the instance or some other large group of issues, and also that retrying later (e.g. in a few hours) it can succeed?

I see :/ > From the logs, it appears to happen in bursts Does that mean that when it happens, it happens across the instance or some other large group of issues, and also that retrying later (e.g. in a few hours) it can succeed?

fnetX commented 2023年06月30日 03:13:56 +02:00

I matched the occurences with access logs. It happens on the same routes, so the bursts probably come from a single user that retries again and again, eventually gives up.

For example, a burst of problems appeared today, all in the GitNex repo. I suppose this was you :)

I matched the occurences with access logs. It happens on the same routes, so the bursts probably come from a single user that retries again and again, eventually gives up. For example, a burst of problems appeared today, all in the GitNex repo. I suppose this was you :)

wxiaoguang commented 2023年06月30日 06:25:51 +02:00

The problem is that the "Comment" and "Label" db code is quite complex and not transaction-safe (no strict locking order).

No easy fix at the moment, but in 1.21 the forms will be submitted by AJAX then no content would be lost anymore ..... (as the log says: restarting transaction by re-submit)

The problem is that the "Comment" and "Label" db code is quite complex and not transaction-safe (no strict locking order). No easy fix at the moment, but in 1.21 the forms will be submitted by AJAX then no content would be lost anymore ..... (as the log says: restarting transaction by re-submit)

fnetX commented 2023年06月30日 12:51:30 +02:00

@wxiaoguang I think AJAX is no solution here. The issue persisted for about 20 retries over 10 minutes, so AJAX would need to be really stubborn and retry for a long time.

As an aside, will issue comments break for users without javascript? 🤔

@wxiaoguang I think AJAX is no solution here. The issue persisted for about 20 retries over 10 minutes, so AJAX would need to be really stubborn and retry for a long time. As an aside, will issue comments break for users without javascript? 🤔

MPeter commented 2023年06月30日 13:46:57 +02:00

No easy fix at the moment, but in 1.21 the forms will be submitted by AJAX then no content would be lost anymore

I think no content should be lost even right now, because pressing the browser back button gets me back to the comments page, with my comment still in the editor. The browser saves the editor contents somewhere, probably.

But on the other hand that change is very interesting, cant wait for it! Right now commenting (and other actions) means reloading the whole page, and with a long thread on a very slow connection.. that takes time.

> No easy fix at the moment, but in 1.21 the forms will be submitted by AJAX then no content would be lost anymore I think no content should be lost even right now, because pressing the browser back button gets me back to the comments page, with my comment still in the editor. The browser saves the editor contents somewhere, probably. But on the other hand that change is very interesting, cant wait for it! Right now commenting (and other actions) means reloading the whole page, and with a long thread on a very slow connection.. that takes time.

MPeter commented 2023年06月30日 13:49:00 +02:00

@wxiaoguang I think AJAX is no solution here. The issue persisted for about 20 retries over 10 minutes, so AJAX would need to be really stubborn and retry for a long time.

Not just AJAX, but the user would also need to be patient to not close the tab when its not working.

> @wxiaoguang I think AJAX is no solution here. The issue persisted for about 20 retries over 10 minutes, so AJAX would need to be really stubborn and retry for a long time. Not just AJAX, but the user would also need to be patient to not close the tab when its not working.

wxiaoguang commented 2023年06月30日 15:27:45 +02:00

I think AJAX is no solution here.

Yup, it's not the best solution, it's only a workaround for losing contents.

The issue persisted for about 20 retries over 10 minutes, so AJAX would need to be really stubborn and retry for a long time.

There could be 2 cases IMO:

1. The single user's submit causes deadlock, then there might be a serious problem in the code.
2. There are multiple users submitting requests at the same time, then the non-strict transaction locking causes the problem.

I have no idea about what's the root problem at the moment.

As an aside, will issue comments break for users without javascript? 🤔

There are already many components depending on JS, I do not think a user could make "changes (POST)" in all cases without JS nowadays, but they could still view (read) the pages without JS in many cases.

For example, IIRC the "close comment" button was 100% JS from very long time ago.

> I think AJAX is no solution here. Yup, it's not the best solution, it's only a workaround for losing contents. > The issue persisted for about 20 retries over 10 minutes, so AJAX would need to be really stubborn and retry for a long time. There could be 2 cases IMO: 1. The single user's submit causes deadlock, then there might be a serious problem in the code. 2. There are multiple users submitting requests at the same time, then the non-strict transaction locking causes the problem. I have no idea about what's the root problem at the moment. > As an aside, will issue comments break for users without javascript? 🤔 There are already many components depending on JS, I do not think a user could make "changes (POST)" in all cases without JS nowadays, but they could still view (read) the pages without JS in many cases. For example, IIRC the "close comment" button was 100% JS from very long time ago.

sploinga commented 2023年07月04日 17:10:29 +02:00

I just got the same issue on the gadgetbridge repository. Tried to submit a new device request, and got a 500 error.

I just got the same issue on the gadgetbridge repository. Tried to submit a new device request, and got a 500 error.

joserebelo commented 2023年07月04日 19:25:41 +02:00

I can confirm that I also get a 500 error trying to open an issue in the Gadgetbridge repository.

Failed request was at 2023年07月04日 17:24:00 UTC, I am assuming it's the same issue - could someone confirm from the logs?

I can confirm that I also get a 500 error trying to open an issue in the Gadgetbridge repository. Failed request was at 2023年07月04日 17:24:00 UTC, I am assuming it's the same issue - could someone confirm from the logs?

fnetX commented 2023年07月04日 23:27:52 +02:00

@sploinga @joserebelo The only problems we can see is that your content is apparently too long. Did you try to post massive amounts of text? We cannot find a correlation to a deadlock. We'll investigate the other issue tomorrow.

@sploinga @joserebelo The only problems we can see is that your content is apparently too long. Did you try to post massive amounts of text? We cannot find a correlation to a deadlock. We'll investigate the other issue tomorrow.

joserebelo commented 2023年07月04日 23:31:07 +02:00

@fnetX I copy-pasted the same issue message that @sploinga was trying to submit and did not even consider that. It does include a somewhat large log file inline that should be causing the problem.

Should not be related with this issue, apologies for the noise.

@fnetX I copy-pasted the same issue message that @sploinga was trying to submit and did not even consider that. It does include a somewhat large log file inline that should be causing the problem. Should not be related with this issue, apologies for the noise.

davidak commented 2023年07月05日 13:48:12 +02:00

I also got a 500 error response when trying to comment at URL davidak/nixos-config#17.

Text was:

https://github.com/NixOS/nixpkgs/issues/102397 systemd

(55 characters, so not too long)

After some time and many tries, it worked. But sending this also failed often.

I also got a 500 error response when trying to comment at URL https://codeberg.org/davidak/nixos-config/issues/17. Text was: >https://github.com/NixOS/nixpkgs/issues/102397 systemd (55 characters, so not too long) After some time and many tries, it worked. But sending this also failed often.

wxiaoguang commented 2023年07月05日 14:32:16 +02:00

It needs to take a look at the MariaDB's innodb transaction error log, to see how the deadlock happens.

It needs to take a look at the MariaDB's innodb transaction error log, to see how the deadlock happens.