Migration potentially stuck (running 6+ hours, I think) #1542

ancientmarinerdev commented 2024年04月18日 19:50:23 +02:00

Comment

https://codeberg.org/ancientmarinerdev/VoxeLibre

I tried to a migration again to see if this was working, I tried to migrate releases, issues, PRs etc.

Source is this:

https://git.minetest.land/MineClone2/MineClone2
It's been migrating issues for possibly over 6 hours now and I suspect it may never complete.

Past Failures

Historically, when we've tried to migrate or dump this (as was requested) in the backend, it gets stuck on a particular issue and going in a repeating loop. So this could still be the issue. It seemed to be on the comment reactions.

We originally tried and it failed raising a ticket previously:

#603

So there is some relevant context here. Attached is an old log where it got stuck.

https://git.minetest.land/api/v1/repos/mineclone2/mineclone2/issues/3306/comments?limit=50&page=14

was part of what it called. I'm not sure why it went to page 14. There is a ghost user on this page, and I'm not sure if that is having an impact:

https://git.minetest.land/MineClone2/MineClone2/issues/3306

As I cannot see the log, I'm assuming you'll have that log in the background and can see what is happening, but any support on identifying the issue, and how to move on from this will be massively be appreciated. We do not know how long we'll have the hosting for so it represents a risk and we'd like to migrate as soon as we can.

Thanks,

AncientMariner

### Comment https://codeberg.org/ancientmarinerdev/VoxeLibre I tried to a migration again to see if this was working, I tried to migrate releases, issues, PRs etc. Source is this: https://git.minetest.land/MineClone2/MineClone2 It's been migrating issues for possibly over 6 hours now and I suspect it may never complete. **Past Failures** Historically, when we've tried to migrate or dump this (as was requested) in the backend, it gets stuck on a particular issue and going in a repeating loop. So this could still be the issue. It seemed to be on the comment reactions. We originally tried and it failed raising a ticket previously: https://codeberg.org/Codeberg/Community/issues/603 So there is some relevant context here. Attached is an old log where it got stuck. https://git.minetest.land/api/v1/repos/mineclone2/mineclone2/issues/3306/comments?limit=50&page=14 was part of what it called. I'm not sure why it went to page 14. There is a ghost user on this page, and I'm not sure if that is having an impact: https://git.minetest.land/MineClone2/MineClone2/issues/3306 As I cannot see the log, I'm assuming you'll have that log in the background and can see what is happening, but any support on identifying the issue, and how to move on from this will be massively be appreciated. We do not know how long we'll have the hosting for so it represents a risk and we'd like to migrate as soon as we can. Thanks, AncientMariner

fnetX commented 2024年04月18日 21:23:45 +02:00

Offtopic

Yet another Mineclone2 fork? 😉

Hmm, the migrations haven't improved a lot with the reactions issue. It is higher on the roadmap, I'd say, but it's quite difficult to achieve migration compatibility when interconnecting Gitea and Forgejo instances: Gitea just doesn't provide the reaction data, and diverging with the API to solve the issue would either require a lot more code, or living with a certain level of incompatibility.

Could you cancel and recreate the migration? Forgejo was restarted about 2 hours after the migration started, so maybe it would have completed if the process wasn't interrupted. It is worth a try before researching workarounds.

I apologize for the inconvenience caused by this.

<details><summary>Offtopic</summary> Yet another Mineclone2 fork? 😉 </details> Hmm, the migrations haven't improved a lot with the reactions issue. It is higher on the roadmap, I'd say, but it's quite difficult to achieve migration compatibility when interconnecting Gitea and Forgejo instances: Gitea just doesn't provide the reaction data, and diverging with the API to solve the issue would either require a lot more code, or living with a certain level of incompatibility. Could you cancel and recreate the migration? Forgejo was restarted about 2 hours after the migration started, so maybe it would have completed if the process wasn't interrupted. It is worth a try before researching workarounds. I apologize for the inconvenience caused by this.

ancientmarinerdev commented 2024年04月19日 00:47:55 +02:00

Offtopic

Yet another Mineclone2 fork? 😉

Hmm, the migrations haven't improved a lot with the reactions issue. It is higher on the roadmap, I'd say, but it's quite difficult to achieve migration compatibility when interconnecting Gitea and Forgejo instances: Gitea just doesn't provide the reaction data, and diverging with the API to solve the issue would either require a lot more code, or living with a certain level of incompatibility.

Could you cancel and recreate the migration? Forgejo was restarted about 2 hours after the migration started, so maybe it would have completed if the process wasn't interrupted. It is worth a try before researching workarounds.

I apologize for the inconvenience caused by this.

This isn't a MineClone2 fork. This is the original MineClone2 project that has been renamed and that is trying to move to Codeberg still but hasn't had much luck. :)

Appreciate the sentiment, I will re-kick this off.

On the topic of compatibility, I would have thought the format has not changed too seriously and we really want to move to Forgejo. I appreciate you don't want to support Gitea code for ever, and I might be wide of the mark, but I expect if our data was in Forgejo and was going to Forgejo, it would still fail as it was failing long before the hard fork. The problem I suspect is that it is having an issue, and the level of error handling isn't ideal. It's better to abandon 1 troublesome issue and carry on. The problem right now is we cannot move at all without losing all issues and PRs. That is huge. If we could migrate everything but reactions, we'd probably take that. Ideally, if it just ignores the reactions that fail and move on, that would get us 99% to where we need. We'd prefer 95-99% of our data to 0%, so this is quite a significant issue.

I've kicked this off here, the same way, but this time I'm doing it to the location I intended (used wrong org).

https://codeberg.org/VoxeLibre/VoxeLibre

> <details><summary>Offtopic</summary> > > Yet another Mineclone2 fork? 😉 > > </details> > > Hmm, the migrations haven't improved a lot with the reactions issue. It is higher on the roadmap, I'd say, but it's quite difficult to achieve migration compatibility when interconnecting Gitea and Forgejo instances: Gitea just doesn't provide the reaction data, and diverging with the API to solve the issue would either require a lot more code, or living with a certain level of incompatibility. > > Could you cancel and recreate the migration? Forgejo was restarted about 2 hours after the migration started, so maybe it would have completed if the process wasn't interrupted. It is worth a try before researching workarounds. > > I apologize for the inconvenience caused by this. > This isn't a MineClone2 fork. This is the original MineClone2 project that has been renamed and that is trying to move to Codeberg still but hasn't had much luck. :) Appreciate the sentiment, I will re-kick this off. On the topic of compatibility, I would have thought the format has not changed too seriously and we really want to move to Forgejo. I appreciate you don't want to support Gitea code for ever, and I might be wide of the mark, but I expect if our data was in Forgejo and was going to Forgejo, it would still fail as it was failing long before the hard fork. The problem I suspect is that it is having an issue, and the level of error handling isn't ideal. It's better to abandon 1 troublesome issue and carry on. The problem right now is we cannot move at all without losing all issues and PRs. That is huge. If we could migrate everything but reactions, we'd probably take that. Ideally, if it just ignores the reactions that fail and move on, that would get us 99% to where we need. We'd prefer 95-99% of our data to 0%, so this is quite a significant issue. I've kicked this off here, the same way, but this time I'm doing it to the location I intended (used wrong org). https://codeberg.org/VoxeLibre/VoxeLibre

ancientmarinerdev commented 2024年04月19日 12:44:26 +02:00

12 hours later, and it's still running. I'm guess it's stuck. Any logs or idea how it is failing?

12 hours later, and it's still running. I'm guess it's stuck. Any logs or idea how it is failing?

fnetX commented 2024年04月19日 18:42:32 +02:00

@ancientmarinerdev yeah unfortunately I cannot really find an indicator of where it is failing. There are some Git processes still running, so maybe it even got stuck earlier than the comment migration.

I'm going to cancel it, because we need to deploy another software update, which would interrupt the progress anyway.

I was wondering if we can do a workaround: You try to migrate things step by step (only issues, then only pull requests etc) into separate repositories, and we see if we can somehow edit the repo id in the database to merge this into one repository later.

@ancientmarinerdev yeah unfortunately I cannot really find an indicator of where it is failing. There are some Git processes still running, so maybe it even got stuck earlier than the comment migration. I'm going to cancel it, because we need to deploy another software update, which would interrupt the progress anyway. I was wondering if we can do a workaround: You try to migrate things step by step (only issues, then only pull requests etc) into separate repositories, and we see if we can somehow edit the repo id in the database to merge this into one repository later.

ancientmarinerdev commented 2024年04月20日 03:09:47 +02:00

@ancientmarinerdev yeah unfortunately I cannot really find an indicator of where it is failing. There are some Git processes still running, so maybe it even got stuck earlier than the comment migration.

I'm going to cancel it, because we need to deploy another software update, which would interrupt the progress anyway.

I was wondering if we can do a workaround: You try to migrate things step by step (only issues, then only pull requests etc) into separate repositories, and we see if we can somehow edit the repo id in the database to merge this into one repository later.

That's an interesting idea. Might need to wait until next week to try that, but happy to go through that process.

> @ancientmarinerdev yeah unfortunately I cannot really find an indicator of where it is failing. There are some Git processes still running, so maybe it even got stuck earlier than the comment migration. > > I'm going to cancel it, because we need to deploy another software update, which would interrupt the progress anyway. > > I was wondering if we can do a workaround: You try to migrate things step by step (only issues, then only pull requests etc) into separate repositories, and we see if we can somehow edit the repo id in the database to merge this into one repository later. > > That's an interesting idea. Might need to wait until next week to try that, but happy to go through that process.

the-real-herowl commented 2024年05月07日 14:50:19 +02:00

Here I'm attempting to migrate everything aside of the issues:

https://codeberg.org/the-real-herowl/VoxeLibre

Here I'm attempting to migrate everything aside of the issues: https://codeberg.org/the-real-herowl/VoxeLibre

the-real-herowl commented 2024年05月07日 14:53:58 +02:00

And here a migration of everything except for Issues and PRs succeeded withing 1 minute: https://codeberg.org/the-real-herowl/VL2

And here a migration of everything except for Issues and PRs succeeded withing 1 minute: https://codeberg.org/the-real-herowl/VL2

the-real-herowl commented 2024年05月07日 21:24:57 +02:00

Here I'm attempting to migrate everything aside of the issues:

https://codeberg.org/the-real-herowl/VoxeLibre

Frozen on PRs for over 6 hours. Doesn't look normal.

> Here I'm attempting to migrate everything aside of the issues: > > https://codeberg.org/the-real-herowl/VoxeLibre Frozen on PRs for over 6 hours. Doesn't look normal.

the-real-herowl commented 2024年05月08日 17:19:11 +02:00

Here I'm attempting to migrate everything aside of the issues:

https://codeberg.org/the-real-herowl/VoxeLibre

I canceled this after over 24 hours. Looks like all it was doing was effectively DoSing the gitea instance.

> Here I'm attempting to migrate everything aside of the issues: > > https://codeberg.org/the-real-herowl/VoxeLibre I canceled this after over 24 hours. Looks like all it was doing was effectively DoSing the gitea instance.

btlogy commented 2024年06月27日 15:13:32 +02:00

I think I've run into the same migration issue from Gitea v1.20.6: : I observe a sort of loop for issues with more than 50 comments!?

Jun 27 12:22:10 <hostname> gitea[182217]: 2024年06月27日 12:22:10 ...eb/routing/logger.go:102:func1() [I] router: completed GET /api/v1/repos/orgname/reponame/issues/2/comments?limit=50&page=1 for <ip>:0, 200 OK in 10.6ms @ repo/issue_comment.go:27(repo.ListIssueComments) 
Jun 27 12:22:11 <hostname> gitea[182217]: 2024年06月27日 12:22:11 ...eb/routing/logger.go:102:func1() [I] router: completed GET /api/v1/repos/orgname/reponame/issues/1/comments?limit=50&page=1 for <ip>:0, 200 OK in 15.6ms @ repo/issue_comment.go:27(repo.ListIssueComments) 
Jun 27 12:22:13 <hostname> gitea[182217]: 2024年06月27日 12:22:13 ...eb/routing/logger.go:102:func1() [I] router: completed GET /api/v1/repos/orgname/reponame/issues/1/comments?limit=50&page=2 for <ip>:0, 200 OK in 16.5ms @ repo/issue_comment.go:27(repo.ListIssueComments) 
Jun 27 12:22:16 <hostname> gitea[182217]: 2024年06月27日 12:22:16 ...eb/routing/logger.go:102:func1() [I] router: completed GET /api/v1/repos/orgname/reponame/issues/1/comments?limit=50&page=3 for <ip>:0, 200 OK in 17.5ms @ repo/issue_comment.go:27(repo.ListIssueComments) 
Jun 27 12:22:18 <hostname> gitea[182217]: 2024年06月27日 12:22:18 ...eb/routing/logger.go:102:func1() [I] router: completed GET /api/v1/repos/orgname/reponame/issues/1/comments?limit=50&page=4 for <ip>:0, 200 OK in 16.4ms @ repo/issue_comment.go:27(repo.ListIssueComments) 
Jun 27 12:22:21 <hostname> gitea[182217]: 2024年06月27日 12:22:21 ...eb/routing/logger.go:102:func1() [I] router: completed GET /api/v1/repos/orgname/reponame/issues/1/comments?limit=50&page=5 for <ip>:0, 200 OK in 15.8ms @ repo/issue_comment.go:27(repo.ListIssueComments) 
...

There is 60 comments on issue 1 and only 6 on issue 2.

Using curl against the Gitea API, I've found out that it does not seem to honor the page number: no matter which page I ask, it keeps sending the first 50 comments. I suppose this could explain the loop.

I've looked for such an issue in the Gitea tracker, but I've not found any obvious hit.
I'll first update to v.1.22 and test again.

I think I've run into the same migration issue from Gitea v1.20.6: : I observe a sort of loop for issues with more than 50 comments!? ``` Jun 27 12:22:10 <hostname> gitea[182217]: 2024年06月27日 12:22:10 ...eb/routing/logger.go:102:func1() [I] router: completed GET /api/v1/repos/orgname/reponame/issues/2/comments?limit=50&page=1 for <ip>:0, 200 OK in 10.6ms @ repo/issue_comment.go:27(repo.ListIssueComments) Jun 27 12:22:11 <hostname> gitea[182217]: 2024年06月27日 12:22:11 ...eb/routing/logger.go:102:func1() [I] router: completed GET /api/v1/repos/orgname/reponame/issues/1/comments?limit=50&page=1 for <ip>:0, 200 OK in 15.6ms @ repo/issue_comment.go:27(repo.ListIssueComments) Jun 27 12:22:13 <hostname> gitea[182217]: 2024年06月27日 12:22:13 ...eb/routing/logger.go:102:func1() [I] router: completed GET /api/v1/repos/orgname/reponame/issues/1/comments?limit=50&page=2 for <ip>:0, 200 OK in 16.5ms @ repo/issue_comment.go:27(repo.ListIssueComments) Jun 27 12:22:16 <hostname> gitea[182217]: 2024年06月27日 12:22:16 ...eb/routing/logger.go:102:func1() [I] router: completed GET /api/v1/repos/orgname/reponame/issues/1/comments?limit=50&page=3 for <ip>:0, 200 OK in 17.5ms @ repo/issue_comment.go:27(repo.ListIssueComments) Jun 27 12:22:18 <hostname> gitea[182217]: 2024年06月27日 12:22:18 ...eb/routing/logger.go:102:func1() [I] router: completed GET /api/v1/repos/orgname/reponame/issues/1/comments?limit=50&page=4 for <ip>:0, 200 OK in 16.4ms @ repo/issue_comment.go:27(repo.ListIssueComments) Jun 27 12:22:21 <hostname> gitea[182217]: 2024年06月27日 12:22:21 ...eb/routing/logger.go:102:func1() [I] router: completed GET /api/v1/repos/orgname/reponame/issues/1/comments?limit=50&page=5 for <ip>:0, 200 OK in 15.8ms @ repo/issue_comment.go:27(repo.ListIssueComments) ... ``` There is 60 comments on issue 1 and only 6 on issue 2. Using curl against the Gitea API, I've found out that it does not seem to honor the `page` number: no matter which page I ask, it keeps sending the first 50 comments. I suppose this could explain the loop. I've looked for such an issue in the Gitea tracker, but I've not found any obvious hit. I'll first update to v.1.22 and test again.

fnetX commented 2024年06月27日 16:14:38 +02:00

If you have control over the remote instance, I suggest upgrading to Forgejo to ease debugging. We won't investigate issues originating from Gitea.

If you have control over the remote instance, I suggest upgrading to Forgejo to ease debugging. We won't investigate issues originating from Gitea.

btlogy commented 2024年06月27日 16:23:30 +02:00

Thank you for your feedback @fnetX.
I totally understand you won't investigate this. No worries.

Gitea is a strong requirement for the migration project I'm working on (Trac > trac2gitea > Gitea > Codeberg).
And I'm pretty sure this is a bug on Gitea side (maybe also present on Forgejo, but I'll not go that path yet).
I was merely reporting here to help other potentially impacted users.

I'll dig into the Gitea code and drop a message here if I find the culprit or the fix.

Thank you for your feedback @fnetX. I totally understand you won't investigate this. No worries. Gitea is a strong requirement for the migration project I'm working on (Trac > trac2gitea > Gitea > Codeberg). And I'm pretty sure this is a bug on Gitea side (maybe also present on Forgejo, but I'll not go that path yet). I was merely reporting here to help other potentially impacted users. I'll dig into the Gitea code and drop a message here if I find the culprit or the fix.

fnetX commented 2024年06月27日 17:18:07 +02:00

If you are talking about https://github.com/stevejefferson/trac2gitea, from a first glimpse, this should work with Forgejo just the same, but I haven't tried.

If you are talking about https://github.com/stevejefferson/trac2gitea, from a first glimpse, this should work with Forgejo just the same, but I haven't tried.

btlogy commented 2024年06月27日 22:25:27 +02:00

I've forked the latest fork of it: https://github.com/tahoe-lafs/trac2gitea
And we are pretty deep in the migration already: Gitea is our self-hosted target.

However, we are interested in going to SaaS with Codeberg if possible.
That's what I'm testing.

I've updated our Gitea to v1.21.11 (NixOS-stable) and the "bug" is still present :-(

Now, I may have talked too fast about this lying on the Gitea API...
I'm not 100% sure this size=50&page=x is even part of that API!?
I'll dig it tomorrow...

I've forked the latest fork of it: https://github.com/tahoe-lafs/trac2gitea And we are pretty deep in the migration already: Gitea is our self-hosted target. However, we are interested in going to SaaS with Codeberg if possible. That's what I'm testing. I've updated our Gitea to v1.21.11 (NixOS-stable) and the "bug" is still present :-( Now, I may have talked too fast about this lying on the Gitea API... I'm not 100% sure this `size=50&page=x` is even part of that API!? I'll dig it tomorrow...

n0toose commented 2024年06月28日 00:42:02 +02:00

Something that may be worth a shot is trying to host your own Gitea or Forgejo instance (it's just an SQLite binary, I believe it's manageable to the average person that knows their way around the command line and you seem to be doing things in that direction) and trying to reproduce this problem - this would make it more clear as to whether this is a Codeberg-specific problem or not.

Something that may be worth a shot is trying to host your own Gitea or Forgejo instance (it's just an SQLite binary, I believe it's manageable to the average person that knows their way around the command line and you seem to be doing things in that direction) and trying to reproduce this problem - this would make it more clear as to whether this is a Codeberg-specific problem or not.

btlogy commented 2024年07月02日 17:28:26 +02:00

I've been digging in the code, and it seems to me that the Codeberg/Forgejo migration uses the page/limit options to call to Gitea API where this paging feature has not been implemented.

• Forgejo migration GiteaDownloader/GetComments
• Gitea API GET /repos/{owner}/{repo}/issues/{index}/comments

This API call is documented the same way by swagger from the Codeberg API itself.
And paging does exist for other call like GET /repos/{owner}/{repo}/issues for instance.

Also I expect the migration from Forgejo to Codeberg to have the same issue, because it seems like the ForgejoDownloader is just an instance of GiteaDownloader!?

If anyone here think my reasoning make sense: where should I create the issue to report this: https://codeberg.org/forgejo/forgejo/issues?

I've been digging in the code, and it seems to me that the Codeberg/Forgejo migration uses the page/limit options to call to Gitea API where this paging feature has not been implemented. * Forgejo migration [`GiteaDownloader/GetComments`](https://codeberg.org/forgejo/forgejo/src/commit/cc80e661531794fff7f8a336eaaefdb7e3bd3956/services/migrations/gitea_downloader.go#L461-L505) * Gitea API [`GET /repos/{owner}/{repo}/issues/{index}/comments`](https://github.com/go-gitea/gitea/blob/3bd87fb170ce0b82d1d79c876b279259168d842b/routers/api/v1/repo/issue_comment.go#L26-L120) This API call is documented the same way by swagger from the Codeberg API [itself](https://codeberg.org/api/swagger#/issue/issueGetComments). And paging does exist for other call like [`GET /repos/{owner}/{repo}/issues`](https://codeberg.org/api/swagger#/issue/issueListIssues) for instance. Also I expect the migration from Forgejo to Codeberg to have the same issue, because it seems like the [`ForgejoDownloader`](https://codeberg.org/forgejo/forgejo/src/tag/v1.21.11-0/services/migrations/forgejo_downloader.go) is just an instance of `GiteaDownloader`!? If anyone here think my reasoning make sense: where should I create the issue to report this: https://codeberg.org/forgejo/forgejo/issues?

btlogy commented 2024年07月02日 17:36:05 +02:00

Something that may be worth a shot is trying to host your own Gitea or Forgejo instance ... and trying to reproduce this problem

That what I'm doing already: I'm hosting this Gitea instance and testing the migration to Codeberg.

I could go the extra mile by hosting a Forgejo instance and test the migration from there, but I feel like it's going to cost me a few hours and not give me more info about the root cause.

There is also some uncertainty about which code is actually running on Codeberg: I do not understand why swagger reports Forgejo API - 7.0.0-368-09be816934+gitea-1.21.11?

> Something that may be worth a shot is trying to host your own Gitea or Forgejo instance ... and trying to reproduce this problem That what I'm doing already: I'm hosting this Gitea [instance](https://code.lafs.eval.latfa.net/tahoe-lafs/trac-2024年05月23日) and testing the migration to Codeberg. I could go the extra mile by hosting a Forgejo instance and test the migration from there, but I feel like it's going to cost me a few hours and not give me more info about the root cause. There is also some uncertainty about which code is actually running on Codeberg: I do not understand why [swagger](https://codeberg.org/api/swagger) reports `Forgejo API - 7.0.0-368-09be816934+gitea-1.21.11`?

n0toose commented 2024年07月02日 17:46:07 +02:00

There is also some uncertainty about which code is actually running on Codeberg: I do not understand why swagger reports Forgejo API - 7.0.0-368-09be816934+gitea-1.21.11?

What is the thing that looks weird to you? Codeberg uses a custom version of Forgejo and +gitea-1.21.11 is also useful so as to know which version of the Gitea API is compatible with Forgejo.

> There is also some uncertainty about which code is actually running on Codeberg: I do not understand why swagger reports Forgejo API - 7.0.0-368-09be816934+gitea-1.21.11? What is the thing that looks weird to you? Codeberg uses a custom version of Forgejo and `+gitea-1.21.11` is also useful so as to know which version of the Gitea API is compatible with Forgejo.

n0toose commented 2024年07月02日 17:47:08 +02:00

I've been digging in the code, and it seems to me that the Codeberg/Forgejo migration uses the page/limit options to call to Gitea API where this paging feature has not been implemented.

Now that's interesting... (but I'm not sure what to say at this point in time)

> I've been digging in the code, and it seems to me that the Codeberg/Forgejo migration uses the page/limit options to call to Gitea API where this paging feature has not been implemented. Now that's interesting... (but I'm not sure what to say at this point in time)

btlogy commented 2024年07月02日 19:27:06 +02:00

There is also some uncertainty about which code is actually running on Codeberg: I do not understand why swagger reports Forgejo API - 7.0.0-368-09be816934+gitea-1.21.11?

What is the thing that looks weird to you? Codeberg uses a custom version of Forgejo and +gitea-1.21.11 is also useful so as to know which version of the Gitea API is compatible with Forgejo.

Oh, that makes sense! Now I get it. Thank you.

> > There is also some uncertainty about which code is actually running on Codeberg: I do not understand why swagger reports Forgejo API - 7.0.0-368-09be816934+gitea-1.21.11? > > What is the thing that looks weird to you? Codeberg uses a custom version of Forgejo and `+gitea-1.21.11` is also useful so as to know which version of the Gitea API is compatible with Forgejo. Oh, that makes sense! Now I get it. Thank you.

btlogy commented 2024年07月02日 19:47:03 +02:00

I've been digging in the code, and it seems to me that the Codeberg/Forgejo migration uses the page/limit options to call to Gitea API where this paging feature has not been implemented.

Now that's interesting... (but I'm not sure what to say at this point in time)

I suppose it would not hurt if I create an issue on forgejo and maybe try to reproduce the problem filling 51 comments via the gitea_downloader_test.go...

> > I've been digging in the code, and it seems to me that the Codeberg/Forgejo migration uses the page/limit options to call to Gitea API where this paging feature has not been implemented. > > Now that's interesting... (but I'm not sure what to say at this point in time) I suppose it would not hurt if I create an issue on [forgejo](https://codeberg.org/forgejo/forgejo/issues) and maybe try to reproduce the problem filling 51 comments via the [gitea_downloader_test.go](https://codeberg.org/forgejo/forgejo/src/branch/forgejo/services/migrations/gitea_downloader_test.go)...

btlogy commented 2024年07月25日 14:51:53 +02:00

Damn, I'm a bit more confused about where this possible bug would come from now!

I've tested multiple migration of a simple repository containing a single issue with 51 comments and I've managed to reproduce the loop only when the data are flowing from Gitea:

Provider	API	Version	Flow	Provider	API	Version	Result
self-hosted	Gitea	1.21.11	▶️	codeberg.org	Forgejo	7.0.0-417+gitea-1.21.11	➰
self-hosted	Gitea	1.21.11	▶️	forgejo.org	Forgejo	9.0.0-dev+gitea-1.22.0	➰
forgejo.org	Forgejo	9.0.0-dev+gitea-1.22.0	▶️	self-hosted	Gitea	1.21.11	✅
forgejo.org	Forgejo	9.0.0-dev+gitea-1.22.0	▶️	codeberg.org	Forgejo	7.0.0-417+gitea-1.21.11	✅
codeberg.org	Forgejo	7.0.0-417+gitea-1.21.11	▶️	self-hosted	Gitea	1.21.11	💥
codeberg.org	Forgejo	7.0.0-417+gitea-1.21.11	▶️	forgejo.org	Forgejo	9.0.0-dev+gitea-1.22.0	💥

However, for the 2 last test scenarios, the migration flatly fails with this suspicious error message:

PANIC whilst trying to do migrate task: runtime error: invalid memory address or nil pointer dereference

This issue looks likely different, but could it still be related to the pagination (now between 2 instances of Forgejo too)?

I'm not sure where to look now, I've to sleep on it. Any idea would be welcome.

Damn, I'm a bit more confused about where this possible bug would come from now! I've tested multiple migration of a simple [repository](https://codeberg.org/btlogy/51-comments-issue) containing a single issue with 51 comments and I've managed to reproduce the loop only when the data are flowing from Gitea: | Provider | API | Version | Flow | Provider | API | Version | Result | |---|---|---|---|---|---|---|---| | self-hosted | Gitea | 1.21.11| ▶️ | codeberg.org | Forgejo | 7.0.0-417+gitea-1.21.11 | ➰ | | self-hosted | Gitea | 1.21.11| ▶️ | forgejo.org | Forgejo | 9.0.0-dev+gitea-1.22.0 | ➰ | | forgejo.org | Forgejo | 9.0.0-dev+gitea-1.22.0 | ▶️ | self-hosted | Gitea | 1.21.11 | ✅ | | forgejo.org | Forgejo | 9.0.0-dev+gitea-1.22.0 | ▶️ | codeberg.org | Forgejo | 7.0.0-417+gitea-1.21.11 | ✅ | | codeberg.org | Forgejo | 7.0.0-417+gitea-1.21.11 | ▶️ | self-hosted | Gitea | 1.21.11 | 💥 | | codeberg.org | Forgejo | 7.0.0-417+gitea-1.21.11 | ▶️ | forgejo.org | Forgejo | 9.0.0-dev+gitea-1.22.0 | 💥 | However, for the 2 last test scenarios, the migration flatly fails with this suspicious error message: ``` PANIC whilst trying to do migrate task: runtime error: invalid memory address or nil pointer dereference ``` This issue looks likely different, but could it still be related to the pagination (now between 2 instances of Forgejo too)? I'm not sure where to look now, I've to sleep on it. Any idea would be welcome.

fnetX commented 2024年07月25日 22:36:55 +02:00

If you have this error message on a self-hosted instance, please attach as much information as you can (e.g. the full log or at least the stacktrace of this specific issue). The logfiles should contain more information and allow the developers to fix the issue.

Feel free to post it here, we can take care of forwarding appropriately.

If you have this error message on a self-hosted instance, please attach as much information as you can (e.g. the full log or at least the stacktrace of this specific issue). The logfiles should contain more information and allow the developers to fix the issue. Feel free to post it here, we can take care of forwarding appropriately.

btlogy commented 2024年07月26日 08:06:29 +02:00

Thank you for your answer @fnetX
The last panic error above is happening between codeberg.org and forgejo.org: I do not have any logs.

Actually, I've managed to reproduce this panic error using only codeberg.org on Fri Jul 26 08:01:51 AM CEST 2024:

image

Thank you for your answer @fnetX The last panic error above is happening between codeberg.org and forgejo.org: I do not have any logs. Actually, I've managed to reproduce this panic error using only codeberg.org on Fri Jul 26 08:01:51 AM CEST 2024: 

Michieal commented 2024年08月06日 16:40:49 +02:00

Probably a silly question... but, if we had the data out of the database from the Gitea instance for the project, and did a self-hosted Forgejo instance, could we copy between the two databases? I mean, I've not delved into the code of either project... but... does anyone know if this is a feasible idea?
It looks like if we had a forgejo instance, then that could be pushed over to Codeberg, and things would be able to move forward.

thoughts?

Probably a silly question... but, if we had the data out of the database from the Gitea instance for the project, and did a self-hosted Forgejo instance, could we copy between the two databases? I mean, I've not delved into the code of either project... but... does anyone know if this is a feasible idea? It looks like if we had a forgejo instance, then that could be pushed over to Codeberg, and things would be able to move forward. thoughts?

btlogy commented 2024年08月19日 14:19:49 +02:00

I've managed to reproduce this loop issue while migrating an empty repo with a single issue with a description and 50 comments from Forgejo to Forgejo using v7.0.4 (self-hosted installed a few weeks ago).
But then I've failed to reproduce this on the demo server which currently runs v7.0.7 (https://v7.next.forgejo.org).

The good news is this likely means this loop issue has been fixed between 7.0.4 and 7.0.7.
However, I was not able to quickly pin point the relevant commit(s) in Forgejo, which may help to understand what's wrong when migrating from Gitea.
I'm still looking around...
In the meantime, I'll likely try to migrate my repository to Codeberg via Forgejo v7.0.7 instead of Gitea v1.21.11. I bet this will work.

I've managed to reproduce this loop issue while migrating an empty repo with a single issue with a description and 50 comments from Forgejo to Forgejo using v7.0.4 (self-hosted installed a few weeks ago). But then I've failed to reproduce this on the demo server which currently runs v7.0.7 (https://v7.next.forgejo.org). The good news is this likely means this loop issue has been fixed between 7.0.4 and 7.0.7. However, I was not able to quickly pin point the relevant commit(s) in Forgejo, which may help to understand what's wrong when migrating from Gitea. I'm still looking around... In the meantime, I'll likely try to migrate my repository to Codeberg via Forgejo v7.0.7 instead of Gitea v1.21.11. I bet this will work.

btlogy commented 2024年08月20日 10:48:30 +02:00

Unfortunately, it's a bit more blurry in term of which version has a fix for this loop issue!
I'm now running v7.0.7 on a self-hosted instance, and I can still reproduce the bug there.
But I can not reproduce it in v7.next which is apparently running today this commit from yesterday: 684c3106b4.

Interestingly, I can complete the migration of the same repo from v7.next to v7.0.7, but enter the loop issue when trying the other way around!
Which seems to mean the fix is lying in the recent commits related to the API rather than to the migration service.

But based on the 4 related PRs that have been merged between v7.0.7 and 684c3106b4, it does not make sense to me: none of those would explain the change of behavior!

• forgejo/forgejo#4917
• forgejo/forgejo#4925
• forgejo/forgejo#4950
• forgejo/forgejo#5005

I fail to understand why the migration is happily dealing with lack of pagination support for this API call in one case, but not the other!?

Holly molly: I've forgot this difference in behavior could be explained by the configuration settings for the API:

MAX_RESPONSE_ITEMS: 50: Max number of items in a page.

More testing...

Unfortunately, it's a bit more blurry in term of which version has a fix for this loop issue! I'm now running v7.0.7 on a self-hosted instance, and I can still reproduce the bug there. But I can not reproduce it in v7.next which is apparently running today this commit from yesterday: [684c3106b4](https://codeberg.org/forgejo/forgejo/commit/684c3106b44c76cd7872f8bd4c5e6de49701dda0). Interestingly, I can complete the migration of the same repo from v7.next to v7.0.7, but enter the loop issue when trying the other way around! Which seems to mean the fix is lying in the recent commits related to the API rather than to the migration service. But based on the 4 related PRs that have been merged between [v7.0.7 and 684c3106b4]([url](https://codeberg.org/forgejo/forgejo/compare/v7.0.7...684c3106b44c76cd7872f8bd4c5e6de49701dda0)), it does not make sense to me: none of those would explain the change of behavior! - https://codeberg.org/forgejo/forgejo/pulls/4917 - https://codeberg.org/forgejo/forgejo/pulls/4925 - https://codeberg.org/forgejo/forgejo/pulls/4950 - https://codeberg.org/forgejo/forgejo/pulls/5005 I fail to understand why the migration is happily dealing with lack of pagination support for this API call in one case, but not the other!? Holly molly: I've forgot this difference in behavior could be explained by the configuration settings for the [API](https://forgejo.org/docs/latest/admin/config-cheat-sheet/#api-api): > MAX_RESPONSE_ITEMS: 50: Max number of items in a page. More testing...

btlogy commented 2024年08月20日 14:11:29 +02:00

I still can not reproduce this loop issue with 100 comments on https://v7.next.forgejo.org, which reports to be configure with MAX_RESPONSE_ITEMS=50 anyway (https://v7.next.forgejo.org/api/v1/settings/api). This is very annoying, because I'd rather not fill a bug report as long as I fail to reproduce it there...

However: changing this settings on my self-hosted Gitea instance seems to be a valid workaround for my migration to Codeberg. 🎉
(ping @Michieal, @the-real-herowl and @ancientmarinerdev).

In our case, I had to query the Gitea DB to count the number of comments per issues and restarted Gitea with the highest value: MAX_RESPONSE_ITEMS=250 (see the doc here).

The migration tool first grab this max value and no longer falls into the pagination loop...
However, I'm now bumping into some other problem (e.g.: Error 1213 (40001): Deadlock found when trying to get lock; try restarting transaction). ;-( - (see #1323)
But we have a huge amount of issues (>4k), so I might hit something else (the load indeed).

BTW: The PANIC error described above seems to be related with something else than the paging. And possibly triggered by some user matching issue. I'm not sure I will dig that one out.

I still can not reproduce this loop issue with 100 comments on https://v7.next.forgejo.org, which reports to be configure with MAX_RESPONSE_ITEMS=50 anyway (https://v7.next.forgejo.org/api/v1/settings/api). This is very annoying, because I'd rather not fill a bug report as long as I fail to reproduce it there... However: changing this settings on my self-hosted Gitea instance seems to be a valid workaround for my migration to Codeberg. :tada: (ping @Michieal, @the-real-herowl and @ancientmarinerdev). In our case, I had to query the Gitea DB to count the number of comments per issues and restarted Gitea with the highest value: `MAX_RESPONSE_ITEMS=250` (see the doc [here](https://forgejo.org/docs/latest/admin/config-cheat-sheet/#api-api)). The migration tool first grab this max value and no longer falls into the pagination loop... However, I'm now bumping into some other problem (e.g.: `Error 1213 (40001): Deadlock found when trying to get lock; try restarting transaction`). ;-( - (see https://codeberg.org/Codeberg/Community/issues/1323) But we have a huge amount of issues (>4k), so I might hit something else (the load indeed). BTW: The `PANIC` error described above seems to be related with something else than the paging. And possibly triggered by some user matching issue. I'm not sure I will dig that one out.

btlogy commented 2024年08月21日 08:12:49 +02:00

I can confirm this work-around is helping: our issues with more than 50 comments have been successfully migrated to Codeberg. 🎉

I can confirm this work-around is helping: our issues with more than 50 comments have been successfully migrated to Codeberg. 🎉

fnetX commented 2024年08月21日 11:28:46 +02:00

@btlogy this is very good to know so we can help others with similar problems.

It would be interesting to know if the migration would also succeed if the setting is set to less than the maximum amount of issues, because it could just be that the higher amount of response items is speeding things up enough ... but don't worry about finding out if your time is better spent elsewhere.

@btlogy this is very good to know so we can help others with similar problems. It would be interesting to know if the migration would also succeed if the setting is set to less than the maximum amount of issues, because it could just be that the higher amount of response items is speeding things up enough ... but don't worry about finding out if your time is better spent elsewhere.

btlogy commented 2024年08月30日 15:26:37 +02:00

@btlogy this is very good to know so we can help others with similar problems.

It would be interesting to know if the migration would also succeed if the setting is set to less than the maximum amount of issues, because it could just be that the higher amount of response items is speeding things up enough ... but don't worry about finding out if your time is better spent elsewhere.

It systematically fails when the page limit is below the number of comment.
I'll open a bug report.

> @btlogy this is very good to know so we can help others with similar problems. > > It would be interesting to know if the migration would also succeed if the setting is set to less than the maximum amount of issues, because it could just be that the higher amount of response items is speeding things up enough ... but don't worry about finding out if your time is better spent elsewhere. It systematically fails when the page limit is below the number of comment. I'll open a bug report.

patdyn commented 2025年11月04日 09:58:13 +01:00

Shall we close this as well?

Shall we close this as well?