migration from github seems to use incorrect pagination requests #1934

martenson commented 2025年05月15日 11:05:17 +02:00

Comment

I am in the process of creating a mirror for https://github.com/galaxyproject/galaxy/ and am encountering this issue, which seems to be caused by improper pagination handling for larger github projects.

error while listing repos: GET https://api.github.com/repos/galaxyproject/galaxy/issues?direction=asc&page=218&per_page=45&sort=created&state=all: 422 Pagination with the page parameter is not supported for large datasets, please use cursor based pagination (after/before) []

### Comment I am in the process of creating a mirror for `https://github.com/galaxyproject/galaxy/` and am encountering this issue, which seems to be caused by improper pagination handling for larger github projects. ``` error while listing repos: GET https://api.github.com/repos/galaxyproject/galaxy/issues?direction=asc&page=218&per_page=45&sort=created&state=all: 422 Pagination with the page parameter is not supported for large datasets, please use cursor based pagination (after/before) [] ```

martenson commented 2025年05月16日 08:19:55 +02:00

Another problem with migrating issues, not sure if related to the above.

Migrating from https://github.com/galaxyproject/galaxy failed.
Error 1062 (23000): Duplicate entry '508493-1' for key 'UQE_issue_repo_index'

Another problem with migrating issues, not sure if related to the above. ``` Migrating from https://github.com/galaxyproject/galaxy failed. Error 1062 (23000): Duplicate entry '508493-1' for key 'UQE_issue_repo_index' ```

Gusted commented 2025年05月16日 16:37:36 +02:00

Is there a good reason to make a mirror for such a huge repository (including issues)? It's very likely to hit the new storage limits.

Is there a good reason to make a mirror for such a huge repository (including issues)? It's very likely to hit the [new storage limits](https://blog.codeberg.org/new-storage-limits-on-codeberg-what-you-need-to-know.html).

martenson commented 2025年05月16日 16:45:44 +02:00

I guess the question is whether codeberg wants to host large and complex software projects. I'm sure we'd be fine with paying for some extra storage or service but if this is a hard limit and large projects are off the table that would also be good to have confirmed.

I guess the question is whether codeberg wants to host large and complex software projects. I'm sure we'd be fine with paying for some extra storage or service but if this is a hard limit and large projects are off the table that would also be good to have confirmed.

Gusted commented 2025年05月16日 16:48:53 +02:00

@martenson wrote in #1934 (comment):

I guess the question is whether codeberg wants to host large and complex software projects

I am not sure how it's related, unless galaxy is in the process or considering to move to Codeberg. This request read more like someone wanting to create a mirror of a large repository (this happens quite frequently with the linux kernel) without having any meaning behind it.

@martenson wrote in https://codeberg.org/Codeberg/Community/issues/1934#issuecomment-4594175: > I guess the question is whether codeberg wants to host large and complex software projects I am not sure how it's related, unless galaxy is in the process or considering to move to Codeberg. This request read more like someone wanting to create a mirror of a large repository (this happens quite frequently with the linux kernel) without having any meaning behind it.

martenson commented 2025年05月16日 16:55:10 +02:00

I am one of the Galaxy maintainers and am exploring whether codeberg could be a reasonable swap-in for our code infrastructure and what is missing. This is my private initiative, I have no mandate to speak for the project on this topic.

I am one of the Galaxy maintainers and am exploring whether codeberg could be a reasonable swap-in for our code infrastructure and what is missing. This is my private initiative, I have no mandate to speak for the project on this topic.

Gusted commented 2025年05月16日 17:03:06 +02:00

I see, thank you for your clarification. It should be possible to start a migration without migrating the issues and pull requests, I am not sure how easy it is to modify the migration code to use this new cursor pagination for the Github migration.

I see, thank you for your clarification. It should be possible to start a migration without migrating the issues and pull requests, I am not sure how easy it is to modify the migration code to use this new cursor pagination for the Github migration.

fnetX commented 2025年05月16日 20:52:28 +02:00

Thank you for your report, and we would be happy to welcome you to Codeberg.

Without a very close look, it seems like GitHub has changed their limits regarding export of large projects and now requires a different method (the reasoning for this is not clear for me, I can't see a performance or otherwise technical difference between having a page and page size and defining start and end manually).

The result is the same. I doubt that there is human resources to modify our migration code in the near future, so I suppose you are trapped with GitHub for the moment and a migration to Codeberg is currently not possible. We hope you check back later, maybe in a year or two?

If you want to try a migration sooner, your best options are:

• Use a custom importer script. There are quite some community tools around in various programming languages and quality, that allow migrating and synchronizing projects with special needs.
• Try to use another platform as a stop gap. Maybe some other code hosting platforms allows a migration from GitHub, and we allow a migration from there. I didn't check the platforms recently, though.
• Try to fix the issue and either submit a patch to Forgejo, or temporarily run your own instance with a hotfix for the issue. You can migrate from there to Codeberg as well.
• Migrate without your issues. Often, the year-long list of issues that no one will actually implement might use a fresh start anyway. It allows to focus on the most important issues today. Priorities change over time.

Thank you for your report, and we would be happy to welcome you to Codeberg. Without a very close look, it seems like GitHub has changed their limits regarding export of large projects and now requires a different method (the reasoning for this is not clear for me, I can't see a performance or otherwise technical difference between having a page and page size and defining start and end manually). The result is the same. I doubt that there is human resources to modify our migration code in the near future, so I suppose you are trapped with GitHub for the moment and a migration to Codeberg is currently not possible. We hope you check back later, maybe in a year or two? If you want to try a migration sooner, your best options are: * Use a custom importer script. There are quite some community tools around in various programming languages and quality, that allow migrating and synchronizing projects with special needs. * Try to use another platform as a stop gap. Maybe some other code hosting platforms allows a migration from GitHub, and we allow a migration from there. I didn't check the platforms recently, though. * Try to fix the issue and either submit a patch to Forgejo, or temporarily run your own instance with a hotfix for the issue. You can migrate from there to Codeberg as well. * Migrate without your issues. Often, the year-long list of issues that no one will actually implement might use a fresh start anyway. It allows to focus on the most important issues today. Priorities change over time.