Codeberg to Codeberg Repo Migration Fails #1498

unfa commented 2024年02月28日 21:22:59 +01:00

Comment

Hi!

I'm trying to archive a large repository (migrate it and mark as archived) before the original gets cleaned up with a long history or commits, PRs and Issues removed.

The archived repo later could be migrated again to private instance for archival and deleted from Codeberg to free used resources there

I have seen a few different error messages while migrating and I clicked "Retry" multiple times, but soon the error message stopped changing and it seems like the process is completely stuck on releases.

Migrating from https://codeberg.org/liblast/liblast failed.

Error 1062 (23000): Duplicate entry '193773-0.1.9-1_hotfix_pre-alpha' for key 'UQE_release_n'

I'm using an API key to migrate everything I can, to preserver a complete copy. Before the main repository will be purged of the old stuff.

The reason for this rather than archiving the main repo and creating a clean one is the adders would change, and people watching the project would get mighty confused.

If you think what I'm trying to do is a bad idea, tell me what else I could do, though the technical issue with migration is still there :D

image

### Comment Hi! I'm trying to archive a large repository (migrate it and mark as archived) before the original gets cleaned up with a long history or commits, PRs and Issues removed. The archived repo later could be migrated again to private instance for archival and deleted from Codeberg to free used resources there I have seen a few different error messages while migrating and I clicked "Retry" multiple times, but soon the error message stopped changing and it seems like the process is completely stuck on releases. > Migrating from https://codeberg.org/liblast/liblast failed. > > Error 1062 (23000): Duplicate entry '193773-0.1.9-1_hotfix_pre-alpha' for key 'UQE_release_n' I'm using an API key to migrate everything I can, to preserver a complete copy. Before the main repository will be purged of the old stuff. The reason for this rather than archiving the main repo and creating a clean one is the adders would change, and people watching the project would get mighty confused. If you think what I'm trying to do is a bad idea, tell me what else I could do, though the technical issue with migration is still there :D 

fnetX commented 2024年03月04日 00:41:04 +01:00

Hi, I'm sorry to hear you are having trouble with this, I hoped someone would find the time to look into this.

Liblast is one of the largest projects on Codeberg (in the useful category, everything larger is mostly abuse), and there is a high chance of edge cases and scaling issues that have not been discovered before, simply before no one used the software at this scale yet.

The obviously easiest option would be to rename and archive the current repo and create a new one. With a prominent notice, this should help inform people about the situation.

The second best option would be to schedule with someone (@Gusted, if you have a minute) to look into the situation and see if there is an easy fix or workaround for the problem.

Another option would be to let me manually fiddle with the database and reassign your issues, releases etc to another repo id, so that there is no need to do a round-trip with the API. It's not a good option and I'll need to test-drive this on a staging machine, but it is an option if there is no better solution found, and I hope we manage to make you an offer.

Oh, and looking at the error message: Maybe a workaround is to remove that specific offending release and try again?

Hi, I'm sorry to hear you are having trouble with this, I hoped someone would find the time to look into this. Liblast is one of the largest projects on Codeberg (in the useful category, everything larger is mostly abuse), and there is a high chance of edge cases and scaling issues that have not been discovered before, simply before no one used the software at this scale yet. The obviously easiest option would be to rename and archive the current repo and create a new one. With a prominent notice, this should help inform people about the situation. The second best option would be to schedule with someone (@Gusted, if you have a minute) to look into the situation and see if there is an easy fix or workaround for the problem. Another option would be to let me manually fiddle with the database and reassign your issues, releases etc to another repo id, so that there is no need to do a round-trip with the API. It's not a good option and I'll need to test-drive this on a staging machine, but it *is* an option if there is no better solution found, and I hope we manage to make you an offer. Oh, and looking at the error message: Maybe a workaround is to remove that specific offending release and try again?

ashimokawa commented 2024年03月07日 16:02:16 +01:00

@unfa

Oh, you want to migrate from codeberg to codeberg in order to also migrate issues and PRs, right?

I have never tried it, will do.

@unfa Oh, you want to migrate from codeberg to codeberg in order to also migrate issues and PRs, right? I have never tried it, will do.

unfa commented 2024年03月08日 17:42:41 +01:00

Hey, thanks for the help, sorry for my latency.

@fnetX Yes, the easiest option would be to archive the repo as is, rename it and make a new one taking it's place. The problem with that is that all the stars and people watching the repository are now pointing at a dead archived repo, and aren't going to see the updated state of the project.

The second option sounds great, though I think scheduling time might be tricky on my part.

@ashimokawa Yes, the point is to make an archival copy a Codeberg repository including all the metadata associated with it, and after that's done the source repository can get reorganized and cleaned up from legacy stuff.

Hey, thanks for the help, sorry for my latency. @fnetX Yes, the easiest option would be to archive the repo as is, rename it and make a new one taking it's place. The problem with that is that all the stars and people watching the repository are now pointing at a dead archived repo, and aren't going to see the updated state of the project. The second option sounds great, though I think scheduling time might be tricky on my part. @ashimokawa Yes, the point is to make an archival copy a Codeberg repository including all the metadata associated with it, and after that's done the source repository can get reorganized and cleaned up from legacy stuff.

ashimokawa commented 2024年03月09日 01:06:03 +01:00

@unfa

The problem with that is that all the stars and people watching the repository are now pointing at a dead archived repo, and aren't going to see the updated state of the project.

As @fnetX mentioned we could reassign watches and stars to the new repo with direct database manipulation.

The database error (duplicated release) in your screenshot looks really weird, so it would be great if we could test that.
Maybe you do not need to be involved, if we can reproduce by doing the same migration to a test organization.

Did the error occur right away or only after a long time of waiting?

@unfa >The problem with that is that all the stars and people watching the repository are now pointing at a dead archived repo, and aren't going to see the updated state of the project. As @fnetX mentioned we could reassign watches and stars to the new repo with direct database manipulation. The database error (duplicated release) in your screenshot looks really weird, so it would be great if we could test that. Maybe you do not need to be involved, if we can reproduce by doing the same migration to a test organization. Did the error occur right away or only after a long time of waiting?

unfa commented 2024年03月09日 12:37:25 +01:00

@ashimokawa If manually reassigning watches and stars is possible, that'd probably be the best course of action.

Now that I think of it, it also includes collaborators. I'm trying to think of anything else that would trouble project members and contributors from following their work as usual.
CI configuration (action runners) is another one that'd I'd like to carry on from the legacy repo to the new one.

I hope that's not too much to ask?
I worry the project is already a burden to the platform :D

@ashimokawa If manually reassigning watches and stars is possible, that'd probably be the best course of action. Now that I think of it, it also includes collaborators. I'm trying to think of anything else that would trouble project members and contributors from following their work as usual. CI configuration (action runners) is another one that'd I'd like to carry on from the legacy repo to the new one. I hope that's not too much to ask? I worry the project is already a burden to the platform :D

ashimokawa commented 2024年03月13日 21:28:39 +01:00

@unfa

Okay, how shall we proceed? You move the repo in the way you want, and then tell which information I should carry over? Do you agree @fnetX ?

Regarding CI, if there is data on ci.codeberg.org it is better to ask @crapStone or @6543 since I am not familiar with that part of codeberg. ;)

@unfa Okay, how shall we proceed? You move the repo in the way you want, and then tell which information I should carry over? Do you agree @fnetX ? Regarding CI, if there is data on ci.codeberg.org it is better to ask @crapStone or @6543 since I am not familiar with that part of codeberg. ;)

unfa commented 2024年03月13日 22:42:12 +01:00

@ashimokawa @fnetX Yeah, I guess I'll just rename the repo now and mark it as archived. Then I'll create a new one giving it the name of the original one. And after that I'll post here and I'll need your help to amend some metadata :)

Thank you x100 BTW, this is incredible that you are open to doing stuff like this for your users <3

@ashimokawa @fnetX Yeah, I guess I'll just rename the repo now and mark it as archived. Then I'll create a new one giving it the name of the original one. And after that I'll post here and I'll need your help to amend some metadata :) Thank you x100 BTW, this is incredible that you are open to doing stuff like this for your users <3

unfa commented 2024年03月14日 01:19:55 +01:00

Ok, the old repo is archived, the new repo is live in its place.
https://codeberg.org/Liblast/Liblast-legacy

Ok, the old repo is archived, the new repo is live in its place. https://codeberg.org/Liblast/Liblast-legacy

ashimokawa commented 2024年03月14日 08:59:28 +01:00

@unfa
I moved watches and stars so far,

how about projects, and issues?
PRs are proable not easy/good to do because they probably depend on some objects being referenced that are no longer in your new cleaned up repo.

I guess our timezone is different, so sorry for the delays!

@unfa I moved watches and stars so far, how about projects, and issues? PRs are proable not easy/good to do because they probably depend on some objects being referenced that are no longer in your new cleaned up repo. I guess our timezone is different, so sorry for the delays!

unfa commented 2024年03月14日 10:52:02 +01:00

@ashimokawa Thank you so much! I was asleep so I wasn't waiting or anything :D

I think that actually it might be good to leave these behind. I can set up CI anew myself. I've already assigned contributors anew as well.

Since we're doing this rewrite, we also want to take the opportunity to clean up a lot of stuff and impose some more structure.

This is completely offtopic, but sadly the built-in Wiki is not very useful without a way to easily upload images to it.Keeping them in the repo itself means the wiki isn't self-contained and also there's 2 commits that need to happen to update the wiki + some weird URL gymnastics to embed images.

I tried cloning the Wiki repo and working on it as a separate repo (which would've been great) but I had no way of accessing images uploaded there to embed them in the documents.

So we've decided to keep documentation in it's own folder in the repo itself.

@ashimokawa Thank you so much! I was asleep so I wasn't waiting or anything :D I think that actually it might be good to leave these behind. I can set up CI anew myself. I've already assigned contributors anew as well. Since we're doing this rewrite, we also want to take the opportunity to clean up a lot of stuff and impose some more structure. This is completely offtopic, but sadly the built-in Wiki is not very useful without a way to easily upload images to it.Keeping them in the repo itself means the wiki isn't self-contained and also there's 2 commits that need to happen to update the wiki + some weird URL gymnastics to embed images. I tried cloning the Wiki repo and working on it as a separate repo (which would've been great) but I had no way of accessing images uploaded there to embed them in the documents. So we've decided to keep documentation in it's own folder in the repo itself.

ashimokawa commented 2024年03月14日 12:09:51 +01:00

@unfa

How about this, regarding the wiki:

https://github.com/go-gitea/gitea/issues/4690#issuecomment-581665577

@unfa How about this, regarding the wiki: https://github.com/go-gitea/gitea/issues/4690#issuecomment-581665577

unfa commented 2024年03月20日 23:37:08 +01:00

Hey, thank you for your help!
I've got all the repo stuff figured out.

Hey, thank you for your help! I've got all the repo stuff figured out.

goetz commented 2025年03月14日 09:33:54 +01:00

I'm running into this issue also while trying to migrate repos to codeberg from github.

As a test I also migrated to a local Forgejo instance, which works fine.

Is this something to get looked into soon, as this really hinders using codeberg.

I'm running into this issue also while trying to migrate repos to codeberg from github. As a test I also migrated to a local Forgejo instance, which works fine. Is this something to get looked into soon, as this really hinders using codeberg.

Gusted commented 2025年03月14日 17:41:28 +01:00

@goetz Please open a new issue.

@goetz Please open a new issue.

Gusted commented 2025年03月14日 17:41:43 +01:00

Also closing as it seems the original issue was resolved.

Also closing as it seems the original issue was resolved.