fix channel cloning #7

abanink commented 2025年02月16日 22:44:08 +01:00

Working to make the channel cloning work between 2 forte sites.
This is still work in progress.

Working to make the channel cloning work between 2 forte sites. This is still work in progress.

streams commented 2025年02月23日 21:00:02 +01:00

Looks fine so far. We can save a little bit of unnecessary code since there is no channel_guid (this is only used for zot6/nomad - and forte is ActivityPub only). On the imports, you won't have to check for it or use it, so instead of adding a checking for empty value, it can just be removed from the queries.

There shouldn't be too much more to do to get import working. Maybe this is just wishful thinking; but there haven't been many schema changes outside the 'channel' table and moving all the private keys to the 'sign' table.

A small (300x300) profile photo is provided in the export file as binary data, just so that the channel is created with one. Later, we'll pull in photos from the original site, which will get the original full-size photo and several different resolutions. The photos and other content are fetched from the original site using OWA.

I may have done this already, but when fetching photos or other remote content, we'll need to generate a signature and this will need to get the signing key from the 'sign' table rather than from 'channel' data or from 'pconfig'; which we did previously.

That's the only other thing I'm aware of which might need investigation/fixing to get import working. So let me know if you encounter something unexpected.

Someplace further down the road, I'd like to be able to import channels from streams or hubzilla. but those will require a lot of work translating portable_id's (aka channel_hash, aka xchan_hash...) on all the tables and converting the channel itself from Zot6/Nomad to nomadic ActivityPub. At the moment, we're only importing from forte to keep the task manageable, but I'm mentioning this just so you're aware and so we don't do anything that will preclude or interfere with that future work.

Another future effort you should be aware of is that eventually I'd like to provide the ability to import ActivityPub posts and content from other projects like Mastodon. In order to do this, a 'locator' table was created to store any URIs which changed during import. It will have just an old URI and a new URI for everything that needs a new URI/URL. This way we can change the 'id' of an ActivityPub post, and then check when importing a comment or Announce/boost of that post to quickly find the new value to replace the 'inReplyTo' (or object->id) field. There are other examples, but that's the easiest to understand the underlying concept. Feel free to use that table if you encounter something that requires similar treatment.

Looks fine so far. We can save a little bit of unnecessary code since there is no channel_guid (this is only used for zot6/nomad - and forte is ActivityPub only). On the imports, you won't have to check for it or use it, so instead of adding a checking for empty value, it can just be removed from the queries. There shouldn't be too much more to do to get import working. Maybe this is just wishful thinking; but there haven't been many schema changes outside the 'channel' table and moving all the private keys to the 'sign' table. A small (300x300) profile photo is provided in the export file as binary data, just so that the channel is created with one. Later, we'll pull in photos from the original site, which will get the original full-size photo and several different resolutions. The photos and other content are fetched from the original site using OWA. I may have done this already, but when fetching photos or other remote content, we'll need to generate a signature and this will need to get the signing key from the 'sign' table rather than from 'channel' data or from 'pconfig'; which we did previously. That's the only other thing I'm aware of which might need investigation/fixing to get import working. So let me know if you encounter something unexpected. Someplace further down the road, I'd like to be able to import channels from streams or hubzilla. but those will require a lot of work translating portable_id's (aka channel_hash, aka xchan_hash...) on all the tables and converting the channel itself from Zot6/Nomad to nomadic ActivityPub. At the moment, we're only importing from forte to keep the task manageable, but I'm mentioning this just so you're aware and so we don't do anything that will preclude or interfere with that future work. Another future effort you should be aware of is that eventually I'd like to provide the ability to import ActivityPub posts and content from other projects like Mastodon. In order to do this, a 'locator' table was created to store any URIs which changed during import. It will have just an old URI and a new URI for everything that needs a new URI/URL. This way we can change the 'id' of an ActivityPub post, and then check when importing a comment or Announce/boost of that post to quickly find the new value to replace the 'inReplyTo' (or object->id) field. There are other examples, but that's the easiest to understand the underlying concept. Feel free to use that table if you encounter something that requires similar treatment.

abanink commented 2025年02月25日 18:39:19 +01:00

@streams

Thanks for the background info.

Messed up a bit but ultimately removed the channel_guid check meanwhile.

Not sure if I can consider the channel cloning as done now. I haven't played with cloned channels on streams much so I'm not sure the current outcome is ok.

As an example, I created a new channel "src" on hell.eenoog.org like this:
Screenshot From 2025年02月25日 17-35-12.png

After uploading a profile picture, it looks like this:
Screenshot From 2025年02月25日 17-33-34.png

This is the record in the db:

MariaDB [forte]> select channel_address,channel_hash from channel where channel_hash like '%hfUS';
+-----------------+----------------------------------------------------------+
| channel_address | channel_hash |
+-----------------+----------------------------------------------------------+
| srcnick | did:key:z6MknxGL3HGvDZc5XuYdFXsKJAWeTUyybaAJuAUco2AZhfUS |
+-----------------+----------------------------------------------------------+
1 row in set (0.001 sec)

I now clone the channel to heaven.eenoog.org, choose a new name 'targetnick', and the result looks like this:
Screenshot From 2025年02月25日 17-34-17.png

And the new record in the DB:

MariaDB [forte]> select channel_address,channel_hash from channel where channel_hash like '%hfUS';
+-----------------+----------------------------------------------------------+
| channel_address | channel_hash |
+-----------------+----------------------------------------------------------+
| targetnick | did:key:z6MknxGL3HGvDZc5XuYdFXsKJAWeTUyybaAJuAUco2AZhfUS |
+-----------------+----------------------------------------------------------+
1 row in set (0,001 sec)

I'll gladly continue to fix once I'm sure to understand the expected outcome for the cloned channel, at least for the channel name and identity/ies.

Could you clarify please?

@streams Thanks for the background info. Messed up a bit but ultimately removed the channel_guid check meanwhile. Not sure if I can consider the channel cloning as done now. I haven't played with cloned channels on streams much so I'm not sure the current outcome is ok. As an example, I created a new channel "src" on hell.eenoog.org like this:  After uploading a profile picture, it looks like this:  This is the record in the db: ``` MariaDB [forte]> select channel_address,channel_hash from channel where channel_hash like '%hfUS'; +-----------------+----------------------------------------------------------+ | channel_address | channel_hash | +-----------------+----------------------------------------------------------+ | srcnick | did:key:z6MknxGL3HGvDZc5XuYdFXsKJAWeTUyybaAJuAUco2AZhfUS | +-----------------+----------------------------------------------------------+ 1 row in set (0.001 sec) ``` I now clone the channel to heaven.eenoog.org, choose a new name 'targetnick', and the result looks like this:  And the new record in the DB: ``` MariaDB [forte]> select channel_address,channel_hash from channel where channel_hash like '%hfUS'; +-----------------+----------------------------------------------------------+ | channel_address | channel_hash | +-----------------+----------------------------------------------------------+ | targetnick | did:key:z6MknxGL3HGvDZc5XuYdFXsKJAWeTUyybaAJuAUco2AZhfUS | +-----------------+----------------------------------------------------------+ 1 row in set (0,001 sec) ``` I'll gladly continue to fix once I'm sure to understand the expected outcome for the cloned channel, at least for the channel name and identity/ies. Could you clarify please?

streams commented 2025年02月25日 21:21:37 +01:00

Yes that looks correct, except perhaps for the profile photo. I don't know if you got that far and it may have been cached. As noted earlier this is exported as a base64url_encode() blob of the 300x300 resolution profile photo.

It also set the 'Identities' correctly; so overall it's looking good.

[Edit: The nickname is actually set correctly. I think you'll find that you have two hubloc records for this channel_hash (stored in that table as hubloc_hash). One for the old location and one for the new location. There will be a single 'xchan' record, and the xchan_addr and xchan_url will point to whichever hubloc record is set with 'hubloc_primary' = '1' -- which implies the new channel was created as a backup channel and you're mainly at the old channel. The xchan record contains the address and location that will generally be shown to others if there is more than one location available.

Hope this explanation helps. ]

Yes that looks correct, except perhaps for the profile photo. I don't know if you got that far and it may have been cached. As noted earlier this is exported as a base64url_encode() blob of the 300x300 resolution profile photo. It also set the 'Identities' correctly; so overall it's looking good. [Edit: The nickname is actually set correctly. I think you'll find that you have two hubloc records for this channel_hash (stored in that table as hubloc_hash). One for the old location and one for the new location. There will be a single 'xchan' record, and the xchan_addr and xchan_url will point to whichever hubloc record is set with 'hubloc_primary' = '1' -- which implies the new channel was created as a backup channel and you're mainly at the old channel. The xchan record contains the address and location that will generally be shown to others if there is more than one location available. Hope this explanation helps. ]

abanink commented 2025年02月25日 21:35:17 +01:00

Thanks.
I'll just work on the profile photo and then I propose to close this PR.

Thanks. I'll just work on the profile photo and then I propose to close this PR.

abanink commented 2025年02月25日 21:43:30 +01:00

@streams wrote in #7 (comment):

Yes that looks correct, except perhaps for the profile photo. I don't know if you got that far and it may have been cached. As noted earlier this is exported as a base64url_encode() blob of the 300x300 resolution profile photo.

It also set the 'Identities' correctly; so overall it's looking good.

[Edit: The nickname is actually set correctly. I think you'll find that you have two hubloc records for this channel_hash. One for the old location and one for the new location. There will be a single 'xchan' record, and the xchan_addr and xchan_url will point to whichever hubloc record is set with 'hubloc_primary' = '1' -- which implies the new channel was created as a backup channel and you're mainly at the old channel. The xchan record contains the address and location that will generally be shown to others if there is more than one location available.

Hope this explanation helps. ]

Yes, sir!

This is on the src hub:

MariaDB [forte]> select xchan_addr,xchan_hash,xchan_url from xchan where xchan_hash like '%hfUS';
+-------------------------+----------------------------------------------------------+--------------------------------------------------------------------------------------------------------------+
| xchan_addr | xchan_hash | xchan_url |
+-------------------------+----------------------------------------------------------+--------------------------------------------------------------------------------------------------------------+
| srcnick@hell.eenoog.org | did:key:z6MknxGL3HGvDZc5XuYdFXsKJAWeTUyybaAJuAUco2AZhfUS | https://hell.eenoog.org/.well-known/apgateway/did:key:z6MknxGL3HGvDZc5XuYdFXsKJAWeTUyybaAJuAUco2AZhfUS/actor |
+-------------------------+----------------------------------------------------------+--------------------------------------------------------------------------------------------------------------+
MariaDB [forte]> select hubloc_hash,hubloc_primary,hubloc_id_url from hubloc where hubloc_hash like '%hfUS';
+----------------------------------------------------------+----------------+--------------------------------------------------------------------------------------------------------------+
| hubloc_hash | hubloc_primary | hubloc_id_url |
+----------------------------------------------------------+----------------+--------------------------------------------------------------------------------------------------------------+
| did:key:z6MknxGL3HGvDZc5XuYdFXsKJAWeTUyybaAJuAUco2AZhfUS | 1 | https://hell.eenoog.org/.well-known/apgateway/did:key:z6MknxGL3HGvDZc5XuYdFXsKJAWeTUyybaAJuAUco2AZhfUS/actor |
+----------------------------------------------------------+----------------+--------------------------------------------------------------------------------------------------------------+

And this is on the target hub:

MariaDB [forte]> select xchan_addr,xchan_hash,xchan_url from xchan where xchan_hash like '%hfUS';
+-------------------------+----------------------------------------------------------+--------------------------------------------------------------------------------------------------------------+
| xchan_addr | xchan_hash | xchan_url |
+-------------------------+----------------------------------------------------------+--------------------------------------------------------------------------------------------------------------+
| srcnick@hell.eenoog.org | did:key:z6MknxGL3HGvDZc5XuYdFXsKJAWeTUyybaAJuAUco2AZhfUS | https://hell.eenoog.org/.well-known/apgateway/did:key:z6MknxGL3HGvDZc5XuYdFXsKJAWeTUyybaAJuAUco2AZhfUS/actor |
+-------------------------+----------------------------------------------------------+--------------------------------------------------------------------------------------------------------------+
MariaDB [forte]> select hubloc_hash,hubloc_primary,hubloc_id_url from hubloc where hubloc_hash like '%hfUS';
+----------------------------------------------------------+----------------+----------------------------------------------------------------------------------------------------------------+
| hubloc_hash | hubloc_primary | hubloc_id_url |
+----------------------------------------------------------+----------------+----------------------------------------------------------------------------------------------------------------+
| did:key:z6MknxGL3HGvDZc5XuYdFXsKJAWeTUyybaAJuAUco2AZhfUS | 1 | https://hell.eenoog.org/.well-known/apgateway/did:key:z6MknxGL3HGvDZc5XuYdFXsKJAWeTUyybaAJuAUco2AZhfUS/actor |
| did:key:z6MknxGL3HGvDZc5XuYdFXsKJAWeTUyybaAJuAUco2AZhfUS | 0 | https://heaven.eenoog.org/.well-known/apgateway/did:key:z6MknxGL3HGvDZc5XuYdFXsKJAWeTUyybaAJuAUco2AZhfUS/actor |
+----------------------------------------------------------+----------------+----------------------------------------------------------------------------------------------------------------+

@streams wrote in https://codeberg.org/fortified/forte/pulls/7#issuecomment-2874902: > Yes that looks correct, except perhaps for the profile photo. I don't know if you got that far and it may have been cached. As noted earlier this is exported as a base64url_encode() blob of the 300x300 resolution profile photo. > > It also set the 'Identities' correctly; so overall it's looking good. > > [Edit: The nickname is actually set correctly. I think you'll find that you have two hubloc records for this channel_hash. One for the old location and one for the new location. There will be a single 'xchan' record, and the xchan_addr and xchan_url will point to whichever hubloc record is set with 'hubloc_primary' = '1' -- which implies the new channel was created as a backup channel and you're mainly at the old channel. The xchan record contains the address and location that will generally be shown to others if there is more than one location available. > > Hope this explanation helps. ] Yes, sir! This is on the src hub: ``` MariaDB [forte]> select xchan_addr,xchan_hash,xchan_url from xchan where xchan_hash like '%hfUS'; +-------------------------+----------------------------------------------------------+--------------------------------------------------------------------------------------------------------------+ | xchan_addr | xchan_hash | xchan_url | +-------------------------+----------------------------------------------------------+--------------------------------------------------------------------------------------------------------------+ | srcnick@hell.eenoog.org | did:key:z6MknxGL3HGvDZc5XuYdFXsKJAWeTUyybaAJuAUco2AZhfUS | https://hell.eenoog.org/.well-known/apgateway/did:key:z6MknxGL3HGvDZc5XuYdFXsKJAWeTUyybaAJuAUco2AZhfUS/actor | +-------------------------+----------------------------------------------------------+--------------------------------------------------------------------------------------------------------------+ MariaDB [forte]> select hubloc_hash,hubloc_primary,hubloc_id_url from hubloc where hubloc_hash like '%hfUS'; +----------------------------------------------------------+----------------+--------------------------------------------------------------------------------------------------------------+ | hubloc_hash | hubloc_primary | hubloc_id_url | +----------------------------------------------------------+----------------+--------------------------------------------------------------------------------------------------------------+ | did:key:z6MknxGL3HGvDZc5XuYdFXsKJAWeTUyybaAJuAUco2AZhfUS | 1 | https://hell.eenoog.org/.well-known/apgateway/did:key:z6MknxGL3HGvDZc5XuYdFXsKJAWeTUyybaAJuAUco2AZhfUS/actor | +----------------------------------------------------------+----------------+--------------------------------------------------------------------------------------------------------------+ ``` And this is on the target hub: ``` MariaDB [forte]> select xchan_addr,xchan_hash,xchan_url from xchan where xchan_hash like '%hfUS'; +-------------------------+----------------------------------------------------------+--------------------------------------------------------------------------------------------------------------+ | xchan_addr | xchan_hash | xchan_url | +-------------------------+----------------------------------------------------------+--------------------------------------------------------------------------------------------------------------+ | srcnick@hell.eenoog.org | did:key:z6MknxGL3HGvDZc5XuYdFXsKJAWeTUyybaAJuAUco2AZhfUS | https://hell.eenoog.org/.well-known/apgateway/did:key:z6MknxGL3HGvDZc5XuYdFXsKJAWeTUyybaAJuAUco2AZhfUS/actor | +-------------------------+----------------------------------------------------------+--------------------------------------------------------------------------------------------------------------+ MariaDB [forte]> select hubloc_hash,hubloc_primary,hubloc_id_url from hubloc where hubloc_hash like '%hfUS'; +----------------------------------------------------------+----------------+----------------------------------------------------------------------------------------------------------------+ | hubloc_hash | hubloc_primary | hubloc_id_url | +----------------------------------------------------------+----------------+----------------------------------------------------------------------------------------------------------------+ | did:key:z6MknxGL3HGvDZc5XuYdFXsKJAWeTUyybaAJuAUco2AZhfUS | 1 | https://hell.eenoog.org/.well-known/apgateway/did:key:z6MknxGL3HGvDZc5XuYdFXsKJAWeTUyybaAJuAUco2AZhfUS/actor | | did:key:z6MknxGL3HGvDZc5XuYdFXsKJAWeTUyybaAJuAUco2AZhfUS | 0 | https://heaven.eenoog.org/.well-known/apgateway/did:key:z6MknxGL3HGvDZc5XuYdFXsKJAWeTUyybaAJuAUco2AZhfUS/actor | +----------------------------------------------------------+----------------+----------------------------------------------------------------------------------------------------------------+ ```

abanink commented 2025年03月01日 11:06:44 +01:00

@streams
as discussed, let's close this PR.
We'll fix remaining channel cloning issues in a new PR.

@streams as discussed, let's close this PR. We'll fix remaining channel cloning issues in a new PR.