Codeberg-CI/feedback
672
80

plans for automatic "release" uploads or "pages" builds? #3

Closed
opened 2021年09月24日 00:43:25 +02:00 by ccoenen · 15 comments
Member
Copy link

currently, I do not see a way to get anything out of woodpecker easily. I saw all kinds of setups where people rsynced their stuff out of the containers, but that setup is tedious, and you also need to store a private key in woodpecker.

Two use cases that will probably come up often are:

  • build a static site from a static site generator and host the result on codeberg pages
  • build a binary and host the result from the project's releases page

both of which are common enough that there should be a "short" way to do it.

Is something like this in the works? (or perhaps even finished, but I just missed it?)

currently, I do not see a way to get anything out of woodpecker _easily_. I saw all kinds of setups where people `rsynced` their stuff out of the containers, but that setup is tedious, and you also need to store a private key in woodpecker. Two use cases that will probably come up often are: - build a static site from a static site generator and host the result on codeberg pages - build a binary and host the result from the project's releases page both of which are common enough that there should be a "short" way to do it. Is something like this in the works? (or perhaps even finished, but I just missed it?)
ccoenen changed title from (削除) plans for automatic "release" uploads? (削除ここまで) to plans for automatic "release" uploads or "pages" builds? 2021年09月24日 00:43:46 +02:00
Member
Copy link

I also feel the need for such a thing. Therefore I asked for info on the fediverse which got boosted and favoured but no hints so far:
https://hofra.rocks/web/statuses/106665478927100781

I also feel the need for such a thing. Therefore I asked for info on the fediverse which got boosted and favoured but no hints so far: https://hofra.rocks/web/statuses/106665478927100781
Member
Copy link

There are plugins for drone, that might work: http://plugins.drone.io/

Upstream (Woodpecker) has to decide how to handle plugins. It might not be the best idea to depend on drone plugins.

https://woodpecker-ci.github.io/docs/usage/plugins/plugins

There are plugins for drone, that might work: http://plugins.drone.io/ - http://plugins.drone.io/appleboy/drone-git-push/ - http://plugins.drone.io/drone-plugins/drone-gitea-release/ Upstream (Woodpecker) has to decide how to handle plugins. It might not be the best idea to depend on drone plugins. https://woodpecker-ci.github.io/docs/usage/plugins/plugins
Author
Member
Copy link

I was also referred to
https://github.com/woodpecker-ci/woodpecker/issues/335 by @6543. I will take a look at this sometime in the future. But I thought it would be good to collect all the info in one ticket for now for easier reference.

I was also referred to https://github.com/woodpecker-ci/woodpecker/issues/335 by @6543. I will take a look at this sometime in the future. But I thought it would be good to collect all the info in one ticket for now for easier reference.
Owner
Copy link

yes this will be handled as now by plugins - but eventualy with a better ui ...

also ref to plugin "market place" https://github.com/woodpecker-ci/woodpecker/issues/315

yes this will be handled as now by plugins - but eventualy with a better ui ... also ref to plugin "market place" https://github.com/woodpecker-ci/woodpecker/issues/315
  • build a static site from a static site generator and host the result on codeberg pages

I did something like that with Drone CI before: Codeberg/Community#498

The issue has links to Hexo, Hugo and Jekyll as examples. They work basically with appleboy/drone-git-push to push the generated static site content back to repo and has some tricks to stop seeing Drone commits as changes so that it doesn't run the build again. This worked perfectly with Pages v1, so I assume it should also work with v2. .drone.yml on those projects has the details.

I'm trying to port these to woodpecker and Pages v2, but I'm getting:

...
+ git remote add deploy git@codeberg.org:hexo-test/pages.git
+ git push deploy HEAD:pages
load pubkey "/root/.ssh/id_rsa": invalid format
Warning: Permanently added 'codeberg.org,193.26.156.135' (ECDSA) to the list of known hosts.
Load key "/root/.ssh/id_rsa": invalid format
git@codeberg.org: Permission denied (publickey).
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
exit status 128

I have something like this in .woodpecker.yml:

...
 push_commit:
 image: appleboy/drone-git-push
 settings:
 branch: pages
 remote: git@codeberg.org:hexo-test/pages.git
 force: false
 commit: true
 commit_message: "Woodpecker build ${CI_COMMIT_SHA:0:7}"
 author_name: "Adnan ___"
 author_email: "adnan360@___.com"
 secrets:
 - source: ssh_key
 target: plugin_ssh_key
 # To show green check in case previous step exits.
 when:
 status:
 - success

I tried to store the ssh private key as secrect on Woodpecker UI (with new lines replaced with \n by using cat ~/.ssh/id_adnan360_codeberg_ci | awk '{printf "%s\\n", 0ドル}') as both ssh_key and SSH_KEY but failed. I used ssh-keygen -m PEM -t rsa -P "" -C "adnan360@___.com" -f ~/.ssh/id_adnan360_codeberg_ci to generate the key because in case of Drone it only worked if I did it like this. Put the public key in Settings SSH / GPG keys. The key shows "No recent activity" under it. Which is a bit odd.

I can't cat to see what's inside /root/.ssh/id_rsa because Woodpecker doesn't let me mix settings and commands. Not sure what's wrong with my setup.

> - build a static site from a static site generator and host the result on codeberg pages I did something like that with Drone CI before: https://codeberg.org/Codeberg/Community/issues/498 The issue has links to Hexo, Hugo and Jekyll as examples. They work basically with `appleboy/drone-git-push` to push the generated static site content back to repo and has some tricks to stop seeing Drone commits as changes so that it doesn't run the build again. This worked perfectly with Pages v1, so I assume it should also work with v2. .drone.yml on those projects has the details. I'm trying to port these to woodpecker and Pages v2, but I'm getting: ``` ... + git remote add deploy git@codeberg.org:hexo-test/pages.git + git push deploy HEAD:pages load pubkey "/root/.ssh/id_rsa": invalid format Warning: Permanently added 'codeberg.org,193.26.156.135' (ECDSA) to the list of known hosts. Load key "/root/.ssh/id_rsa": invalid format git@codeberg.org: Permission denied (publickey). fatal: Could not read from remote repository. Please make sure you have the correct access rights and the repository exists. exit status 128 ``` I have something like this in `.woodpecker.yml`: ``` ... push_commit: image: appleboy/drone-git-push settings: branch: pages remote: git@codeberg.org:hexo-test/pages.git force: false commit: true commit_message: "Woodpecker build ${CI_COMMIT_SHA:0:7}" author_name: "Adnan ___" author_email: "adnan360@___.com" secrets: - source: ssh_key target: plugin_ssh_key # To show green check in case previous step exits. when: status: - success ``` I tried to store the ssh private key as secrect on Woodpecker UI (with new lines replaced with `\n` by using `cat ~/.ssh/id_adnan360_codeberg_ci | awk '{printf "%s\\n", 0ドル}'`) as [both](https://github.com/woodpecker-ci/woodpecker/issues/466#issuecomment-962934933) `ssh_key` and `SSH_KEY` but failed. I used `ssh-keygen -m PEM -t rsa -P "" -C "adnan360@___.com" -f ~/.ssh/id_adnan360_codeberg_ci` to generate the key because in case of Drone it only worked if I did it like this. Put the public key in Settings SSH / GPG keys. The key shows "No recent activity" under it. Which is a bit odd. I can't `cat` to see what's inside `/root/.ssh/id_rsa` because Woodpecker doesn't let me mix settings and commands. Not sure what's wrong with my setup.

When I logged in to Woodpecker today, I noticed the secret value input field is multiline now. I deleted the old ones and created the secret again (with name ssh_key). This time without escaping with \n, and it seems to be working now.

...
+ git remote add deploy git@codeberg.org:hexo-test/pages.git
+ git push deploy HEAD:pages --force
load pubkey "/root/.ssh/id_rsa": invalid format
Warning: Permanently added 'codeberg.org,193.26.156.135' (ECDSA) to the list of known hosts.
remote:
remote: Create a new pull request for 'pages':
remote: https://codeberg.org/hexo-test/pages/compare/main...pages
remote:
remote: . Processing 1 references 
...

The issue seems to have been fixed.

EDIT: Include secret name

When I logged in to Woodpecker today, I noticed the secret value input field is multiline now. I deleted the old ones and created the secret again (with name `ssh_key`). This time without escaping with `\n`, and it seems to be working now. ``` ... + git remote add deploy git@codeberg.org:hexo-test/pages.git + git push deploy HEAD:pages --force load pubkey "/root/.ssh/id_rsa": invalid format Warning: Permanently added 'codeberg.org,193.26.156.135' (ECDSA) to the list of known hosts. remote: remote: Create a new pull request for 'pages': remote: https://codeberg.org/hexo-test/pages/compare/main...pages remote: remote: . Processing 1 references ... ``` The issue seems to have been fixed. EDIT: Include secret name
Member
Copy link

FYI, Codeberg/pages-server#35 contains mostly information relevant for this issue as well, it seems like it was a duplicate in the wrong place.

FYI, https://codeberg.org/Codeberg/pages-server/issues/35 contains mostly information relevant for this issue as well, it seems like it was a duplicate in the wrong place.

build a binary and host the result from the project's releases page

I had exactly this problem and wrote a simple Go application which can be used to upload them to the gitea/codeberg releases: https://codeberg.org/qwerty287/gitea-release-attacher (docker container: https://hub.docker.com/r/qwerty287/gitea-release-attacher)

It's idea is to upload any file (the binary) to the latest release (do not create a release, supports uploading to a specified release ID/tag name), and delete the binary from an old CI run (from the releases). I may implement features to keep old binaries. Example config for woodpecker is the repo itself: https://codeberg.org/qwerty287/gitea-release-attacher/src/branch/main/.woodpecker/build.yml#L9-L20

> build a binary and host the result from the project's releases page I had exactly this problem and wrote a simple Go application which can be used to upload them to the gitea/codeberg releases: https://codeberg.org/qwerty287/gitea-release-attacher (docker container: https://hub.docker.com/r/qwerty287/gitea-release-attacher) It's idea is to upload any file (the binary) to the latest release (do not create a release, supports uploading to a specified release ID/tag name), and delete the binary from an old CI run (from the releases). I may implement features to keep old binaries. Example config for woodpecker is the repo itself: https://codeberg.org/qwerty287/gitea-release-attacher/src/branch/main/.woodpecker/build.yml#L9-L20
Member
Copy link

build a binary and host the result from the project's releases page

I had exactly this problem and wrote a simple Go application which can be used to upload them to the gitea/codeberg releases: https://codeberg.org/qwerty287/gitea-release-attacher (docker container: https://hub.docker.com/r/qwerty287/gitea-release-attacher)

It's idea is to upload any file (the binary) to the latest release (do not create a release, supports uploading to a specified release ID/tag name), and delete the binary from an old CI run (from the releases). I may implement features to keep old binaries. Example config for woodpecker is the repo itself: https://codeberg.org/qwerty287/gitea-release-attacher/src/branch/main/.woodpecker/build.yml#L9-L20

This is cool.

I am not yet convinced of the plugin idea, because if the plugin does not work (for me), i need (YET ANOTHER!) account (this time on docker hub) to publish the plugin, if i understand this correctly.

I have the same thing for releases although we are not using it now and push to a repo. The release publisher just uses curl and the gitea api (most likele like you)... remove old release:
https://codeberg.org/Freeyourgadget/fdroid-repo-config/src/branch/master/repoconfig/remove_old_nightly.sh

attach to a release: https://codeberg.org/Freeyourgadget/fdroid-repo-config/src/branch/master/repoconfig/publish_nightly.sh

> > build a binary and host the result from the project's releases page > > I had exactly this problem and wrote a simple Go application which can be used to upload them to the gitea/codeberg releases: https://codeberg.org/qwerty287/gitea-release-attacher (docker container: https://hub.docker.com/r/qwerty287/gitea-release-attacher) > > It's idea is to upload any file (the binary) to the latest release (do not create a release, supports uploading to a specified release ID/tag name), and delete the binary from an old CI run (from the releases). I may implement features to keep old binaries. Example config for woodpecker is the repo itself: https://codeberg.org/qwerty287/gitea-release-attacher/src/branch/main/.woodpecker/build.yml#L9-L20 This is cool. I am not yet convinced of the plugin idea, because if the plugin does not work (for me), i need (YET ANOTHER!) account (this time on docker hub) to publish the plugin, if i understand this correctly. I have the same thing for releases although we are not using it now and push to a repo. The release publisher just uses curl and the gitea api (most likele like you)... remove old release: https://codeberg.org/Freeyourgadget/fdroid-repo-config/src/branch/master/repoconfig/remove_old_nightly.sh attach to a release: https://codeberg.org/Freeyourgadget/fdroid-repo-config/src/branch/master/repoconfig/publish_nightly.sh
Member
Copy link

I mean, ideally such plugins can become officially supported for Woodpecker itself. I would love to have an official plugin or docker container for the tea command, then a command like tea releases create --tag "v1.0" --title v1.0 --target main --asset whatever.bin could be used for the releases and we don't have to reinvent the wheel for each task there could be with Gitea.

I mean, ideally such plugins can become officially supported for Woodpecker itself. I would love to have an official plugin or docker container for the [`tea`](https://gitea.com/gitea/tea) command, then a command like `tea releases create --tag "v1.0" --title v1.0 --target main --asset whatever.bin` could be used for the releases and we don't have to reinvent the wheel for each task there could be with Gitea.

Not sure if this is helpful for anyone else, but I ended up doing the following:

  • Create .tar.gz archive in a create-archive step in Woodpecker
  • Call shellscript that creates a new release in Codeberg + uploads the tar as an attachment
  • Have local shellscript for doing deploy (obviously optional, mainly for services)

Details look something like this:

Relevant woodpecker definition for create-archive and create-release:

 create-archive:
 image: debian
 when:
 branch: master
 event: push
 commands:
 - apt-get update && apt-get install --yes binutils upx
 - export RELEASE_DIR=ditzes-$CI_COMMIT_SHA
 - mkdir $RELEASE_DIR
 - cp src-tauri/target/release/api $RELEASE_DIR/ditzes-api
 - cp src-tauri/target/release/cli $RELEASE_DIR/ditzes-cli
 - strip $RELEASE_DIR/ditzes-api
 - upx $RELEASE_DIR/ditzes-api
 - strip $RELEASE_DIR/ditzes-cli
 - upx $RELEASE_DIR/ditzes-cli
 - tar cvzf ditzes-$CI_COMMIT_SHA-linux.tar.gz $RELEASE_DIR
 create-release:
 image: debian
 when:
 branch: master
 event: push
 secrets: [ API_TOKEN ]
 commands:
 - apt-get update && apt-get --yes install jq curl
 - export RELEASE_TAR=ditzes-$CI_COMMIT_SHA-linux.tar.gz
 - ./codeberg-release.sh

CapableWeb/ditzes@6063dc53c6/.woodpecker.yml (L10-L35)

codeberg-release.sh is basically two curl calls (with API_TOKEN stored as a secret via the Woodpecker UI): CapableWeb/ditzes@106405d0fa/codeberg-release.sh

deploy-release.sh can be called from any instance that has access to the instance you want to deploy to, with the commit as the argument. Downloads the archive, scps the files over and restarts the service: CapableWeb/ditzes@106405d0fa/deploy-release.sh

A bit hacky solution, but works for my simple purposes.

Not sure if this is helpful for anyone else, but I ended up doing the following: - Create .tar.gz archive in a `create-archive` step in Woodpecker - Call shellscript that creates a new release in Codeberg + uploads the tar as an attachment - Have local shellscript for doing deploy (obviously optional, mainly for services) Details look something like this: Relevant woodpecker definition for `create-archive` and `create-release`: ``` create-archive: image: debian when: branch: master event: push commands: - apt-get update && apt-get install --yes binutils upx - export RELEASE_DIR=ditzes-$CI_COMMIT_SHA - mkdir $RELEASE_DIR - cp src-tauri/target/release/api $RELEASE_DIR/ditzes-api - cp src-tauri/target/release/cli $RELEASE_DIR/ditzes-cli - strip $RELEASE_DIR/ditzes-api - upx $RELEASE_DIR/ditzes-api - strip $RELEASE_DIR/ditzes-cli - upx $RELEASE_DIR/ditzes-cli - tar cvzf ditzes-$CI_COMMIT_SHA-linux.tar.gz $RELEASE_DIR create-release: image: debian when: branch: master event: push secrets: [ API_TOKEN ] commands: - apt-get update && apt-get --yes install jq curl - export RELEASE_TAR=ditzes-$CI_COMMIT_SHA-linux.tar.gz - ./codeberg-release.sh ``` https://codeberg.org/CapableWeb/ditzes/src/commit/6063dc53c61ffc2a73b2b20ceac678987500003a/.woodpecker.yml#L10-L35 **codeberg-release.sh** is basically two curl calls (with API_TOKEN stored as a secret via the Woodpecker UI): https://codeberg.org/CapableWeb/ditzes/src/commit/106405d0fa7893ab4e574e3dbacb8670584ed0f8/codeberg-release.sh *deploy-release.sh* can be called from any instance that has access to the instance you want to deploy to, with the commit as the argument. Downloads the archive, scps the files over and restarts the service: https://codeberg.org/CapableWeb/ditzes/src/commit/106405d0fa7893ab4e574e3dbacb8670584ed0f8/deploy-release.sh A bit hacky solution, but works for my simple purposes.
Member
Copy link

I faced some problems while setting up SSR deploy to codeberg-pages with codeberg-ci and trying to replicate what I used to have with gh-pages. Reading here helped but also made a bit of confusion at some point. At the end is working like a charm, so I'm here just to give a full positive feedback and clarify it's easily doable.

The major issue was solved by using two separate repositories (source and target like it's suggested in codeberg-pages docs) and then I was able to use the image appleboy/drone-git-push straightforwardly.

 deploy:
 image: appleboy/drone-git-push
 secrets:
 - git_ssh_key
 settings:
 remote_name: origin-git
 remote: git@codeberg.org:username/pages.git
 branch: main
 commit: true
 commit_message: '[skip ci] publish to pages'
 ssh_key:
 from_secret: git_ssh_key
I faced some problems while setting up SSR deploy to codeberg-pages with codeberg-ci and trying to replicate what I used to have with gh-pages. Reading here helped but also made a bit of confusion at some point. At the end is working like a charm, so I'm here just to give a full positive feedback and clarify it's easily doable. The major issue was solved by using two separate repositories (source and target like it's suggested in codeberg-pages docs) and then I was able to use the image appleboy/drone-git-push straightforwardly. ``` deploy: image: appleboy/drone-git-push secrets: - git_ssh_key settings: remote_name: origin-git remote: git@codeberg.org:username/pages.git branch: main commit: true commit_message: '[skip ci] publish to pages' ssh_key: from_secret: git_ssh_key ```
Member
Copy link

There is probably also already a ready Woodpecker plugin
https://woodpecker-ci.org/plugins/Gitea%20Release

There is probably also already a ready Woodpecker plugin https://woodpecker-ci.org/plugins/Gitea%20Release

There is probably also already a ready Woodpecker plugin
https://woodpecker-ci.org/plugins/Gitea%20Release

for reference: yes, it works like a charm. I created a new access token named codeberg_access_token (under applications in the codeberg user settings) with probably more write access than it'd need and used it like so:

image:woodpeckerci/plugin-gitea-release
> There is probably also already a ready Woodpecker plugin > https://woodpecker-ci.org/plugins/Gitea%20Release for reference: yes, it works like a charm. I created a new access token named `codeberg_access_token` (under applications in the codeberg user settings) with probably more write access than it'd need and used it like so: https://codeberg.org/tectogen/tectogen/src/commit/789daf0cbc82e222bd64d62f65e4906a46da88a9/.woodpecker.yml#L25
Member
Copy link

Seems solved -> closing.

Seems solved -> closing.
Sign in to join this conversation.
No Branch/Tag specified
main
No results found.
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
13 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
Codeberg-CI/feedback#3
Reference in a new issue
Codeberg-CI/feedback
No description provided.
Delete branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?