Deployment of Documentation Website #36

lhinderberger commented 2020年08月23日 14:50:20 +02:00

As discussed in Codeberg/Blog#3 we now need a secure method of building and deploying the new Documentation website.

I'm currently working on a Dockerfile for isolated builds of the site, without having to run npm directly on our production servers. I'll write a WIP PR once it's ready for comments.

Other ideas are welcome too! :)

As discussed in Codeberg/Blog#3 we now need a secure method of building and deploying the new Documentation website. I'm currently working on a Dockerfile for isolated builds of the site, without having to run npm directly on our production servers. I'll write a WIP PR once it's ready for comments. Other ideas are welcome too! :)

lhinderberger commented 2020年08月24日 19:02:27 +02:00

An approach using Docker is now ready for review in #40

An approach using Docker is now ready for review in #40

hw commented 2020年08月24日 19:55:34 +02:00

Nice! Will test+integrate asap. Is this using relative paths, so that we can scp/rsync the build to production webroot?

Nice! Will test+integrate asap. Is this using relative paths, so that we can scp/rsync the build to production webroot?

lhinderberger commented 2020年08月24日 19:59:08 +02:00

The files in the build output assume they're at the root of their subdomain (so, links look like this: /assets/css/codeberg-docs.css). Thus as long as the build output gets deployed to its own subdomain, I think it should work 👍

The files in the build output assume they're at the root of their subdomain (so, links look like this: `/assets/css/codeberg-docs.css`). Thus as long as the build output gets deployed to its own subdomain, I think it should work 👍

hw commented 2020年08月24日 21:29:37 +02:00

ok, maybe it could be that simple: https://docs.codeberg.org/

In 9c17c4d3c3 we add a script deploy.sh, which builds+deploys the content to the pages repo of the docs org. Seems simple enough, ... do you think this is a viable approach?

ok, maybe it could be that simple: https://docs.codeberg.org/ In https://codeberg.org/Codeberg/Documentation/commit/9c17c4d3c30715d9934b92b924e156ae982c9791 we add a script `deploy.sh`, which builds+deploys the content to the pages repo of the docs org. Seems simple enough, ... do you think this is a viable approach?

hw commented 2020年08月24日 21:33:50 +02:00

fwiw, html sources are deployed to https://codeberg.org/docs/, where we can browse them for debugging (currently some links seem broken?)

fwiw, html sources are deployed to https://codeberg.org/docs/, where we can browse them for debugging (currently some links seem broken?)

lhinderberger commented 2020年08月25日 08:15:59 +02:00

The broken links seem to be already resolved - when browsing docs.codeberg.org I get a working website :)

The broken links seem to be already resolved - when browsing docs.codeberg.org I get a working website :)

lhinderberger commented 2020年08月25日 08:18:35 +02:00

Oh, wait - when clicking links in the text (and only in the text) it adds a docs/ prefix to the URL.

It seems it gets redirected whenever a trailing slash is missing. Maybe some remaining server-side configuration from when it was still under the docs/ subdirectory?

Oh, wait - when clicking links in the text (and only in the text) it adds a `docs/` prefix to the URL. It seems it gets redirected whenever a trailing slash is missing. Maybe some remaining server-side configuration from when it was still under the `docs/` subdirectory?

lhinderberger commented 2020年08月25日 08:20:51 +02:00

In 9c17c4d3c3 we add a script deploy.sh...

Great that it works now! What was wrong with the containerized solution - so that I know what to do better the next time? Or will both be combined?

> In https://codeberg.org/Codeberg/Documentation/commit/9c17c4d3c30715d9934b92b924e156ae982c9791 we add a script `deploy.sh`... Great that it works now! What was wrong with the containerized solution - so that I know what to do better the next time? Or will both be combined?

hw commented 2020年08月25日 08:34:02 +02:00

Oh, wait - when clicking links in the text (and only in the text) it adds a docs/ prefix to the URL.

It seems it gets redirected whenever a trailing slash is missing. Maybe some remaining server-side configuration from when it was still under the docs/ subdirectory?

Yes, these are no valid URLs (pretty much all web servers won't serve them). Github pages introduced this, over time more and more toolkits rely on those and are not really portable anymore to static webhosting. To make it easier for people we mimic most of the behavior (see Codeberg/Community#177 (comment) -- and the implementation in https://codeberg.org/Codeberg/build-deploy-gitea/src/branch/production/var/www/pages/index.php for details). It works quite nicely for common content, subdomain URL rewriting seems to break it tho.

Of course it would be best not to rely on non-standardized or undefined server behavior. Is it possible to configure eleventy to emit proper URLs?

Alternatively, we might need to tweak fallback/catch cases in pages/index.php to handle these cases too.

> Oh, wait - when clicking links in the text (and only in the text) it adds a `docs/` prefix to the URL. > > It seems it gets redirected whenever a trailing slash is missing. Maybe some remaining server-side configuration from when it was still under the `docs/` subdirectory? Yes, these are no valid URLs (pretty much all web servers won't serve them). Github pages introduced this, over time more and more toolkits rely on those and are not really portable anymore to static webhosting. To make it easier for people we mimic most of the behavior (see https://codeberg.org/Codeberg/Community/issues/177#issuecomment-56020 -- and the implementation in https://codeberg.org/Codeberg/build-deploy-gitea/src/branch/production/var/www/pages/index.php for details). It works quite nicely for common content, subdomain URL rewriting seems to break it tho. Of course it would be best not to rely on non-standardized or undefined server behavior. Is it possible to configure eleventy to emit proper URLs? Alternatively, we might need to tweak fallback/catch cases in pages/index.php to handle these cases too.

hw commented 2020年08月25日 08:39:10 +02:00

In 9c17c4d3c3 we add a script deploy.sh...

Great that it works now! What was wrong with the containerized solution - so that I know what to do better the next time? Or will both be combined?

The current docker file does not provide any build environment but just calls "npm install && npm run build", thus it does not really seem to add any functionality?

In production docker does not provide sufficient isolation. Running container in VM otoh does not provide benefit either ... and the non-containerized build environment should be easier to debug for contributors?

All this said, it just seemed like a simple+efficient solution for a quick start ;) if anything else works better: all improvements are welcome!

> > In https://codeberg.org/Codeberg/Documentation/commit/9c17c4d3c30715d9934b92b924e156ae982c9791 we add a script `deploy.sh`... > > Great that it works now! What was wrong with the containerized solution - so that I know what to do better the next time? Or will both be combined? > The current docker file does not provide any build environment but just calls "npm install && npm run build", thus it does not really seem to add any functionality? In production docker does not provide sufficient isolation. Running container in VM otoh does not provide benefit either ... and the non-containerized build environment should be easier to debug for contributors? All this said, it just seemed like a simple+efficient solution for a quick start ;) if anything else works better: all improvements are welcome!

lhinderberger commented 2020年08月25日 08:40:11 +02:00

I'm afraid I can't follow. Could you give an example of such an invalid URL and why it is invalid?

Wouldn't it be easier to configure the docs.codeberg.org subdomain in the webserver you use and point it to its own webroot, without Codeberg Pages?

I'm afraid I can't follow. Could you give an example of such an invalid URL and why it is invalid? Wouldn't it be easier to configure the docs.codeberg.org subdomain in the webserver you use and point it to its own webroot, without Codeberg Pages?

hw commented 2020年08月25日 08:45:09 +02:00

The issue is nicely explained in https://github.com/jekyll/jekyll/pull/3452

The issue is nicely explained in https://github.com/jekyll/jekyll/pull/3452

hw commented 2020年08月25日 08:46:07 +02:00

Codeberg/build-deploy-gitea@058b2a961d resolves the issue :)

Should work now?

https://codeberg.org/Codeberg/build-deploy-gitea/commit/058b2a961da7ab65d92bfc62d98f8363160b10df resolves the issue :) Should work now?

lhinderberger commented 2020年08月25日 08:49:47 +02:00

it does not really seem to add any functionality?

For one thing, it puts everything inside a container. And with bind mounts you can configure which data goes in and out (please see examples in the PR's README). Also, it stores the result of npm install in the image, thus saving some time and traffic at the actual build (something we could implement for deploy.sh too).

In production docker does not provide sufficient isolation.

How so? Is it possible for non-root npm scripts to escape the Container? Wouldn't that be a major security hole for a project like Docker? 🤔

> it does not really seem to add any functionality? For one thing, it puts everything inside a container. And with bind mounts you can configure which data goes in and out (please see examples in the PR's README). Also, it stores the result of `npm install` in the image, thus saving some time and traffic at the actual build (something we could implement for `deploy.sh` too). > In production docker does not provide sufficient isolation. How so? Is it possible for non-root npm scripts to escape the Container? Wouldn't that be a major security hole for a project like Docker? 🤔

lhinderberger commented 2020年08月25日 08:51:41 +02:00

Yes, it works now. 👍

Yes, it works now. 👍

hw commented 2020年08月25日 08:55:43 +02:00

it does not really seem to add any functionality?

For one thing, it puts everything inside a container. And with bind mounts you can configure which data goes in and out (please see examples in the PR's README). Also, it stores the result of npm install in the image, thus saving some time and traffic at the actual build (something we could implement for deploy.sh too).

The npm setup+build took a few seconds only here, far less than a typical docker build? This was just to get deployment going, whatever works best is welcome!

In production docker does not provide sufficient isolation.

How so? Is it possible for non-root npm scripts to escape the Container? Wouldn't that be a major security hole for a project like Docker? 🤔

Yes, there have been a number of CVEs in the past.

> > it does not really seem to add any functionality? > > For one thing, it puts everything inside a container. And with bind mounts you can configure which data goes in and out (please see examples in the PR's README). Also, it stores the result of `npm install` in the image, thus saving some time and traffic at the actual build (something we could implement for `deploy.sh` too). > The npm setup+build took a few seconds only here, far less than a typical docker build? This was just to get deployment going, whatever works best is welcome! > > In production docker does not provide sufficient isolation. > > How so? Is it possible for non-root npm scripts to escape the Container? Wouldn't that be a major security hole for a project like Docker? 🤔 > Yes, there have been a number of CVEs in the past.

lhinderberger commented 2020年08月25日 08:58:48 +02:00

The issue is nicely explained in https://github.com/jekyll/jekyll/pull/3452

Are we stripping the .html extension though? Because the leaf nodes of the build documentation site are index.htmls in appropriately named directories, not something.htmls that rely on the extension being stripped. I still don't get where we have invalid URLs?

> The issue is nicely explained in https://github.com/jekyll/jekyll/pull/3452 Are we stripping the `.html` extension though? Because the leaf nodes of the build documentation site are `index.html`s in appropriately named directories, not `something.html`s that rely on the extension being stripped. I still don't get where we have invalid URLs?

hw commented 2020年08月25日 09:00:11 +02:00

The issue is nicely explained in https://github.com/jekyll/jekyll/pull/3452

Are we stripping the .html extension though? Because the leaf nodes of the build documentation site are index.htmls in appropriately named directories, not something.htmls that rely on the extension being stripped. I still don't get where we have invalid URLs?

Hm, you are right, this is a different issue. Sorry for confusion

> > The issue is nicely explained in https://github.com/jekyll/jekyll/pull/3452 > > Are we stripping the `.html` extension though? Because the leaf nodes of the build documentation site are `index.html`s in appropriately named directories, not `something.html`s that rely on the extension being stripped. I still don't get where we have invalid URLs? Hm, you are right, this is a different issue. Sorry for confusion

hw commented 2020年08月25日 09:02:43 +02:00

Fwiw we need some similar handling in the main branch too (with subdomain support currently tested on https://codeberg.eu/ and https://codeberg-test.org/), right now https://docs.codeberg.eu/ should work, but https://docs.codeberg-test.org/ probably won't.

Fwiw we need some similar handling in the main branch too (with subdomain support currently tested on https://codeberg.eu/ and https://codeberg-test.org/), right now https://docs.codeberg.eu/ should work, but https://docs.codeberg-test.org/ probably won't.

lhinderberger commented 2020年08月25日 09:02:59 +02:00

The npm setup+build took a few seconds only here

I meant repeated website builds using the already created container...

Yes, there have been a number of CVEs in the past.

So containers were out of the questions in the first place? 😐

Also: How do we automate+isolate with deploy.sh? Will it run on the local development machine and push to pages, without ever running on the server? Then the question would be how to automate this.

> The npm setup+build took a few seconds only here I meant repeated website builds using the already created container... > Yes, there have been a number of CVEs in the past. So containers were out of the questions in the first place? 😐 Also: How do we automate+isolate with `deploy.sh`? Will it run on the local development machine and push to pages, without ever running on the server? Then the question would be how to automate this.

hw commented 2020年08月25日 09:12:13 +02:00

The npm setup+build took a few seconds only here

I meant repeated website builds using the already created container...

oh, I had the workflow described in the readme in mind, npm run serve for development+testing, and npm build (via deploy.sh) for release. If this was misunderstanding, please rework as needed, the relevant few lines for deployment should be easily transferable to whatever workflow is most appropriate... ;)

Whatever works best.

Yes, there have been a number of CVEs in the past.

So containers were out of the questions in the first place? 😐

Also: How do we automate+isolate with deploy.sh? Will it run on the local development machine and push to pages, without ever running on the server? Then the question would be how to automate this.

Currently manual release after review by maintainers (@lhinderberger and @n for example, you can also merge PRs) is intended. Assuming we build some kind of trigger/action/CI/CD support at some point in time, the action could automatically get triggered there (And of course this would need bulletproof isolation, as arbitrary code from arbitrary sources+repositories is executed in CI).

> > The npm setup+build took a few seconds only here > > I meant repeated website builds using the already created container... > oh, I had the workflow described in the readme in mind, `npm run serve` for development+testing, and `npm build` (via `deploy.sh`) for release. If this was misunderstanding, please rework as needed, the relevant few lines for deployment should be easily transferable to whatever workflow is most appropriate... ;) Whatever works best. > > Yes, there have been a number of CVEs in the past. > > So containers were out of the questions in the first place? 😐 > > Also: How do we automate+isolate with `deploy.sh`? Will it run on the local development machine and push to pages, without ever running on the server? Then the question would be how to automate this. > Currently manual release after review by maintainers (@lhinderberger and @n for example, you can also merge PRs) is intended. Assuming we build some kind of trigger/action/CI/CD support at some point in time, the action could automatically get triggered there (And of course this would need bulletproof isolation, as arbitrary code from arbitrary sources+repositories is executed in CI).

lhinderberger commented 2020年08月25日 09:24:02 +02:00

You have understood the workflow correctly - it's just a bit of an optimization to separate npm install from the deploy process so that it isn't repeated each time we deploy (it takes a few seconds each time it's called and as dependency versions are locked, there is no advantage in running npm install once dependencies are already installed).

I'll write a PR that unifies deploy.sh and package.json so the workflow for deployment would be:

npm install
npm run deploy

and just npm run deploy for repeated deployments (with deploy automatically calling build but not calling install)

Regarding isolation+automation: Okay 👍 And CI is a good keyword - let's wait with automating until we have a solid CI solution in place :)

You have understood the workflow correctly - it's just a bit of an optimization to separate `npm install` from the deploy process so that it isn't repeated each time we deploy (it takes a few seconds each time it's called and as dependency versions are locked, there is no advantage in running `npm install` once dependencies are already installed). I'll write a PR that unifies `deploy.sh` and `package.json` so the workflow for deployment would be: ``` npm install npm run deploy ``` and just `npm run deploy` for repeated deployments (with `deploy` automatically calling `build` but not calling `install`) Regarding isolation+automation: Okay :thumbsup: And CI is a good keyword - let's wait with automating until we have a solid CI solution in place :)

lhinderberger commented 2020年08月25日 09:26:55 +02:00

you can also merge PRs

Are we allowed to merge anything we think isn't controversial/doesn't need additional feedback (thus, simple additional documentation, typos, bug fixes etc?)? :)

> you can also merge PRs Are we allowed to merge anything we think isn't controversial/doesn't need additional feedback (thus, simple additional documentation, typos, bug fixes etc?)? :)

lhinderberger commented 2020年08月25日 09:29:56 +02:00

Closing, as this is now live on https://docs.codeberg.org

Thank you everyone!

Closing, as this is now live on https://docs.codeberg.org Thank you everyone!

Codeberg/Documentation

Codeberg/Documentation

Codeberg/Documentation