to comply with the 1-month release cycle

If I may nitpick this a bit, I wouldn't say "comply with 1-month release cycle." We want a 1-month cycle and we're shooting for it, but with a small team size, it wouldn't be perfect. I would say we will do what we can to make that happen but it may not occassionally.

@jsjoeio You are doing a tremendous job. Thank you and your team for all your work.

We're also open to other ideas! We would love any support for adding more tests to increasing the code quality or other suggestions the community has.

I could do some manual smoke tests before any release – just ping me to do so. If you do not get an answer within 48 hours (depending on my availability) simply go ahead.
i️ My focus is on the applicability in the field of Data Science, i.e. proxied JupyterLab integration, working terminal as well as compatibility with extensions ms-python.python, ms-toolsai.jupyter, Ikuyadeu.r and julialang.language-julia.

We appreciate that but we can always do better! <3

I could do some manual smoke tests before any release – just ping me to do so. If you do not get an answer within 48 hours (depending on my availability) simply go ahead.

Appreciate you offering that! I'm wondering though how we might automate it. I guess Step 1 would be simply adding an e2e test to make sure you can install the Jupyter and Python extensions. We'd have to pin the versions though so then we could include .vsix files in source code and not rely on a connection for the tests to pass.

@code-asher any thoughts on these e2e tests and making sure code-server works for data scientists?

I guess Step 1 would be simply adding an e2e test to make sure you can install the Jupyter and Python extensions. We'd have to pin the versions though so then we could include .vsix files in source code and not rely on a connection for the tests to pass.

I don't see how installing extensions should fail. A .vsix file is merely a .zip file containing some additional files next to the extension [folder] itself.
i️ Using the CLI (code-server --install-extension) always installs the latest (and possibly incompatible) version of an extension.
👉 See also #5031 and microsoft/vscode#153201.

@code-asher any thoughts on these e2e tests and making sure code-server works for data scientists?

IMHO That's beyond the scope of a stable release of code-server. I am happy to provide pre-releases on demand at https://vscode-r.jupyter.b-data.ch. (To look into #5335 for example)

FYI @fritterhoff

I don't see how installing extensions should fail. A .vsix file is merely a .zip file containing some additional files next to the extension [folder] itself.

Ah sorry, my message was confusing. I meant if you do code-server --install-extension <ext-name> it will require an internet connection vs. code-server --install-extension <vsix file> doesn't so less likely for tests to flake.

IMHO That's beyond the scope of a stable release of code-server.

Maybe! But if we have community members and potential customers relying on Jupyter to work, we need to make sure it doesn't break between releases.

I think we have a few problems.

1. Not enough manual testing

This testing is tedious and I have done a very poor job smoke testing releases ever since my primary focus was moved away from code-server so I am largely to blame here.

Updating Code can seem benign but doing so often breaks code-server unexpectedly and due to the architecture there is no guarantee that something which works when running via yarn watch (which is what we use when updating Code or making changes) works with a full build for production.

That said, I think if we test all the flows touched by our patches behind a reverse proxy before each release we should be in a good position. After each Code update we should do the same (with a full build, not yarn watch as that has caused us to miss multiple significant regressions).

Flows not touched by our patches are not as critical I think.

We might benefit from our own version of https://vscode-r.jupyter.b-data.ch/ that gets deployed on each PR and gets linked in the PR comments to lower the friction of testing a PR or a release.

Maybe release candidates (as pre-releases) would be a good call. We could require a minimum of a week before promoting the candidate to full release (assuming no release blockers) or something like that.

2. Not enough automated testing

If we find ourselves with time to write tests we might want to focus on e2e tests since I think for code-server they are more valuable than any other test even though they do not contribute to code coverage.

E2e tests should focus on covering patched flows.

3. Not marking regressions as release blockers.

Pretty straightforward. If an issue reports a regression the next release should not continue without resolving the issue. No exceptions IMO. We could use labels and/or put it on the milestone.

4. Keeping up to date with Code remains difficult.

I think we struggle to allocate time for updates and updating always takes longer than we think. The dream is to just update the submodule and away we go but it is almost never that easy.

We have talked about including code-server in planning sessions but we have not followed through.

Long-term we should think about upstreaming as much as possible and see if we can eventually deprecate code-server in favor of upstream's code-server. I think we should have an easier go of it now that they seem to be documenting ways to self-host (although it is still in private preview). Maybe we can start by trying upstream's code-server in Coder OSS.

tl;dr
Most important takeaways:

1. We need to test all patched flows on each release
2. Regressions must block releases

Secondary takeaways:

1. Test all patched flows with a full build after updates, not just with yarn watch
2. Prioritize e2e tests over other types
3. Automatic PR deploys would be nice
4. Try release candidates
5. Try to deprecate in favor of upstream
6. Try out upstream code-server in Coder OSS or elsewhere
7. Include code-server in planning sessions